Commit Graph

1668 Commits

Author SHA1 Message Date
Anders Kaseorg d56a5bdd88 dev_settings: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 9650188dfe backends: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Alexandra Ciobica f109dcce9c auth: Add logic for avatars to the GitHub auth email selection page.
For the emails that are associated to an existing account in an
organisation, the avatars will be displayed in the email selection
page.  This includes avatar data in what is passed to the page.

Added `avatar_urls` to the context in `test_templates.py`.
2019-08-08 11:12:51 -07:00
Alexandra Ciobica d4ccd73ae3 auth: Remove `@users.noreply.github.com` from the email selection list.
Apparently GitHub changed the email address for these; we need to
update our code accordingly.

One cannot receive emails on the username@users.noreply.github.com, so
if someone tries creating an account with this email address, that
person would not be able to verify the account.
2019-08-08 11:12:51 -07:00
Tim Abbott b223159264 settings: Fix avatars directory being created in populate_db.
We were incorrectly setting LOCAL_UPLOADS_DIR to the empty string in
this code path, which would result in upload files being logged to the
root directory of the repository.

Fixes #12909.
2019-08-07 12:03:59 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 5ab64daecc mypy: Remove type ignore by defining ProfileDataElement using TypedDict. 2019-08-06 23:24:56 -07:00
Rishi Gupta 8167b5b645 email: Remove special integration doc behavior when gateway not set.
The current code looks like it's trying to redirect /integrations/doc/email
to /integrations when EMAIL_GATEWAY_PATTERN is not set.

I think it doesn't currently do this. The test for that pathway has a bug:
self.get_doc('integrations/doc-html/email', subdomain='zulip') needs a
leading slash, and putting the slash back in results in the test failing.

This redirection is not really desired behavior -- better is to
unconditionally show that the email integration exists, and just point the
user to https://zulip.readthedocs.io/en/latest/production/email-gateway.html
(this is done in a child commit).
2019-08-05 17:33:05 -07:00
Mateusz Mandera 8e0b6fa6cf populate_db: Use separate realm for system bots, like in production.
This commit alone breaks things, needs to be merged with the follow-up
ones.

welcome-bot is removed from the explicit list, because it already is in
settings.INTERNAL_BOTS.
2019-07-31 15:58:39 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Wyatt Hoodes ef02de4834 public_export: Add endpoint for returning all REALM_EXPORTED objects. 2019-07-26 15:52:02 -07:00
Wyatt Hoodes 55ad0d316a settings: Fix typing for LDAP_EMAIL_ATTR.
This is already defined in `prod_settings_template`.
2019-07-25 17:41:01 -07:00
Wyatt Hoodes 77068f3e1e settings: Fix typing for AUTHENTICATION_BACKENDS. 2019-07-25 17:41:01 -07:00
Anders Kaseorg 2b33822de9 storage: Stop using django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg fd7803e7f4 settings: Unset STATIC_ROOT in development.
Django’s default FileSystemFinder disallows STATICFILES_DIRS from
containing STATIC_ROOT (by raising an ImproperlyConfigured exception),
because STATIC_ROOT is supposed to be the result of collecting all the
static files in the project, not one of the potentially many sources
of static files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott 05fdbd5224 settings: Fix typing for EXTERNAL_HOST.
The previous logic has a reassignment of EXTERNAL_HOST temporarily to
Optional[str], which we don't want.
2019-07-23 17:22:14 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Tim Abbott 40e8fd24ff portico: Enable default ToS for development environment. 2019-07-22 17:43:43 -07:00
Mateusz Mandera be0a024355 integration docs: Document new email integration behaviors.
Tweaked by tabbott for cleaner, more precise English.
2019-07-22 15:52:08 -07:00
Harshit Bansal bf14a0af4d auth: Migrate google auth to python-social-auth.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend.  These are:

* GoogleMobileOauth2Backend, the ancient code path for mobile
  authentication last used by the EOL original Zulip Android app.

* The `finish_google_oauth2` code path in zerver/views/auth.py, which
  was the webapp (and modern mobile app) Google authentication code
  path.

This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
2019-07-21 20:51:34 -07:00
Wyatt Hoodes 9b69c1f7fe Merge pull request #12833 from whoodes/clean-up-new-user-bot
bots: Replace remnants of new-user-bot with default-bot@zulip.com.
2019-07-20 14:38:05 -07:00
Anders Kaseorg afa251de5d dev_urls: Serve static files with staticfiles_urlpatterns.
The test_docs change is because Django runs test cases with DEBUG =
False, which ordinarily means it doesn’t serve /static during tests.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Anders Kaseorg 239aefca74 settings: Use production webpack assets when PIPELINE_ENABLED in development.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Anders Kaseorg 1bb6634be1 settings: Fix storage settings for (DEBUG and PIPELINE_ENABLED) case.
The documentation suggests that you can get the dev server to use
production assets by setting PIPELINE_ENABLED = True, but that
resulted in Django being unable to find any static files because
FileSystemFinder was missing from STATICFILES_FINDERS.  Using the
production storage configuration in this case reduces the number of
possible configurations and seems to result in things being less
broken.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Wyatt Hoodes 1418b8b05f test_server: Set LOCAL_UPLOADS_DIR in set_up_django.
For the Casper and API tests, we need to allow setting
LOCAL_UPLOADS_DIR, which is normally managed within test_runner.py.
2019-07-17 16:14:20 -07:00
Wyatt Hoodes 5f20caa6e0 test_upload: Refactor test_upload output to new filepath.
We write a function to set the `LOCAL_UPLOADS_DIR` path depending
on whether tests are being run in parallel or serial mode.
2019-07-12 17:35:44 -07:00
Wyatt Hoodes ff156c1f8a test_runner: Create file structure for test-backend filepath rework.
We create a path structure in the from:
 `/var/<uuid>/test-backend/run_1234567/worker_N/`

A settings attribute, `TEST_WORKER_DIR`, was created as a worker's
directory for a given `test-backend` run's file storage.  The
appropirate path is created in `setup_test_environment`, while each
workers subdirectory is created within `init_worker`.

This allows a test class to write to `settings.TEST_WORKER_DIR`,
populating the appropirate directory of a given worker.  Also
providing the long-term approach to clean up filesystem access
in the backend unit tests.
2019-07-12 17:27:18 -07:00
Mateusz Mandera 3a4f8ed737 docs: Move prose email mirror instructions out of settings template.
This should make it easier to understand the documentation.
2019-07-10 13:33:25 -07:00
Tim Abbott 6758b016ac auth: Fix social auth URL pattern for AzureAD.
Apparently, only the login URL was correctly updated to allow dashes
in the name of the social backend.
2019-07-09 14:18:21 -07:00
Mateusz Mandera 8c987900ca retention: Move archive vacuuming days setting to DEFAULT_SETTINGS.
This was incorrectly placed in "Email settings", and accidentally not
overridable.
2019-07-08 16:35:53 -07:00
Hemanth V. Alluri 74a72fc422 openapi: Add regex-to-openapi url pattern matching in the test.
With this, the automated validation test will now be able to
work with URL patterns containing regular expressions.
2019-07-08 12:34:31 -07:00
Anders Kaseorg f47f98e2f1 storage: Remove AddHeaderMixin.
We should be doing this inside webpack, which would be a lot less
hacky.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:15:46 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
vinitS101 04f3fce761 ldap: Fix LDAP avatar synchronization to check if avatar has changed.
When "manage.py sync_ldap_user_data" is run, user avatars are now only
updated if they have changed in LDAP.

Fixes #12381.
2019-07-02 17:52:48 -07:00
Anders Kaseorg 5f3d36ae87 tex: Bundle KaTeX CLI using Webpack.
We don’t need a hacked copy anymore.  We run the installed version out
of node_modules in development, and a Webpack-bundled version of that
in production.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Anders Kaseorg b0be0d5285 settings_account: Use webpack asynchronous require to load zxcvbn.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Hemanth V. Alluri 7e6bcbeac0 openapi: Allow us to specify an endpoint as being undocumented in urls.
In each url of urls.py, if we want to mark an endpoint as being
intentionally undocumented, then in the kwargs instead of directly
mapping like 'METHOD': 'zerver.views.package.foo', we can provide
a tag called 'intentionally_undocumented' and map like:
'METHOD': ('zerver.views.package.foo', {'intentionally_undocumented'}).

If an endpoint is marked as intentionally undocumented, but we find
some OpenAPI documentation for it then we'll throw an error, in which
case either the 'intentionally_undocumented' tag should be removed or
the faulty documentation should be removed.
2019-07-02 16:34:16 -07:00
Hemanth V. Alluri e5cb3cabef openapi: Add intentionally_undocumentated parameter to REQ.
This will allow us to mark a REQ variable as intentionally
undocumented. With this, we can remove some of the endpoints marked
as "buggy" even though they're not actually buggy, we just needed to
specify certain parameters as intentionally undocumented (e.g. the
stream_id for the /users/me/subscriptions/muted_topics endpoint.)

Any REQ variable with intentionally_undocumentated set to True
will not be added to the arguments_map data structure.

For some of the other "buggy" endpoints, we would want to mark the
entire endpoint as being undocumented intentionally via. the urls.py
file.
2019-07-02 16:30:33 -07:00
Anders Kaseorg 0334be7053 zproject: Remove unused template tags minified_js, static.
`minified_js` was obsoleted by the Webpack migration and `static` has
never been used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:18:23 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Anders Kaseorg 02d19d48fc tooling: Remove static/node_modules symlink.
It appears not to have been useful and makes it marginally harder to
reason about how module resolution works.  Paths to static content in
node_modules should be resolved through Webpack instead.

(This node_modules symlink was originally created in the pre-webpack world
where all of our static asset paths were based in static/.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-27 14:42:16 -07:00
Mateusz Mandera 7950aaea1e retention: Add code for deleting old archive data. 2019-06-26 12:24:47 -07:00
Wyatt Hoodes 0d20e0e9d5 test-backend: Move `test-upload` dirs to `var/<uuid>/test-backend`. 2019-06-24 12:23:09 -07:00
Wyatt Hoodes 3c1982ecab wsgi.py: Fix broken error ouput for django.setup() call.
It's common for a broken settings.py file to result in Django not
starting, thus never writing to `/var/log/zulip/errors.log`.  Such
behavior can be discouraging when the server 500s without a traceback
to accompany it.  To fix this, we simply catch any exceptions in
django.setup(), if raised, and log the exception appropriately.

Comments tweaked by tabbott.

Closes: #7032.
2019-06-24 12:19:51 -07:00
Shubham Padia 80a3651cf3 auth: Let user choose emails in GitHub auth.
Previously, our Github authentication backend just used the user's
primary email address associated with GitHub, which was a reasonable
default, but quite annoying for users who have several email addresses
associated with their GitHub account.

We fix this, by adding a new screen where users can select which of
their (verified) GitHub email addresses to use for authentication.

This is implemented using the "partial" feature of the
python-social-auth pipeline system.

Each email is displayed as a button. Clicking on that button chooses
the email. The email value is stored in a hidden input above the
button. The `primary_email` is displayed on top followed by
`verified_non_primary_emails`. Backend name is also passed as
`backend` to the template, which in our case is GitHub.

Fixes #9876.
2019-06-23 21:27:04 -07:00
Tim Abbott ff38d125b1 settings: Configure LDAP avatar synchronization in dev environment.
This should make it a lot more convenient to do manual testing of
these common LDAP configuration options.
2019-06-17 13:18:36 -07:00
Vishnu Ks cc91e6cb94 auth: Allow setting GOOGLE_OAUTH2_CLIENT_ID from dev-secrets.
This makes it much more convenient to use Google/GitHub authentication
in a Zulip development environment for testing; one only has to set it
up once.
2019-06-14 10:16:26 -07:00
vinitS101 a6eda858d0 ldap: Fix avatar sync not working with the S3 backend.
This fixes an issue that caused LDAP synchronization to fail for
avatars.  The problem occurred due to the lack of a 'name' attribute
on the BytesIO object that we pass to the upload backend (which is
only used in the S3 backend for computing Content-Type).

Fixes #12411.
2019-06-13 15:12:13 -07:00
Mateusz Mandera cbee5beeac retention: Log progress through the archiving process. 2019-06-13 11:02:11 -07:00
Tim Abbott f5375adf8f settings: Improve example for RABBITMQ_HOST.
We prefer 127.0.0.1 over localhost for RabbitMQ because of weird IPv6
issues, so we should avoid mentioning localhost as an example.
2019-06-06 16:41:15 -07:00