Commit Graph

1643 Commits

Author SHA1 Message Date
Wyatt Hoodes 5f20caa6e0 test_upload: Refactor test_upload output to new filepath.
We write a function to set the `LOCAL_UPLOADS_DIR` path depending
on whether tests are being run in parallel or serial mode.
2019-07-12 17:35:44 -07:00
Wyatt Hoodes ff156c1f8a test_runner: Create file structure for test-backend filepath rework.
We create a path structure in the from:
 `/var/<uuid>/test-backend/run_1234567/worker_N/`

A settings attribute, `TEST_WORKER_DIR`, was created as a worker's
directory for a given `test-backend` run's file storage.  The
appropirate path is created in `setup_test_environment`, while each
workers subdirectory is created within `init_worker`.

This allows a test class to write to `settings.TEST_WORKER_DIR`,
populating the appropirate directory of a given worker.  Also
providing the long-term approach to clean up filesystem access
in the backend unit tests.
2019-07-12 17:27:18 -07:00
Mateusz Mandera 3a4f8ed737 docs: Move prose email mirror instructions out of settings template.
This should make it easier to understand the documentation.
2019-07-10 13:33:25 -07:00
Tim Abbott 6758b016ac auth: Fix social auth URL pattern for AzureAD.
Apparently, only the login URL was correctly updated to allow dashes
in the name of the social backend.
2019-07-09 14:18:21 -07:00
Mateusz Mandera 8c987900ca retention: Move archive vacuuming days setting to DEFAULT_SETTINGS.
This was incorrectly placed in "Email settings", and accidentally not
overridable.
2019-07-08 16:35:53 -07:00
Hemanth V. Alluri 74a72fc422 openapi: Add regex-to-openapi url pattern matching in the test.
With this, the automated validation test will now be able to
work with URL patterns containing regular expressions.
2019-07-08 12:34:31 -07:00
Anders Kaseorg f47f98e2f1 storage: Remove AddHeaderMixin.
We should be doing this inside webpack, which would be a lot less
hacky.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:15:46 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
vinitS101 04f3fce761 ldap: Fix LDAP avatar synchronization to check if avatar has changed.
When "manage.py sync_ldap_user_data" is run, user avatars are now only
updated if they have changed in LDAP.

Fixes #12381.
2019-07-02 17:52:48 -07:00
Anders Kaseorg 5f3d36ae87 tex: Bundle KaTeX CLI using Webpack.
We don’t need a hacked copy anymore.  We run the installed version out
of node_modules in development, and a Webpack-bundled version of that
in production.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Anders Kaseorg b0be0d5285 settings_account: Use webpack asynchronous require to load zxcvbn.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Hemanth V. Alluri 7e6bcbeac0 openapi: Allow us to specify an endpoint as being undocumented in urls.
In each url of urls.py, if we want to mark an endpoint as being
intentionally undocumented, then in the kwargs instead of directly
mapping like 'METHOD': 'zerver.views.package.foo', we can provide
a tag called 'intentionally_undocumented' and map like:
'METHOD': ('zerver.views.package.foo', {'intentionally_undocumented'}).

If an endpoint is marked as intentionally undocumented, but we find
some OpenAPI documentation for it then we'll throw an error, in which
case either the 'intentionally_undocumented' tag should be removed or
the faulty documentation should be removed.
2019-07-02 16:34:16 -07:00
Hemanth V. Alluri e5cb3cabef openapi: Add intentionally_undocumentated parameter to REQ.
This will allow us to mark a REQ variable as intentionally
undocumented. With this, we can remove some of the endpoints marked
as "buggy" even though they're not actually buggy, we just needed to
specify certain parameters as intentionally undocumented (e.g. the
stream_id for the /users/me/subscriptions/muted_topics endpoint.)

Any REQ variable with intentionally_undocumentated set to True
will not be added to the arguments_map data structure.

For some of the other "buggy" endpoints, we would want to mark the
entire endpoint as being undocumented intentionally via. the urls.py
file.
2019-07-02 16:30:33 -07:00
Anders Kaseorg 0334be7053 zproject: Remove unused template tags minified_js, static.
`minified_js` was obsoleted by the Webpack migration and `static` has
never been used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:18:23 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Anders Kaseorg 02d19d48fc tooling: Remove static/node_modules symlink.
It appears not to have been useful and makes it marginally harder to
reason about how module resolution works.  Paths to static content in
node_modules should be resolved through Webpack instead.

(This node_modules symlink was originally created in the pre-webpack world
where all of our static asset paths were based in static/.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-27 14:42:16 -07:00
Mateusz Mandera 7950aaea1e retention: Add code for deleting old archive data. 2019-06-26 12:24:47 -07:00
Wyatt Hoodes 0d20e0e9d5 test-backend: Move `test-upload` dirs to `var/<uuid>/test-backend`. 2019-06-24 12:23:09 -07:00
Wyatt Hoodes 3c1982ecab wsgi.py: Fix broken error ouput for django.setup() call.
It's common for a broken settings.py file to result in Django not
starting, thus never writing to `/var/log/zulip/errors.log`.  Such
behavior can be discouraging when the server 500s without a traceback
to accompany it.  To fix this, we simply catch any exceptions in
django.setup(), if raised, and log the exception appropriately.

Comments tweaked by tabbott.

Closes: #7032.
2019-06-24 12:19:51 -07:00
Shubham Padia 80a3651cf3 auth: Let user choose emails in GitHub auth.
Previously, our Github authentication backend just used the user's
primary email address associated with GitHub, which was a reasonable
default, but quite annoying for users who have several email addresses
associated with their GitHub account.

We fix this, by adding a new screen where users can select which of
their (verified) GitHub email addresses to use for authentication.

This is implemented using the "partial" feature of the
python-social-auth pipeline system.

Each email is displayed as a button. Clicking on that button chooses
the email. The email value is stored in a hidden input above the
button. The `primary_email` is displayed on top followed by
`verified_non_primary_emails`. Backend name is also passed as
`backend` to the template, which in our case is GitHub.

Fixes #9876.
2019-06-23 21:27:04 -07:00
Tim Abbott ff38d125b1 settings: Configure LDAP avatar synchronization in dev environment.
This should make it a lot more convenient to do manual testing of
these common LDAP configuration options.
2019-06-17 13:18:36 -07:00
Vishnu Ks cc91e6cb94 auth: Allow setting GOOGLE_OAUTH2_CLIENT_ID from dev-secrets.
This makes it much more convenient to use Google/GitHub authentication
in a Zulip development environment for testing; one only has to set it
up once.
2019-06-14 10:16:26 -07:00
vinitS101 a6eda858d0 ldap: Fix avatar sync not working with the S3 backend.
This fixes an issue that caused LDAP synchronization to fail for
avatars.  The problem occurred due to the lack of a 'name' attribute
on the BytesIO object that we pass to the upload backend (which is
only used in the S3 backend for computing Content-Type).

Fixes #12411.
2019-06-13 15:12:13 -07:00
Mateusz Mandera cbee5beeac retention: Log progress through the archiving process. 2019-06-13 11:02:11 -07:00
Tim Abbott f5375adf8f settings: Improve example for RABBITMQ_HOST.
We prefer 127.0.0.1 over localhost for RabbitMQ because of weird IPv6
issues, so we should avoid mentioning localhost as an example.
2019-06-06 16:41:15 -07:00
Tim Abbott 72bc497f7d urls: Fix URL definitions with missing ^/$ symbols.
This is mostly for consistency, though I also found it while writing
code to parse our regular expressions for validation purposes.
2019-06-06 12:58:10 -07:00
Puneeth Chaganti 717d1e504d ldap: Turn off the AUTH_LDAP_ALWAYS_UPDATE_USER setting.
The `AUTH_LDAP_ALWAYS_UPDATE_USER` is `True` by default, and this would sync the
attributes defined in the `AUTH_LDAP_USER_ATTR_MAP` to the user profile. But,
the default code in `django-auth-ldap` would work correctly only for `full_name`
field. This commit disables the setting by default, in favour of using the
`sync_ldap_user_data` script as a cron job.
2019-06-02 11:24:19 -07:00
Puneeth Chaganti a1590c613e url preview: Enable server level setting for url embed previews.
This significantly simplifies the process for enabling this feature in
new organizations.
2019-05-31 15:37:03 -07:00
Anders Kaseorg 802d3dbbf4 authenticate: Use keyword-only parameters.
Since positional arguments are interpreted differently by different
backends in Django's authentication backend system, it’s safer to
disallow them.

This had been the motivation for previously declaring the parameters
with default values when we were on Python 2, but that was not super
effective because Python has no rule against positional default
arguments and that convention for our authentication backends was
solely enforced by code review.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:49:54 -07:00
Anders Kaseorg 082f23a659 authenticate: Remove default values for required parameters.
It is now the caller’s responsibility to check that realm is not None.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:47:22 -07:00
Anders Kaseorg 725582850f login_or_register_remote_user: Remove unused invalid_subdomain parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:47:22 -07:00
vinitS101 4c8f9f07d7 uploads: Add comment explaining how to disable uploads.
Added a comment to explain how to disable file uploads, and what other
changes that will cause.
2019-05-24 14:47:04 -07:00
Mayank Madan d5e61e20e5 dev tooling: Add "create new user" and "create new realm" buttons.
Significantly tweaked by tabbott to clean up and expand the tests.

Fixes: #6018.
2019-05-21 15:03:37 -07:00
Hemanth V. Alluri 8214d65336 devtools: Add support for send_all in the integrations dev panel.
Using this feature a reviewer can easily send and view all fixture
messages for any given integration - with only JSON fixtures that is.
2019-05-20 12:29:10 -07:00
Hemanth V. Alluri bae8295c52 devtools: Add integrations dev panel.
This commit adds a new developer tool: The "integrations dev panel"
which will serve as a replacement for the send_webhook_fixture_message
management command as a way to test integrations with much greater ease.
2019-05-15 13:07:44 -07:00
Rishi Gupta c40eb524de org settings: Change a few references to Jitsi to Jitsi Meet.
Jitsi Meet is the correct name for the product we integrate with. There is
one other reference to Jitsi, but it's in the db and will require a
migration.
2019-05-08 15:10:21 -07:00
Puneeth Chaganti b82ae987ef digest: Enable server level setting for sending digest emails. 2019-05-08 14:39:12 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
Puneeth Chaganti da33b72848 url preview: Use in-memory caching in dev environment. 2019-05-06 12:37:32 -07:00
Shubham Padia 8c01f396db settings: Add AVATAR_CHANGES_DISABLED server setting.
Fixes #12132.
Realm setting to disable avatar changes is already present.
The `AVATAR_CHANGES_DISABLED` setting now follows the same
2-setting model as `NAME_CHANGES_DISABLED`.
2019-05-03 12:58:51 -07:00
Eeshan Garg a73e8109b7 webhooks: Remove the legacy GitHub integration.
The github-services model for how GitHub would send requests to this
legacy integration is no longer available since earlier in 2019.
Removing this integration also allows us to finally remove
authenticated_api_view, the legacy authentication model from 2013 that
had been used for this integration (and other features long since
upgraded).

A few functions that were used by the Beanstalk webhook are moved into
that webhook's implementation directly.
2019-04-27 15:13:44 -07:00
Wyatt Hoodes d4715f23d7 public_export: Add backend API endpoint for triggering export.
An endpoint was created in zerver/views.  Basic rate-limiting was
implemented using RealmAuditLog.  The idea here is to simply log each
export event as a realm_exported event.  The number of events
occurring in the time delta is checked to ensure that the weekly
limit is not exceeded.

The event is published to the 'deferred_work' queue processor to
prevent the export process from being killed after 60s.

Upon completion of the export the realm admin(s) are notified.
2019-04-26 17:24:29 -07:00
Tim Abbott d1b8497afb settings: Fix comment explaining NOREPLY_EMAIL_ADDRESS.
This documentation comment incorrectly claimed that we don't use
NOREPLY_EMAIL_ADDRESS when ADD_TOKENS_TO_NOREPLY_ADDRESS=True.

Fixes #12169.
2019-04-23 14:46:39 -07:00
Tim Abbott 435477c703 settings: Document changing LOCAL_UPLOADS_DIR.
The main point here is that you should use a symlink rather than
changing it, since it's more maintenance work to update our nginx
configuration to use an alternative path than to just create a
symbolic link.

Fixes #12157.
2019-04-22 14:33:01 -07:00
Shubham Padia 7743fa5297 auth: Redirect deactivated user to /login when attempting social login. (#12130) 2019-04-17 12:28:57 -07:00
Shubham Padia 3c09f226a4 auth: Redirect deactivated users with error for social auth backend.
Fixes #11937.

Also extracts the error message for a deactivated account to
`DEACTIVATED_ACCOUNT_ERROR`.
2019-04-13 19:58:15 -07:00
Tim Abbott 701cc524b0 settings: Set a default for SSO_APPEND_DOMAIN.
This means that if a user comments out the field in
/etc/zulip/settings.py, they won't get an error.
2019-04-12 10:48:05 -07:00
Tim Abbott 54915b76c7 urls: Clarify comment explaining /accounts/login URLs. 2019-03-25 14:12:51 -07:00
Hemanth V. Alluri 8239a3514a context_processors: Extract keys from zulip_default_context.
Previously, we had some expensive-to-calculate keys in
zulip_default_context, especially around enabled authentication
backends, which in total were a significant contributor to the
performance of various logged-out pages.  Now, these keys are only
computed for the login/registration pages where they are needed.

This is a moderate performance optimization for the loading time of
many logged-out pages.

Closes #11929.
2019-03-25 14:05:36 -07:00
Tim Abbott ca04b4f243 settings: Move production noreply templates.
These previously lived in Optional settings, which generally caused
users to not read it.

(Also do a bit of reorganization of the "optional settings" area).
2019-03-21 17:10:11 -07:00