Commit Graph

9026 Commits

Author SHA1 Message Date
Tim Abbott 32f24bff8d stripe: Fix exception handling for suppressed event types. 2018-12-15 14:14:09 -08:00
Tim Abbott 930e65d1be push: Include type in add-push-notification events.
This should make us able to clean up the logic for this in the future
(right now, we still need to do the .get() for backwards compatibility).
2018-12-15 13:58:52 -08:00
ishanrai05 4105fb683b notifications: Optimize push notifications code path in tests.
This checks if push_notification_enabled() is set to false in
handle_push_notification and adds an early return statement.

This is a significant performance optimization for our unit tests
because the push notifications code path does a number of database
queries, and this migration means we don't end up doing those queries
the hundreds of times we send PMs or mentions in our tests where we're
not trying to test the push notifications functionality.

This should also have a small message sending scalability improvement
for any Zulip servers without push notifications enabled.

Tweaked by tabbott to fix a few small issues.

Fixes #10895.
2018-12-15 11:12:43 -08:00
Tim Abbott a63eae48cc test_push_notifications: Fix leak that can leak to test flakes.
While reviewing #11012, I discovered a nondeterministic result for
test_signup, which I tracked down to specifically this triple of tests
failing when run in this order:

test-backend GCMSuccessTest \
  zerver.tests.test_push_notifications.TestAPNs.test_get_apns_client \
  zerver.tests.test_signup.LoginTest.test_register

with a query count mismatch like this:

expected length: 73
actual length: 79

Comparing the list of queries, it's clear that test_register was
seeing `push_notifications_enabled()` returning True in this test order.

It's not clear why GCMSuccessTest was required here (it was!), but
further debugging determined the problem was that
`test_get_apns_client` left the _apns_client initialization system in
a state where get_apns_client would return a non-None value, resulting
in push_notifications_enabled() returning True for future tests.

The immediate fix is to just reset the `_apns_client` and
`_apns_client_initializedstate` state properly after the test runs;
but arguably we should do a larger refactor to make this less
fragile.
2018-12-15 11:12:43 -08:00
Rishi Gupta 111eda604b portico: Add /atlassian for Zulip users migrating from HipChat. 2018-12-14 23:42:47 -08:00
Roger Souza 69da22d998 api docs: Document the custom emoji upload endpoint.
Tweaked by tabbott to fix some English phrasing and make the file
pointer thing require less Python knowledge.

Fixes: #10746.
2018-12-14 14:19:28 -08:00
Tim Abbott 67981725ec send_email: Fix migration code path for ScheduledEmail.
The previous migration code path was broken in two ways:

* ScheduledEmail objects generally contain a `None` value for
  whichever of `to_user_id` and `to_email` isn't in use; this could
  result in us sending a [None] to send_email(), which doesn't make
  sense.

* We were calling handle_send_email_format_changes in the wrong order
  with respect to the JSON loading process.

Thanks to Tom Daff for the report!
2018-12-14 12:46:31 -08:00
Tim Abbott 8c30c36006 realm filters: Expand set of characters allowed in prefixes.
Our list of allowed characters in realm filter patterns has long been
too string; fix this by extending the pattern.

Also, extend the tests to have examples of actual strings one would
use with the patterns, for clarity.

Fixes #10953, fixes #6835.
2018-12-14 11:33:13 -08:00
Tim Abbott 626e191201 ldap: Add support for automatic user deactivation/reactivation.
As part of this, extend our documentation on synchronizing data from
Active Directory.
2018-12-13 16:24:15 -08:00
Tim Abbott 81271b0d20 sync_ldap_user_data: Process deactivated users as well.
Technically, we will only need to process deactivated users for the
purpose of reactivating them (and can ignore, e.g., name changes).
But it's simplest to just process them unconditionally.
2018-12-13 16:24:15 -08:00
Tim Abbott 0a5221a819 ldap: Extract dev_ldap_directory.py.
This gets what is fundamentally unit testing code out of backends.py.
2018-12-13 16:24:15 -08:00
Tim Abbott 5dd646f33f ldap: Add support for syncing avatar images from LDAP.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.

This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.

Fixes #286.
2018-12-13 13:39:22 -08:00
Rishi Gupta 8a95526ced billing: Always transition to Realm.LIMITED via do_change_plan_type.
Fixes a bug in import_realm where secondary attributes like message
visibility weren't being set, and also makes bugs like this less likely in
the future.

Also, putting the plan_type change at the end of import_realm, so that
future restrictions to LIMITED realms don't affect the import process.
2018-12-13 13:26:24 -08:00
Rishi Gupta b245c661da billing: Change do_change_plan_type to take a realm instead of a user.
More often than not, changes in plan type are not directly due to user
action.
2018-12-13 13:26:24 -08:00
Tim Abbott e7746809d2 stripe: Fix exception handler for suppressed events.
Apparently, we incorrectly placed the try/except block around the
common code, not the code that can actually raise these exceptions.
2018-12-13 10:22:19 -08:00
Tim Abbott a6ca95dfc4 slack import: Fix all messages being imported to one channel.
This was an ugly variable-escape-from-loop regression introduced in
e59ff6e6db.
2018-12-12 17:54:37 -08:00
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Tim Abbott d6217eb862 slack import: Fix empty values for custom profile fields.
The Slack import process would incorrectly issue
CustomProfileFieldValue entries with a value of "" for users who
didn't have a given CustomProfileField (especially common for the
"skype" and "phone" fields).  This had no user-visible effect, but
certainly added some clutter in the database.
2018-12-12 12:58:27 -08:00
Tim Abbott 5d5b1afd2d rate_limiter: Improve handling of deadlocks.
We should rate-limit users when our rate limiter deadlocks trying to
increment its count; we also now log at warning level (so it doesn't
send spammy emails) and include details on the user and route was, so
that we can properly investigate whether the rate-limiting on the
route was in error.
2018-12-12 12:19:01 -08:00
Tim Abbott 8a11c94a2d fakeldap: Add thumbnailPhoto/jpegPhoto for testing avatar syncing. 2018-12-12 11:23:33 -08:00
Tim Abbott e9900b2bdf gitter: Do something reasonable with invalid fullnames. 2018-12-12 10:07:52 -08:00
Tim Abbott 27f3919259 push_notifications: Also treat ConnectionError as a retry.
It's a bug that the HTTP20 library isn't catching these, but we just
have to work around it.
2018-12-12 09:40:51 -08:00
Tim Abbott 914ae88c62 push_notifications: Expand APNS main try/except block.
We should be catching exceptions trying to send the notice as well as
in the result section.
2018-12-12 09:40:51 -08:00
Roman Godov 9c8eeaed85 digest_email: Add endpoint for rendering digest to the web.
Adds "/digest/" endpoint for rendering content of digest email
to the web.

Fixes #9974
2018-12-11 13:38:30 -08:00
Tim Abbott 15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Tim Abbott b7127a395b server_settings: Use a reasonable default when User-Agent not specified.
Apparently, some Zulip clients don't send a User-Agent in their
requests to /server_settings, and this makes that not 500.
2018-12-11 11:30:11 -08:00
Tim Abbott 2a39852e91 auth: Add logging for which client does /server_settings requests.
This is generally helpful for debugging.
2018-12-11 11:30:11 -08:00
Tim Abbott 88368996fb decorator: Rename remote_server_request to skip_update_user_activity.
This makes it reasonable to call process_client from other contexts,
like /server_settings, where we don't have an authenticated
UserProfile object.
2018-12-11 11:30:11 -08:00
Tim Abbott 14bfa74069 registration: Enable copying profile settings in production.
Now that we've styled this feature properly, this makes it possible to
copy various user-preferences type profile data in production when
making a new account with the same email address as an existing
account.
2018-12-10 16:55:07 -08:00
Max Nussenbaum 536de666df register: Style avatar that shows when importing settings.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.

Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
2018-12-10 15:40:30 -08:00
Jack Zhang be9b6a6dee compatibility: Add a compatibility check to api_get_server_settings.
This should make it convenient for the mobile app to present errors of
the form "Your Zulip app is not new enough for this Zulip server".
2018-12-10 15:06:09 -08:00
Jack Zhang 8711fe7bae compatibility: Respond with HTTP 400 when no User-Agent header.
Prior to this commit, a request to /compatibility without a User-Agent
header would result in the server 500ing.
2018-12-10 15:04:34 -08:00
Tim Abbott a1d99adce7 auth: Handle SSO_APPEND_DOMAIN in remote_user SSO for mobile.
Apparently, while the main code path through
login_or_register_remote_user was correctly calling
remote_user_to_email(username) to get a proper email address for
situations where auth username != email (i.e. when SSO_APPEND_DOMAIN
is set), we neglected to do so in the mobile_flow_otp corner case.

Fixes #11005.
2018-12-10 10:36:15 -08:00
neiljp (Neil Pilgrim) 63768858ff dependencies: Upgrade mypy to version 0.650.
Broaden the type of the AbstractEnum __reduce_ex__ parameter to object; this
matches the parameter type specified in the latest enum.pyi file in typeshed.

Fixes #10996.
2018-12-09 22:20:12 -08:00
Marco Burstein ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
rht a1ff44a230 refactor: Add a helper function to create s3 buckets.
This refactor makes upgrading boto to boto3 easier.
Based on 43d2f6286c
2018-12-07 13:58:11 -08:00
rht a0dbcde063 export_files_from_s3: Move saving s3 object to local file to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
rht 0ddb242583 export_files_from_s3: get s3 object info in dict to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
rht 1cecf0f142 export_files_from_s3: Move checking for s3 oject's metadata to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
Joshua Pan 441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott 5c2e7a169f test_events: Use a better assertion when comparing event count. 2018-12-06 17:43:26 -08:00
Tim Abbott 7e6f30caab actions: Extract send_user_email_update_event.
This will make it easy for us to reuse this event-sending logic for
the email_address_visibility change code path.
2018-12-06 16:26:46 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 9d32857c37 realm: Add new Realm.email_address_visibility field.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
2018-12-06 15:56:30 -08:00
Rishi Gupta 82ee496efc webhooks/stripe: Update fixture for customer_subscription_created. 2018-12-06 11:29:16 -08:00
Rishi Gupta 76aebac568 webhooks/stripe: Format timestamps as dates. 2018-12-06 11:29:16 -08:00
Rishi Gupta 027d5e90c5 webhooks/stripe: Update how we handle updated events.
Also more explicitly ignores the Stripe Connect related events in the
'account' category.
2018-12-06 11:29:16 -08:00
Rishi Gupta 5fb683e788 webhooks/stripe: Use exceptions instead of json_success to exit main body. 2018-12-06 11:29:16 -08:00
Tim Abbott fc1c146d31 export: Remove assertion on current working directory.
This command hasn't made deep assumptions about CWD for a long time,
and this enables users to run it through a symlink (etc.).

Fixes #10961.
2018-12-06 11:05:40 -08:00