Commit Graph

3911 Commits

Author SHA1 Message Date
Tim Abbott 626e191201 ldap: Add support for automatic user deactivation/reactivation.
As part of this, extend our documentation on synchronizing data from
Active Directory.
2018-12-13 16:24:15 -08:00
Tim Abbott 0a5221a819 ldap: Extract dev_ldap_directory.py.
This gets what is fundamentally unit testing code out of backends.py.
2018-12-13 16:24:15 -08:00
Rishi Gupta 8a95526ced billing: Always transition to Realm.LIMITED via do_change_plan_type.
Fixes a bug in import_realm where secondary attributes like message
visibility weren't being set, and also makes bugs like this less likely in
the future.

Also, putting the plan_type change at the end of import_realm, so that
future restrictions to LIMITED realms don't affect the import process.
2018-12-13 13:26:24 -08:00
Rishi Gupta b245c661da billing: Change do_change_plan_type to take a realm instead of a user.
More often than not, changes in plan type are not directly due to user
action.
2018-12-13 13:26:24 -08:00
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott 5d5b1afd2d rate_limiter: Improve handling of deadlocks.
We should rate-limit users when our rate limiter deadlocks trying to
increment its count; we also now log at warning level (so it doesn't
send spammy emails) and include details on the user and route was, so
that we can properly investigate whether the rate-limiting on the
route was in error.
2018-12-12 12:19:01 -08:00
Tim Abbott 27f3919259 push_notifications: Also treat ConnectionError as a retry.
It's a bug that the HTTP20 library isn't catching these, but we just
have to work around it.
2018-12-12 09:40:51 -08:00
Tim Abbott 914ae88c62 push_notifications: Expand APNS main try/except block.
We should be catching exceptions trying to send the notice as well as
in the result section.
2018-12-12 09:40:51 -08:00
Roman Godov 9c8eeaed85 digest_email: Add endpoint for rendering digest to the web.
Adds "/digest/" endpoint for rendering content of digest email
to the web.

Fixes #9974
2018-12-11 13:38:30 -08:00
Tim Abbott 15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Tim Abbott 14bfa74069 registration: Enable copying profile settings in production.
Now that we've styled this feature properly, this makes it possible to
copy various user-preferences type profile data in production when
making a new account with the same email address as an existing
account.
2018-12-10 16:55:07 -08:00
neiljp (Neil Pilgrim) 63768858ff dependencies: Upgrade mypy to version 0.650.
Broaden the type of the AbstractEnum __reduce_ex__ parameter to object; this
matches the parameter type specified in the latest enum.pyi file in typeshed.

Fixes #10996.
2018-12-09 22:20:12 -08:00
Marco Burstein ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
rht a1ff44a230 refactor: Add a helper function to create s3 buckets.
This refactor makes upgrading boto to boto3 easier.
Based on 43d2f6286c
2018-12-07 13:58:11 -08:00
rht a0dbcde063 export_files_from_s3: Move saving s3 object to local file to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
rht 0ddb242583 export_files_from_s3: get s3 object info in dict to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
rht 1cecf0f142 export_files_from_s3: Move checking for s3 oject's metadata to a separate function.
This refactor makes upgrading boto to boto3 easier.
Based on 24bf813e8a
2018-12-07 11:37:46 -08:00
Joshua Pan 441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott 7e6f30caab actions: Extract send_user_email_update_event.
This will make it easy for us to reuse this event-sending logic for
the email_address_visibility change code path.
2018-12-06 16:26:46 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott fc1c146d31 export: Remove assertion on current working directory.
This command hasn't made deep assumptions about CWD for a long time,
and this enables users to run it through a symlink (etc.).

Fixes #10961.
2018-12-06 11:05:40 -08:00
Tim Abbott 460249704e send_email: Fix cleanup for incorrectly migrated jobs.
This logic, unfortunately, had a typo.
2018-12-05 16:30:28 -08:00
Tim Abbott 1adc40f014 import: Deduplicate functions for uploading to S3/files.
We've had a long stream of bugs existed because only one of these two
code paths was tested (usually the local uploads backend).  By
deduplicating these functions, we ensure that this category of bugs no
longer happens.

Following my recent refactor, this is just a straightforward merge,
with code for one or the other backend ending up inside an if
statement.
2018-12-05 16:15:01 -08:00
Tim Abbott c9b801efde import: Use the s3_path attribute for path_maps unconditionally.
While the s3_path is almost always the same as the path, structurally,
`path` is the location in the export object, whereas s3_path is the
URL path.
2018-12-05 16:15:01 -08:00
Tim Abbott f4c5a45f4f import: Fix S3 paths for imported avatar PNG.
Previously, we were incorrectly importing avatar PNGs to a filename
without the .png extension, resulting in them effectively not being
imported.

This was mitigated by the fact that we imported the originals and ran
the appropriate `ensure_` functions, but still a bug.
2018-12-05 16:15:01 -08:00
Tim Abbott 412dc8dcda import: Set last_modified in import_uploads_local.
This has no effect other than to make the S3 and local code paths more
nearly identical.
2018-12-05 16:15:01 -08:00
Tim Abbott d8d0492d64 import: Restructure uploads path logic to be more similar.
This is preparation for future deduplication of the two redundant
uploads backends.
2018-12-05 16:15:01 -08:00
Tim Abbott 671ceccd78 import: Deduplicate medium avatars special logic.
This requires a bit of care with upload_backend to avoid breaking how
we mock that class in our tests.
2018-12-05 16:15:01 -08:00
Tim Abbott 36b43a6d7a import: Deduplicate first block of import_uploads logic. 2018-12-05 16:15:01 -08:00
Tim Abbott f80bab58c0 import_realm: Add progress indicator for importing uploads.
This makes it easier to see how we're doing when uploading a very
large number of files.
2018-12-05 16:15:01 -08:00
Greg Price 60de598cb5 user_agent: Use re.X to make the regex a little more readable. 2018-12-05 14:58:13 -08:00
Tim Abbott 380231af9d push_notifications: Add tests for BrokenPipeError case.
This was missing in d723dbfef7.
2018-12-05 10:44:25 -08:00
Tim Abbott b47535d8bb push notifications: Fix exception when handling deleted messages.
If a user deletes message between when it triggered a potential push
notification for another user, and when that notification was actually
sent, we'd end up with a situation where the `Message` table didn't
have an entry for the requested message ID.

The clean fix for this is to still throw an exception in the event
that the message doesn't exist at all, but if it exists in
ArchivedMessage, don't throw a user-facing exception.
2018-12-05 10:38:37 -08:00
Tim Abbott d723dbfef7 push_notifications: Handle BrokenPipeError from APNS.
This seems to happen when Apple is having a partial outage on some of
their APNS shards; it should be treated like other networking errors
connecting to APNS (with an automatic retry).
2018-12-05 10:02:18 -08:00
Anders Kaseorg 9ba860b4f2 Avoid double redirects to /login and then to /login/.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-04 16:12:58 -08:00
Tim Abbott 02a79b677b send_email: Extract handle_email_format_changes and use.
Apparently, we have a second code path where we might try to call
send_email library functions on old data, namely in the
queue_processors codebase.  So we apply the same migration logic here.
2018-12-04 16:08:18 -08:00
Eeshan Garg 901100bebf webhooks/jira: Decode topics on our end.
The problem addressed in 006e47198c
resurfaces with topics when a user provides a custom topic in the
webhook URL.
2018-12-03 17:37:36 -08:00
Tim Abbott fa59573a65 mypy: Fix mypy errors in send_email logic.
mypy should be able to figure this out given the xor just above, but
it's not surprising that it doesn't.
2018-12-03 16:30:48 -08:00
Raymond Akornor 92dc3637df send_email: Add support for multiple recipients.
This adds a function that sends provided email to all administrators
of a realm, but in a single email. As a result, send_email now takes
arguments to_user_ids and to_emails instead of to_user_id and
to_email.

We adjust other APIs to match, but note that send_future_email does
not yet support the multiple recipients model for good reasons.

Tweaked by tabbott to modify `manage.py deliver_email` to handle
backwards-compatibily for any ScheduledEmail objects already in the
database.

Fixes #10896.
2018-12-03 15:12:11 -08:00
Tim Abbott 4c1e80032a push notifications: Fix unnecessary warning in development.
This prevents the warning about push notifications not being
registered for from being printed in development environment startup
by default.  In development, that's the expected state, and we don't
need to spam up the output with that notice.
2018-12-03 12:03:56 -08:00
Tim Abbott d96624490e bulk_add_subscriptions: Fix confusing access to user_profile.realm.
Previously, we frequently accessed user_profile.realm from outside the
loops that interact with UserProfile objects.  This variable reuse
outside the loop could be confusing and should be a style/lint
violation.

While in this case, the behavior was correct (in that all users in the
loops were within the same realm), extracting a separate `realm`
variable significantly clarifies what's going on here.
2018-12-03 10:38:10 -08:00
Tim Abbott 5696ad5490 sessions: Suppress coverage error until we can debug. 2018-12-01 10:08:57 -08:00
Tim Abbott 18e2118fc8 Revert "sessions: Stop using the cached_db sessions engine."
This reverts commit 2fa77d9d54.

Further investigation has determined that this did not fix the
password-reset problem described in the previous commit message;
meanwhile, it causes other problems.  We still need to track down the
root cause of the original password-reset bug.
2018-11-30 15:54:32 -08:00
Tim Abbott d0f71881f4 docs: Add detailed documentation on the process for sending messages.
This has long been something missing from our suite of documentation.
2018-11-29 16:25:35 -08:00
Vishnu Ks f3033207b9 emails: Cover all LDAP cases when emailing login details.
This provides a nice user experience for folks where we do know what
their LDAP credentials are.

Though we need to fix #10917 before the content in the email with be
correct.
2018-11-29 12:14:22 -08:00
Rohitt Vashishtha c4e50a34d3 bugdown: Refactor get_user to get_user_by_name.
Also adds a warning against the use of this function.
2018-11-29 10:19:08 -08:00
Hemanth V. Alluri 6b1d724f5c zerver: Add bugdown rendering for text custom profile fields.
This is the first step of letting users use Zulip markdown in their
SHORT_TEXT and LONG_TEXT custom profile fields, so that they can
include emphasis, links, etc.

This doesn't include any frontend logic yet, however.
2018-11-29 09:47:32 -08:00
Anders Kaseorg 1d15d72775 zerver/lib/export.py: Avoid shelling out for cp, rm, ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:28:17 -08:00
Rohitt Vashishtha 681368b937 bugdown: Refactor get_possible_mentions_info and related functions.
This commit changes the return type of get_possible_mentions_info  to a
list instead of a dict, thus disposing off the hacky logic of storing
users with duplicate full names with name|id keys that made the code
obfuscated.

The other functions continue to use the dicts as before, however, there
are minor variable changes where needed in accordance with the updated
definition of get_possible_mentions_info.
2018-11-28 14:07:52 -08:00
Tim Abbott e4946dd182 bugdown: Rename full_names to mention_texts.
This is another straight variable rename, which will help clarify the
upcoming commits.
2018-11-28 14:07:23 -08:00