Commit Graph

5985 Commits

Author SHA1 Message Date
Umair Khan d95d34a66a Retry email failures in missed-message emails queue.
Fixes #6518.
2017-10-03 10:35:07 -07:00
Umair Khan 7d6ddaad26 mypy: Use Dict instead of Mapping in queues. 2017-10-03 10:35:07 -07:00
Umair Khan 60b8cba7df django: Bump version to 1.11.5. 2017-10-03 08:27:06 -07:00
Tim Abbott 7581e0e1c5 settings: Remove remaining references to REALMS_HAVE_SUBDOMAINS. 2017-10-02 16:43:54 -07:00
Tim Abbott 1d72629dc4 subdomains: Hardcode REALMS_HAVE_SUBDOMAINS=True. 2017-10-02 16:42:43 -07:00
Tim Abbott 95a0a4a06f subdomains: Update AuthBackendTest for subdomains always on.
This is separate from the main subdomains commit mainly for
readability of the history.
2017-10-02 16:32:11 -07:00
Tim Abbott 26a5332828 subdomains: Update some tests to run with subdomains.
Specifically, our request middleware adds an extra database query for
these code paths, to verify whether the subdomain is valid.
2017-10-02 16:32:11 -07:00
Tim Abbott 13bb546ddd subdomains: Hardcode can_add_realm_domains=True.
It was already always True if REALMS_HAVE_SUBDOMAINS, and now that's
the only supported configuration.
2017-10-02 16:32:10 -07:00
Tim Abbott 66658bbf25 subdomains: Hardcode REALMS_HAVE_SUBDOMAINS in API docs. 2017-10-02 16:32:10 -07:00
Tim Abbott 262a0692ab subdomains: Delete completely_open helper function.
We no longer use this anyway.
2017-10-02 16:32:10 -07:00
Tim Abbott e9f13a91e7 subdomains: Remove get_unique_non_system_realm.
This functionality is no longer required with REALMS_HAVE_SUBDOMAINS
always True.
2017-10-02 16:32:10 -07:00
Tim Abbott e6f8032972 subdomains: Remove get_unique_open_realm code paths.
Since we no longer support !REALMS_HAVE_SUBDOMAINS in production,
these no longer make sense.
2017-10-02 16:32:10 -07:00
Tim Abbott 99e4ceebd4 test_auth_backends: Extract REALMS_HAVE_SUBDOMAINS overrides.
This will make the diff a lot smaller when we hardcode
REALMS_HAVE_SUBDOMAINS=True.
2017-10-02 16:32:09 -07:00
Tim Abbott 83a226ad3d do_send_message: Replace get_user_profile_by_email.
get_system_bot is the new way to do this sort of thing.
2017-10-02 16:15:36 -07:00
Tim Abbott bd066fe0f1 test_subdomain_restrictions: Update to assume subdomains. 2017-10-02 15:32:39 -07:00
Tim Abbott e3c395f247 webhooks: Remove use of get_user_profile_by_email.
This function is deprecated.
2017-10-02 15:28:44 -07:00
Eeshan Garg 32bd609361 webhooks/yo: Migrate to check_send_private_message. 2017-10-02 15:27:26 -07:00
Eeshan Garg d20b57c20d webhooks/teamcity: Migrate to check_send_private_message. 2017-10-02 15:27:26 -07:00
Eeshan Garg 763437cc9c backend: Introduce check_send_private_message.
check_send_private_message utilizes Addressee.for_user_profile()
to send a private message to a user_profile.
2017-10-02 15:27:26 -07:00
Steve Howell 2be713a7e4 Rename get_userids_for_missed_messages().
We rename this function to get_active_presence_idle_userids().
2017-10-02 15:19:28 -07:00
Steve Howell e660428c21 Rename missed_message_userids to presence_idle_userids. 2017-10-02 15:19:28 -07:00
Steve Howell 32d5e297a3 Rename get_idle_userids to filter_presence_idle_userids.
We have two different concepts of "idle", and this function
is based on the "presence" aspect of idleness.  There is also
idleness in terms of a user having no current client
descriptors accepting messages, and we check that later in
the process for things like sending missed message emails.
2017-10-02 15:19:28 -07:00
Tim Abbott e5592fe89f test_signup: Remove REALMS_HAVE_SUBDOMAINS check.
This test actually is just as correct with
REALMS_HAVE_SUBDOMAINS=True.
2017-10-02 13:50:07 -07:00
Tim Abbott 9622da6b16 test_signup: De-inline REALMS_HAVE_SUBDOMAINS=True checks.
These are now just declarations at the top of functions.
2017-10-02 13:50:06 -07:00
Tim Abbott edc10e8f51 test_decorators: de-inline REALMS_HAVE_SUBDOMAINS with statements.
This will significantly shrink the diff when we toggle
REALMS_HAVE_SUBDOMAINS soon.
2017-10-02 13:49:53 -07:00
Tim Abbott 5a27a4cf1c decorator: Improve wrong-subdomain logging output.
Often, it's helpful to see both values in errors.
2017-10-02 13:31:20 -07:00
Vishnu Ks 6b9755c854 email_log: Make generate_email work with subdomains.
This fixes one of our last few test failures with
REALMS_HAVE_SUBDOMAINS.
2017-10-02 12:27:27 -07:00
Tim Abbott 6327f30c76 test_decorators: Remove test_login_deactivated_realm.
This test was a duplicate of test_login_deactivated in LoginTest.
2017-10-02 11:42:15 -07:00
derAnfaenger d1afab7199 Replace deprecated Logging.warn calls with Logging.warning. 2017-10-02 11:11:42 +02:00
Tim Abbott 7d63928fe1 test_signup: Update deactivate realm tests for subdomains.
We add a new test to make sure we cover the potential race condition
with the realm being deactivated after starting the registration
process.
2017-10-01 23:52:08 -07:00
Tim Abbott 651b1882e6 test_outgoing_webhook_system: Set REALMS_HAVE_SUBDOMAINS=True. 2017-10-01 22:47:15 -07:00
Eeshan Garg 86c2c7ad34 webhooks: Migrate to check_send_stream_message.
This commit migrates all webhooks to use check_send_stream_message
instead of check_send_message. The only two webhooks that still
use check_send_message are our yo and teamcity webhooks. They
both use check_send_message for private messages.
2017-09-30 17:48:55 -07:00
Eeshan Garg 24aff0d0a2 backend: Introduce check_send_stream_message.
check_send_stream_message is a simpler version of
check_send_message for sending messages where the addressee is
a stream. Instead of relying on Addressee.legacy_build,
check_send_stream_message uses Addressee.for_stream. Consequently,
it eschews many of check_send_message's kwargs that aren't needed
when the intended recipient of a message is a stream.
2017-09-30 17:48:55 -07:00
Umair Khan 69ccc8ce0e ldap: Show helpful message when realm is None. 2017-09-30 10:18:25 -07:00
Robert Hönig 9d4bffb59d outgoing webhook: Make exception handling more granular and verbose.
Fixes #6127.
2017-09-30 10:14:28 -07:00
Robert Hönig 93ebd1660f outgoing webhook: Make notify_bot_owner mypy annotation more specific. 2017-09-30 10:12:31 -07:00
Robert Hönig ba598def0b outgoing webhook: Don't retry on 50x response.
This reverts one retry case introduced in 53a8b2a.
2017-09-30 10:12:31 -07:00
rht 6dd94becb3 zerver/management: Remove unused optparse import. 2017-09-30 09:22:08 -07:00
Tim Abbott f0c8e75f57 settings: Remove obsolete VERBOSE_SUPPORT_OFFERS.
We removed the actual functionality of emails differing depending on
this setting several months ago.
2017-09-29 22:35:12 -07:00
Tim Abbott 94ba678f5b email_mirror: Clean up EMAIL_GATEWAY_EXAMPLE setting.
This isn't something that a user can ever modify, so it doesn't belong
in DEFAULT_SETTINGS.  While we're at it, we align the appearance of
the email gateway in the docs with whether this setting in the docs
will be valid.
2017-09-29 21:32:26 -07:00
Rishi Gupta 0335d8dca7 authentication: Update error message for deactivated user. 2017-09-29 12:32:46 -07:00
Harshit Bansal 5b5bcce098 emoji: Migrate bugdown emoji to use sprite sheets.
This commit switches to use sprite sheets for rendering emojis
in all the remaining places, i.e., message bodies and composebox
typeahead. This commit also includes some changes to notifications.py
file so that the spans used for rendering emojis can be converted
to corresponding image tags so that we don't break the emoji rendering
in missed message emails since we can't use sprite sheets there.

As part of switching the bugdown system to use sprite sheets, we need
to switch the name_to_codepoint mappings to match the new sprite
sheets.  This has the side effect of fixing a bunch of emoji like
numbers and flag emoji in the emoji pickers.

Fixes: #3895.
Fixes: #3972.
2017-09-29 11:14:34 -07:00
Greg Price 2a832386dc queue: Move more received-event logs to DEBUG level.
These can spew a lot of noise when starting the dev server, and aren't
in a super helpful form for normal server administration anyway.
2017-09-28 18:32:42 -07:00
Greg Price 0d5f77cf86 logging: Abbreviate log-level names to 4 characters.
These are long enough to still be self-explanatory (the only one I'm
at all in doubt about there is DEBG; I avoided "DBUG" because it reads
"BUG" which suggests a high-priority message, and those are the
opposite of that), while saving a good bit of horizontal space
vs. padding everything to the 8 characters of "CRITICAL".

Also add a linter exception to allow easy-to-read alignment here,
similar to several existing exceptions for other alignment cases.
2017-09-28 18:26:39 -07:00
Greg Price 412f6e70d8 logging: Add option to show the module a log message came from. 2017-09-28 18:26:39 -07:00
Greg Price cc288dacc5 logging: Abbreviate logger name for some extremely common logs.
These are the records that we print on each web request.
2017-09-28 18:26:39 -07:00
Greg Price 71b8e7e8d4 logging: Attribute each log message to the logger it came through.
This also gives us a place to hang the originating module, if we write a bit
of logic to work that out; sadly it doesn't come out of the box, only
the filename (which is likely to have a bunch of noise that just shows the
path to the deployment or virtualenv.)
2017-09-28 18:26:39 -07:00
Greg Price bfc245574c logging: Show ms with a decimal point, not comma.
The rest of these messages in the server logs are in English;
the numbers should be localized to match.
2017-09-28 18:26:39 -07:00
Greg Price 0163920577 logging: Make our own Formatter class.
This doesn't yet do much, but it gives us a suitable place to
add code to customize how log messages are displayed, beyond what
a format string passed to the default formatter can do.
2017-09-28 18:26:39 -07:00
Greg Price e0a5e6fade logging: Explain quirks of Python logging config, and apply a style.
This should make it a little easier to understand our logging config
and make changes to it with confidence.

Many of these items that are now redundant used to be required when we
were setting disable_existing_loggers to True (before 500d81bf2), in
order to exempt those loggers from being cleared out.  Now they're not.

One bit of test code needed a tweak to how it got its hands on the
AdminZulipHandler instance; it can do it from the list on the root
logger just as well as on the `django` logger.
2017-09-28 18:26:39 -07:00
Greg Price f129dc4f72 LDAP: Restore an except clause and add test to cover it.
Most of the paths leading through this except clause were cut in
73e8bba37 "ldap auth: Reassure django_auth_ldap".  The remaining one
had no test coverage -- the case that leads to it had a narrow unit
test, but no test had the exception actually propagate here.  As a
result, the clause was mistakenly cut, in commit
8d7f961a6 "LDAP: Remove now-impossible except clause.", which could
lead to an uncaught exception in production.

Restore the except clause, and add a test for it.
2017-09-28 18:26:39 -07:00
rht f22daeba1a Cleanup rabbitmq queues when doing dev db rebuild.
Modified by tabbott to cleanup the argument handling in the modified
purge_queue management command.

Fixes #1335.
2017-09-28 18:18:54 -07:00
Tim Abbott 630ecf9669 queue: Move user presence log events to DEBUG log level.
These are pretty noisy, and generally of little interest, since
presence requests come with a server log entry anyway.
2017-09-28 14:35:55 -07:00
Steve Howell aaaaa66d4c refactor: Move default_sending_stream logic to Addressee.
Having Addressee take care of setting stream_name to
sender.default_sending_stream.name makes us able to have
the invariant that stream_name is never None when the
message type is 'stream', which will help for mypy, among
other things.

One thing to be aware of is that Addressee does do a little
bit of validation work, and this adds yet another JsonableError
exception.  I don't view this as a bad thing, just something to
know.
2017-09-28 12:14:08 -07:00
Eeshan Garg da3c008872 webhooks/trello: Remove trello/view/templates.py.
TRELLO_MESSAGE_TEMPLATE and TRELLO_SUBJECT_TEMPLATE are
redundant. This commit removes them. Now, subjects don't end
in periods. And where a period is necessary in the message body,
one is appended at the end of the specific template for that
message.
2017-09-28 10:58:07 -07:00
Marco Matarazzo a6b83def8f webhooks/trello: Support updating Card descriptions.
We now support events that add, change or remove Trello Card
descriptions.

Minor fixes by eeshangarg.
2017-09-28 10:58:07 -07:00
Marco Matarazzo 765f8e7d41 webhooks/trello: Display Card comment's text in notifications.
Now, when someone makes a comment on a Trello Card, the text
of the comment is quoted in the notification.

Minor fixes by eeshangarg.
2017-09-28 10:58:07 -07:00
rht 4975122338 zerver/migrations: Remove from __future__ import *. 2017-09-28 10:16:18 -07:00
rht 426548097e zerver/tornado: Remove from __future__ import *. 2017-09-28 10:14:03 -07:00
rht 9f944d9449 zerver/lib: Remove from __future__ import *. 2017-09-28 10:14:03 -07:00
Greg Price 7b8f725707 APNs: Accept 1.6-format payloads in bouncer.
This is just enough of a quick fix to work with a stock Zulip 1.6
server.  We should really also make this robust to arbitrary input
from the remote Zulip server, even though it'll be a little tedious.
2017-09-28 10:01:16 -07:00
Tim Abbott 0744961afe docs: Add more comprehensive documentation on error reporting.
Fixes #2232.
2017-09-27 19:47:08 -07:00
rht 241e318eba zerver/tornado: remove `import six`. 2017-09-27 19:10:28 -07:00
rht 035ed93111 zerver/lib: remove `import six`. 2017-09-27 19:10:28 -07:00
rht c2290b0328 Remove the rest of print_function. 2017-09-27 18:06:47 -07:00
rht 32650c8fdc zerver/management: Remove print_function. 2017-09-27 18:05:45 -07:00
rht 26f5d9a32c zerver/tests: Remove print_function. 2017-09-27 18:05:45 -07:00
rht b8aa92194c zerver/tornado: Remove print_function. 2017-09-27 18:05:45 -07:00
rht 2e12fe5e2e zerver/lib: Remove print_function. 2017-09-27 18:05:45 -07:00
rht e4c824ed61 zerver/views: Remove print_function. 2017-09-27 18:05:45 -07:00
Vishnu Ks 17cc32f168 settings: Show error or success message when deleting uploaded files.
Fixes #6519
2017-09-27 17:50:22 -07:00
Vishnu Ks 94e099eaab auth: Make multiuse invite link work with oAuth2.
This works by attaching to the user's session the multi-use invitation
key, allowing that to be used in the Google/GitHub auth flows.
2017-09-27 17:47:40 -07:00
Steve Howell de0b47fd4e Always notify service bots about stream mentions.
Before this change, we were only triggering service bots
for stream mentions when the bot was subscribed to the
stream.
2017-09-27 17:22:12 -07:00
rht daef6a7dbc zerver/tests: Remove `import six`. 2017-09-27 17:07:40 -07:00
rht a7803cbf1b zerver/tornado: remove `import six`.
Tweaked by tabbott to remove the changes in
zerver/tornado/handlers.py, which are hard to test.
2017-09-27 17:07:08 -07:00
rht 0fa3c1d876 zerver/management: Remove `import six`. 2017-09-27 17:06:16 -07:00
rht 2ae47525d3 zerver/worker: Remove `import six`. 2017-09-27 17:06:16 -07:00
rht 106375bd68 zerver/views: remove `import six`. 2017-09-27 17:03:03 -07:00
Tim Abbott 28eaf5620e emails: Use common_context for email change notifications.
This also lets us remove `realm_uri`.
2017-09-27 16:48:18 -07:00
Vishnu Ks dfaa939af2 emails: Store the compiled emails in separate directory. 2017-09-27 16:48:18 -07:00
Vishnu Ks caaa90f22d emails: Extract CSS from email_base_default. 2017-09-27 16:48:18 -07:00
Vaida Plankyte 207e06d558 frontend: Add header & body illustrations to emails. 2017-09-27 16:32:43 -07:00
Steve Howell 1b518f1983 Return mentioned users in get_user_info_for_message_updates().
The dictionary result for get_user_info_for_message_updates()
now has a `mention_user_ids` field that is a set of user ids
who were mentioned in a message.
2017-09-27 16:01:50 -07:00
Steve Howell 646abb57b7 refactor: Extract get_user_info_for_message_updates.
We'll want to expand this to get users that were mentioned in
the prior message, but this commit is just a refactoring.
2017-09-27 16:01:50 -07:00
Steve Howell 5abf52de71 Extract get_idle_userids().
This function will help us send missed-message mails for
updates, in a future commit.
2017-09-27 16:01:50 -07:00
Tim Abbott 06e6b3abe1 event_queue: Fixed stream_push_notify code path in missedmessage_hook.
This finishes implementing the stream_push_notify feature, which
previously didn't work if you were not idle when the message was sent.
2017-09-27 15:57:48 -07:00
Tim Abbott 2b1ea1555c event_queue: Simplify API of maybe_enqueue_notifications. 2017-09-27 15:57:47 -07:00
Tim Abbott fa461c6a4f missedmessage_hook: Fix name of ClientDescription argument.
This was previously incorrectly named to suggest it was the EventQueue
object.
2017-09-27 15:57:47 -07:00
Tim Abbott d6732d0781 event_queue: Fix handling of private messages in missedmessage_hook.
Previously, we didn't correctly trigger email/push notifications for
PMs in these cases.
2017-09-27 15:57:47 -07:00
Tim Abbott bfa8f0efdb tests: Add an end-to-end test for missedmessage_hook.
This checks what arguments it passes into the enqueuing function.

Note, however, that the arguments are wrong for various cases, we'll
update the tests as we fix those bugs.
2017-09-27 15:57:47 -07:00
Tim Abbott 71fc8363b4 tests: Add a test suite for maybe_enqueue_notifications.
This ensures that as we expand the logic for under what circumstances
email and push notifications should be sent, we can be confident about
this code path always doing the right thing.
2017-09-27 15:55:29 -07:00
Tim Abbott 830d3e8d07 event_queue: Use maybe_enqueue_notifications in missedmessage_hook.
This fixes a problem introduced in the recent refactoring where
`triggers` would not be set correctly when a push or email
notification was triggered by missedmessage_hook.

Fixes #6612.
2017-09-27 15:55:29 -07:00
Tim Abbott 9749826e9c event_queue: Refactor code for checking if a message was mentioned.
Now, the two code paths do the same thing for this check.

It seems like there may be more work to do here, in that
wildcard_mentioned messages seem to not be eligible for sending
email/push notifications.  We probably want to add some logic there
for the user doing the mention to control whether or not it does.
2017-09-27 15:55:29 -07:00
Tim Abbott f68089fbe1 event_queue: Extract maybe_enqueue_notifications.
Conceptually, this code path should be shared between the
process_message handler and the garbage-collection hook.
2017-09-27 15:55:29 -07:00
Tim Abbott c3911e3fb6 test_classes: Improve error handling for send_message. 2017-09-27 15:55:29 -07:00
rht 2949d1c1e8 zerver: Remove the rest of absolute_import. 2017-09-27 10:02:39 -07:00
rht 7115a29bee zerver/webhooks: Remove absolute_import. 2017-09-27 10:00:39 -07:00
rht e239e97351 zerver/management: Remove absolute_import. 2017-09-27 10:00:39 -07:00
rht 41d8db373a zerver/worker: Remove absolute_import. 2017-09-27 10:00:39 -07:00
rht 1e87a4b68c zerver/tests: Remove absolute_import. 2017-09-27 10:00:39 -07:00