Commit Graph

2063 Commits

Author SHA1 Message Date
Tim Abbott e6f8032972 subdomains: Remove get_unique_open_realm code paths.
Since we no longer support !REALMS_HAVE_SUBDOMAINS in production,
these no longer make sense.
2017-10-02 16:32:10 -07:00
Tim Abbott 99e4ceebd4 test_auth_backends: Extract REALMS_HAVE_SUBDOMAINS overrides.
This will make the diff a lot smaller when we hardcode
REALMS_HAVE_SUBDOMAINS=True.
2017-10-02 16:32:09 -07:00
Tim Abbott bd066fe0f1 test_subdomain_restrictions: Update to assume subdomains. 2017-10-02 15:32:39 -07:00
Steve Howell 2be713a7e4 Rename get_userids_for_missed_messages().
We rename this function to get_active_presence_idle_userids().
2017-10-02 15:19:28 -07:00
Steve Howell e660428c21 Rename missed_message_userids to presence_idle_userids. 2017-10-02 15:19:28 -07:00
Tim Abbott e5592fe89f test_signup: Remove REALMS_HAVE_SUBDOMAINS check.
This test actually is just as correct with
REALMS_HAVE_SUBDOMAINS=True.
2017-10-02 13:50:07 -07:00
Tim Abbott 9622da6b16 test_signup: De-inline REALMS_HAVE_SUBDOMAINS=True checks.
These are now just declarations at the top of functions.
2017-10-02 13:50:06 -07:00
Tim Abbott edc10e8f51 test_decorators: de-inline REALMS_HAVE_SUBDOMAINS with statements.
This will significantly shrink the diff when we toggle
REALMS_HAVE_SUBDOMAINS soon.
2017-10-02 13:49:53 -07:00
Tim Abbott 5a27a4cf1c decorator: Improve wrong-subdomain logging output.
Often, it's helpful to see both values in errors.
2017-10-02 13:31:20 -07:00
Vishnu Ks 6b9755c854 email_log: Make generate_email work with subdomains.
This fixes one of our last few test failures with
REALMS_HAVE_SUBDOMAINS.
2017-10-02 12:27:27 -07:00
Tim Abbott 6327f30c76 test_decorators: Remove test_login_deactivated_realm.
This test was a duplicate of test_login_deactivated in LoginTest.
2017-10-02 11:42:15 -07:00
derAnfaenger d1afab7199 Replace deprecated Logging.warn calls with Logging.warning. 2017-10-02 11:11:42 +02:00
Tim Abbott 7d63928fe1 test_signup: Update deactivate realm tests for subdomains.
We add a new test to make sure we cover the potential race condition
with the realm being deactivated after starting the registration
process.
2017-10-01 23:52:08 -07:00
Tim Abbott 651b1882e6 test_outgoing_webhook_system: Set REALMS_HAVE_SUBDOMAINS=True. 2017-10-01 22:47:15 -07:00
Umair Khan 69ccc8ce0e ldap: Show helpful message when realm is None. 2017-09-30 10:18:25 -07:00
Robert Hönig 9d4bffb59d outgoing webhook: Make exception handling more granular and verbose.
Fixes #6127.
2017-09-30 10:14:28 -07:00
Harshit Bansal 5b5bcce098 emoji: Migrate bugdown emoji to use sprite sheets.
This commit switches to use sprite sheets for rendering emojis
in all the remaining places, i.e., message bodies and composebox
typeahead. This commit also includes some changes to notifications.py
file so that the spans used for rendering emojis can be converted
to corresponding image tags so that we don't break the emoji rendering
in missed message emails since we can't use sprite sheets there.

As part of switching the bugdown system to use sprite sheets, we need
to switch the name_to_codepoint mappings to match the new sprite
sheets.  This has the side effect of fixing a bunch of emoji like
numbers and flag emoji in the emoji pickers.

Fixes: #3895.
Fixes: #3972.
2017-09-29 11:14:34 -07:00
Greg Price e0a5e6fade logging: Explain quirks of Python logging config, and apply a style.
This should make it a little easier to understand our logging config
and make changes to it with confidence.

Many of these items that are now redundant used to be required when we
were setting disable_existing_loggers to True (before 500d81bf2), in
order to exempt those loggers from being cleared out.  Now they're not.

One bit of test code needed a tweak to how it got its hands on the
AdminZulipHandler instance; it can do it from the list on the root
logger just as well as on the `django` logger.
2017-09-28 18:26:39 -07:00
Greg Price f129dc4f72 LDAP: Restore an except clause and add test to cover it.
Most of the paths leading through this except clause were cut in
73e8bba37 "ldap auth: Reassure django_auth_ldap".  The remaining one
had no test coverage -- the case that leads to it had a narrow unit
test, but no test had the exception actually propagate here.  As a
result, the clause was mistakenly cut, in commit
8d7f961a6 "LDAP: Remove now-impossible except clause.", which could
lead to an uncaught exception in production.

Restore the except clause, and add a test for it.
2017-09-28 18:26:39 -07:00
Greg Price 7b8f725707 APNs: Accept 1.6-format payloads in bouncer.
This is just enough of a quick fix to work with a stock Zulip 1.6
server.  We should really also make this robust to arbitrary input
from the remote Zulip server, even though it'll be a little tedious.
2017-09-28 10:01:16 -07:00
rht 26f5d9a32c zerver/tests: Remove print_function. 2017-09-27 18:05:45 -07:00
Vishnu Ks 17cc32f168 settings: Show error or success message when deleting uploaded files.
Fixes #6519
2017-09-27 17:50:22 -07:00
Vishnu Ks 94e099eaab auth: Make multiuse invite link work with oAuth2.
This works by attaching to the user's session the multi-use invitation
key, allowing that to be used in the Google/GitHub auth flows.
2017-09-27 17:47:40 -07:00
Steve Howell de0b47fd4e Always notify service bots about stream mentions.
Before this change, we were only triggering service bots
for stream mentions when the bot was subscribed to the
stream.
2017-09-27 17:22:12 -07:00
rht daef6a7dbc zerver/tests: Remove `import six`. 2017-09-27 17:07:40 -07:00
Vishnu Ks dfaa939af2 emails: Store the compiled emails in separate directory. 2017-09-27 16:48:18 -07:00
Steve Howell 1b518f1983 Return mentioned users in get_user_info_for_message_updates().
The dictionary result for get_user_info_for_message_updates()
now has a `mention_user_ids` field that is a set of user ids
who were mentioned in a message.
2017-09-27 16:01:50 -07:00
Steve Howell 646abb57b7 refactor: Extract get_user_info_for_message_updates.
We'll want to expand this to get users that were mentioned in
the prior message, but this commit is just a refactoring.
2017-09-27 16:01:50 -07:00
Tim Abbott 06e6b3abe1 event_queue: Fixed stream_push_notify code path in missedmessage_hook.
This finishes implementing the stream_push_notify feature, which
previously didn't work if you were not idle when the message was sent.
2017-09-27 15:57:48 -07:00
Tim Abbott 2b1ea1555c event_queue: Simplify API of maybe_enqueue_notifications. 2017-09-27 15:57:47 -07:00
Tim Abbott d6732d0781 event_queue: Fix handling of private messages in missedmessage_hook.
Previously, we didn't correctly trigger email/push notifications for
PMs in these cases.
2017-09-27 15:57:47 -07:00
Tim Abbott bfa8f0efdb tests: Add an end-to-end test for missedmessage_hook.
This checks what arguments it passes into the enqueuing function.

Note, however, that the arguments are wrong for various cases, we'll
update the tests as we fix those bugs.
2017-09-27 15:57:47 -07:00
Tim Abbott 71fc8363b4 tests: Add a test suite for maybe_enqueue_notifications.
This ensures that as we expand the logic for under what circumstances
email and push notifications should be sent, we can be confident about
this code path always doing the right thing.
2017-09-27 15:55:29 -07:00
rht 1e87a4b68c zerver/tests: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Tim Abbott f0f0176517 test_auth_backends: Clean up Google auth subdomains handling.
This makes GoogleSubdomainLoginTest consistently access subdomains the
standard way, replacing the original hacky approach it had that
predated the library.
2017-09-26 22:20:07 -07:00
Steve Howell b340b28055 Extract get_service_bot_events().
There are several reasons to extract this function:

    * It's easy to unit test without extensive mocking.
    * It will show up when we profile code.
    * It is something that you can mostly ignore for
      most messages.

The main reason to extract this, though, is that we are about
to do some fairly complex splicing of data for the use case
of mentioning service bots on streams they are not subscribed to,
and we want to localize the complexity.
2017-09-26 18:49:03 -07:00
Robert Hönig 43422fa6f2 outgoing webhook: Notify bot owner on failure response. 2017-09-26 16:11:27 -07:00
Vishnu Ks 7fd4a71b7f test_emails: Generate emails instead of hardcoding.
Tweaked by tabbott to use require_GET.

Fixes #6344.
2017-09-26 15:02:47 -07:00
Vishnu Ks f58c87917e emails: Log emails that are queued or sent in dev environment.
Tweaked by tabbott to add some comments and clarify the code.
2017-09-26 15:02:47 -07:00
Alena Volkova abe4196232 urls: Change the method for adding alert words from PUT to POST. 2017-09-26 14:00:51 -07:00
Alena Volkova ca687e01d7 urls: Remove the old POST endpoint for alert words. 2017-09-26 14:00:51 -07:00
Tim Abbott e5ea598fe3 test_email_change: Cover the case of email already existing.
This fixes master failing tests.
2017-09-26 11:16:12 -07:00
Tim Abbott 0d2d9fc30d test_signup: Add checks for messages sent on invited user signup.
Previously, we didn't have a test that verified that these notices
were sent.
2017-09-25 13:58:41 -07:00
Tim Abbott f010ed117b Migrate several Google auth tests to subdomains test class.
The plan is to have everything expect subdomains, so it makes sense to
move these tests to the subdomains-only test class and style.

Most of the remaining GoogleLoginTest tests are now either duplicates
or basic API-level tests where subdomains are irrelevant.
2017-09-25 07:29:18 -07:00
Tim Abbott c62befe655 test_messages: Fix invalid subdomain in mirrored message test. 2017-09-25 07:20:58 -07:00
Tim Abbott 4674af0894 outgoing_webhook: Fix broken way of accessing realm.uri.
Previously, this accessed realm.uri via trying to use
zulip_default_context.  That doesn't make any sense, because
zulip_default_context expects an HttpRequest object, and those are
nowhere in sight in the code path.  We do, however, have the outgoing
webhook bot user involved in the event, and that's the object to
access realm.uri from here.
2017-09-25 07:05:02 -07:00
Tim Abbott 5159151e5f test_signup: Remove unnecessary realm_name/realm_subdomain arguments.
These arguments are only intended to be used for realm creation, and
they make the code more confusing.

We need to make a few changes after doing this, because some tests
were relying on these extra arguments causing the form to not submit
for their error handling.

We don't apply these changes to the LDAP tests, since fixing those
seems complicated.
2017-09-25 06:45:56 -07:00
Tim Abbott 163da73d2d test_signup: Hardcode REALMS_HAVE_SUBDOMAINS for mirror dummy tests.
This also fixes the fact that this code path was randomly setting a
bunch of fields only used in realm creation in the registration form.
2017-09-25 06:14:38 -07:00
Robert Hönig 15a1bf2b58 outgoing webhook: Log all non-200 responses. 2017-09-25 06:00:42 -07:00
Eeshan Garg 11b7d219bd integrations: Render bots' docs alongside integration docs.
This commit implements support for rendering static files in
under static/generated/bots/ in the same manner as we render
our webhooks/integration documentation. Said static files are
generated by tools/setup/generate_zulip_bots_static_files.py
during provisioning.
2017-09-22 15:19:20 -07:00
Eeshan Garg 390a1fec92 zulip_bots: Generate static files during provisioning.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.

This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
2017-09-22 15:14:42 -07:00
Tim Abbott de9f00208d test_notifications: Fix unnecessary use of dirname. 2017-09-22 10:17:54 -07:00
Tim Abbott 3d63f981a1 test_notifications: Clarify code for new MissedMessages test. 2017-09-22 10:16:24 -07:00
Harshit Bansal e4c1aa37f3 notifications: Add tests for `relative_to_full_url()` function.
Fixes: #6544.
2017-09-22 10:04:33 -07:00
Vishnu Ks b4fedaa765 backend: Add support for multiuse user invite link. 2017-09-22 07:56:53 -07:00
Tim Abbott dc843efd56 RealmAuditLog: Add support for stringifying entries. 2017-09-22 07:09:37 -07:00
Rishi Gupta 6ec3595b77 emails: Change enqueue_welcome_emails to take a user rather than user_id. 2017-09-22 06:20:33 -07:00
Rishi Gupta a7c8770f97 emails: Move enqueue_welcome_emails outside of signups queue.
The only thing this queue should do is sign you up for the newsletter, since
it is only populated if newsletter_data is not None.
2017-09-22 06:20:33 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Tim Abbott 7554958316 test_signup: Add a test for invite emails being cleared. 2017-09-21 06:15:11 -07:00
Tim Abbott f706f657c0 signup: Fix invitation emails not being cleared properly.
Previously, invitation reminder emails were only being cleared after a
successful signup if newsletter_data was available, since that was the
circumstance in which we were calling the relevant queue processor
code.  Now, we (1) clear them when a human user finishes signing up
and (2) correctly clear them using the 'address' field of
ScheduleEmail, not user_id.
2017-09-21 06:15:11 -07:00
Steve Howell 9110ff935e tests: Restore 100% coverage to views/messages.py.
A recent change moved some untested code into messages.py, which
requires 100% coverage.  Now we test bogus stream names in
exclude_muting_conditions.
2017-09-20 13:02:22 -07:00
Steve Howell 428d3027c2 Only require ids for finding DefaultStream objects.
We don't need full Realm objects to find DefaultStream
objects for a realm.  So now a few functions related to
adding/removing default streams use realm_id for lookups.

Similarly, we don't need a full Stream object to find
out if a stream exists in DefaultStream, so we do id
lookups there as well.

This sets us up to use thinner objects in callers.
2017-09-20 10:31:33 -07:00
Steve Howell 8ad7133351 Cache active_user_ids() more directly.
We now have a dedicated cache for active_user_ids() that only
stores a list of user_ids.

Before this commit, active_user_ids() used a cache of UserProfile
dictionaries, so it incurred unnecessary deserialization costs for
all the user fields that it sliced away in a list comprehension.

Because the cache is skinnier here, we also need to invalidate it
less frequently.  Basically, all we care about is new users, realm
deactivations, and user deactivations.

It's hard to measure how much this will improve performance, because
the speedup for any operation here is pretty minor, but we use this
function a lot, so hopefully it will make the overall system more
healthy.
2017-09-20 10:31:33 -07:00
Steve Howell cad3a35b6a Only require realm_id for active_user_ids().
This is mostly a preparatory commit for an upcoming optimization
related to stream data, but it probably does save us an
occasional DB hop to the realm table.
2017-09-20 10:31:33 -07:00
Steve Howell 0966bf1a48 Simplify get_stream_cache_key().
Before this commit, we could pass in either a Realm object
or a realm_id to get_stream_cache_key().  Now we consistently
pass it a realm_id.
2017-09-20 10:31:33 -07:00
Steve Howell 0e24e6bdfa Add test_num_queries_with_streams() for home page. 2017-09-20 10:31:33 -07:00
Steve Howell 9773750375 tests: Count queries for home page. 2017-09-20 10:31:33 -07:00
Harshit Bansal 122167148a notifications: Fix the misalignment of emojis with the message text.
Fixes: #6579.
2017-09-20 10:30:35 -07:00
kunall17 1436d558a3 APNs: Add message body to notification payload.
(Edited by greg to leave the badge logic as is for now.)
2017-09-18 16:37:02 -07:00
invinciblycool 5475c5cedb tests: Replace test_user_agent_parsing with a normal test.
Previously, this was its own separate test script; now it's a normal
part of the test suite.

Tweaked by tabbott to use a proper test method.

Fixes #6327.
2017-09-16 03:56:18 -07:00
Tim Abbott a1ddd934b4 bots: Fix calculation of bot domain with REALMS_HAVE_SUBDOMAINS.
Previously, the bot domain was calculated correctly in most
circumstances, but if you were using the root domain, it would be
e.g. ".chat.zulip.org", not "chat.zulip.org".  We fix this, with
perhaps more use of setting REALMS_HAVE_SUBDOMAINS than would be ideal
if we weren't about to set that True unconditionally.
2017-09-16 02:49:15 -07:00
Tim Abbott 0d5badb71a urls: Remove robots.txt blocking Zulip serves from search engines.
This was added back in 2012 and seems to have been accidentally not
updated since then.
2017-09-16 01:59:40 -07:00
Tim Abbott e1498988d9 test_choose_realm: Hardcode REALMS_HAVE_SUBDOMAINS.
This is the only case that'll be important in the future, and this
is a nice checkpoint on the path to making REALMS_HAVE_SUBDMAINS=True.
2017-09-15 13:11:12 -07:00
Tim Abbott 064c8e39d6 backends: convert GitHub auth tests to consistently use zulip subdomain. 2017-09-15 13:09:02 -07:00
Tim Abbott f1133ab53e test_notifications: Fix HTML tests to assume REALMS_HAVE_SUBDOMAINS.
This is a step towards making REALMS_HAVE_SUBDOMAINS the default.
2017-09-15 12:28:38 -07:00
Steve Howell 1553dc00e0 Introduce StreamRecipient class.
This class encapsulates the mapping of stream ids to
recipient ids, and it is optimized for bulk use and
repeated use (i.e. it remembers values it already fetched).

This particular commit barely improves the performance
of gather_subscriptions_helper, but it sets us up for
further optimizations.

Long term, we may try to denormalize stream_id on to the
Subscriber table or otherwise modify the database so we
don't have to jump through hoops to do this kind of mapping.
This commit will help enable those changes, because we
isolate the mapping to this one new class.
2017-09-15 10:44:32 -07:00
Tim Abbott 15781dbc83 server_settings: Add email auth related features to data sent to clients.
This should make it possible for the mobile app to correctly allow
non-email addresses as usernames exactly when it makes sense to do so.
2017-09-15 10:30:20 -07:00
Supermanu 5f41f3c3cb backends.py: Expose backends that require email usernames 2017-09-15 10:29:02 -07:00
Supermanu 28beddfd76 backends.py: Enable auth with any ldap attributes as username.
This commit enables user to authenticate with any attribute set in
AUTH_LDAP_USER_SEARCH given that LDAP_EMAIL_ATTR is set to an email
attributes in the ldap server. Thus email and username can be
completely unrelated.

With some tweaks by tabbott to squash in the documentation and make it
work on older servers.
2017-09-15 10:28:41 -07:00
Rishi Gupta b9c8747bd0 hotspots: Use tutorial_status to manage whether we show hotspots. 2017-09-15 04:14:52 -07:00
Rishi Gupta ac48772732 hotspots.js: Fix timeout millisecond conversion. 2017-09-15 04:14:52 -07:00
Rishi Gupta a8deedbbb6 hotspots: Replace stream_settings with intro_streams and intro_topics. 2017-09-15 04:14:52 -07:00
Rishi Gupta 0677bd2a6d hotspots: Rename and update new_topic_button. 2017-09-15 04:14:52 -07:00
Rishi Gupta 4a383544af hotspots: Rename and update click_to_reply. 2017-09-15 04:14:52 -07:00
Vishnu Ks 22bab1c7ec events: Add upload_quota to initial state data. 2017-09-15 01:27:27 -07:00
Vishnu Ks 0ce62cd3e1 events: Add total_uploads_size to initial state data. 2017-09-15 01:27:20 -07:00
Steve Howell 7159f34bed Avoid getting realm emoji unless there is emoji syntax. 2017-09-15 01:09:08 -07:00
Steve Howell 8a941d03ae Avoid querying streams unless there are stream links. 2017-09-15 01:09:08 -07:00
Steve Howell c3032a7fe8 Avoid looking up emails when rendering messages.
We now fetch email -> id mappings for messages ONLY if
it potentially uses the !avatar(foo@example.com) syntax.
2017-09-15 01:09:08 -07:00
Steve Howell 4e7fce60ee Add possible_mentions() to speed up rendering.
We now triage message content for possible mentions before
going to the cache/DB to get name info.  This will create an
extra data hop for messages with mentions, but it will save
a fairly expensive cache lookup for most messages.  (This will
be especially helpful for large realms.)

[Note that we need a subsequent commit to actually make the speedup
happen here, since avatars also cause us to look up all users in
the realm.]
2017-09-15 01:09:08 -07:00
Tim Abbott 5722237f59 push: Rename received_pm to private_message.
This is a clearer name for this now more broadly used interface.
2017-09-14 05:41:37 -07:00
Sarah 97571a203d push: Add new formatting for stream message push and add tests.
This should make the push notifications for messages to streams with
the new stream push notifications setting enabled make sense.
2017-09-14 05:41:37 -07:00
Sarah c3a8138f74 user_settings: Add push notifications for all stream messages.
Add setting to enable push notifications for all stream messages.
2017-09-14 05:41:37 -07:00
Steve Howell d723be125a Optimize get_recipient_info() for sending messages.
This commit makes get_recipient_info() faster by never creating
Django ORM objects.  We use the ORM to create a values query
instead, and then we iterate over the rows to create various
collections of ids.

In order to avoid lots of code duplication, this commit unifies
how we query UserProfile for PMs and streams.  Prior to this
commit we were getting "wide" UserProfile objects out of
our memcached cache.  Now we just go to the database with our
list of userids.  The new approach at worst adds one hop to the
database for PMs, which aren't really a performance bottleneck
(compared to streams).  And the new approach actually saves a
hop when both partners aren't in cache (plus we don't pay the
penalty of hitting the cache itself).

The performance improvement here is easy to measure for messages
to streams with many users, even with all the other activity
that goes on inside do_send_messages().  I took test_performance()
in test_messages.py, set num_extra_users to 3000, and consistently
measured a ~20% speedup in do_send_messages().

This commit also eliminates fetching of emails.  We probably
could have done that in a prior commit, but in this commit it
is very explicit that we don't need it.  While removing email
from the query is a no-brainer, it actually had a negigible
impact on performance.  Almost all the savings here comes from
not create UserProfile objects.
2017-09-12 04:22:55 -07:00
Steve Howell 82b2bd8b65 Take user_ids in get_userids_for_missed_messages().
This helps us phase out the need for getting lots of UserProfile
objects.
2017-09-12 04:22:55 -07:00
Steve Howell ba397b5109 Use user_ids, not full objects, in render path.
There is no reason for either render_incoming_message() or
render_markdown() to require full UserProfile objects just to
triage alert words.

By only asking for user_ids, we save extra queries in two
callpaths and we make it easier to start using user_ids in
do_send_messages().
2017-09-12 04:22:55 -07:00
Steve Howell 262abe41ab Add a performance test for do_send_messages(). 2017-09-12 04:22:55 -07:00
Steve Howell b6bb7f2b1e Fix bug where we hard code realm for PushDeviceToken.
This had no test coverage, which is part of the reason it went
undetected, plus many instances probably only have one realm
with realm_id=1.
2017-09-08 12:32:17 -07:00
Steve Howell 69203c1c81 fix_unreads: Remove commit() call in fix().
The commit() call in fix() breaks migrations and tests (unless you
mock) due to outer transactions.

We now explicitly call commit() from the management command.
2017-09-07 07:06:03 -07:00