Commit Graph

5752 Commits

Author SHA1 Message Date
Tim Abbott b4aa1336e7 logging: Avoid logging certain common 404 errors to error logs.
This should help keep /var/log/zulip/errors.log relatively clean on
production Zulip servers.
2017-09-16 08:07:06 -07:00
invinciblycool 5475c5cedb tests: Replace test_user_agent_parsing with a normal test.
Previously, this was its own separate test script; now it's a normal
part of the test suite.

Tweaked by tabbott to use a proper test method.

Fixes #6327.
2017-09-16 03:56:18 -07:00
Aditya Bansal ddf2075228 Make migration 0032 use an old version of user_avatar_path.
This fixes upgrading from very old Zulip servers (e.g. 1.4.3) all the
way to current.

Fixes: #6516.
2017-09-16 03:27:02 -07:00
Tim Abbott a33c98f23e push_notifications: Require the message.triggers fields to exist.
The code was already assuming it implicitly, and making it explicit
improves the readability.
2017-09-16 03:14:56 -07:00
Steve Howell c4b17f2f80 Optimize get_recipient_info() with get_ids_for().
The get_ids_for() function produces a 2x speedup for
1000 users.
2017-09-16 03:07:13 -07:00
Tim Abbott a1ddd934b4 bots: Fix calculation of bot domain with REALMS_HAVE_SUBDOMAINS.
Previously, the bot domain was calculated correctly in most
circumstances, but if you were using the root domain, it would be
e.g. ".chat.zulip.org", not "chat.zulip.org".  We fix this, with
perhaps more use of setting REALMS_HAVE_SUBDOMAINS than would be ideal
if we weren't about to set that True unconditionally.
2017-09-16 02:49:15 -07:00
Tim Abbott 0d5badb71a urls: Remove robots.txt blocking Zulip serves from search engines.
This was added back in 2012 and seems to have been accidentally not
updated since then.
2017-09-16 01:59:40 -07:00
Tim Abbott e1498988d9 test_choose_realm: Hardcode REALMS_HAVE_SUBDOMAINS.
This is the only case that'll be important in the future, and this
is a nice checkpoint on the path to making REALMS_HAVE_SUBDMAINS=True.
2017-09-15 13:11:12 -07:00
Tim Abbott 064c8e39d6 backends: convert GitHub auth tests to consistently use zulip subdomain. 2017-09-15 13:09:02 -07:00
Tim Abbott f1133ab53e test_notifications: Fix HTML tests to assume REALMS_HAVE_SUBDOMAINS.
This is a step towards making REALMS_HAVE_SUBDOMAINS the default.
2017-09-15 12:28:38 -07:00
Steve Howell 84041d3195 Use itertools.groupby in bulk_get_subscriber_user_ids().
This results in about a 20% speedup by making more O(N)
things happen in C vs. Python.
2017-09-15 10:44:32 -07:00
Steve Howell 24b9f72b22 Use raw SQL in bulk_get_subscriber_user_ids().
This leads to more than a 2x speedup when tested with
20k+ total subscribers.  (For large realms with lots of default
streams, this function deals with LOTS of data, so it is important
to optimize.)
2017-09-15 10:44:32 -07:00
Steve Howell 1553dc00e0 Introduce StreamRecipient class.
This class encapsulates the mapping of stream ids to
recipient ids, and it is optimized for bulk use and
repeated use (i.e. it remembers values it already fetched).

This particular commit barely improves the performance
of gather_subscriptions_helper, but it sets us up for
further optimizations.

Long term, we may try to denormalize stream_id on to the
Subscriber table or otherwise modify the database so we
don't have to jump through hoops to do this kind of mapping.
This commit will help enable those changes, because we
isolate the mapping to this one new class.
2017-09-15 10:44:32 -07:00
Steve Howell fc2e485ca7 Sort emails in gather_subscriptions().
This helps makes the tests more deterministic.
2017-09-15 10:44:32 -07:00
Tim Abbott 15781dbc83 server_settings: Add email auth related features to data sent to clients.
This should make it possible for the mobile app to correctly allow
non-email addresses as usernames exactly when it makes sense to do so.
2017-09-15 10:30:20 -07:00
Supermanu ac79bbfe08 login: Enable non-email username to login. 2017-09-15 10:30:19 -07:00
Supermanu 5f41f3c3cb backends.py: Expose backends that require email usernames 2017-09-15 10:29:02 -07:00
Supermanu 28beddfd76 backends.py: Enable auth with any ldap attributes as username.
This commit enables user to authenticate with any attribute set in
AUTH_LDAP_USER_SEARCH given that LDAP_EMAIL_ATTR is set to an email
attributes in the ldap server. Thus email and username can be
completely unrelated.

With some tweaks by tabbott to squash in the documentation and make it
work on older servers.
2017-09-15 10:28:41 -07:00
Rishi Gupta b9c8747bd0 hotspots: Use tutorial_status to manage whether we show hotspots. 2017-09-15 04:14:52 -07:00
Rishi Gupta ac48772732 hotspots.js: Fix timeout millisecond conversion. 2017-09-15 04:14:52 -07:00
Rishi Gupta 21f076ac5d hotspots.py: Clean up SEND_ALL codepath.
Moves SEND_ALL to inside get_next_hotspots, since it is not something other
files should call.
Also changes the delay to 0s, and gates the code behind an
`if settings.DEVELOPMENT`.
2017-09-15 04:14:52 -07:00
Rishi Gupta a8deedbbb6 hotspots: Replace stream_settings with intro_streams and intro_topics. 2017-09-15 04:14:52 -07:00
Rishi Gupta 0677bd2a6d hotspots: Rename and update new_topic_button. 2017-09-15 04:14:52 -07:00
Rishi Gupta 4a383544af hotspots: Rename and update click_to_reply. 2017-09-15 04:14:52 -07:00
Steve Howell 51d4f16fe0 Ignore inactive users in get_recipient_info().
We were mostly excluding inactive users before this fix, but
now we completely ignore them.

This potentially changes some of the data we return from
get_recipient_info(), but the extra user ids before this fix
were effectively ignored by the caller.
2017-09-15 03:08:52 -07:00
Steve Howell 1759137e4f Don't queue feedback unless the bot is active.
The prior code would queue up feedback messages even if the
feedback bot was deactivated, which was just due to oversight
most likely.  (People probably rarely disable the feedback bot,
but they should have that option.)
2017-09-15 03:08:52 -07:00
Vishnu Ks 2993ae5433 models: Include create_time in Attachment to_dict method. 2017-09-15 01:27:28 -07:00
Vishnu Ks e34931971b models: Include size in Attachment to_dict method. 2017-09-15 01:27:28 -07:00
Vishnu Ks 22bab1c7ec events: Add upload_quota to initial state data. 2017-09-15 01:27:27 -07:00
Vishnu Ks 0ce62cd3e1 events: Add total_uploads_size to initial state data. 2017-09-15 01:27:20 -07:00
Tim Abbott f1a0f23d23 bugdown: Document the mention fetches code block. 2017-09-15 01:21:04 -07:00
Steve Howell 7159f34bed Avoid getting realm emoji unless there is emoji syntax. 2017-09-15 01:09:08 -07:00
Steve Howell 8a941d03ae Avoid querying streams unless there are stream links. 2017-09-15 01:09:08 -07:00
Steve Howell c3032a7fe8 Avoid looking up emails when rendering messages.
We now fetch email -> id mappings for messages ONLY if
it potentially uses the !avatar(foo@example.com) syntax.
2017-09-15 01:09:08 -07:00
Steve Howell 4e7fce60ee Add possible_mentions() to speed up rendering.
We now triage message content for possible mentions before
going to the cache/DB to get name info.  This will create an
extra data hop for messages with mentions, but it will save
a fairly expensive cache lookup for most messages.  (This will
be especially helpful for large realms.)

[Note that we need a subsequent commit to actually make the speedup
happen here, since avatars also cause us to look up all users in
the realm.]
2017-09-15 01:09:08 -07:00
Steve Howell 7a4c3c1a5c Make mentions regex more strict for all/everyone.
We only want `@all` and `@everyone` as shorthands.  For user
names we want askerisks: `@**Steve Howell**`.
2017-09-15 01:09:08 -07:00
Steve Howell 6a625eef66 Clean up model imports for bugdown/__init__.py.
We broke some circular dependencies a while back, so we can
move a bunch of imports to the top of the file.
2017-09-15 01:09:08 -07:00
Eeshan Garg a041a23199 webhooks/trello: Ignore when Board background is changed.
I feel like getting notifications about a board's background being
changed isn't very useful information and could interrupt the flow
of other important information such as Card changes or movement,
so I think we should not support this event and
should simply ignore such payloads in the future.
2017-09-15 01:02:47 -07:00
Tim Abbott d06cb4e4fd event_queue: Make path forward for push/email notifications clearer.
This is a nonfunctional refactor, designed primarily to make it
simpler to extend this code path when we later add support for
controlling whether email notifications go out on stream messages.
2017-09-15 01:01:11 -07:00
Tim Abbott d0e8163f13 event_queue: Remove some unnecessary parenthesis. 2017-09-15 01:01:11 -07:00
Tim Abbott e085af3324 Fix stream_push_notify feature to not send emails.
Previously, due to a logic bug, this feature would also send email
notifications for all messages on the stream, which is definitely not
the intent.  The recent refactoring we just did makes the logic more
obvious.
2017-09-15 01:01:11 -07:00
Tim Abbott 22ea2a5858 event_queue: Separate email and push notification loops. 2017-09-15 01:01:11 -07:00
Tim Abbott 7fa0325fb5 event_queue: Refactor notified logic. 2017-09-15 01:01:11 -07:00
Greg Price c4b506998f tornado: Disable routine logging in dev.
This creates a lot of logging noise, and also causes confusion
for new contributors when something isn't working as they expect
and they aren't sure if this message is normal or an error.
2017-09-14 12:38:57 -07:00
Tim Abbott e8f835d852 migrations: Fix migration 0041 failures for long attachment filenames.
We should have done this a long time ago, but better late than never.
Basically, this migration would crash in the event that there were any
attachments with particularly long names.  The fix is the next
migration, 0042; we just inline it here to avoid that crash.
2017-09-14 07:00:07 -07:00
Tim Abbott 5722237f59 push: Rename received_pm to private_message.
This is a clearer name for this now more broadly used interface.
2017-09-14 05:41:37 -07:00
Sarah 97571a203d push: Add new formatting for stream message push and add tests.
This should make the push notifications for messages to streams with
the new stream push notifications setting enabled make sense.
2017-09-14 05:41:37 -07:00
Sarah c3a8138f74 user_settings: Add push notifications for all stream messages.
Add setting to enable push notifications for all stream messages.
2017-09-14 05:41:37 -07:00
Steve Howell 41e3a819da Inline get_recipient_user_ids() into two callers.
This sets us up a subsequent commit where we need more data
from the Subscription table to build recipient info, so the
function boundary doesn't work any more for get_recipient_info,
which is part of the heavily optimized send-message
path.

We used to share code here with typing notifications, but
typing notifications need a lot less data than the
send-message path, so it's useful to decouple these two
things.  The idioms that are duplicated here are pretty simple
one-liners.
2017-09-14 05:13:58 -07:00
Steve Howell ac61c48964 Optimize get_status_dict_by_realm().
This change optimizes get_status_dict_by_realm() by
introducing query_for_ids(), which quickly computes
an "IN" clause for user ids.

This change also inlines the `two_weeks_ago` check, but
that is just for clarity, not performance.
2017-09-14 04:22:02 -07:00