Commit Graph

20806 Commits

Author SHA1 Message Date
Greg Price 7c5f3d7be1 deps: Refactor tools/update-locked-requirements to save repeating the filenames.
My first version of this just replaced the repeated list of two output
files with an array variable, but I decided `"${outputs[@]}"` was too
much to ask people to understand, and the alternative of `$outputs`,
unquoted, encourages bad habits of shell programming.  So just handle
one file at a time; the only at all expensive part here is `pip-compile`.

I am tempted to move this to Python, but holding back.
2017-09-15 00:23:20 -07:00
Greg Price a171f23d63 deps: Remove a redundant `deactivate`.
This has no effect except on the running shell; so at the end
of a script, it has no meaningful effect at all.
2017-09-15 00:23:20 -07:00
Greg Price b362e21ebb deps: Simplify tools/update-locked-requirements slightly.
This makes the code a little more concise and also more uniform,
treating `future` the same in prod and in dev.  The non-uniformity
looks like an oversight in 2be8a793e, one of the commits that updated
this code for the Python 3-only world.
2017-09-15 00:23:20 -07:00
Greg Price bab2078705 deps: Explain the `-e` oddness, and handle it uniformly. 2017-09-15 00:23:20 -07:00
Greg Price 806220cafa deps: Organize tools/update-locked-requirements a bit further.
A couple of remarks still reflected the Python 2 world; and the
"Remove the editable flag" comment was confusingly above a line
that it looks like it could be talking about, but isn't.
2017-09-15 00:23:20 -07:00
Sampriti Panda e98d7a6ee8 Fix issues with left-sidebar stream search and multiple words.
If you typed in more than one word for a stream with multiple words in
it's name, it would not show up in the search list. This fixes that
and adds some more tests covering the entire functionality of the
filter.
2017-09-15 00:22:59 -07:00
Greg Price c4b506998f tornado: Disable routine logging in dev.
This creates a lot of logging noise, and also causes confusion
for new contributors when something isn't working as they expect
and they aren't sure if this message is normal or an error.
2017-09-14 12:38:57 -07:00
Cynthia Lin 5be0124e0b left sidebar: Add cursor pointer change on hover to Streams icons. 2017-09-14 12:37:27 -07:00
Eeshan Garg 0721fc851a meetings: Switch integrations meeting to Fridays 9 am (PST). 2017-09-14 10:18:15 -07:00
Cynthia Lin d7fbae2410 left-sidebar.css: Remove .brand selector.
This selector is unused (there is no element with class `brand` in element with ID `left-sidebar`).
2017-09-14 10:18:03 -07:00
Supermanu 184bfe7cfd docs: Expand the french translation style guide.
As the french word organisation is used, the related line is excluded
from the linter spell check.
2017-09-14 08:35:51 -07:00
Brock Whittaker 25304ddeee Fix padding to match new font.
This commit fixes the padding on several things that are now not
vertically centered because the new font has slightly different
heights.
2017-09-14 07:38:44 -07:00
Cory Lynch f4a09b7758 recent_senders: Add sort by stream message recency.
If both users haven't posted in the current topic, then
as a second order sort, check which user has posted first
in the stream as a whole.

Fixes part of #5956; we still need to sort by sending in the
organization.
2017-09-14 07:20:52 -07:00
Tim Abbott e8f835d852 migrations: Fix migration 0041 failures for long attachment filenames.
We should have done this a long time ago, but better late than never.
Basically, this migration would crash in the event that there were any
attachments with particularly long names.  The fix is the next
migration, 0042; we just inline it here to avoid that crash.
2017-09-14 07:00:07 -07:00
Vishnu Ks d8f8e29ff5 frontend: Show link to delete files in quota exceeded error. 2017-09-14 06:04:09 -07:00
Vishnu Ks 9c70da1a78 docs: Remove misleading fullstop from vagrant plugin command. 2017-09-14 06:01:11 -07:00
Joshua Pan 0fd9f0bcc9 hotkeys: Make 'v' hotkey toggle. 2017-09-14 05:56:55 -07:00
Joshua Pan 42b05912f8 hotkeys: Make '?' hotkey toggle. 2017-09-14 05:56:55 -07:00
Tim Abbott 5722237f59 push: Rename received_pm to private_message.
This is a clearer name for this now more broadly used interface.
2017-09-14 05:41:37 -07:00
Sarah 97571a203d push: Add new formatting for stream message push and add tests.
This should make the push notifications for messages to streams with
the new stream push notifications setting enabled make sense.
2017-09-14 05:41:37 -07:00
Sarah c3a8138f74 user_settings: Add push notifications for all stream messages.
Add setting to enable push notifications for all stream messages.
2017-09-14 05:41:37 -07:00
Steve Howell 41e3a819da Inline get_recipient_user_ids() into two callers.
This sets us up a subsequent commit where we need more data
from the Subscription table to build recipient info, so the
function boundary doesn't work any more for get_recipient_info,
which is part of the heavily optimized send-message
path.

We used to share code here with typing notifications, but
typing notifications need a lot less data than the
send-message path, so it's useful to decouple these two
things.  The idioms that are duplicated here are pretty simple
one-liners.
2017-09-14 05:13:58 -07:00
Cynthia Lin 191bb4c20f navbar: Remove text-decoration from brand logo link. 2017-09-14 04:57:40 -07:00
Robert Hönig 29d512a4a8 docs: Move `zulip_bots` installation steps to new section. 2017-09-14 04:48:53 -07:00
Robert Hönig bc4627e61a docs: Add reference to zulip-bot-output script. 2017-09-14 04:48:10 -07:00
Robert Hönig 6bfbbaa0ce docs: Clarify zulip_bots repository install. 2017-09-14 04:48:10 -07:00
Steve Howell ac61c48964 Optimize get_status_dict_by_realm().
This change optimizes get_status_dict_by_realm() by
introducing query_for_ids(), which quickly computes
an "IN" clause for user ids.

This change also inlines the `two_weeks_ago` check, but
that is just for clarity, not performance.
2017-09-14 04:22:02 -07:00
Steve Howell aade317d87 Extract UserPresence.get_status_dicts_for_rows().
The prior version of this function was passed in a QuerySet, which
made it difficult to effectively profile the callers, and there
is really no compelling reason to pass in a query any more.
2017-09-14 04:22:02 -07:00
Robert Hönig 5294e10801 custom linter rules test: Load rules in test setup. 2017-09-14 04:17:03 -07:00
Robert Hönig 402d7fca50 test_linter_custom_check.py: Fix typo in docstring. 2017-09-14 04:17:03 -07:00
Umair Khan 1f93c06b76 i18n: Optimize get_language_list().
compilemessages command now does all the heavy lifting by creating a
language_name_map.json file under locale directory. This file is used
by get_language_list to retrieve the require information.

Fixes: #6486
2017-09-14 02:28:58 -07:00
Umair Khan df70fa962e linter: Improve i18n related regexes.
There are four regexes which try to ensure that the i18n strings are
properly captured.

1) The one which disallows multiline strings.
```
i18n\.t\([^)]+[^,\{\)]$

// Disallows:
i18n.t('some '
       + 'text');
```

2) The one which disallows concatenation within argument to i18n.t():
```
i18n\.t\([\'\"].+?[\'\"]\s*\+

// Disallows:
i18n.t("some " + "text");
```

3) There are two which disallow concatenation with i18n.t():
```
i18n\.t\(.+\).*\+

// Disallows:
i18n.t('some text') +

\+.*i18n\.t\(.+\)

// Disallows:
+ i18n.t('some text')
```

The ideal case is that you try to bring the string argument to the
i18n.t() on one line. In case this is not possible, you can do the
following:

```
var1 = i18n.t("Some text to be translated");
var2 = i18n.t("Some more text to be translated");
complete = var1 + var2;
2017-09-13 12:58:27 -07:00
Robert Hönig 221afb1492 docs: Describe memcache flush with a 'normal permission' command. 2017-09-12 08:27:58 -07:00
Tim Abbott 600b164130 setup-apt-repo: Fix failure to rerun properly on provision failure.
Apparently, the refactoring to make this script only run when changes
are present was buggy, in that if `apt-get update` failed, running
provision against wouldn't rerun `apt-get update`, resulting in a
broken state that requires expertise to fix.  This closes that gap, by
using a stamp file to ensure we always successfully update apt before
proceeding.

It doesn't fix existing installations.
2017-09-12 07:08:54 -07:00
Tim Abbott 12a0359999 static: Remove accidentally committed avatar image. 2017-09-12 06:55:49 -07:00
Robert Hönig 52a55a37ad docs: Remove legacy Python2/3 compatibility code. 2017-09-12 06:35:29 -07:00
Robert Hönig 211fbdf59b docs: Add additional reason why mocking is useful. 2017-09-12 06:35:29 -07:00
Steve Howell c953759486 Get unread messages from the server in the web app.
The server sends down lists of unread message ids in various
buckets, and we now use those on the client to provide more
complete counts of unread messages.
2017-09-12 06:02:40 -07:00
Steve Howell 6c90940f84 performance: Add UserMessageLite class.
This speeds up sending messages significantly.

For 1000 users, this speeds up create_user_messages from
0.652s to 0.0558s, so basically a 10x speedup.
2017-09-12 04:22:55 -07:00
Steve Howell 811fcf51ee Extract create_user_messages.
The logic to create UserMessage rows when you create a message
is very self-contained, and it's helpful to be able to profile it.
2017-09-12 04:22:55 -07:00
Steve Howell 7fbffb8e30 Optimize bulk inserts for UserMessage rows.
Avoiding ORM overhead makes inserting UserMessage rows
about 15 times faster.
2017-09-12 04:22:55 -07:00
Steve Howell d723be125a Optimize get_recipient_info() for sending messages.
This commit makes get_recipient_info() faster by never creating
Django ORM objects.  We use the ORM to create a values query
instead, and then we iterate over the rows to create various
collections of ids.

In order to avoid lots of code duplication, this commit unifies
how we query UserProfile for PMs and streams.  Prior to this
commit we were getting "wide" UserProfile objects out of
our memcached cache.  Now we just go to the database with our
list of userids.  The new approach at worst adds one hop to the
database for PMs, which aren't really a performance bottleneck
(compared to streams).  And the new approach actually saves a
hop when both partners aren't in cache (plus we don't pay the
penalty of hitting the cache itself).

The performance improvement here is easy to measure for messages
to streams with many users, even with all the other activity
that goes on inside do_send_messages().  I took test_performance()
in test_messages.py, set num_extra_users to 3000, and consistently
measured a ~20% speedup in do_send_messages().

This commit also eliminates fetching of emails.  We probably
could have done that in a prior commit, but in this commit it
is very explicit that we don't need it.  While removing email
from the query is a no-brainer, it actually had a negigible
impact on performance.  Almost all the savings here comes from
not create UserProfile objects.
2017-09-12 04:22:55 -07:00
Steve Howell d00c001b5f Create get_recipient_info().
This function returns a summary of recipient data for a message
that's being sent.  It's mostly just moving code into the
old function called get_recipient_user_profiles().
2017-09-12 04:22:55 -07:00
Steve Howell b562dedb53 Avoid using email to detect that the feedback bot is addressed.
This commit is necessary to prevent bringing back emails from the
DB for all N recipients of a message just to see if the feedback
bot is being invoked.
2017-09-12 04:22:55 -07:00
Steve Howell 6f0289ae79 do_send_messages(): Extract internal push_notify_user_ids set.
This is one more step toward not needing UserProfile objects.
2017-09-12 04:22:55 -07:00
Steve Howell 82b2bd8b65 Take user_ids in get_userids_for_missed_messages().
This helps us phase out the need for getting lots of UserProfile
objects.
2017-09-12 04:22:55 -07:00
Steve Howell 06c388774f do_send_messages(): Clean up service bot code.
We calculate `service_bot_tuples` earlier in the function, so that
we don't need "full" UserProfile objects later in the function.

This is part of consolidating code that basically just needs to
triage user_ids.
2017-09-12 04:22:55 -07:00
Steve Howell a22a22966f do_send_messages(): Create UserMessage objects with user_id.
This starts to phase out the need for UserProfile objects in
do_send_messages().  UserProfile objects are expensive to create
for large streams with lots of users.  The objects in the code
before this commit aren't even full UserProfile objects.

This change mostly sets up future performance improvements, but
we also get a minor speedup here when we run a test with 3000
stream subscribers.
2017-09-12 04:22:55 -07:00
Steve Howell ba397b5109 Use user_ids, not full objects, in render path.
There is no reason for either render_incoming_message() or
render_markdown() to require full UserProfile objects just to
triage alert words.

By only asking for user_ids, we save extra queries in two
callpaths and we make it easier to start using user_ids in
do_send_messages().
2017-09-12 04:22:55 -07:00
Steve Howell 9e8c24168d Extract get_typing_user_profiles().
This function is essentially a copy of get_recipient_user_profiles,
which is about to go away. The new function enforces the contract of
typing indicators, which is that they don't apply to streams, which
allows us to use a relatively simple approach for getting user
profile objects.

We are diverging this code, because the send-message path needs
more optimizations.
2017-09-12 04:22:55 -07:00