Message.get_raw_db_rows is moved to MessageDict, since its
implementation details are highly coupled to other methods
in MessageDict.
And then sew_messages_and_reactions comes along for the
ride.
We eventually want to move Reaction.get_raw_db_rows to there
as well.
Add this field to the Stream model will prevent us from having
to look at realm data for several types of stream operations, which
can be prone to either doing extra database lookups or making
our cached data bloated.
Going forward, we'll set stream.is_zephyr to True whenever the
realm's string id is "zephyr".
This removes sender names from the message cache, since
they aren't guaranteed to be valid, and they're inexpensive
to add.
This commit will make the message cache entries smaller
by removing sender___full_name and sender__short_name
fields.
Then we add in the sender fields to the message payloads
by doing a query against the unique sender ids of the
messages we are processing.
This change leads to 2 extra database hops for most of
our message-related codepaths. The reason there are 2 hops
instead of 1 is that we basically re-calculate way too
much data to get a no-markdown dictionary.
We now have a dedicated cache for active_user_ids() that only
stores a list of user_ids.
Before this commit, active_user_ids() used a cache of UserProfile
dictionaries, so it incurred unnecessary deserialization costs for
all the user fields that it sliced away in a list comprehension.
Because the cache is skinnier here, we also need to invalidate it
less frequently. Basically, all we care about is new users, realm
deactivations, and user deactivations.
It's hard to measure how much this will improve performance, because
the speedup for any operation here is pretty minor, but we use this
function a lot, so hopefully it will make the overall system more
healthy.
Previously, the bot domain was calculated correctly in most
circumstances, but if you were using the root domain, it would be
e.g. ".chat.zulip.org", not "chat.zulip.org". We fix this, with
perhaps more use of setting REALMS_HAVE_SUBDOMAINS than would be ideal
if we weren't about to set that True unconditionally.
This change optimizes get_status_dict_by_realm() by
introducing query_for_ids(), which quickly computes
an "IN" clause for user ids.
This change also inlines the `two_weeks_ago` check, but
that is just for clarity, not performance.
The prior version of this function was passed in a QuerySet, which
made it difficult to effectively profile the callers, and there
is really no compelling reason to pass in a query any more.
Avoid a join to UserProfile here speeds up the query from
86ms -> 28ms when you analyze it with about 2000 mobile users
in a 5000-user realm.
We also avoid some code duplication here, since we filter
UserPresence for the same group of users as we filter
PushDeviceToken.
This avoids an O(N-squared) hit during presence queries. The speedup
here is probably negligible compared to everything else going on, but
sets are more semantically correct, anyway.
Before this commit, postgres would choose a non-optimal query
plan to find all presence rows belonging to a realm. We now
do an extra query to get the list of relevant user_ids, which allows
the next query to take advantage of UserPresence's index on
user_profile_id.
Here is the query plan for the offending query (this particular query isn't
verbatim from the code, but it's representative of the problem):
explain analyze
select client_id
from zerver_userpresence
INNER JOIN zerver_userprofile ON
zerver_userprofile.id = zerver_userpresence.user_profile_id
WHERE
zerver_userprofile.is_active and
zerver_userprofile.realm_id = 3;
Hash Join (cost=149.66..506.82 rows=5007 width=4) (actual time=48.834..121.215 rows=5007 loops=1)
Hash Cond: (zerver_userprofile.id = zerver_userpresence.user_profile_id)
-> Seq Scan on zerver_userprofile (cost=0.00..260.11 rows=5369 width=4) (actual time=0.009..24.322 rows=5021 loops=1)
Filter: (is_active AND (realm_id = 3))
Rows Removed by Filter: 3
-> Hash (cost=87.07..87.07 rows=5007 width=8) (actual time=48.789..48.789 rows=5010 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 196kB
-> Seq Scan on zerver_userpresence (cost=0.00..87.07 rows=5007 width=8) (actual time=0.007..24.355 rows=5010 loops=1)
Total runtime: 145.063 ms
You can see above that we're filtering on realm_id instead of using an index.
When you decompose the query into two queries, the total time is about 100ms, for a
savings of 33%. I imagine the savings would be even greater on an instance with lots
of realms. This was tested on dev with one really large realm and one tiny realm.
We were using `.order_by('user_profile_id', '-timestamp') in our
UserPresence query in get_status_dicts_for_query.
We don't need a full sort to produce the dictionary of statuses.
In fact the whole operation in Python is still O(N):
- divvy rows up to be per-user in an O(N) pass
- find max row for the 'aggregated' entry in an O(n) pass
per user
The one minor annoyance of this fix is that datetime_to_timestamp
is lossy, so if you naively call to_presence_dict before finding
the "max" row, you get test flakes if rows are created during the
same second. I decided to avoid calling to_presence_dict so there
are fewer moving parts, but there's still the ugly step of having
to remove the "dt" field from the final results.
This commit completely switches us over to using a
dedicated model called MutedTopic to track which topics
a user has muted.
This includes the necessary migrations to create the
table and populate it from legacy data in UserProfile.
A subsequent commit will actually remove the old field
in UserProfile.
This never made sense to be a flag on the UserMessage table, since
it's not per-user state. And in fact it doesn't need to be in a
database at all, since it's easily computed from content anyway.
Fixes#1099.
Previously, realm.uri and realm.host didn't support using a subdomain
of the empty string (""), aka using the root domain.
Also, since we're already accessing self.subdomain, we don't need to
check REALMS_HAVE_SUBDOMAINS again.
This is the first part of a larger migration to convert Zulip's
reactions storage to something based on the codepoint, not the emoji
name that the user typed in, so that we don't need to worry about
changes in the names we're using breaking the emoji storage.
zerver/message.py used it in this way previously when the type was not
a stream, so the type has been set to match usage and implementation.
Also added docstring to clarify this for the specific function.
We are adding a new list of unread message ids grouped by
conversation to the queue registration result. This will allow
clients to show accurate unread badges without needing to load an
unbound number of historic messages.
Jason started this commit, and then Steve Howell finished it.
We only identify conversations using stream_id/user_id info;
we may need a subsequent version that includes things like
stream names and user emails/names for API clients that don't
have data structures to map ids -> attributes.
Steve Howell
Tim Abbott
rht
Vishnu Ks
Vishnu Ks
Sarah
neiljp (Neil Pilgrim)
Umair Khan
Steve Howell
Jason Michalski
Harshit Bansal
vaibhav