Before, it was trying to use connection.queries, but Django
could pull the rug out from under us. Now we monkeypatch
the CursorDebugWrapper methods instead.
(imported from commit 25d5bab47673f2b66a6325f48d33e66c31055ab3)
When we send a message, we send some presence information to Tornado
to help it figure out how to generate emails for idle recipients of
a message. This change limits the presence info to being the
intersection of present users and recipients of the message. It is
just an internal optimization to avoid queueing up unneeded data.
The history behind this feature is that I implemented it a while
back, but I think I made a rebase mistake that sent all the presence
data over the wire, despite having code to filter on recipients.
It was mostly harmless, just leading to some inefficiency which is
now fixed.
(imported from commit 7c8e97705afb299c67b99053909e952fbc823551)
When you load the activity report, it will just show summary
counts for realms, but if you click on a realm, you will see
details about users in the realms. You can also click "Show all"
to see an interleaved view of realms and users.
(imported from commit b106557b1fae64d525071afc124b5a8aed319086)
Add rows to the activity report that roll up counts for all
users on each realm, to go along with individual users.
(imported from commit 8104f3ef7fbe406fe0fd2ba1bb00ce76a1ccbee5)
The overall message charset may be null or not match the part's
charset. Even though it's unclear from the documentation,
experimentally using the charset for a message part seems to give you
the charset even for non-multipart emails.
(imported from commit 0e1d23073f4c53041f9760e66a6635f8a94893d1)
For a 4-person stream, we were hitting the DB 8 times, and 4 of
those queries were to lazily get user.email for the 4 recipients
due to upstream code using only(). I added user_profile__email
to the only() call.
I believe this regression started 9/18, and after pushing this
to prod, we would should look at this graph:
https://stats1.zulip.net/graphs/8274cd84588
(imported from commit 70629cb69fe5955c674ba76482609dfe78e5faaf)
This is useful in debugging when you just want to discard all the
messages in a queue because they have the wrong structure.
(imported from commit 8559ac74f11841430b4d0c801d5506ebcb74c3eb)
Instead of collapsing muted messages, just hide them altogether
in view where it makes sense to hide them.
(imported from commit 1c2c987ff302ceb135a025753cf421b4de1aea71)
Use stream.num_subscribers() in check_if_a_bot_is_sending_a_message_to_an_empty_stream().
The num_subscribers() function using Django's count() method, which returns
a single row, vs. len() on an iterator of query rows.
(imported from commit 6157fe248945e9288ee71d8cc39fb6dda4e9a247)
This tests that a bot's owner gets sent a message if the bot
sends a message to a stream with no subscribers. (Presumably
the message will be a PM; we could make the test more precise
in the future.)
(imported from commit 0aaf931a90cb9c7bc3fde8ac545c6b6ad0a55668)
This includes:
* Merging the pull request and issue subject creation functions
* Factoring out pull request and issue content creation
(imported from commit 9cf90e999482a1998431e6483788522101607167)
Some bots created by us do not have owners. Don't try to send a
message to the nonexistent owner.
(imported from commit ab952eccd7d6c4728e9477a106142214b5c81ca9)
Instead just rely on the 2-minute delay in the management command to
batch conversations.
We've had people report being confused or thinking the feature was
broken when they didn't get e-mails because of our rate-limiting, so
let's see if this is not too overwhelming.
(imported from commit 706ddb07b906b5c2edea1159c04acc2ee6f06e29)
Warn inside these functions when you get data on streams that you
are not subscribed to:
add_subscriber
remove_subscriber
user_is_subscribed
The back end should be smart enough not to spam us with subscriber
info that we don't care about.
(imported from commit b27644be2abc37c11ddff884ef392ea208bd1bd3)
Don't send peer_add notifications to users who are already
getting add notifications, because they will already know
about subscribers.
(imported from commit 726b54ae0e30b71440b17d9c51b026872ea96218)
It only grabs the user_profile_id column now. This leads to a
speedup of about 16x between grabbing large ORM objects vs.
small 1-column dictionaries.
(imported from commit 95150bff3fdcbe250b04f014062224af42a6644f)
Splitting out notify_peers() will give us flexibility for cleaning
up how we notify peers for bulk adds.
(imported from commit e108fa2c432cc1fe54d788c58c82c983e0f2394e)
If you expand subscribers on your settings page, you will now see
a query like this in your postgres logs:
SELECT "zerver_userprofile"."email"
FROM "zerver_subscription" INNER JOIN "zerver_recipient" ON ("zerver_subscription"."recipient_id" = "zerver_recipient"."id") INNER JOIN "zerver_userprofile" ON ("zerver_subscription"."user_profile_id" = "zerver_userprofile"."id") WHERE ("zerver_recipient"."type" = 2 AND "zerver_subscription"."active" = true AND "zerver_recipient"."type_id" = 40 AND "zerver_userprofile"."is_active" = true )
The join's still complicated, but the list of fields is one instead of 40+.
(imported from commit 48de1f888193a4d23fcea52d0b633d134e4a3ff7)
get_subscribers_backend() now calls the new get_subscriber_emails()
function, which just queries the email field:
"zerver_userprofile"."email"
...instead of querying about 40 fields that it never uses.
I was able to verify the query slimming by watching my postgres server log.
Also, you can verify that the ORM does roughly 16x less work using values():
>>> def f(): return [sub.user_profile.email for sub in list(Subscription.objects.all().select_related())]
...
>>> def g(): return [row['user_profile__email'] for row in list(Subscription.objects.all().values('user_profile__email'))]
...
>>> def timeit(func): t = time.time(); func(); return time.time() - t
...
>>> timeit(f)
0.045198917388916016
>>> timeit(g)
0.002752065658569336
(imported from commit a69f690a96d076b323fdfc2f4821b0548bdfac7f)
LinkPattern returned a string which contained a placeholder if the URL was
considered invalid. AtomicLinkPattern wrapped this in an AtomicString,
where the placeholder doesn't get removed properly.
m.group(0) is always incorrect because python-markdown modifies your regex
to include more than you specified (this is why part of the message got
duplicated).
(imported from commit 576bdf09c2b677cf4bc56484c363eb05f2110158)
* Remove the action from the topic and add the issue title
* Only show the issue body on open or reopen
(imported from commit f08eb40f36122d2498fe0c36a69df9e606296ff3)
We have to read the data anyway, and we don't have a convenient file
handle for uploads from attachments sent through the e-mail gateway.
(imported from commit 86260a4eaceef85c82707929a80558e11dc54ef6)
The get_status_dict_by_realm helper gets called whenever our
realm user_presences cache expires, and it used to query these fields:
"zerver_userpresence"."id", "zerver_userpresence"."user_profile_id", "zerver_userpresence"."client_id", "zerver_userpresence"."timestamp", "zerver_userpresence"."status", "zerver_userprofile"."id", "zerver_userprofile"."password", "zerver_userprofile"."last_login", "zerver_userprofile"."is_superuser", "zerver_userprofile"."email", "zerver_userprofile"."is_staff", "zerver_userprofile"."is_active", "zerver_userprofile"."is_bot", "zerver_userprofile"."date_joined", "zerver_userprofile"."bot_owner_id", "zerver_userprofile"."full_name", "zerver_userprofile"."short_name", "zerver_userprofile"."pointer", "zerver_userprofile"."last_pointer_updater", "zerver_userprofile"."realm_id", "zerver_userprofile"."api_key", "zerver_userprofile"."enable_desktop_notifications", "zerver_userprofile"."enable_sounds", "zerver_userprofile"."enter_sends", "zerver_userprofile"."enable_offline_email_notifications", "zerver_userprofile"."last_reminder", "zerver_userprofile"."rate_limits", "zerver_userprofile"."avatar_source", "zerver_userprofile"."tutorial_status", "zerver_userprofile"."onboarding_steps", "zerver_userprofile"."invites_granted", "zerver_userprofile"."invites_used", "zerver_userprofile"."alert_words", "zerver_userprofile"."muted_topics", "zerver_client"."id", "zerver_client"."name"
Now it queries just the fields it needs:
"zerver_client"."name", "zerver_userpresence"."status", "zerver_userpresence"."timestamp", "zerver_userprofile"."email" FROM "zerver_userpresence"
Also, get_status_dict_by_realm is now namespaced under UserPresence as a static method.
(imported from commit be1266844b6bd28b6c615594796713c026a850a1)
This function gets user presence information, which changes rapidly
and requires a pretty simple query.
(imported from commit f9b9f0f22277335c76eb4371930a4fff2758a240)
The do_send_messages() populates the user_presences data structure
for process_new_message(), so that Tornado code never needs to hit
the database or memcached to get the user presence info.
(imported from commit 194aeaead8fa712297a2ee8aff5aa773b92f1207)
This reduces the number of memcached calls we make in our time-
slice-limited tornado event handler.
(imported from commit 8903ce4ac754ba82d57e04d1b0356be7533edee2)
We create a blueslip error for undefined keys in Dict. This led
to a straightforward change in the unit tests for Dict. For the
unread test, to avoid the blueslip error, we had to be more specific
in setting up a user in one place, but this reduced our coverage,
leading to another small test being added.
(imported from commit 33e14795500d9283de2a7c03c4c58aec11cea4b8)
The exceptions were cryptic before, and they were inconsistent with
the fold_case: false behavior.
(imported from commit a40704d1a22bcdc60d91be832ee3c81eb416c6dd)
There was nothing to ensure that the changes resulting from scrolling
happened before the unread counts were checked. We already had a long
wait there; might as well do those checks after it to ensure that the
DOM is updated.
(imported from commit 0d4014ae6a74dd684521fecabefc4bf79015f842)
These engagement data will be useful both for making pretty graphs of
how addicted our users are as well as for allowing us to check whether
a new deployment is actually using the product or not.
This measures "number of minutes during which each user had checked
the app within the previous 15 minutes". It should correctly not
count server-initiated reloads.
It's possible that we should use something less aggressive than
mousemove; I'm a little torn on that because you really can check the
app for new messages without doing anything active.
This is somewhat tested but there are a few outstanding issues:
* Mobile apps don't report these data. It should be as easy as having
them send in update_active_status queries with new_user_input=true.
* The semantics of this should be better documented (e.g. the
management script should print out the spec above)x.
(imported from commit ec8b2dc96b180e1951df00490707ae916887178e)
The test will fail if a new attribute is added to the structure that
gather_subscriptions() returns. It should only be concerned with the
subscription's color.
(imported from commit fd5bad97bbce2544e0078ee029f54d4e45da9c15)
We found that since bugdown processes are threaded, the cost of
doing a db query in a markdown processor is quite high---each
thread must start up a new db connection including a SSL handshake
etc. We should strive to keep our rendering pipeline free of mandatory
DB queries.
(imported from commit 555066bd03da6c681b74ce6137acc264eb41c55d)
It is triggered by specifying the "language" of a code block to
"quote" or "quoted":
Hamlet said:
~~~ quote
To be or **not** to be.
That is the question
~~~
(imported from commit 847a0602e335e9f2955e32d9955adf8ac8de068c)
Steve Howell
Tim Abbott
Jessica McKellar
Zev Benjamin
Leo Franchi
Waseem Daher
Kevin Mehall
Scott Feeney
acrefoot
acrefoot