There were a couple of bugs in the security checks that resulted in
IRC mirroring of stream messages not working.
(imported from commit 31ac732461a733c1c993f77356053d4f88c67177)
Previously we were having the database do the matching on sender email
address, which resulted in an unnecessary join.
(imported from commit 70bf791a00b7d5965ef977e45b4a0eccbd3402a0)
I believe with this change the log lines will fit much better into
Zulip, and the Client string was I suspect rarely important for
responding to slow queries (and is always available in the main log
anyway).
(imported from commit ad56f446bf3fb96a14a56b825f46c1dad9b6babe)
Summary blocks can contain hundreds of messages. When the rendering window
code didn't take this into account, it would lead to all kinds of
unpleasant behavior when you scroll.
Trac #1888
Unfortunately, this replaces a subtraction with a function that iterates
through all the messages.
(imported from commit 9259a246946cd968a8725c38ff5ef2d4b4793717)
The queued email gets deleted if the user signs up before it gets sent.
Otherwise, they are reminded in 2 days that they still haven't signed up.
This addressses Trac #1812
(imported from commit c1bdc09c03ac576b08986e56994de72d52fd293b)
clear_followup_emails_queue now filters by from_email too
send_local_email_template_with_delay passes the template_payload into the subject template
(imported from commit 8044fe2ebad90a9d6d5c67cdfdd08801760fd7f7)
The current version should only be used for testing; for example,
if you want to create a bunch of streams for stress testing, you
can run this in a loop.
(imported from commit ec51a431fb9679fc18379e4c6ecdba66bc75a395)
We need to run the schema migration manually using
"CREATE INDEX CONCURRENTLY upper_stream_name_idx ON zerver_stream ((upper(name)));"
since we need CONCURRENTLY and I seem to recall that doesn't work with South.
This significantly improves the uncached performance of get_stream()
(e.g. from 32ms to 9ms). At present, this codepath is not used
particularly heavily since we do cache the stream names and do most of
our filtering by recipient ID, but the index isn't expensive and does
provide a significant improvement in the uncached case.
(imported from commit 4d28dc2e9a02d0602861b165393d90ed18f5f4c8)
We need to run the schema migration manually using
"CREATE INDEX CONCURRENTLY upper_subject_idx ON zerver_message ((upper(subject)));"
since we need CONCURRENTLY and I seem to recall that doesn't work with South.
Apparently our existing indexes on subject/topic weren't being used in
our narrowing queries, because we do case-insensitive search.
This substantially improves our database performance around
stream+topic narrows. See before and after query plans below from my
test instance.
humbug=# explain analyze SELECT "zerver_message"."id" FROM "zerver_message" WHERE ("zerver_message"."recipient_id" = 38 AND UPPER(zerver_message.subject) = 'TEST' AND "zerver_message"."id" <= 348495 ) ORDER BY "zerver_message"."id" DESC LIMIT 50;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=13510.61..13510.71 rows=41 width=4) (actual time=32.952..32.958 rows=2 loops=1)
-> Sort (cost=13510.61..13510.71 rows=41 width=4) (actual time=32.946..32.947 rows=2 loops=1)
Sort Key: id
Sort Method: quicksort Memory: 25kB
-> Bitmap Heap Scan on zerver_message (cost=237.99..13509.51 rows=41 width=4) (actual time=2.357..32.912 rows=2 loops=1)
Recheck Cond: (recipient_id = 38)
Filter: ((id <= 348495) AND (upper((subject)::text) = 'TEST'::text))
-> Bitmap Index Scan on zephyr_message_recipient_id (cost=0.00..237.98 rows=8221 width=0) (actual time=1.178..1.178 rows=10354 loops=1)
Index Cond: (recipient_id = 38)
Total runtime: 33.049 ms
(10 rows)
humbug=# explain analyze SELECT "zerver_message"."id" FROM "zerver_message" WHERE ("zerver_message"."recipient_id" = 38 AND UPPER(zerver_message.subject) = 'TEST' AND "zerver_message"."id" <= 348495 ) ORDER BY "zerver_message"."id" DESC LIMIT 50;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=435.11..435.22 rows=41 width=4) (actual time=4.998..4.999 rows=2 loops=1)
-> Sort (cost=435.11..435.22 rows=41 width=4) (actual time=4.997..4.997 rows=2 loops=1)
Sort Key: id
Sort Method: quicksort Memory: 25kB
-> Bitmap Heap Scan on zerver_message (cost=275.63..434.02 rows=41 width=4) (actual time=4.981..4.984 rows=2 loops=1)
Recheck Cond: ((upper((subject)::text) = 'TEST'::text) AND (recipient_id = 38))
Filter: (id <= 348495)
-> BitmapAnd (cost=275.63..275.63 rows=41 width=0) (actual time=4.954..4.954 rows=0 loops=1)
-> Bitmap Index Scan on upper_subject_idx (cost=0.00..37.38 rows=1744 width=0) (actual time=2.972..2.972 rows=27457 loops=1)
Index Cond: (upper((subject)::text) = 'TEST'::text)
-> Bitmap Index Scan on zephyr_message_recipient_id (cost=0.00..237.98 rows=8221 width=0) (actual time=0.855..0.855 rows=10354 loops=1)
Index Cond: (recipient_id = 38)
Total runtime: 5.049 ms
(13 rows)
(imported from commit 1f4815ccb0691053ff8d505149482dbc74153fb3)
It makes the event queue return all messages on public streams, rather
than only the user's subscriptions. It's meant for use with chat bots.
(imported from commit 12d7e9e9586369efa7e7ff9eb060f25360327f71)
By far the common case for get_old_messages is the home view loading
queries, for which we have raw queries. This patch substantially
improves those queries using the observation that we weren't actually
using the zerver_message table that we were joining with.
I actually expect this to result in a noticable performance
improvement for loading of the homepage.
(imported from commit 12807e5a74eb63275b2523a5f62fd901ab632f0f)
Deployment instructions: I think all the queue workers get
restarted automatically, so there is probably nothing special
to do here in the deploy itself, but we will want to monitor
it closely, and the change should make our number of locks go
down.
QueueProcessingWorker.start() now calls consume_and_commit(),
which ensures that we don't hold locks after work actions
by using Django's commit_on_success() decorator.
Obviously, workers that override start() will not call consume_and_commit()
through this code path. SlowQueryWorker calls commit_on_success()
in its start() method now, and I hope to address MissedMessageWorker soon.
(imported from commit f3f38a7f45730eee8f3b5794371ba5b994017676)
These are some queries on API usage, desktop usage, and
Android usage that would be of interest to Waseem. These
will eventually be subsumed into /activity, but some interim
data issues may make them easier to keep separate for now.
(imported from commit 697a8496cbf4447d557a3fc89f64c1c4d3e67e70)
In order to support iOS Push Notifications, we need to keep track
of a device's unique APNS Token. These are delivered to our iOS
code after registering for remote notifications
(imported from commit bbe34483e1380dc20a1c93e3ffa1fcfdb9087e67)
Use the commit_on_success() context manager around the call
to internal_send_message() inside of SlowQueryWorker's polling
loop, so that the pending SELECT statement from
get_status_dict_by_realm() gets committed. If we don't do
this, postgres will hold locks on zerver_userprofile, and other
tables, for a long time, which can interfere with migrations.
This is an interim solution until we switch postgres's default
commit behavior. Right now the default transaction isolation
is "read committed," so SELECT statements lead to AccessShareLocks
that do no get closed until the transaction finishes.
(imported from commit f72aeffbbe71a731e327459f15bd7dbebaf9e0b8)
Trac #1162
The process_fence method replaces code blocks with placeholders, so
indexes stored before the replacement are incorrect. However, because
the closed code blocks have been replaced, we can simply search the
whole string for any remaining opening code block markers.
(imported from commit 6a9e6924840f8f3ca5175da7c52a905e27c1fabd)
I added filter() statements to do_update_message_flags().
Here is some context:
Steve Howell: Case 1, have AND clause to reduce work for DB.
humbug=> update zerver_usermessage set flags = (flags & ~1) where id > 9000;
UPDATE 382
humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
count
-------
382
(1 row)
humbug=> explain analyze update zerver_usermessage set flags = (flags | 1) where (flags & 1) = 0;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Update on zerver_usermessage (cost=0.00..266.85 rows=47 width=27) (actual time=5.727..5.727 rows=0 loops=1)
-> Seq Scan on zerver_usermessage (cost=0.00..266.85 rows=47 width=27) (actual time=0.045..2.751 rows=382 loops=1)
Filter: ((flags & 1::bigint) = 0)
Rows Removed by Filter: 9000
Total runtime: 5.759 ms
(5 rows)
humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
count
-------
0
(1 row)
Leo Franchi: Sounds reasonable, but I know way less than zev about DBs so I'll defer to his judgement :)
Steve Howell: Case 2, how the code works now:
humbug=> update zerver_usermessage set flags = (flags & ~1) where id > 9000;
UPDATE 382
humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
count
-------
382
(1 row)
humbug=> explain analyze update zerver_usermessage set flags = (flags | 1);
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------
Update on zerver_usermessage (cost=0.00..243.28 rows=9382 width=27) (actual time=362.075..362.075 rows=0 loops=1)
-> Seq Scan on zerver_usermessage (cost=0.00..243.28 rows=9382 width=27) (actual time=0.008..6.138 rows=9382 loops=1)
Total runtime: 362.105 ms
(3 rows)
humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
count
-------
0
(1 row)
Steve Howell: In both trials, we set it up so that only 382 of 9382 rows need to be updated. The first trial runs about 63x as fast. The second trial, if my theory is correct, is doing 24x as many writes as it needs. Both trials are reading all 9382 rows.
Steve Howell: The expense of the update statement seems to be proportional to the number of rows you "update", not the number of rows that you actually change.
Steve Howell: For now I created #1869.
Zev Benjamin: That sounds like a reasonable explanation. The disk IO can be expensive
(imported from commit d9090daee1f81cad76c430de0956f9bd504da075)
Handled by the queue processor for signups. Added a management command
that accomplishes the same task, in case it's needed for manually added users,
or in case we goof and need to remove queued emails for a given user.
This addresses Trac #1807
(imported from commit 6727b82a07fa6a3ea3d827860c9e60fd0602297a)
We want to avoid opening a DB connection in the markdown thread
as its DB connection might live for a long time
(imported from commit 7700b2ca793ee5e9add7f071b92f22a4bf576b3d)
This will hopefully incentivize people to click one and get back into
the app.
We'll also need this for digest emails.
(imported from commit 57191c3fcca3b12df93a81e4692bb7eb8ccc83b2)
This requires renaming the account in Google Apps at the time we
deploy this; we'll probably want to do this during off hours to avoid
any user-visible downtime.
This also updates some related email addresses.
(imported from commit fce7629b359a4f278bbf7815c8d177a8fa0484fe)
This may require just doing an mv on the home directory, plus changing
the home directory in /etc/passwd. It should of course be done carefully.
(imported from commit 660997d897ee6d33563af74f0fc5d4267a911755)
A few "slow" tests aren't as slow any more, for whatever reason,
so we're setting a higher bar going forward.
(imported from commit 642137cebb7826f4512b5635da9d7b75bd5c35f4)
The text of manual links are already AtomicStrings, so linkified strings
should be too.
Moves emoji detection to happen after linkification, so the emoji rule
won't look at links.
(imported from commit 9c56bce6a0e873b398255e0762dfb312a4a9a64e)
InlinePatterns should return None on failure, not text that may
have placeholders in it.
(imported from commit f9d8d22b2b8cfa7a92ecf3e52a6c76b48e6f0175)
We want the UserActivity.query field to reflect the name of the
function for REST calls, not the URL, and we accomplish this by
setting request._query to target_function.__name__.
(imported from commit 9df05fef0dffb34483b182b95f8cbc4409083eed)
If request._query is set in the call to update_user_activity(),
we will use that instead of request.META['PATH_INFO'] for
the query field of the UserActivity row we write.
(imported from commit fcee30098e1c7c5cb4195a1e5905fc7b88af804f)
This results in some small behavior changes. First, if a user
has both malformed JSON and an invalid API key, they will now
be informed of the invalid API key, not the malformed JSON,
because the decorator wrapping code executing first. Second,
we call process_client(), which basically builds us a
UserActivity record with the client "API".
(imported from commit fadb523db9bdc82984bdae61833c5c99f1ebd1c0)
This is for webhook API endpoints that only get passed in an api_key,
not an email. An example would be api_jira_webhook, and some of
the code is borrowed from there. The rest of the code is from
authenticated_api_view().
(imported from commit b5b2a4ea52f9b317f00357ef3142c76534fabf20)
Add the number of person-minutes for the last 24 hours to the
realm report on the main tab of /activity.
(imported from commit 2ff46eacc4c8276ab0407fc6ff9f28f5137f1ed2)
When decoding an operand, a + can be converted to a space
only if the operand is not an email address.
(imported from commit 08fc36a579bbe6409137c60c0fa9579fe3ab2c43)
This tab shows how long each user has been on during the last 24
hours, using data from UserActivityInterval. Much of the code
is borrowed from analyze_user_activity.py, but in this version
we set the time interval to be the last 24 hours and sort by
realm and email. I also ensure that it only executes one
query to get all the data (and there's test coverage for that).
(imported from commit 7a2b80f52679054b03c5f5f42b2cda07d5599432)
Waseem is ok with removing the client-specific tabs on the
main /activity page. This reduces the number of queries from
25 to 1. We might eventually restore some of that logic, but
we will do it more efficiently. A lot of the data for
non-website clients is kind of unreliable, anyway.
The page looks kind of funny with only one tab, but that
will be fixed in the next commit.
(imported from commit 54f08f89d5242ad3e045d8ca0d97b86617c15380)
When we don't already have old messages in cache, we need to
fetch data from the database and create dictionaries for the
cache. This commit makes that process work in 50ms, instead
of 130ms, for the data set in test_bulk_message_fetching(),
which is 602 records. Before this commit we had about 132
microseconds of unnecessary churn per message, because we
were fetching DB fields we didn't need and incurring the cost
of the Django ORM. Now we use values() to get only the columns
we need, and we take advantage of previous commits that make
our code less OO and more function-driven, so we can pass the
values directly to build_message_dict() without having to create
objects.
A couple caveats on this commit:
1) I haven't been able to get good measurements on the overall
effect on get_old_messages_backend(). If you kill the cache to
force DB queries, you introduce noise related to sessions and
user profiles.
2) Look at the long comment in this commit related to
re-rendering messages in this codepath. The problem precedes
this commit.
(imported from commit dcb64aa9416f0e9583355ddd6dc3adfa746b9fc7)
Only call a function on the message object in the unfortunate
situation that we are rendering new content in to_dict_uncached().
Long term, it would be nice if this function didn't have side
effects, and we had a better strategy for upgrading rendered
content when bugdown versions change.
(imported from commit 2a323f52af37a6d651c171cb8234fbfa3d25d561)
This function doesn't require the whole UserProfile object to
create the avatar url, and we call it from Message.to_dict_uncached().
(imported from commit e814caab101c4fedd1ba66df041a3408014e4085)
For a bunch of self-dot references, move them to the top. (This
is kind of funny out of context, but it sets us up for future
refactorings.)
(imported from commit 4ebc1c44a633d86772df1828c51180707769c3dc)
If this line of code were ever called, it would crash anyway,
because it would be an unknown type, and Recipient.type_name()
would raise a KeyError.
(imported from commit db38c5f71fb2f0b044a832eb88e53fceb0d8a9cf)
This is a variation of get_display_recipient that takes
values instead of an object, so that it is decoupled from
the Django object system.
(imported from commit 25bed43ecd62f1fe0176d517b7003e7f4c78bc37)
If it's ok for the tests to use memcached, it should be ok
for them to use the in-process cache too.
(imported from commit be43879c3c48f3780317fd5b4139b44d4a1f0ed3)
This is a harmless extraction designed to allow subclasses override
the behavior of how rendered content gets saved.
(imported from commit 9df4ed9f86c857897fcb5f2b6781bfc5a0813766)
The realm should always be the realm of the stream, and we should
always pass in a stream rather than sometimes passing in a stream name
and other times passing in a stream.
(imported from commit a098d6ed3db218a37c1b6b7c956e847c316c2d13)
There is a scenario where we call process_read_message()
for a message that we haven't recorded as unread before.
I'm not sure how it happens, but I put back code to
guard against crashing. The regression happened in
5752458c821.
(imported from commit 5ce15d2e236b738b445ed88f1733aa0612be0ff3)
We have been persisting muting preferences on the back end for
a while, but we haven't been adding them to page_params for the
client to have at reload/startup time.
(imported from commit d9ca68aa0e4d22bfb0e6ce67fc0bc63981175c8b)
Update get_counts() so that it ignores counts for muted topics
when calculating stream/home unread counts.
(imported from commit 9b4e4da4346c225c535e97d709d3dee032603cc5)
The indirection was more confusing than helpful, especially
since the function had side effects, despite its getter-like
name.
(imported from commit 85d9cf642b4177f62488136f0e0f7f6c9304942e)
Empirically, we only get these for malformed emails where the charset
specified in a message part header does not match the true encoding of
the part. I checked what the resulting Zulip looked like for the
original offender, and it looked find with ignoring errors.
(imported from commit ac6ba65b611cb22d4ec547b75a585abce6fc50b0)
We now bulk-fetch subscription information once from the database
and use it throughout bulk_add_subscriptions in order to avoid
hitting the db O(streams) times.
On my machine this shaved the accounts_register API call from making
66 queries to making 37 queries.
(imported from commit 5dd5ad3f50b2a6edf85b5f1d55ebd697a1c60647)
We have a handy bulk_add_subscriptions function to make cases
like this fast, so lets use it.
On my machine this reduces the number of db queries during account_register
from 112 to 66.
(imported from commit 21a6b31d0f229998d095735b8c581a50ca6aab66)
It shows domains and how many active users they have. A
user is consider active if they have done something at least
as active as updating their pointer in the last day. Domains
with no meaningful activity in the last two weeks are excluded
from the report.
(imported from commit 700cecfc7f1732e9ac3ea590177da18f75b01303)
A small functional change here was to eliminate an enormous "Usage"
headline that was already implicit from the tabs. It would have
complicated the refactoring to try to preserve it, and I don't think
anyone will miss it.
Extracting this template will give us a little more flexibility
to customize future tabs in the /activity page.
(imported from commit bdb0b7030c8ec1e20d4451dc059830c3f5ea7632)
We are still showing the same data points, but the logic to drill
down on details for a particular realm is now all server side,
not client side, and we are smarter about omitting fields. In
summary mode, we don't show empty Name or Email columns. In
detailed mode, we show the realm as a headline instead of a column.
In this version you do lose the ability to see all system users in
the same view, but Waseem is ok with this.
(imported from commit edd2e646ab4cf5783ea64232d0cd621debece8d4)
Before, it was trying to use connection.queries, but Django
could pull the rug out from under us. Now we monkeypatch
the CursorDebugWrapper methods instead.
(imported from commit 25d5bab47673f2b66a6325f48d33e66c31055ab3)
When we send a message, we send some presence information to Tornado
to help it figure out how to generate emails for idle recipients of
a message. This change limits the presence info to being the
intersection of present users and recipients of the message. It is
just an internal optimization to avoid queueing up unneeded data.
The history behind this feature is that I implemented it a while
back, but I think I made a rebase mistake that sent all the presence
data over the wire, despite having code to filter on recipients.
It was mostly harmless, just leading to some inefficiency which is
now fixed.
(imported from commit 7c8e97705afb299c67b99053909e952fbc823551)
When you load the activity report, it will just show summary
counts for realms, but if you click on a realm, you will see
details about users in the realms. You can also click "Show all"
to see an interleaved view of realms and users.
(imported from commit b106557b1fae64d525071afc124b5a8aed319086)
Add rows to the activity report that roll up counts for all
users on each realm, to go along with individual users.
(imported from commit 8104f3ef7fbe406fe0fd2ba1bb00ce76a1ccbee5)
The overall message charset may be null or not match the part's
charset. Even though it's unclear from the documentation,
experimentally using the charset for a message part seems to give you
the charset even for non-multipart emails.
(imported from commit 0e1d23073f4c53041f9760e66a6635f8a94893d1)
For a 4-person stream, we were hitting the DB 8 times, and 4 of
those queries were to lazily get user.email for the 4 recipients
due to upstream code using only(). I added user_profile__email
to the only() call.
I believe this regression started 9/18, and after pushing this
to prod, we would should look at this graph:
https://stats1.zulip.net/graphs/8274cd84588
(imported from commit 70629cb69fe5955c674ba76482609dfe78e5faaf)
This is useful in debugging when you just want to discard all the
messages in a queue because they have the wrong structure.
(imported from commit 8559ac74f11841430b4d0c801d5506ebcb74c3eb)
Instead of collapsing muted messages, just hide them altogether
in view where it makes sense to hide them.
(imported from commit 1c2c987ff302ceb135a025753cf421b4de1aea71)
Use stream.num_subscribers() in check_if_a_bot_is_sending_a_message_to_an_empty_stream().
The num_subscribers() function using Django's count() method, which returns
a single row, vs. len() on an iterator of query rows.
(imported from commit 6157fe248945e9288ee71d8cc39fb6dda4e9a247)
This tests that a bot's owner gets sent a message if the bot
sends a message to a stream with no subscribers. (Presumably
the message will be a PM; we could make the test more precise
in the future.)
(imported from commit 0aaf931a90cb9c7bc3fde8ac545c6b6ad0a55668)
This includes:
* Merging the pull request and issue subject creation functions
* Factoring out pull request and issue content creation
(imported from commit 9cf90e999482a1998431e6483788522101607167)
Some bots created by us do not have owners. Don't try to send a
message to the nonexistent owner.
(imported from commit ab952eccd7d6c4728e9477a106142214b5c81ca9)
Instead just rely on the 2-minute delay in the management command to
batch conversations.
We've had people report being confused or thinking the feature was
broken when they didn't get e-mails because of our rate-limiting, so
let's see if this is not too overwhelming.
(imported from commit 706ddb07b906b5c2edea1159c04acc2ee6f06e29)
Warn inside these functions when you get data on streams that you
are not subscribed to:
add_subscriber
remove_subscriber
user_is_subscribed
The back end should be smart enough not to spam us with subscriber
info that we don't care about.
(imported from commit b27644be2abc37c11ddff884ef392ea208bd1bd3)
Don't send peer_add notifications to users who are already
getting add notifications, because they will already know
about subscribers.
(imported from commit 726b54ae0e30b71440b17d9c51b026872ea96218)
It only grabs the user_profile_id column now. This leads to a
speedup of about 16x between grabbing large ORM objects vs.
small 1-column dictionaries.
(imported from commit 95150bff3fdcbe250b04f014062224af42a6644f)
Splitting out notify_peers() will give us flexibility for cleaning
up how we notify peers for bulk adds.
(imported from commit e108fa2c432cc1fe54d788c58c82c983e0f2394e)
If you expand subscribers on your settings page, you will now see
a query like this in your postgres logs:
SELECT "zerver_userprofile"."email"
FROM "zerver_subscription" INNER JOIN "zerver_recipient" ON ("zerver_subscription"."recipient_id" = "zerver_recipient"."id") INNER JOIN "zerver_userprofile" ON ("zerver_subscription"."user_profile_id" = "zerver_userprofile"."id") WHERE ("zerver_recipient"."type" = 2 AND "zerver_subscription"."active" = true AND "zerver_recipient"."type_id" = 40 AND "zerver_userprofile"."is_active" = true )
The join's still complicated, but the list of fields is one instead of 40+.
(imported from commit 48de1f888193a4d23fcea52d0b633d134e4a3ff7)
get_subscribers_backend() now calls the new get_subscriber_emails()
function, which just queries the email field:
"zerver_userprofile"."email"
...instead of querying about 40 fields that it never uses.
I was able to verify the query slimming by watching my postgres server log.
Also, you can verify that the ORM does roughly 16x less work using values():
>>> def f(): return [sub.user_profile.email for sub in list(Subscription.objects.all().select_related())]
...
>>> def g(): return [row['user_profile__email'] for row in list(Subscription.objects.all().values('user_profile__email'))]
...
>>> def timeit(func): t = time.time(); func(); return time.time() - t
...
>>> timeit(f)
0.045198917388916016
>>> timeit(g)
0.002752065658569336
(imported from commit a69f690a96d076b323fdfc2f4821b0548bdfac7f)
LinkPattern returned a string which contained a placeholder if the URL was
considered invalid. AtomicLinkPattern wrapped this in an AtomicString,
where the placeholder doesn't get removed properly.
m.group(0) is always incorrect because python-markdown modifies your regex
to include more than you specified (this is why part of the message got
duplicated).
(imported from commit 576bdf09c2b677cf4bc56484c363eb05f2110158)
* Remove the action from the topic and add the issue title
* Only show the issue body on open or reopen
(imported from commit f08eb40f36122d2498fe0c36a69df9e606296ff3)
We have to read the data anyway, and we don't have a convenient file
handle for uploads from attachments sent through the e-mail gateway.
(imported from commit 86260a4eaceef85c82707929a80558e11dc54ef6)
The get_status_dict_by_realm helper gets called whenever our
realm user_presences cache expires, and it used to query these fields:
"zerver_userpresence"."id", "zerver_userpresence"."user_profile_id", "zerver_userpresence"."client_id", "zerver_userpresence"."timestamp", "zerver_userpresence"."status", "zerver_userprofile"."id", "zerver_userprofile"."password", "zerver_userprofile"."last_login", "zerver_userprofile"."is_superuser", "zerver_userprofile"."email", "zerver_userprofile"."is_staff", "zerver_userprofile"."is_active", "zerver_userprofile"."is_bot", "zerver_userprofile"."date_joined", "zerver_userprofile"."bot_owner_id", "zerver_userprofile"."full_name", "zerver_userprofile"."short_name", "zerver_userprofile"."pointer", "zerver_userprofile"."last_pointer_updater", "zerver_userprofile"."realm_id", "zerver_userprofile"."api_key", "zerver_userprofile"."enable_desktop_notifications", "zerver_userprofile"."enable_sounds", "zerver_userprofile"."enter_sends", "zerver_userprofile"."enable_offline_email_notifications", "zerver_userprofile"."last_reminder", "zerver_userprofile"."rate_limits", "zerver_userprofile"."avatar_source", "zerver_userprofile"."tutorial_status", "zerver_userprofile"."onboarding_steps", "zerver_userprofile"."invites_granted", "zerver_userprofile"."invites_used", "zerver_userprofile"."alert_words", "zerver_userprofile"."muted_topics", "zerver_client"."id", "zerver_client"."name"
Now it queries just the fields it needs:
"zerver_client"."name", "zerver_userpresence"."status", "zerver_userpresence"."timestamp", "zerver_userprofile"."email" FROM "zerver_userpresence"
Also, get_status_dict_by_realm is now namespaced under UserPresence as a static method.
(imported from commit be1266844b6bd28b6c615594796713c026a850a1)
This function gets user presence information, which changes rapidly
and requires a pretty simple query.
(imported from commit f9b9f0f22277335c76eb4371930a4fff2758a240)
The do_send_messages() populates the user_presences data structure
for process_new_message(), so that Tornado code never needs to hit
the database or memcached to get the user presence info.
(imported from commit 194aeaead8fa712297a2ee8aff5aa773b92f1207)
This reduces the number of memcached calls we make in our time-
slice-limited tornado event handler.
(imported from commit 8903ce4ac754ba82d57e04d1b0356be7533edee2)
We create a blueslip error for undefined keys in Dict. This led
to a straightforward change in the unit tests for Dict. For the
unread test, to avoid the blueslip error, we had to be more specific
in setting up a user in one place, but this reduced our coverage,
leading to another small test being added.
(imported from commit 33e14795500d9283de2a7c03c4c58aec11cea4b8)
The exceptions were cryptic before, and they were inconsistent with
the fold_case: false behavior.
(imported from commit a40704d1a22bcdc60d91be832ee3c81eb416c6dd)
There was nothing to ensure that the changes resulting from scrolling
happened before the unread counts were checked. We already had a long
wait there; might as well do those checks after it to ensure that the
DOM is updated.
(imported from commit 0d4014ae6a74dd684521fecabefc4bf79015f842)
These engagement data will be useful both for making pretty graphs of
how addicted our users are as well as for allowing us to check whether
a new deployment is actually using the product or not.
This measures "number of minutes during which each user had checked
the app within the previous 15 minutes". It should correctly not
count server-initiated reloads.
It's possible that we should use something less aggressive than
mousemove; I'm a little torn on that because you really can check the
app for new messages without doing anything active.
This is somewhat tested but there are a few outstanding issues:
* Mobile apps don't report these data. It should be as easy as having
them send in update_active_status queries with new_user_input=true.
* The semantics of this should be better documented (e.g. the
management script should print out the spec above)x.
(imported from commit ec8b2dc96b180e1951df00490707ae916887178e)
The test will fail if a new attribute is added to the structure that
gather_subscriptions() returns. It should only be concerned with the
subscription's color.
(imported from commit fd5bad97bbce2544e0078ee029f54d4e45da9c15)
We found that since bugdown processes are threaded, the cost of
doing a db query in a markdown processor is quite high---each
thread must start up a new db connection including a SSL handshake
etc. We should strive to keep our rendering pipeline free of mandatory
DB queries.
(imported from commit 555066bd03da6c681b74ce6137acc264eb41c55d)
It is triggered by specifying the "language" of a code block to
"quote" or "quoted":
Hamlet said:
~~~ quote
To be or **not** to be.
That is the question
~~~
(imported from commit 847a0602e335e9f2955e32d9955adf8ac8de068c)
The new version is more accurate (doesn't rely on UserMessage IDs
being sorted, which they aren't necessarily) and simpler.
(imported from commit 671dd89dc8881ae2dcb8d0e804fd65458e074a29)
This will allow mail with an implicit destination (mailing lists, complex
forwarding) to be received by our system.
See http://support.google.com/a/bin/answer.py?hl=en&answer=2368151 for
documentation of Google's behaviour that adds this header.
(imported from commit f8fd500e3c27e12af5941c63c91d5c796a2cd24a)
Previously the email gateway had to be the only address in a recipient
field or we'd mis-parse the recipient.
This commit also makes the mirror correctly handle addresses of the
form "Jessica McKellar <jesstess@zulip.com>".
(imported from commit 7435f2b59b8f47dc599cc869f64597a730af7d12)
The mirror will use INBOX when deployed and Test locally. Send an
e-mail to the Test mailbox by including the word "localhost" in the
subject; a GMail filter will place it in Test on receipt.
(imported from commit bacf9a9554c8c5e1f3ec8497761edf2c15d3745d)
Previously, when added to an invite-only stream, you got notifications
like this:
Hi there! We thought you'd like to know that Some Person just subscribed
you to the invite-only stream 'Secret stuff'
You can see historical content on a non-invite-only stream by narrowing
to it.
Note that the second line is irrelevant and confusing in light of the
first!
This commit leaves this out, and also, to make sure I didn't mess
anything up (and because I needed to change the tests anyway), adds a
test for invite-only stream notifications.
(imported from commit 49c333629c78fc06f6d2f1ec8a627c6d38e7716a)
It was getting hard to follow and is going to get more complicated
with a new super user check in a later commit.
(imported from commit 8d5cfa960824519d87ce0f09aab3a120ba9ef357)
An important part of this is updating the various caches that cache
the display_recipient.
(imported from commit 888bf54fd205516cf31a25ba3f4e45ad11bbd4d5)
This cache was created to make recipient lookups within a single
request (e.g. when fetching old messages) cheaper. To support stream
name changes, we need to invalidate this cache on every request so
that users get a consistent view of the name change.
(imported from commit 801051b9f6a108c1f50be7eca9a1242d661919b1)
This shows up when you're not running a Zephyr mirroring bot and lets
you use Webathena to have us run it. Obviously needs more docs.
Current problems include:
* supervisorctl reload ends up recreating /var/run/supervisor.sock
with the wrong permissions, so it only works once in a row before
you need to chmod that.
* /etc/supervisor/conf.d needs to be humbug-writeable; this is a clear
local root vulnerability
* This uses SSH and thus is kinda slow.
(imported from commit 7029979615ffd50b10f126ce2cf9a85a5eefd7a2)
Before this it was [deleted]. Using parens is consistent with how we put
in (no topic) if you don't specify a topic.
(imported from commit 931c06a1096cf7b0d226336cbe82535abd2e6032)
This is experimental, for staging only. There might be a better
way to model this than dueling force_expand/force_collapse flags,
but it works for now. The code in collapse_recipient_group()
could also be DRYed up relative to expand_summary_row().
(imported from commit 107151d1ecd640970fb7700d41278a003bd1abaa)
I believe the test data to be incorrect, but we'll find out when
we've captured some real data.
(imported from commit ae32af0436b6273d0b2a60b2c55047aace6adcad)
I was saying bar.d in places where I wasn't really specifically
testing the .d feature, and it was distracting and just an
unintentional consequence of copy/paste.
(imported from commit 7b137b28cb33c72b83f02fe1d2961c5c6accc263)
This change will allow us to test the muting feature on
staging. Any topic named "muted" will automatically be
muted. You can also mute any other topic on the console:
muting.mute_topic('devel', 'ios');
current_msg_list.rerender();
More UI around this experiment will be coming soon, as well
as support for muting entire streams.
The muting module keeps track of which topics are muted, but a
user can expand muted messages, and once that happens, the
messages are marked with the "force_expand" flag that gets
persisted to the back end.
Muted messages are rendered in similar fashion to the summarized
rows, and as part of unifying some of that code, we have
made it so that expanding a summarized section doesn't remove
individual flags related to summaries; instead, the messages
get the force_expand flag set.
(imported from commit acee4190e63813d46850415c41ff8ebfae4a6953)
I regressed this recently, thinking that all our operators are
strings, but I forgot about the "near:" operator used in the
"Narrow messages around this time" feature. The user facing
symptom was that the search bar showed up empty instead
of saying near:50, which might actually be the better
behavior, but it certainly was not intentional. :)
(imported from commit fcb93cecbe9a052bb9bc1af7fcac5aecaba5aafb)
I'm trying to move well-isolated methods out of narrow.js, so that
narrow.js is more strongly focused on UI/ajax interactions and
big, heavy lifting stuff. The logical home for parse/unparse
seemed to be Filter, and they brought along two private methods
with them. The big code moves involved trivial follow ups
like s/exports/Filter/.
(imported from commit ace0fe5aa1c7abce0334d079ba9eb8d9a57bd10f)
This is hard to break up into separate commits; sorry about that.
Before this commit we had 3 tests:
1. Claims to check that an unread count was 3, but actually doesn't.
2. Checks that scrolling down causes the left-sidebar stream unread
count to decrease.
3. Claims to check that unread counts are correct, but actually doesn't.
From talking to Leo, it seems that he originally tried to actually do
what tests 1 and 3 claim to do, but found it too fragile to check an
exact unread count because of font sizes, layout, etc.
We now have 4 tests. For each of the stream sidebar and user sidebar, we
test that:
1. Scrolling down causes the unread count to decrease
2. Logging out and back in again leaves the unread count unchanged
I've removed the two bogus tests and some other code that didn't seem to
serve a purpose.
(imported from commit 9f8e4b521e2765099510426d0b7e2960885e6f19)
You used to have to call casper.test.done(N) where N was the number of
tests run. This is no longer required and is deprecated in CasperJS 1.1.
(imported from commit 0de9ecb1930cbce416fa02c24a882e926cdc8e87)
Have ui.set_presence_list() only touch the presence list.
Before this change, it was calling update_unread_counts(), which
has a bunch of side effects unrelated to the presence list.
(imported from commit 690f754d78874a03fa36f8ff8765d5a63e431d28)
Previously it only provided the list of all public streams; now it
allows one to specify any union of some of the following:
* all public streams
* all streams the user subscribed to
(the most relevant being the union of those two, which is what we want
for the "streams" page).
Or:
* all streams in realm (superuser only)
The manual task required is that when this is pushed to prod, we need
to also deploy the new sync-public-streams version to zmirror.
(imported from commit 27848b8bd136e2777f399b7d05b2fdcec35e4e21)
Github's documentation is a little unclear on what data is actually
sent and in some cases conflicts with the examples in the
github-services repo. The hope here is to capture a good sample of
different events to develop against and then use it in testing.
(imported from commit 356935894b87ca21591f5682c15f9ed27289a5b6)
The functions add_dependencies() and set_global() are convenience
methods that allow you to modify the global namespace while
the current file is running but then have it be cleaned up
by index.js when you're done.
(imported from commit f75b8a10c19f773a8d2d3a8fa4bc39b1679566fe)
This is like Python's dict.setdefault. I don't love the name, but
the consistency is nice.
We have lots of places where we do things like:
if (! dict.has('foo')) {
dict.set('foo', []);
}
var arr = dict.get('foo');
arr.push(3);
We can now write:
var arr = dict.setdefault('foo', []);
arr.push(3);
(imported from commit b8933809c69ba47ec346ed51d53966793403e56c)
This helps make our statuses more meaningful and should resolve trac #1534.
As part of this, we lower OFFLINE_THRESHOLD_SECS to 1.1̅6 minutes and
mark the user as idle after 5 minutes.
(imported from commit ee6b1ad203554a84b11e16c4c6195be9df5bcf4f)
For syncing streams between Zephyr and Zulip, we need to be able to
have the API client send the server a long list of streams, some of
which might be invite-only, and add the ones that it can add and not
the ones it cannot without a bunch of annoying round trips dropping
individual streams one is not authorized to one by one. This argument
makes that possible.
We might find other applications as well.
(imported from commit 9236d185897c42218ab6cac3d8f3ddcb1bbc94e9)
This changes the mit.edu access rules from:
* Susbcriber list and inviting users to streams are unavailable
to
* Susbcriber lists and inviting users to streams are only available
for invite-only streams
streams must still be made invite-only manually.
This both cuts down on the amount of code that is different between
the mit.edu user experience and the standard one, as well as paving
the way for us to invite-only streams for zcrypt.
(imported from commit 24e0e85428608c05c89eeea349338dd392e5489a)
It's likely we'll eventually want to split this into 2 API queries,
but most of the code in get_public_streams_backend is specialized for
the performance requirements of the mit.edu mirroring anyway.
(imported from commit 47ac6586b59467fd64992c573b6e36885167dcc5)
The function narrow.unparse() is used in a bunch of places in
the search suggestion code, and now it no longer lower cases
operands. This change contributes to fixing trac #1659.
(imported from commit 6b44b8a818482b5c8b4f9a45bc7d3a9d21e04eba)
Streams are converted to their "official" names now.
Topics are not canonicalized at all.
All other operands continue to be lowercased.
Since we don't lowercase stream/topic at the parsing stage,
we have to modify the predicate function to do the lowercasing
of stream/topic to enable case-insensitive comparisons. This
is slightly more expensive. The server-side predicate
functions are already case-insensitive.
(imported from commit 286f118c6c3ff9d23b37c7f958cab4c0eacd5feb)
If we have a stream named "Denmark" and we're narrowed to it,
then use "Denmark" as the default stream name in the compose box
even if the narrow operators are lowercase.
(imported from commit e9f06b7307c73231aa887dc95849e0307984e6f0)
This function returns the stream's actual name, if we can get it;
otherwise, it's the identity function.
(imported from commit 7a981adba9632d6c6eba54cb6514a9226d1e83e8)
There are no functional changes; you can still use the shell script
tools/test-js-with-node. It just delegates now to the new index.js to
iterate through all the other .js files in the test directory and run
them. This sets the stage for Istanbul to correctly compute test
coverage.
(imported from commit 6f521c78b7a314d010fa113f9c2c971ab999b637)
Trac #1716
Fix the tests to use real message IDs for the pointer. One helper didn't
use the pointer ID it was passed at all, so the test didn't actually test
what it was supposed to before.
(imported from commit 457bcdb04a6c3873e224b68cd9d79c9a26612010)
This change would allow anyone in the realm to set a topic for a "no topic"
message. As soon as the message topic is set, only the sender can change it again.
(imported from commit 0a91a93b8fd14549965cedc79f45ffd869d82307)
This solves the problem of restoring a tab taking you to the previous
load's initial pointer position.
(imported from commit e5e988af65966b15a9d091064b65f87be3f0d75a)
For now, just do this, and we'll reach out to realms having trouble
manually. We may eventually need to automatically reply to the e-mail,
reach out to a realm admin, etc.
(imported from commit 5c5ac354066f9e9be3fb928e1f8801613c22c1ac)
We do this because the upcoming Bitbucket webhook does not send
enough information for us to send a useful message for deleted
branches or force-pushes.
(imported from commit dcac675f47e10672125caabd6fefa8dc0dc9c86c)
This otherwise causes Unicode bugs. See for example:
>>> import urllib
>>> urllib.quote_plus("hello&world+whats@up?")
'hello%26world%2Bwhats%40up%3F'
>>> urllib.quote_plus(u"faraoné")
/usr/lib/python2.7/urllib.py:1268: UnicodeWarning: Unicode equal
comparison failed to convert both arguments to Unicode - interpreting
them as being unequal
return ''.join(map(quoter, s))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/urllib.py", line 1275, in quote_plus
return quote(s, safe)
File "/usr/lib/python2.7/urllib.py", line 1268, in quote
return ''.join(map(quoter, s))
KeyError: u'\xe9'
>>> urllib.quote_plus(u"faraoné".encode("utf-8"))
'faraon%C3%A9'
(imported from commit 7c13b749bdc7f89e7b767ddd630be0ebce30802c)
Dict.each() allows to iterate through values and keys of a Dict.
The callback function is passed value as the first parameter to
be similar to _.each()'s calling sequence.
(imported from commit e745e8b5d2f167b8b8acf7542b767494e354b037)
This commit CANNOT be deployed until the previous schema change
([schema] models: add an email_token field to Streams) is on prod.
Before applying this schema change, run the populate-stream-tokens
management command to generate tokens for streams that need them.
(imported from commit 7adc81c8c317ec5d59dd59ba42a4dc1a46174007)
This has the amusing side effect of showing all the Zulip bots in the
administration view because none of them have the is_bot set.
(imported from commit cdec19d2109c092018c1f331aa32f345d1587683)
We now show a list of users and allow you to deactivate a user using the
same process as `python manage.py deactivate_user`.
We add a new menu item accessible from the gear icon which will eventually
have much more than just this, but we have a good start here.
Here we also add a property to UserProfile which determines whether you're
eligible to access the administration panel, and then have code which shows
the menu option if so.
This introduces a new JS file, admin.js.
(imported from commit 52296fdedb46b4f32d541df43022ffccfb277297)
Previously, we'd determine unread counts client-side as messages got
loaded, which meant:
a) how many unread messages you were known to have kept changing
b) you could bankrupt yourself and then get the bankruptcy message
again as more messages arrived from the server
(imported from commit 90f1af27b493c720f77d37487d8746749faf157e)
ALLOW_REGISTER was no longer being used in determining whether you could
register for the app, so I've removed it to avoid additional local-dev /
production issues.
This closes#1613.
(imported from commit c928c6d350602d35f745ae1e60d734e4567885fc)
We add a new validator that ensures that people who sign up with @mit.edu
addresses are in fact MIT users.
This closes#1612.
(imported from commit 1e30794b1615dd57cb0e367d1fa186a877253357)
On Debian systems, this is found in the `python-dns` package.
On OS X and others, install "pydns" using your Python package manager.
(imported from commit 17827d0a1d3d72b12945df5563295a1573bfa1ed)
This was previously causing us to generate a traceback every time we
hit a duplicated zephyr due to CC'ing.
(imported from commit 240e1559655d0166dcd864e84649ab97b87a29ad)
1) The class Filter now lives in its own module.
2) The function canonicalized_operators() is now a class method on Filter.
3) The function message_in_home moved to filter.js and became private.
4) Various calling code had to change, of course.
5) Splitting out Filter helped simplify a few tests.
(imported from commit e41d792b46d3d6a30d3bd03db0419f129d0a2a7b)
This is an experiment to try to ease the customer3 onboarding process
given that there will be a website with a public feed of recent Zulip
messages, to ensure that those messages appear for new users.
(imported from commit 31fb59c1800728b5e4d8a5ce7282c7dcedb02b21)
This if statement stopped working when we made the pointer managed via
the events system.
(imported from commit 382ca5bd055ab85048b211068ff3d43a47607f28)
There are also one or two places we don't need to use it for security
purposes, but we do so for consistencey.
(imported from commit aa111f5a22a0e8597ec3cf8504adae66d5fb6768)
util.enforce_arity takes a function and returns a new version which
throws an error if an incorrect number of arguments (as determined by
the function prototype) are passed.
(imported from commit 20e69a6dc7b6f8455726ab4fae8d5b7b04dc4103)
Our API documentation says that we do, and it seems like it could be
useful to clients, so we might as well do it.
(imported from commit c391e4952a09d41df4dc06e3dc6ee094f774822b)
The main changes are:
(1) Changing the input format for the example response so that it is
human-readable and editable
(2) Updating it to use the events API
(imported from commit 308fade9595d6877836d343d2307e3fceff3e7d4)
The e-mail forwarder will use this. Set it to nullable temporarily to
accomodate existing streams; later commits will a) provide a script to
give all streams a token, and b) make the field non-null.
Realm administrators will eventually have a UI to regenerate stream
tokens.
(imported from commit a084d0a7012eb9665e4da095cbc46aa9ef354eaa)
The test-all script now calls the symlink, and the run script
has been cleaned up to be symlink friendly.
(imported from commit 8abb5c1e5744416e94ff843e50c53e0d0f7e1316)
This fixes trac #1660. A deactivated user may still have the
active flag set to "true" on their subscriptions, but this is
just to help future reactivations; they are not actually
valid subscribers while deactivated, so we add UserProfile.is_active
to our filter in the query in get_subscribers_backend().
(imported from commit 8598b2e180faea618371293e42b794898e645004)
We need to be able to let a user through if they are trying to sign up
for a completely open realm like CUSTOMER3.
(imported from commit 1e33ab0ce94545f217739d501e9227dfb48e1123)
It will have new callers soon and that's as fine a place as any to
avoid circular imports.
(imported from commit 089a724e9ad06cb5a51ffe80f1729d789238e5f6)
This includes a hack to preserve humbug/backends.py as a symlink, so
that we don't need to regenerate all our old sessions.
(imported from commit b7918988b31c71ec01bbdc270db7017d4069221d)
When you read messages in a narrow and then un-narrow, collapse
adjacent messages read in the narrow into a summary row that can
be clicked to expand those messages.
Scoped to staging with feature flags.
The implementation of this within our current MessageList is rather ugly.
(imported from commit bcb3a39d8c0c334136fe86318f18ead03f0f50bf)
Now parsed: 🍺,🍺;🍺!
If \w characters surround :foo:, we still say it's NOT an
emoji, but we used to do this for \S characters, so it's loosened up.
(imported from commit 49b33d2f0ffdcfde8947ae411a4addcf4c24af9c)
If you entered "stream:Denmark " in the search box, we would show
you two suggestions for "stream Denmark", despite our duplicate
detection, because we didn't canonicalize the suggestion that is
literally based off the user typed query, and so the other way
of generating the "stream Denmark" suggestion created a duplicate.
Now all the suggestions we generate are canonicalized, so the
generalized duplicate detection can work.
(imported from commit 52bf08ccf9bb2e2260ca8c20690169aead3732ab)
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).
At the time it is deployed, we need to make a few changes directly in
the database:
(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';
(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
Steve Howell
Tim Abbott
Kevin Mehall
acrefoot
Jessica McKellar
Leo Franchi
Zev Benjamin
Luke Faraone
Waseem Daher
Scott Feeney
acrefoot
Allen Rabinovich
Steve Howell
Tim Abbott
Kevin Mehall