Commit Graph

12924 Commits

Author SHA1 Message Date
akshatdalton 806c1a0b8b markdown: Fix flickering of embedded link inside Italic.
This commit fixes a bug in marked.js which caused it to double-escape
HTML when rendering messages of the form: *[text](url)*.

This fixes a bug introduced in
3bdc8bbaa5, where an unnecessary
escape() call was added for the <em> code path, likely just because it
was adjacent to the others that needed it in the file.

Fix this, and add tests to verify that things are still being escaped
once after removing this extra escape.

Fixes #14845.
2020-11-06 10:09:15 -08:00
Steve Howell 5da4332620 minor: Add order-by-id to digest message query.
The order-by-id is now explicit, and I add
comments to explain the select_related tables.
2020-11-06 10:05:46 -08:00
Steve Howell 936171d258 refactor: Extract DigestTopic class.
This gets us away from a lot of dictionary soup.
2020-11-06 10:05:46 -08:00
Steve Howell e8b6c56322 refactor: Simplify get_hot_topics().
The code we deleted here was no longer
doing anything.

Maybe the code was always dead, or maybe it
was written during a time when topics_by_diversity
and topics_by_length actually had different keys.

But now it's clearly cruft.

If we have 4 or more topics, then the code above
it would already have populated the list with 4
elements, and the `if num_convos < 4` condition
would evaluate to False.

And if we had 3 or fewer topics, then we would
have already put all possible topics into our
result, and the `topics_by_diversity[num_convos:4]`
slice would be empty.

It's possible that we should just have a simple
heuristic for topic hotness like `10*num_senders
+ messages`, so we don't have to maintain this
fiddly function, and we can just do something like
`topics_by_score[:4]`.
2020-11-06 10:05:46 -08:00
Steve Howell c5dc9d386f refactor: Use sets of stream_ids for email digests.
I now use sets for stream_ids in more of the digest
code.

As part of this I replaced exclude_subscription_modified_streams
with streams_recently_modified_for_user.

It's easier for the caller to just ask for ids
to delete from its callee than it is to pass
in a set/list to mutate.

The simpler boundary between the functions makes
the tests easier to write--you can see the
`filtered_streams` logic goes away in this diff.

I also make the tests a bit more thorough by using
combinations of Cordelia/Othello and Verona/Denmark
to try to find multiple possible flaws.

And I make the time intervals longer than 1s to
avoid false negatives from slow CI boxes.
2020-11-05 17:42:43 -08:00
Steve Howell 88a57ed4ac bulk digest: Get stream subscriptions in bulk.
If we have multiple users, this reduces the amount
of queries we need to do, because we get all
subscriptions for all users in a single query
to Subscription.

For the single-user case, we are introducing an
extra query hop, but the database is doing
roughly the same work, because we are just breaking
up this complex query into two hops:

    messages =
        select ...  from message
        where recipient__type_id in (
            select stream_id from subscription
            where ...
        )

Now it's more like:

    stream_ids =
        select stream_id from subscription
        where ...

    messages =
        select ... from message
        where recipient__type_id in stream_ids
2020-11-05 09:36:59 -08:00
Steve Howell c83db37161 email digests: Introduce bulk methods for digest.
Note that we are not changing anything semantically
or algorithmically yet.  The only overhead here
for the single-user case is boxing and unboxing
data into single-item dicts and lists.

The interfaces for callers in the view and the
queue processor remain the same for now.
2020-11-05 09:36:59 -08:00
Steve Howell 7c89e46731 minor: Clean up some code formatting. 2020-11-05 09:36:59 -08:00
Steve Howell 4bd02eea19 minor: Use user, not user_profile, in some digest code. 2020-11-05 09:36:59 -08:00
Steve Howell 0e2d02b0a2 digest tests: Count cache tries. 2020-11-05 09:36:59 -08:00
Steve Howell 127f4e1291 digest tests: Add more users to bulk digest test. 2020-11-05 09:36:59 -08:00
Steve Howell 89cb3fa841 digest tests: Localize mocks.
We didn't need the enough-traffic mock.

We also continue to prep for testing multiple users.

I also finally remove a comment that is about to
be addressed (and which inaccurately refers to huddles).
2020-11-05 09:36:59 -08:00
Steve Howell 1ec16dd1da digest tests: Prep to test bulk digests.
All this does, essentially, is put the logic
we used to test for othello inside of a loop.

We'll add more users in the next commit.
2020-11-05 09:36:59 -08:00
Steve Howell e31326c823 refactor: Extract get_digest_context.
This eliminates the union type and boolean parameter,
and it makes it a bit easier to migrate to a
bulk-get approach.
2020-11-05 09:36:59 -08:00
Steve Howell 217967f743 refactor: Extract get_hot_topics.
This extraction will make a bit more sense when
we start doing bulk operations on a realm to
get digests, but even now, it encapsulates the
slightly complex way we cherry-pick the top 4
topics for a user.
2020-11-05 09:36:59 -08:00
Steve Howell 5a6d6f81ff refactor: Extract get_recent_topic_activity. 2020-11-05 09:36:59 -08:00
Steve Howell f987b014b3 refactor: Rename conversation to topic.
Not only is topic shorter, but the name makes
it clear that we're not dealing with abstract
conversations here--we are truly bucketing by
topic.
2020-11-05 09:36:59 -08:00
Steve Howell 6ac3cd3534 refactor: Use list of topics, not tuples. 2020-11-05 09:36:59 -08:00
Steve Howell 878e938a89 minor: Rename conversation_diversity to conversation_senders. 2020-11-05 09:36:59 -08:00
Steve Howell 6dc8250e9a mypy: Add TopicKey type for digests. 2020-11-05 09:36:59 -08:00
Steve Howell 96f6064b18 refactor: Move Messages query down the digest stack.
This prep step is mostly for diff hygiene; the next
commit will make the code a bit nicer.

The original code here had the nice property that
most (but not all) of the DB work happened up
front in `handle_digest_email`, and none of the
DB work was delegated to the callers.  But I
prefer the tradeoff of making the helpers a bit
more cohesive--let them get the data they need.
And we have query-count coverage in our tests,
so there's no real danger of having helpers
down in the stack insidiously doing a bunch of
extra DB hops.
2020-11-05 09:36:59 -08:00
Anders Kaseorg 13c11ec5f3 openapi: Fix escaping in curl command generation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-05 09:36:31 -08:00
Steve Howell c1f134a3a4 performance: Use ORM to fetch sender in render_markdown.
In 709493cd75 (Feb 2017)
I added code to render_markdown that re-fetched the
sender of the message, to detect whether the message is
a bot.

It's better to just let the ORM fetch this.  The
message object should already have sender.

The diff makes it look like we are saving round trips
to the database, which is true in some cases.  For
the main message-send codepath, though, we are only
saving a trip to memcached, since the middleware
will have put our sender's user object into the
cache.  The test_message_send test calls internally
to check_send_stream_message, so it was actually
hitting the database in render_markdown (prior to
my change).
2020-11-05 09:35:15 -08:00
Steve Howell 637f596751 tests: Fix queries_captured to clear cache up front.
Before this change we were clearing the cache on
every SQL usage.

The code to do this was added in February 2017
in 6db4879f9c.

Now we clear the cache just one time, but before
the action/request under test.

Tests that want to count queries with a warm
cache now specify keep_cache_warm=True.  Those
tests were particularly flawed before this change.

In general, the old code both over-counted and
under-counted queries.

It under-counted SQL usage for requests that were
able to pull some data out of a warm cache before
they did any SQL.  Typically this would have bypassed
the initial query to get UserProfile, so you
will see several off-by-one fixes.

The old code over-counted SQL usage to the extent
that it's a rather extreme assumption that during
an action itself, the entries that you put into
the cache will get thrown away.  And that's essentially
what the prior code simulated.

Now, it's still bad if an action keeps hitting the
cache for no reason, but it's not as bad as hitting
the database.  There doesn't appear to be any evidence
of us doing something silly like fetching the same
data from the cache in a loop, but there are
opportunities to prevent second or third round
trips to the cache for the same object, if we
can re-structure the code so that the same caller
doesn't have two callees get the same data.

Note that for invites, we have some cache hits
that are due to the nature of how we serialize
data to our queue processor--we generally just
serialize ids, and then re-fetch objects when
we pop them off the queue.
2020-11-05 09:35:15 -08:00
Tim Abbott eae14baa05 api: URL-quote password when testing authentication API.
The passwords generated for our development environment / test suite
include the `+` character, which needs to be quoted when encoded as an
HTTP POST parameter.

This is hopefully sufficient to fix the CI failures we've seen with
the tests for POST /api/v1/fetch_api_key; I haven't reproduced the
failure so am not completely sure.
2020-11-03 15:55:30 -08:00
YashRE42 967efc32d2 widgets: Remove tictactoe example widget.
Steve asked me to remove this, since the tictactoe game was always
intended as a proof of concept. Now that we have poll and todo
widgets, the sample code for tictactoe has much less value.

We replace the content and type in test_widgets.py to maintain
coverage.
2020-11-03 14:46:39 -08:00
Aman Agrawal 87cdd8433d home: Allow logged out user through home.
We allow user to load webapp without log-in. This is only
be enabled for developed purposes now. Production setups will
see no changes.
2020-11-02 17:07:12 -08:00
shanukun be39672026 api_docs: Document the /fetch-api-key endpoint.
With tweaks by tabbott to document addition details.

Fixes: #16408.
2020-11-02 16:45:42 -08:00
shanukun da9d586254 openapi: Add parameter examples for fetch api key endpoints. 2020-11-02 16:45:42 -08:00
Anders Kaseorg ac5cbf7693 Revert "markdown: Escape lang when echoing back custom non-pygments languages."
This reverts commit 564b199fe6, which
was part of #16308.

Escaping is either required or incorrect; it is never “defensive”.
This escaping is incorrect.  lxml already escapes attributes during
serialization (any other behavior would be a serious bug), and
additional escaping just results in double escaping.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-02 16:23:48 -08:00
akshatdalton 620e9cbf72 markdown: Fix merging of separate quotations.
Initally, when writing two or more quotes, having
a blank line in between them, merges those quotes.
This created confusion especially in "quote and reply".

This commit fixes such issues. Now two or more quotes
having a blank line in between them, will not get merged.

This change is correct both for usability and for improving our
compatibility with CommonMark.

Fixes #14379.
2020-10-30 15:21:15 -07:00
Mateusz Mandera cbeeadab16 delete_realm: Register a post_delete Realm handler.
By registering a post_delete handler to clear appropriate caches in a
nicer way, we can get rid of the ugly flush-memcached call in the
delete_realm command.
2020-10-30 11:43:03 -07:00
Alex Vandiver bff503feb4 delete_realm: Add command to completely remove realms.
This will need some tweaking in upcoming commits.
2020-10-30 11:42:40 -07:00
Anders Kaseorg 3c663e48db url_encoding: Skip unnecessary encode before quote.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg df10b306a6 python: Remove force_bytes.
We are generally good enough at types to know whether a value is str
or bytes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg cc55393671 python: Open text files as text to skip decode operations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 18d0e4664c python: Replace binascii with bytes.hex to skip some decode operations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg aaa7b766d8 python: Use universal_newlines to get str from subprocess.
We can replace ‘universal_newlines’ with ‘text’ when we bump our
minimum Python version to 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 9281dccae4 python: Serialize lxml elements directly to str.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 7c4f68d9cf python: Skip unnecessary decode before BeautifulSoup parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 86e8d81c7f python: Skip unnecessary decode before JSON parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 1802a50cc9 python: Use requests.Response.text instead of decoding content.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Tim Abbott 067cd3a97a docs: Remove incorrect references to chat.zulip.org.
Most of these are Help Center links that should be pointing to the
production Help Center.
2020-10-29 16:46:40 -07:00
Tim Abbott 3b9c726fc6 outgoing_webhook: Avoid logging a bytes string.
This fixes the new assertLogs() tests failing in CI; we fixed the
weird use of bytes in the test, but not in the runtime code.
2020-10-29 15:55:11 -07:00
sahil839 7106069d4d migration: Add migration to remove default status of private streams.
This commit adds migration which removes default status of exisitng
default private streams, i.e. private stream exists but they are no
longer default.
2020-10-29 15:47:34 -07:00
sahil839 b29d39195c streams: Do not allow default streams to be private.
We now do not allow to make a stream private which is already
a default stream.
2020-10-29 15:47:32 -07:00
sahil839 557ca0802c streams: Do not allow private streams to be set as default.
We now do not allow to set a private stream as default.
2020-10-29 15:43:37 -07:00
m-e-l-u-h-a-n cbfd6464a5 logging: replace mock.patch() for logging with assertLogs()
This commit removes mock.patch with assertLogs().

* Adds return value to do_rest_call() in outgoing_webhook.py, to
  support asserting log output in test_outgoing_webhook_system.py.

* Logs are not asserted in test_realm.py because it would require to users
  to be queried using users=User.objects.filter(realm=realm) and the order
  of resulting queryset varies for each run.

* In test_decorators.py, replacement of mock.patch is not done because
  I'm not sure if it's worth the effort to replace it as it's a return
  value of a function.

Tweaked by tabbott to set proper mypy types.
2020-10-29 15:37:45 -07:00
Hemanth V. Alluri 99cf37dc51 drafts: Make the ID of the draft a part of the draft dict.
Then because the ID is now part of the draft dict, we can
(and do) change the structure of the "drafts" parameter
returned from `GET /drafts` from an object (mapping ID to
data) to an array.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
Hemanth V. Alluri 8d59fd2f45 tests/drafts: Simplify create_and_check_drafts_for_success.
Sometimes we don't need to specify the expected_drafts field.
So by removing it, we can reduce the clutter a bit.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
Hemanth V. Alluri e60925b3e8 drafts: Change "timestamp" from float to integer.
Now the timestamp returned in a draft dict will always be an int.
The endpoints will still accept either an int or a float.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
Abhijeet Prasad Bodas e98a8856c7 logging: Add logging in deferred_work queue processor.
Adds logging statements in deferred_work queue consume.
2020-10-29 10:34:53 -07:00
m-e-l-u-h-a-n be7a70e742 logging: Remove unnecessary mock.patch() for logging.
Our test-backend validation confirms that we don't log anything to
stdout in the tests, so the fact that CI passes with this removes
shows there was nothing being logged.
2020-10-28 23:15:27 -07:00
Vishnu KS fdea49742c apps: Use GitHub API for generating the web app download link. 2020-10-28 23:04:14 -07:00
ryanreh99 dfa7ce5637 uploads: Support non-AWS S3-compatible server.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.

We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.

Fixes #16246.
2020-10-28 21:59:07 -07:00
ryanreh99 1c370a975c refactor: Access a bucket by calling `zerver.lib.uploads.get_bucket`. 2020-10-28 21:52:08 -07:00
Alex Vandiver f4eae83542 export: Only include real, active humans in the displayed count. 2020-10-28 18:31:06 -07:00
Alex Vandiver 3037e22f61 tornado: Never use proxies when talking from Django to Tornado.
The `no_proxy` parameter does not work to remove proxying[1]; in this
case, since all requests with this adapter are to the internal Tornado
process, explicitly pass in an empty set of proxies to disable
proxying.

[1] https://github.com/psf/requests/issues/4600
2020-10-28 12:13:04 -07:00
Alex Vandiver 97745688ca docs: Link to the new doc home of the email gateway. 2020-10-28 12:13:04 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Alex Vandiver 142de0f670 queue: Increase default timeout to 30s, from 10s.
Not all of the workers are known to be safe to interrupt; they might
leave inconsistent state.  As such, terminating them with timeouts
should currently only be a last-resort against stalled queues, not a
regular occurrence.
2020-10-27 16:39:31 -07:00
Alex Vandiver c73dd194f0 sentry: Group all worker timeouts together, by queue.
Since the exception can be triggered at arbitrary places in the stack
based on whenever the alarm happens to fire, they do not often group
together.

Explicitly group them together, grouped only by which queue the work
is in.
2020-10-27 16:39:31 -07:00
Anders Kaseorg 1352f2f233 python: Replace manual quote_plus usage with urlencode.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 4e9d587535 python: Pass query parameters as a dict when making GET requests.
This provides automatic URL-encoding.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 41f509170b users: Canonicalize the timezone identifier.
While working on shifting toward native browser time zone APIs
(#16451), it was found that all but very recent Chrome and Node
versions reject certain legacy timezone aliases like US/Pacific
(https://crbug.com/364374).

For now, we only canonicalize the timezone property returned in user
objects and not the timezone setting itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg a8b1691e97 timezone: Convert get_common_timezones cache to lru_cache.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 0b288f92c9 timezone: Remove get_timezone wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 0134112b51 timezone: Remove get_all_timezones wrapper.
Both callers want a set, and pytz already provides all_timezones_set.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Chris Bobbe 500f92169e openapi: Fix the "the the" instances. 2020-10-26 12:50:01 -07:00
Chris Bobbe dd43d4feb8 openapi: Note that a user's `avatar_url` may be missing.
Like it already says in the detail about
`user_avatar_url_field_optional`, but on the field itself [1].

[1] https://github.com/zulip/zulip-mobile/pull/4230#discussion_r493109645
2020-10-26 12:50:01 -07:00
Tim Abbott 6d7cd351a3 events: Optimize creating streams for new users.
During the new user creation code path, there can be no existing
active clients for the user being created, so we can skip the code to
send events to that user's clients.

The tests here reflect that we need to send fewer events, and do fewer
queries that would have been spent computing data for these..

Fixes #16503, combined with the long series of recent changes by Steve
Howell to fix super-linear behavior in this code path.
2020-10-26 12:47:15 -07:00
Steve Howell 88a7a1b002 events: Optimize peer_add/peer_remove for public streams.
We no bulk up peer_add/peer_remove events by user if the
same user has subscribed to multiple streams (and just
that single user).

This mostly optimizes the new-user codepath, but the
algorithm is a bit more general in nature.
2020-10-26 12:33:28 -07:00
Alex Vandiver 7cf737988d queue: Be more explicit about test/real queue division. 2020-10-26 12:32:47 -07:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Steve Howell 3ad1335a97 tests: Clear ContentType cache for user test.
This keeps the number of queries predictable.
2020-10-26 07:18:08 -04:00
Steve Howell 5ef01b3ad8 tests: Fix test_create_user_with_multiple_streams.
This test was flaky due to some date-related
non-determinism.  I make all the Message objects
current to make add_new_user_history reliably
try to bulk-update UserMessage rows to read.
2020-10-26 07:18:08 -04:00
Anders Kaseorg 96dee7e40b decorator: Unfork redirect_to_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-25 14:58:53 -07:00
sahil839 a96811ab58 management: Replace knight command with change_user_role command.
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.

Tweaked by tabbott to improve the logging output and update documentation.

Fixes #16586.
2020-10-25 14:55:08 -07:00
Harsh Srivastava 9b31df009b openapi: Fix excessively large test_events failure output.
Because of the very large `oneOf` clause of the formats of events
possible in Zulip's `GET /events` system, we had issues with
`test-backend` failures for missing documentation for a new event
format being like 1000 lines of output, which was very much unhelpful.

Fix this by limiting the output use only the oneOf variants that are
broadly similar to the actual payload received.

Fixes #16023.
2020-10-23 17:00:17 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg e513b75e86 markdown: Remove handler for old bug with incompatible twitter library.
See commit 8b002040e0 and #86.  The
development environment bug that necessitated this handler has long
been irrelevant.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:30:26 -07:00
Anders Kaseorg b9fd49a2c6 mypy: Correct mistaken *args type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:29:13 -07:00
Anders Kaseorg d295da676b test_message_fetch: Clean up obsolete PGroonga bug workaround.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-22 23:27:23 -07:00
Anders Kaseorg 831d086110 i18n: Fix get_language_translation_data for zh_TW.
Fixes #16600.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-22 16:43:02 -07:00
sahil839 571bb62e3d events: Update subscriber list on peer_add for unsubscribed streams.
We update the subscriber list on peer_add event for unsubscribed
streams as well.
2020-10-22 15:12:32 -07:00
sahil839 733d26aef2 events: Update subscriber list on peer_remove for never subscribed stream.
We now update the subscriber list on peer_remove event for never
subscribed streams also.
2020-10-22 15:12:32 -07:00
sahil839 af9b153ee3 events: Update subscriber list on peer_remove for unsubscribed stream.
We update the subscriber list on peer_remove event for unsubscribed
streams also.
2020-10-22 15:12:32 -07:00
sahil839 709edd29d4 test_events: Fix comment in do_test_subscribe_events.
The comment still pointed to 'vacate' event flow, but
we have removed the vacate event in a9356508ca.
This commit fixes the comment to depict the correct
purpose of below lines, i.e. to test the remove
event flow.
2020-10-22 15:12:32 -07:00
sahil839 e578742b02 test_events: Remove 'realm_user' from event_types in subscription test.
We were including 'realm_user' in event_types along with 'subscription',
but we don't send event of type 'realm_user' when subscribing to a new
stream. This was added in 1c332f5d6a.

This commit removes 'realm_user' from event_types.
2020-10-22 15:12:32 -07:00
sahil839 d0f5537fb2 actions: Modify check_message for handling wildcard_mention_policy setting.
This commit adds enforcement for sending messages containing wildcard
mentions according to wildcard_mention_policy.
2020-10-22 14:46:32 -07:00
sahil839 25f32d461e tests: Add tests for all the values of wildcard_mention_policy. 2020-10-22 12:08:22 -07:00
Mateusz Mandera 48f80fcb0a auth: Expect name in request params in Apple auth.
The name used to be included in the id_token, but this seems to have
been changed by Apple and now it's sent in the `user` request param.

https://github.com/python-social-auth/social-core/pull/483 is the
upstream PR for this - but upstream is currently unmaintained, so we
have to monkey patch.

We also alter the tests to reflect this situation. Tests no longer put
the name in the id_token, but rather in the `user` request param in the
browser flow, just like it happens in reality.

An adaptation has to be made in the native flow - since the name won't
be included by Apple in the id_token anymore, the app, when POSTing
to the /complete/apple/ endpoint,
can (and should for better user experience)
add the `user` param formatted as json of
{"email": "hamlet@zulip.com", "name": {"firstName": "Full", "lastName": "Name"}}
dict. This is also reflected by the change in the
native flow tests.
2020-10-22 12:07:46 -07:00
Steve Howell 7ff3859136 subscriber events: Change schema for peer_add/peer_remove.
We now can send an implied matrix of user/stream tuples
for peer_add and peer_remove events.

The client code basically does this:

    for stream_id in event['stream_ids']:
        for user_id in event['user_ids']:
            update_sub(stream_id, user_id)

We used to send individual events, which gets real
expensive when you are creating new streams. For
the case of copy-to-stream case, we should see
events go from U to 1, where U is the number of users
added.

Note that we don't yet fully optimize the potential
of this schema.  For adding a new user with lots
of default streams, we still send S peer_add events.

And if you subscribe a bunch of users to a bunch of
private streams, we only go from U * S to S; we can't
optimize it down to one event easily.
2020-10-22 11:19:53 -07:00
Anders Kaseorg efa8dd3a47 compilemessages: Sort language list with Unicode Collation Algorithm.
Right now the list of languages in Display settings → Default language
is sorted in an unintuitive order due to the varying case conventions:

British English
Chinese (Taiwan)
Deutsch
English
Hindi
Indonesian (Indonesia)
Lietuviškai
Magyar
Malayalam
Nederlands
Português
Română
Tiếng Việt
Türkçe
català
español
français
galego
italiano
polski
suomi
svenska
česky
Русский
Українська
български
српски
فارسی
தமிழ்
日本語
简体中文
繁體中文
한국어

Fix the sort to use the locale-independent Unicode Collation
Algorithm:

British English
català
česky
Chinese (Taiwan)
Deutsch
English
español
français
galego
Hindi
Indonesian (Indonesia)
italiano
Lietuviškai
Magyar
Malayalam
Nederlands
polski
Português
Română
suomi
svenska
Tiếng Việt
Türkçe
български
Русский
српски
Українська
فارسی
தமிழ்
한국어
日本語
简体中文
繁體中文

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 15:22:18 -07:00
Steve Howell 85ed6f332a performance: Avoid Recipient lookup for stream messages.
All the fields of a stream's recipient object can
be inferred from the Stream, so we just make a local
object.  Django will create a Message object without
checking that the child Recipient object has been
saved.  If that behavior changes in some upgrade,
we should see some pretty obvious symptom, including
query counts changing.

Tweaked by tabbott to add a longer explanatory comment, and delete a
useless old comment.
2020-10-20 11:47:23 -07:00
Steve Howell 7bbcc2ac96 refactor: Compute peers for public streams later.
This saves us a query for edge cases like when
you try to unsubscribe from a public stream
that you have already unsubscribed from.

But this is mostly to prep for upcoming
optimizations.
2020-10-20 11:31:22 -07:00
Steve Howell 363e5d31a6 refactor: Split out public/private logic for peer events.
This doesn't change anything yet, but the goal is
to eventually optimize events for the case where
one user (typically a new user) gets subscribed
to multiple public streams.
2020-10-20 11:31:22 -07:00
Steve Howell 3961e69381 refactor: Extract send_peer_subscriber_events.
We now use the same basic code to send peer_add
and peer_remove events.
2020-10-20 11:31:22 -07:00
Anders Kaseorg 254b904965 markdown: Migrate off deprecated extension registration interface.
Fixes #15205.

https://python-markdown.github.io/change_log/release-3.0/#homegrown-ordereddict-has-been-replaced-with-a-purpose-built-registry
https://python-markdown.github.io/change_log/release-3.0/#md_globals-keyword-deprecated-from-extension-api

The priority numbers are arbitrarily chosen to preserve the existing
order.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 18:31:12 -07:00
akshatdalton 287c4ed2bb markdown: Fix Youtube and Vimeo preview overriding markdown link titles bug.
Initially markdown titles were overridden by Youtube and Vimeo preview titles.
But now it will check if any markdown title is present to replace Youtube or
Vimeo preview titles, if preview of linked websites is enabled.
Fixes #16100
2020-10-19 12:06:13 -07:00
Anders Kaseorg d81a93cdf3 requirements: Upgrade markdown to 3.3.1.
Upstream has slightly changed the whitespace around stashes.  Take
this opportunity to clean up the extra blank lines we were outputting.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg 2ab15149bc forms: Convert percent formatting to "".format.
Caught by upgrading Semgrep.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg f461a64a6b i18n: Fix some ineffective calls to ugettext at top level.
Translation has no effect when we don’t yet know what language we’re
translating for.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-18 14:31:15 -07:00
Anders Kaseorg bba43f35ca i18n: Be deliberate about distinguishing ugettext and ugettext_lazy.
The early str conversions in zerver.models were defeating the point of
ugettext_lazy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-18 14:31:15 -07:00
Steve Howell e6f6f8d45f refactor: Avoid "stream_id" on sub.
There was no need to put "stream_id" on the sub
dictionary here.  It's kinda annoying to introduce
the little helper here, but I feel
that's better than crufting up the sub data
structure.
2020-10-18 14:27:31 -07:00
Steve Howell 628a826aa2 minor: Move code and add comments about three lists. 2020-10-18 14:27:31 -07:00
Steve Howell ffee129a35 refactor: Clean up is_web_public flag.
The is_web_public flag is already in Stream.API_FIELDS,
so there is no reason for all this complicated logic.

There's no reason to hack it on to the subscription
object.
2020-10-18 14:27:31 -07:00
Steve Howell 4dce34ab8b refactor: Simplify call to bulk_get_subscriber_user_ids.
The way we were computing the dictionary was very
convoluted--all we need is a set of subscribed user
ids.
2020-10-18 14:27:31 -07:00
Steve Howell b58152abda refactor: Introduce all_streams_map.
We replace all_streams_id with a map.

We also use it to populate never_subscribed_streams.

And all_streams_map is a superset of stream_hash,
which we will soon kill off as well.
2020-10-18 14:27:31 -07:00
Steve Howell 78384ebf1b minor: Remove confusing parens.
Apparently I put these parens in the code as
part of 73c30774cb
during 2017.

It looks like I extracted is_public during
the middle of my change and forgot to remove
the unnecessary parens.  (The code was correct,
but it makes it look like a tuple if you're
skimming it too quickly.)
2020-10-18 14:27:31 -07:00
Steve Howell d60dd94168 refactor: Extract funcs from gather_subscriptions_helper.
This is a pure code move, apart from a little bit
of quote cleanup and renames:

    user_profile -> user
    stream_dict -> result
2020-10-18 14:27:31 -07:00
Steve Howell 79fcf78143 refactor: Exclude "active" from API_FIELDS.
We just need to make sure the relevant queries
get it for the triage process.
2020-10-18 14:27:31 -07:00
Steve Howell c5769d31f2 minor: Move code for web_public_stream_ids. 2020-10-18 14:27:31 -07:00
Steve Howell 0ca07ffd3c peformance: Eliminate StreamRecipientMap.
That class is an artifact of when Stream
didn't have recipient_id.  Now it's simpler
to deal with stream subscriptions.

We also save a query during page load (and
other places where we get subscriber
info).
2020-10-18 14:27:31 -07:00
Steve Howell 1951d75796 performance: Avoid select_related("realm").
We also move this query up in the function
for some future refactorings.
2020-10-18 14:27:31 -07:00
Steve Howell 2f8ba383ef tests: Test overhead for creating new users. 2020-10-18 14:27:31 -07:00
Mateusz Mandera 716df658fa queue_processors: Don't run test queues with run-dev.py. 2020-10-18 14:07:31 -07:00
Steve Howell 57efe9d81a performance: Streamline list_to_streams.
We take advantage of stream.recipient to simplify
the query's where clause and avoid the need
for select_related("recipient").
2020-10-16 12:58:11 -07:00
Steve Howell e1bcf6124f refactor: Remove recipient from access_stream_by_name. 2020-10-16 12:58:11 -07:00
Steve Howell a51b483f1a performance: Remove recipient from access_stream_by_id.
The Recipient table is now kind of useless for
stream-related operations, since we have
recipient_id on Stream now.
2020-10-16 12:58:11 -07:00
Steve Howell 31622feb87 refactor: Only return sub from access_stream_common.
Let the callers access stream.recipient as needed.
It costs the same, and some of the callers can
actually stop caring about the actual Recipient
object.
2020-10-16 12:58:11 -07:00
Steve Howell bfd6e2b1fd refactor: Use recipient_id to get topic history. 2020-10-16 12:58:11 -07:00
Steve Howell 3685fcc701 refactor: Remove recipient arg for do_mute_topic. 2020-10-16 12:58:11 -07:00
Steve Howell 65dbee4837 minor: Ask for recipient_id, not recipient. 2020-10-16 12:58:11 -07:00
Steve Howell 378062cc83 performance: Avoid call to access_stream_by_id.
We already trust ids that are put on our queue
for deferred work. For example, see the code for
"mark_stream_messages_as_read_for_everyone"

We now pass stream_recipient_id when we queue
up work for do_mark_stream_messages_as_read.

This generally saves about 3 queries per
user when we unsubscribe them from a stream.
2020-10-16 12:58:11 -07:00
Steve Howell 2256d72015 minor: Add comment to subscriber test. 2020-10-16 12:58:11 -07:00
Steve Howell 31eb97ddde performance: Fix do_mark_stream_messages_as_read.
This function no longer asks for data that it
doesn't need.
2020-10-16 12:58:11 -07:00
Steve Howell 6d1f9de7d3 performance: Use SubInfo when removing subscribers.
We get two speedups:

    * The query to get existing subscribers only
      gets the two fields we need.  We no longer
      need all the overhead of user_profile
      and recipient data being returned in the
      query.

    * We avoid Django making extra hops to the
      database to get user info.
2020-10-16 12:58:11 -07:00
Steve Howell 73982f6cc9 refactor: Move SubInfo to stream_subscription.py. 2020-10-16 12:58:11 -07:00
Tim Abbott caa939d2d5 actions: Use transaction.atomic properly when removing subscriptions.
Previously, the transaction.atomic() was not properly scoped to ensure
that RealmAuditLog entries were created in the same transaction,
making it possible for state changes to not be properly recorded in
RealmAuditLog.
2020-10-15 15:12:05 -07:00
Steve Howell 0b91526f28 events: Remove "occupied" semantics for "streams".
When apps like mobile register for "streams", we
will now just use active streams as our baseline,
rather than "occupied" streams.

This means we will send a stream that is active,
even if it happens to have zero occupants.  It's
actually pretty rare that a stream has zero occupants,
and it's not exactly clear that we want to exclude
a non-occupied but otherwise active stream from
our list of streams.

It also happens to be fairly expensive to compute
whether a stream is occupied.

This change only affects API clients (including
possibly our mobile app).  The main webapp never
used the data from this codepath.
2020-10-15 15:12:01 -07:00
Steve Howell b4346d0276 performance: Extract subscribers/peers in bulk.
We replace get_peer_user_ids_for_stream_change
with two bulk functions to get peers and/or
subscribers.

Note that we have three codepaths that care about
peers:

    subscribing existing users:
        we need to tell peers about new subscribers
        we need to tell subscribed user about old subscribers

    unsubscribing existing users:
        we only need to tell peers who unsubscribed

    subscribing new user:
        we only need to tell peers about the new user
        (right now we generate send_event
        calls to tell the new user about existing
        subscribers, but this is a waste
        of effort that we will fix soon)

The two bulk functions are this:

    bulk_get_subscriber_peer_info
    bulk_get_peers

They have some overlap in the implementation,
but there are some nuanced differences that are
described in the comments.

Looking up peers/subscribers in bulk leads to some
nice optimizations.

We will save some memchached traffic if you are
subscribing to multiple public streams.

We will save a query in the remove-subscriber
case if you are only dealing with private streams.
2020-10-15 15:12:01 -07:00
Steve Howell 94e41c71f9 refactor: Use set of ids for altered users. 2020-10-15 15:12:01 -07:00
Steve Howell b894597fa3 refactor: Use sets of stream_ids for helper args. 2020-10-15 15:12:01 -07:00
Steve Howell 3889554977 refactor: Extract send_peer_remove_events. 2020-10-15 15:12:01 -07:00
Steve Howell c73f84f275 tests: Improve tests for unsubscribing multiple users.
Note that the tests now reflect that we have O(N)
behavior for multiple users.
2020-10-15 15:12:01 -07:00
Steve Howell f86823f82f tests: Add cache_tries_captured helper. 2020-10-15 15:12:01 -07:00
Steve Howell ce70d08cbf test_helpers: Use mock.patch.multiple. 2020-10-15 15:12:01 -07:00
Tim Abbott bf66e9c4ab actions: Add transaction.atomic to bulk_add_subs_to_db_with_logging.
This will ensure that we always fully execute the database part of
modifying subscription objects.  In particular, this should prevent
invariant failures like #16347 where Subscription objects were created
without corresponding RealmAuditLog entries.

Fixes #16347.
2020-10-14 11:06:00 -07:00
Steve Howell 5728149e94 performance: Streamline query to add subscribers.
We don't need the select_related('user_profile')
optimization any more, because we just keep
track of user info in our own data structures.

In this codepath we are never actually modifying
users; we just occasionally need their ids or
emails.

This can be a pretty substantive improvement if
you are adding a bunch of users to a stream
who each have a bunch of their own subscriptions.

We could also limit the number of full rows in this
query by adding an extra hop to the DB just to
get colors (using values_list), and then only get
full sub info for the streams that we're adding, rather
than getting every single subscription, in full, for each user.

Apart from finding what colors the user has already
used, the only other reason we need all the columns
in Subscription here is to handle streams that
need to be reactivated.  Otherwise we could do
only("id", "active", "recipient_id", "user_profile_id")
or similar.  Fortunately, Subscription isn't
an overly wide table; it's mostly bool fields.

But by far the biggest thing to avoid is bringing
in all the extra user_profile data.

We have pretty good coverage on query counts here,
so I think this fix is pretty low risk.
2020-10-14 11:03:07 -07:00
Steve Howell 116a441bc5 refactor: Introduce SubInfo class.
This class removes a lot of the annoying tuples
we were passing around.

Also, by including the user everywhere, which
is easily available to us when we make instances
of SubInfo, it sets the stage to remove
select_related('user_profile').
2020-10-14 10:53:10 -07:00
Steve Howell febef45e38 minor: Add comments to do_get_streams. 2020-10-14 10:53:10 -07:00
Steve Howell a9356508ca events: Stop sending occupy/vacate events.
We used to send occupy/vacate events when
either the first person entered a stream
or the last person exited.

It appears that our two main apps have never
looked at these events.  Instead, it's
generally the case that clients handle
events related to stream creation/deactivation
and subscribe/unsubscribe.

Note that we removed the apply_events code
related to these events.  This doesn't affect
the webapp, because the webapp doesn't care
about the "streams" field in do_events_register.

There is a theoretical situation where a
third party client could be the victim of
a race where the "streams" data includes
a stream where the last subscriber has left.
I suspect in most of those situations it
will be harmless, or possibly even helpful
to the extent that they'll learn about
streams that are in a "quasi" state where
they're activated but not occupied.

We could try to patch apply_event to
detect when subscriptions get added
or removed. Or we could just make the
"streams" piece of do_events_register
not care about occupy/vacate semantics.
I favor the latter, since it might
actually be what users what, and it will
also simplify the code and improve
performance.
2020-10-14 10:53:10 -07:00
Steve Howell 1bcb8d8ee8 performance: Avoid computing page_params.streams in webapp.
The query to get "occupied" streams has been expensive
in the past.  I'm not sure how much any recent attempts
to optimize that query have mitigated the issue, but
since we clearly aren't sending this data, there is no
reason to compute it.
2020-10-14 10:53:10 -07:00
Steve Howell 79803f01f4 minor: Format some code in events.py. 2020-10-14 10:53:10 -07:00
Steve Howell 193ca397f9 tests: Include deactivated users for subscribe test. 2020-10-14 10:53:10 -07:00
Aman Agrawal fbf7cb82a7 web_public_guest: Rename to web_public_visitor for clarity.
Using web_public_guest for anonymous users is confusing since
'guest' is actually a logged-in user compared to
web_public_guest which is not logged-in and has only
read access to messages. So, we rename it to
web_public_visitor.
2020-10-13 16:59:52 -07:00
Steve Howell e7a8c7ac48 test: Improve tests for bulk-adding subscribers.
This is a more thorough test of adding multiple
streams for multiple users, including streams
that users have already subscribed to.

The extra queries here are due to the fact
that we call `principal_to_user_profile` in
a loop in the view.  So that's an example
of O(N) overhead.  We may be able to bulk-fetch
these users eventually.
2020-10-13 18:54:55 -04:00
Steve Howell c29ba75135 refactor: Extract send_messages_for_new_subscribers.
This is a pure extraction, except that I remove a
redundant check that `len(principals) > 0`.  Whenever
that value is false, then `new_subscriptions` will
only have one possible entry, which is the current
user, and we skip that in the loop.
2020-10-13 18:54:55 -04:00
Steve Howell 3b338ec32e performance: Optimize filter_stream_authorization.
We no longer do O(N) queries to get existing streams.

This is a somewhat contrived use case--generally, we
are not trying to re-subscribe a user to several
streams.  Still, we want to avoid this.

This commit also makes `test_bulk_subscribe_many`
do more work, and the change to the test helped
me discover this bug.
2020-10-13 18:54:55 -04:00
Anders Kaseorg 6564540d15 docs: Fix some spelling errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Anders Kaseorg dd48dbd912 docs: Add spaces to “check out”, “log in”, “set up”, “sign up” as verbs.
“Checkout”, “login”, “setup”, and “signup” are nouns, not verbs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Steve Howell 598601e8fc stream events: Prevent spurious events.
If a user asks to be subscribed to a stream
that they are already subscribed to, then
that stream won't be in new_stream_user_ids,
and we won't need to send an event for it.

This change makes that happen more automatically.
2020-10-13 11:28:17 -07:00
Steve Howell 18771099e4 performance: Introduce new_stream_user_ids.
Let
    U = number of users to subscribe
    S = number of streams to subscribe

We were technically doing N^3 amount of work
when we sent certain events, or to be more
precise, U * S * S amount of work.  For each
stream, we were looping through a list of tuples
of size U * S to find the users for the stream.

In practice either U or S is usually 1, so the
performance gains here are probably negligible,
especially since the constant factors here
were just slinging around Python data.

But the code is actually more readable now, so
it's a double win.
2020-10-13 11:28:17 -07:00
Steve Howell ebb605319b refactor: Rename stream_map to recipient_id_to_stream.
I want to make a new dict called stream_id_to_stream,
and stream_map would be confusing.
2020-10-13 11:28:17 -07:00
Steve Howell b502957184 refactor: Extract new_recipient_ids local.
We rename needs_new_sub (which sounds like
a boolean!) to new_recipient_ids, and we
calculate it explicitly within the loop, so
that we don't need to worry as much about
subsequent passes through the loop mutating it.

This allows us to also remove recipient_ids,
which in turn lets us remove recipients_map,
albeit with a small tweak for stream_map.

I also introduce the my_subs local, which
I use to more directly populate used_colors,
as well as using it as the loop var.
2020-10-13 11:28:17 -07:00
Steve Howell 766892d8aa import: Reuse get_last_message_id() helper. 2020-10-13 11:28:17 -07:00
Steve Howell 188cc9bb3b minor: Fix user/stream in test_subscriptions. 2020-10-13 11:28:17 -07:00
Steve Howell 9df9934ed6 refactor: Pass realm to bulk_add_subscriptions.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.

This may be overkill--I would also be happy if we
just included the assertions from this commit.
2020-10-13 11:28:17 -07:00
Steve Howell efc931a671 minor: Extract realm local. 2020-10-13 11:28:17 -07:00
Steve Howell b2d0a2efb9 refactor: Extract send_subscription_add_events.
This function now does all the work that we used
to do with notify_subscriptions_added happening
inside a loop.

There's a small fine-tuning here, where we only
get recent traffic on streams that we're actually
sending events for.
2020-10-13 11:28:17 -07:00
Steve Howell 223ce83a0a refactor: Clean up call to notify_subscriptions_added.
We now just pass in all_subscribers_by_stream, rather
than a callback.

We also move sub_tuples_by_user closer to the
loop where we call notify_subscriptions_added.
2020-10-13 11:28:17 -07:00
Steve Howell 811426b345 Extract send_stream_creation_events_for_private_streams.
We can probably avoid passing in users here.
2020-10-12 16:40:37 -07:00
Steve Howell 1cfaef0d1a refactor: Simplify pick_color logic.
This removes the need to jankily mutate
the active flag in the caller, and we don't
need to mutate our subs_by_user either.
2020-10-12 16:40:37 -07:00
Steve Howell 13569ff97a refactor: Eliminate new_subs.
We now just process new subs for a user immediately
within the loop.
2020-10-12 16:40:37 -07:00
Steve Howell 8c70fbde78 refactor: Use subs_to_add in return value.
The subs_to_add is directly related to a var
called new_subs, which I hope to eliminate
soon.
2020-10-12 16:40:37 -07:00
Steve Howell 1afca3d430 minor: Extract local for stream. 2020-10-12 16:40:37 -07:00
Steve Howell 84aa1389d8 Extract bulk_add_subs_to_db_with_logging.
This is a trivial code extraction.
2020-10-12 16:40:37 -07:00
Steve Howell 3ff9ce78ea refactor: Extract send_peer_add_events. 2020-10-12 16:40:37 -07:00
Alex Vandiver f3ba227614 create_user: Strip whitespace from initial password file.
Fixes #12144.
2020-10-11 16:29:00 -07:00
Cody Piersall 5dab6e9d31 emoji-upload: Fix transparency issues on GIF emoji upload.
This preserves the alpha layer on GIF images that need to be resized
before being uploaded.  Two important changes occur here:

1. The new frame is a *copy* of the original image, which preserves the
   GIF info.
2. The disposal method of the original GIF is preserved.  This
   essentially determines what state each frame of the GIF starts from
   when it is drawn; see PIL's docs:
   https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#saving
   for more info.

This resolves some but not all of the test cases in #16370.
2020-10-11 16:23:07 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Anders Kaseorg 7f69c1d3d5 python: Catch specific exceptions from requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 17ac17286c python: Catch specific exceptions from subprocess.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg aabef3d9be python: Catch specific exceptions from orjson.
Followup to #16120.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 234f7245cf export_usermessage_batch: Use os.rename.
This avoids an extra stat call to check whether the target is a
directory.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 83eca256a4 compilemessages: Use polib for get_name_from_po_file.
This also corrects the name of zh_TW from “Chinese” to
“Chinese (Taiwan)”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 1346c5397a zephyr: Use correct shell quoting for ssh.
ssh always runs its command through a shell (after naïvely joining
multiple arguments with spaces), so it needs an extra level of shell
quoting.  This should have no effect because we already validated user
with a regex, but it’s better for escaping to be locally correct in
case the context changes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 82593338ba report: Show Git commit in a way that works for merges.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg c9fec8f021 deliver_scheduled_messages: Don’t do_send_messages inside a transaction.
do_send_messages has side effects outside the database and may not
work reliably if its database effects are reordered by being inside a
transaction.

This also fixes a bug where we were doing the update incorrectly on
the Message table.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:09:22 -07:00
Alex Vandiver c2132a4f9c queue: Drop register_json_consumer / json_drain_queue interface.
Now that all callsites use the same interface, drop the now-unused
ones, and their tests.
2020-10-11 14:19:42 -07:00
Alex Vandiver 5477b9d9a1 queue: Switch tests to start_json_consumer interface. 2020-10-11 14:19:42 -07:00
Alex Vandiver 179c387409 tornado: Switch to start_json_consumer interface. 2020-10-11 14:19:42 -07:00
Alex Vandiver f0b23b0752 queue: Switch non-batch consumer to also use start_json_consumer.
This has no effect on consumption rate, but unifies the codepaths.
Before:
```
$ ./manage.py queue_rate --count 50000
Purging queue...
Enqueue rate: 11187 / sec
Dequeue rate: 4158 / sec
```

After:
```
$ ./manage.py queue_rate --count 50000
Purging queue...
Enqueue rate: 11010 / sec
Dequeue rate: 4113 / sec
```
2020-10-11 14:19:42 -07:00
Alex Vandiver 45c9c3cc30 queue: Monitor user_activity queue, now that it has a consumer.
Since this was using repead individual get() calls previously, it
could not be monitored for having a consumer.  Add it in, by marking
it of queue type "consumer" (the default), and adding Nagios lines for
it.

Also adjust missedmessage_emails to be monitored; it stopped using
LoopQueueProcessingWorker in 5cec566cb9, but was never added back
into the set of monitored consumers.
2020-10-11 14:19:42 -07:00
Alex Vandiver f9358d5330 queue: Switch batch interface to use the channel.consume iterator.
This low-level interface allows consuming from a queue with timeouts.
This can be used to either consume in batches (with an upper timeout),
or one-at-a-time.  This is notably more performant than calling
`.get()` repeatedly (what json_drain_queue does under the hood), which
is "*highly discouraged* as it is *very inefficient*"[1].

Before this change:
```
$ ./manage.py queue_rate --count 10000 --batch
Purging queue...
Enqueue rate: 11158 / sec
Dequeue rate: 3075 / sec
```

After:
```
$ ./manage.py queue_rate --count 10000 --batch
Purging queue...
Enqueue rate: 11511 / sec
Dequeue rate: 19938 / sec
```

[1] https://www.rabbitmq.com/consumers.html#fetching
2020-10-11 14:19:40 -07:00
Alex Vandiver 571f8b8664 queue: Use low-level queue_purge to empty at the end of tests.
This is O(1) at the RabbitMQ API level, and doesn't rely on the code
under test to function correctly during test cleanup.
2020-10-09 20:43:49 -07:00
Alex Vandiver ac0ba21c2c tests: Stop reusing a variable name.
`loopworker_sleep_mock` is a file-level variable used to mock out the
sleep() call in LoopQueueProcessingWorker; don't reuse the variable
name for something else.
2020-10-09 20:42:20 -07:00
Alex Vandiver 754638f673 tests: Refactor test_queue_worker to separate queues. 2020-10-09 20:42:12 -07:00
Alex Vandiver 2547bdbf4a queue: Rename consume_wrapper to a better name. 2020-10-09 20:40:51 -07:00
Alex Vandiver d5a6b0f99a queue: Rename queue_size, and update for all local queues.
Despite its name, the `queue_size` method does not return the number
of items in the queue; it returns the number of items that the local
consumer has delivered but unprocessed.  These are often, but not
always, the same.

RabbitMQ's queues maintain the queue of unacknowledged messages; when
a consumer connects, it sends to the consumer some number of messages
to handle, known as the "prefetch."  This is a performance
optimization, to ensure the consumer code does not need to wait for a
network round-trip before having new data to consume.

The default prefetch is 0, which means that RabbitMQ immediately dumps
all outstanding messages to the consumer, which slowly processes and
acknowledges them.  If a second consumer were to connect to the same
queue, they would receive no messages to process, as the first
consumer has already been allocated them.  If the first consumer
disconnects or crashes, all prior events sent to it are then made
available for other consumers on the queue.

The consumer does not know the total size of the queue -- merely how
many messages it has been handed.

No change is made to the prefetch here; however, future changes may
wish to limit the prefetch, either for memory-saving, or to allow
multiple consumers to work the same queue.

Rename the method to make clear that it only contains information
about the local queue in the consumer, not the full RabbitMQ queue.
Also include the waiting message count, which is used by the
`consume()` iterator for similar purpose to the pending events list.
2020-10-09 20:40:39 -07:00
Alex Vandiver a1ce1aca3b queue: Update comment to be more accurate about import errors. 2020-10-09 20:40:32 -07:00
Alex Vandiver 2d71ca1fb8 email: Remove unused `log_digest_event` function.
Its last callsite was removed in e46cbaffa2.

Also ref #6786.
2020-10-08 20:35:53 -07:00
sahil839 4e8f5b5b31 streams: Change access_stream_for_delete_or_update to also return sub.
We modify access_stream_for_delete_or_update function to return
Subscription object also along with stream. This change will be
helpful in avoiding an extra query to get subscription object in
code for updating subscription role.
2020-10-08 17:07:30 -07:00
Aman Agrawal 8b419c93e4
message_send: Fix old guests being treated as full members.
For streams in which only full members are allowed to post,
we block guest users from posting there.

Guests users were blocked from posting to admin only streams
already. So now, guest users can only post to
STREAM_POST_POLICY_EVERYONE streams.

This is not a new feature but a bugfix which should have
happened when implementing full member stream policy / guest users.
2020-10-08 11:30:11 -07:00
Anders Kaseorg e1a3aa9c1a deliver_scheduled_messages: Exit cleanly on Ctrl+C.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Anders Kaseorg 363374346c management: Use signal.pause to sleep forever.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Anders Kaseorg c7cbdba257 eslint: Fix unicorn/{new-for-builtins,throw-new-error,error-message}.
https://github.com/sindresorhus/eslint-plugin-unicorn/blob/master/docs/rules/new-for-builtins.md
https://github.com/sindresorhus/eslint-plugin-unicorn/blob/master/docs/rules/throw-new-error.md
https://github.com/sindresorhus/eslint-plugin-unicorn/blob/master/docs/rules/error-message.md

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:00:33 -07:00