Commit Graph

12717 Commits

Author SHA1 Message Date
Alex Vandiver 3037e22f61 tornado: Never use proxies when talking from Django to Tornado.
The `no_proxy` parameter does not work to remove proxying[1]; in this
case, since all requests with this adapter are to the internal Tornado
process, explicitly pass in an empty set of proxies to disable
proxying.

[1] https://github.com/psf/requests/issues/4600
2020-10-28 12:13:04 -07:00
Alex Vandiver 97745688ca docs: Link to the new doc home of the email gateway. 2020-10-28 12:13:04 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Alex Vandiver 142de0f670 queue: Increase default timeout to 30s, from 10s.
Not all of the workers are known to be safe to interrupt; they might
leave inconsistent state.  As such, terminating them with timeouts
should currently only be a last-resort against stalled queues, not a
regular occurrence.
2020-10-27 16:39:31 -07:00
Alex Vandiver c73dd194f0 sentry: Group all worker timeouts together, by queue.
Since the exception can be triggered at arbitrary places in the stack
based on whenever the alarm happens to fire, they do not often group
together.

Explicitly group them together, grouped only by which queue the work
is in.
2020-10-27 16:39:31 -07:00
Anders Kaseorg 1352f2f233 python: Replace manual quote_plus usage with urlencode.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 4e9d587535 python: Pass query parameters as a dict when making GET requests.
This provides automatic URL-encoding.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 41f509170b users: Canonicalize the timezone identifier.
While working on shifting toward native browser time zone APIs
(#16451), it was found that all but very recent Chrome and Node
versions reject certain legacy timezone aliases like US/Pacific
(https://crbug.com/364374).

For now, we only canonicalize the timezone property returned in user
objects and not the timezone setting itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg a8b1691e97 timezone: Convert get_common_timezones cache to lru_cache.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 0b288f92c9 timezone: Remove get_timezone wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 0134112b51 timezone: Remove get_all_timezones wrapper.
Both callers want a set, and pytz already provides all_timezones_set.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Chris Bobbe 500f92169e openapi: Fix the "the the" instances. 2020-10-26 12:50:01 -07:00
Chris Bobbe dd43d4feb8 openapi: Note that a user's `avatar_url` may be missing.
Like it already says in the detail about
`user_avatar_url_field_optional`, but on the field itself [1].

[1] https://github.com/zulip/zulip-mobile/pull/4230#discussion_r493109645
2020-10-26 12:50:01 -07:00
Tim Abbott 6d7cd351a3 events: Optimize creating streams for new users.
During the new user creation code path, there can be no existing
active clients for the user being created, so we can skip the code to
send events to that user's clients.

The tests here reflect that we need to send fewer events, and do fewer
queries that would have been spent computing data for these..

Fixes #16503, combined with the long series of recent changes by Steve
Howell to fix super-linear behavior in this code path.
2020-10-26 12:47:15 -07:00
Steve Howell 88a7a1b002 events: Optimize peer_add/peer_remove for public streams.
We no bulk up peer_add/peer_remove events by user if the
same user has subscribed to multiple streams (and just
that single user).

This mostly optimizes the new-user codepath, but the
algorithm is a bit more general in nature.
2020-10-26 12:33:28 -07:00
Alex Vandiver 7cf737988d queue: Be more explicit about test/real queue division. 2020-10-26 12:32:47 -07:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Steve Howell 3ad1335a97 tests: Clear ContentType cache for user test.
This keeps the number of queries predictable.
2020-10-26 07:18:08 -04:00
Steve Howell 5ef01b3ad8 tests: Fix test_create_user_with_multiple_streams.
This test was flaky due to some date-related
non-determinism.  I make all the Message objects
current to make add_new_user_history reliably
try to bulk-update UserMessage rows to read.
2020-10-26 07:18:08 -04:00
Anders Kaseorg 96dee7e40b decorator: Unfork redirect_to_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-25 14:58:53 -07:00
sahil839 a96811ab58 management: Replace knight command with change_user_role command.
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.

Tweaked by tabbott to improve the logging output and update documentation.

Fixes #16586.
2020-10-25 14:55:08 -07:00
Harsh Srivastava 9b31df009b openapi: Fix excessively large test_events failure output.
Because of the very large `oneOf` clause of the formats of events
possible in Zulip's `GET /events` system, we had issues with
`test-backend` failures for missing documentation for a new event
format being like 1000 lines of output, which was very much unhelpful.

Fix this by limiting the output use only the oneOf variants that are
broadly similar to the actual payload received.

Fixes #16023.
2020-10-23 17:00:17 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg e513b75e86 markdown: Remove handler for old bug with incompatible twitter library.
See commit 8b002040e0 and #86.  The
development environment bug that necessitated this handler has long
been irrelevant.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:30:26 -07:00
Anders Kaseorg b9fd49a2c6 mypy: Correct mistaken *args type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:29:13 -07:00
Anders Kaseorg d295da676b test_message_fetch: Clean up obsolete PGroonga bug workaround.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-22 23:27:23 -07:00
Anders Kaseorg 831d086110 i18n: Fix get_language_translation_data for zh_TW.
Fixes #16600.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-22 16:43:02 -07:00
sahil839 571bb62e3d events: Update subscriber list on peer_add for unsubscribed streams.
We update the subscriber list on peer_add event for unsubscribed
streams as well.
2020-10-22 15:12:32 -07:00
sahil839 733d26aef2 events: Update subscriber list on peer_remove for never subscribed stream.
We now update the subscriber list on peer_remove event for never
subscribed streams also.
2020-10-22 15:12:32 -07:00
sahil839 af9b153ee3 events: Update subscriber list on peer_remove for unsubscribed stream.
We update the subscriber list on peer_remove event for unsubscribed
streams also.
2020-10-22 15:12:32 -07:00
sahil839 709edd29d4 test_events: Fix comment in do_test_subscribe_events.
The comment still pointed to 'vacate' event flow, but
we have removed the vacate event in a9356508ca.
This commit fixes the comment to depict the correct
purpose of below lines, i.e. to test the remove
event flow.
2020-10-22 15:12:32 -07:00
sahil839 e578742b02 test_events: Remove 'realm_user' from event_types in subscription test.
We were including 'realm_user' in event_types along with 'subscription',
but we don't send event of type 'realm_user' when subscribing to a new
stream. This was added in 1c332f5d6a.

This commit removes 'realm_user' from event_types.
2020-10-22 15:12:32 -07:00
sahil839 d0f5537fb2 actions: Modify check_message for handling wildcard_mention_policy setting.
This commit adds enforcement for sending messages containing wildcard
mentions according to wildcard_mention_policy.
2020-10-22 14:46:32 -07:00
sahil839 25f32d461e tests: Add tests for all the values of wildcard_mention_policy. 2020-10-22 12:08:22 -07:00
Mateusz Mandera 48f80fcb0a auth: Expect name in request params in Apple auth.
The name used to be included in the id_token, but this seems to have
been changed by Apple and now it's sent in the `user` request param.

https://github.com/python-social-auth/social-core/pull/483 is the
upstream PR for this - but upstream is currently unmaintained, so we
have to monkey patch.

We also alter the tests to reflect this situation. Tests no longer put
the name in the id_token, but rather in the `user` request param in the
browser flow, just like it happens in reality.

An adaptation has to be made in the native flow - since the name won't
be included by Apple in the id_token anymore, the app, when POSTing
to the /complete/apple/ endpoint,
can (and should for better user experience)
add the `user` param formatted as json of
{"email": "hamlet@zulip.com", "name": {"firstName": "Full", "lastName": "Name"}}
dict. This is also reflected by the change in the
native flow tests.
2020-10-22 12:07:46 -07:00
Steve Howell 7ff3859136 subscriber events: Change schema for peer_add/peer_remove.
We now can send an implied matrix of user/stream tuples
for peer_add and peer_remove events.

The client code basically does this:

    for stream_id in event['stream_ids']:
        for user_id in event['user_ids']:
            update_sub(stream_id, user_id)

We used to send individual events, which gets real
expensive when you are creating new streams. For
the case of copy-to-stream case, we should see
events go from U to 1, where U is the number of users
added.

Note that we don't yet fully optimize the potential
of this schema.  For adding a new user with lots
of default streams, we still send S peer_add events.

And if you subscribe a bunch of users to a bunch of
private streams, we only go from U * S to S; we can't
optimize it down to one event easily.
2020-10-22 11:19:53 -07:00
Anders Kaseorg efa8dd3a47 compilemessages: Sort language list with Unicode Collation Algorithm.
Right now the list of languages in Display settings → Default language
is sorted in an unintuitive order due to the varying case conventions:

British English
Chinese (Taiwan)
Deutsch
English
Hindi
Indonesian (Indonesia)
Lietuviškai
Magyar
Malayalam
Nederlands
Português
Română
Tiếng Việt
Türkçe
català
español
français
galego
italiano
polski
suomi
svenska
česky
Русский
Українська
български
српски
فارسی
தமிழ்
日本語
简体中文
繁體中文
한국어

Fix the sort to use the locale-independent Unicode Collation
Algorithm:

British English
català
česky
Chinese (Taiwan)
Deutsch
English
español
français
galego
Hindi
Indonesian (Indonesia)
italiano
Lietuviškai
Magyar
Malayalam
Nederlands
polski
Português
Română
suomi
svenska
Tiếng Việt
Türkçe
български
Русский
српски
Українська
فارسی
தமிழ்
한국어
日本語
简体中文
繁體中文

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 15:22:18 -07:00
Steve Howell 85ed6f332a performance: Avoid Recipient lookup for stream messages.
All the fields of a stream's recipient object can
be inferred from the Stream, so we just make a local
object.  Django will create a Message object without
checking that the child Recipient object has been
saved.  If that behavior changes in some upgrade,
we should see some pretty obvious symptom, including
query counts changing.

Tweaked by tabbott to add a longer explanatory comment, and delete a
useless old comment.
2020-10-20 11:47:23 -07:00
Steve Howell 7bbcc2ac96 refactor: Compute peers for public streams later.
This saves us a query for edge cases like when
you try to unsubscribe from a public stream
that you have already unsubscribed from.

But this is mostly to prep for upcoming
optimizations.
2020-10-20 11:31:22 -07:00
Steve Howell 363e5d31a6 refactor: Split out public/private logic for peer events.
This doesn't change anything yet, but the goal is
to eventually optimize events for the case where
one user (typically a new user) gets subscribed
to multiple public streams.
2020-10-20 11:31:22 -07:00
Steve Howell 3961e69381 refactor: Extract send_peer_subscriber_events.
We now use the same basic code to send peer_add
and peer_remove events.
2020-10-20 11:31:22 -07:00
Anders Kaseorg 254b904965 markdown: Migrate off deprecated extension registration interface.
Fixes #15205.

https://python-markdown.github.io/change_log/release-3.0/#homegrown-ordereddict-has-been-replaced-with-a-purpose-built-registry
https://python-markdown.github.io/change_log/release-3.0/#md_globals-keyword-deprecated-from-extension-api

The priority numbers are arbitrarily chosen to preserve the existing
order.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 18:31:12 -07:00
akshatdalton 287c4ed2bb markdown: Fix Youtube and Vimeo preview overriding markdown link titles bug.
Initially markdown titles were overridden by Youtube and Vimeo preview titles.
But now it will check if any markdown title is present to replace Youtube or
Vimeo preview titles, if preview of linked websites is enabled.
Fixes #16100
2020-10-19 12:06:13 -07:00
Anders Kaseorg d81a93cdf3 requirements: Upgrade markdown to 3.3.1.
Upstream has slightly changed the whitespace around stashes.  Take
this opportunity to clean up the extra blank lines we were outputting.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg 2ab15149bc forms: Convert percent formatting to "".format.
Caught by upgrading Semgrep.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg f461a64a6b i18n: Fix some ineffective calls to ugettext at top level.
Translation has no effect when we don’t yet know what language we’re
translating for.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-18 14:31:15 -07:00
Anders Kaseorg bba43f35ca i18n: Be deliberate about distinguishing ugettext and ugettext_lazy.
The early str conversions in zerver.models were defeating the point of
ugettext_lazy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-18 14:31:15 -07:00
Steve Howell e6f6f8d45f refactor: Avoid "stream_id" on sub.
There was no need to put "stream_id" on the sub
dictionary here.  It's kinda annoying to introduce
the little helper here, but I feel
that's better than crufting up the sub data
structure.
2020-10-18 14:27:31 -07:00