Commit Graph

15186 Commits

Author SHA1 Message Date
Anders Kaseorg f29553d809 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 10:10:06 -07:00
Adam Sah dbd03b5054 email_backends: Fix concurrent backend testing for generate_emails.
Previously, this command would reliably fail:

```
tools/test-backend --skip-provision-check --parallel=3
  zerver.tests.test_email_log.EmailLogTest.test_forward_address_details
  zerver.tests.test_email_log.EmailLogTest.test_generate_and_clear_email_log
  zerver.tests.test_example.TestDevelopmentEmailsLog
```

and now it reliably succeeds. :-)

After hours of fiddling/googling/hair-tearing, I found that
mocking-away Django Connection.send_messages() was the best:

- We're testing Zulip and not Django.
- Mocking at this lower level exercises more of our code.
- EmailLogBackEnd._do_send_messages() helper method added to simplify mocking.

Fixes #21925.
2022-05-03 08:45:51 -07:00
Anders Kaseorg 7acb642fa5 requirements: Upgrade to Tornado 6.
Fixes #8913.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 6fd1a558b7 runtornado: Switch to asyncio event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg c263bfdb41 queue: Use a thread-local Pika connection.
According to the documentation: “Pika does not have any notion of
threading in the code. If you want to use Pika with threading, make
sure you have a Pika connection per thread, created in that thread. It
is not safe to share one Pika connection across threads, with one
exception: you may call the connection method add_callback_threadsafe
from another thread to schedule a callback within an active pika
connection.”

https://pika.readthedocs.io/en/stable/faq.html

This also means that synchronous Django code running in Tornado will
use its own synchronous SimpleQueueClient rather than sharing the
asynchronous TornadoQueueClient, which is unfortunate but necessary as
they’re about to be on different threads.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg c9faefd50e cache: Instantiate only one BMemcached cache backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 5d69dafddb runtornado: Avoid deprecated IOLoop debugging methods.
IOLoop.set_blocking_log_threshold and IOLoop.handle_callback_exception
are removed in Tornado 6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg e4bf7066f3 tornado: Remove instrument_tornado_ioloop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg bded7180f7 tornado: Unfork tornado.autoreload.
We previously forked tornado.autoreload to work around a problem where
it would crash if you introduce a syntax error and not recover if you
fix it (https://github.com/tornadoweb/tornado/issues/2398).

A much more maintainable workaround for that issue, at least in
current Tornado, is to use tornado.autoreload as the main module.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Lauryn Menard a0d483a9a5 openapi: Fix reference link in `/register-queue` documentation.
Fixes a reference link in `realm_enable_spectator_access`
description in `/register-queue` endpoint documentation.
2022-04-29 08:35:25 -07:00
Alex Vandiver bcd179be0e import: Default the S3 metadata user of emoji to the first owner.
This matches the metadata that we store in the database, and means
that the S3 metadatata invariant of always having a `user_profile_id`
in the metadata.

This does not fix existing imports, which may still have missing
`user_profile_id`s.
2022-04-28 16:51:58 -07:00
Tim Abbott 627144b0c7 settings: Simplify setting organization default language.
The changes in the last few commits changed the semantics of the
organization default language to no longer be the primary source of
information for a user's language when creating a new account.

Here, we change the settings UI and /help/ documentation to reflect
this.
2022-04-28 15:03:26 -07:00
Sahil Batra 47c6100da9 onboarding: Override the language of welcome bot PMs with user setting. 2022-04-28 15:03:26 -07:00
Siddharth Asthana 9954db4b59 users: Use browser locale to set the default language of new user.
This commit reads the browser locale during user registration, and
sets it as default language of user if it is supported by Zulip.
Otherwise, it is set to realm's default language.
2022-04-28 15:03:26 -07:00
Siddharth Asthana 807c73ef17 i18n: Add function to get default language for new users.
It returns realm's default language if the browser's locale
is not available or is unsupported.
2022-04-28 15:03:26 -07:00
Sahil Batra 476a217087 i18n: Add get_browser_language_code function.
This commit adds get_browser_language_code function
which returns None if there is no Accept-language
header in the request or Accept-languge header contains
only unsupported languages or all languages (meaning
header having value of '*'). Otherwise it returns the
language with highest weight/quality-value.
2022-04-28 15:03:26 -07:00
Christopher Chong b6f6f6db8d slack_incoming: Process request payload directly in view.py.
slack_incoming webhook previously used has_request_variables to
extract payload from HttpRequest object first, before trying to
access HttpRequest.body again in view.py. This caused an error
when one sends a request without payload - it is forbidden to
read from request data stream twice.

Instead of relying on has_request_variables, this PR extracts
payload depending on content type in view.py directly to avoid
reading request data stream twice.

Fixes #19056.
2022-04-28 14:45:38 -07:00
Aman Agrawal 4e08c737ca home: For web public realms, skip login for spectators.
To provide a smoother experience of accessing a web public stream,
we don't ask user to login unless user directly requests a
`/login` URL.

Fixes #21690.
2022-04-28 12:34:29 -07:00
Tim Abbott bd2dc7358b help: Add redirect to new public access page. 2022-04-28 12:27:13 -07:00
Aman Agrawal 00fffd1681 help: Rename `web-public-streams` page to `public-access-option`. 2022-04-28 12:09:08 -07:00
Alex Vandiver 127108c7d1 workers: Log the exception if the export fails.
We previously just swallowed the exception entirely.
2022-04-28 11:52:47 -07:00
Anders Kaseorg 6331a314d4 Correctly hyphenate “non-”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Anders Kaseorg d58fece832 Correctly hyphenate “web-public”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Chris Bobbe 8ed605de13 openapi: Mention that `profile_data` will be missing for bots.
For why we put this in the schema, instead of at the places where
the schema is consumed, Tim says:

> The code to not include it for bots is present in the common code
> path for formatting user objects for the API, so it should apply
> to all places we have users in the API documentation.

Discussion:
  https://chat.zulip.org/#narrow/stream/412-api-documentation/topic/.60profile_data.60.20in.20.60.2Fregister.60.20response/near/1374737
2022-04-27 22:09:55 -07:00
Aman Agrawal 5ee4f71701 avatar: Add rate limit similar to attachments on medium avatars.
Followup on #20136
2022-04-27 16:51:18 -07:00
Zixuan James Li b4feb673f1 push_notifications: Soft reactivate mentioned users.
Fixes #19861

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Zixuan James Li a8fd9eb701 email_notifications: Soft reactivate mentioned users.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Zixuan James Li d8101de34d notifications: Move user group mentions helpers together.
This refactored `get_mentioned_user_group_name` from
`zerver/lib/email_notifications.py` to
`zerver/lib/notification_data.py` just after
`get_user_group_mentions_data` to indicate the logical
similarity between them.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Sahil Batra 0c7828aeab user_groups: Add get_user_group_direct_members function.
This commit adds get_user_group_direct_members function
which returns a QuerySet of UserProfile and is used in
is_user_in_group function.
2022-04-27 14:17:52 -07:00
Sahil Batra d9d27b459d user_groups: Rename get_user_group_direct_members function.
This commit renames get_user_group_direct_members function to
get_user_group_direct_member_ids as it returns a list of ids
and to avoid it being parallel to get_recursive_group_members,
which returns a QuerySet.
2022-04-27 14:17:52 -07:00
Sahil Batra 0f01e3d327 tests: Add checks for user groups in different realms. 2022-04-27 14:17:52 -07:00
Sahil Batra 4d319ff34c tests: Use high enough number for invalid user group and user ids.
We use 1111 for testing invalid user and user group id cases instead
of 25, since we may have a user group in 25 in test data in future.
2022-04-27 14:17:52 -07:00
Anders Kaseorg 7c247a11a9 test_openapi: Use Python 3.8 typing.{get_origin,get_args}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 098a514599 python: Use Python 3.8 shlex.join function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 4d903dcc94 i18n: Remove size limit on lru_cache for get_language_list.
Really just for consistency, since the size will be at most 1 anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 2afe82ba2e test_link_embed: Convert "".format to f-string.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg b572b18e70 test_digest: Modernize set literal syntax.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 1c228de049 models: Excise NULL from pointlessly nullable fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Anders Kaseorg fd16f97d6b python: Excise None from pointlessly nullable booleans.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Anders Kaseorg af30538a65 forms: Fix type of domain_override parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Mateusz Mandera ee0021de08 models: Change Stream.history_public_to_subscribers value to True.
The default of a Stream is to be public - having
history_public_to_subscribers default to False is inconsistent with
that. The defaults on the model should generally be consistent.
2022-04-27 12:08:01 -07:00
Mateusz Mandera 04fdf3e4d9 import_utils: Fix history_public_to_subscribers being set incorrectly.
history_public_to_subscribers wasn't explicitly set when creating
streams via build_stream, thus relying on the model's default of False.
This lead to public streams being created with that value set to False,
which doesn't make sense.

We can solve this by inferring the correct value based on invite_only in
the build_stream funtion itself - rather than needing to add a flag
argument to it.

This commit also includes a migration to fix public stream with the
wrong history_public_to_subscribers value.

Fixes #21784.
2022-04-27 12:08:01 -07:00
Anders Kaseorg a543dcc8e3 Remove Debian 10 support.
As a consequence:

• Bump minimum supported Python version to 3.8.
• Move Vagrant environment to Ubuntu 20.04, which has Python 3.8.
• Move CI frontend tests to Ubuntu 20.04.
• Move production build test to Ubuntu 20.04.
• Move 3.4 upgrade test to Ubuntu 20.04.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 16:32:02 -07:00
Lauryn Menard 1292338537 frontend: Add `org_type` to realm settings updates and events.
Adds a drop-down menu for updating the organization type in the
`organization_profile_admin` page. Implements front end for
this setting to work / update like other organization profile,
notification and permissions settings.

One special note about this dropdown is that the listed options
should change once an organization has successfully set a type
other than 'unspecified' in the database. To accomplish this
the initial settings overlay build checks the realm_org_type
value in the page_params to select the correct options list,
and when the dropdown value is reset, either for update events
or for discarding changes, the page_params value is again used
to check for whether the 'unspecified' value should be present
as an option in the dropdown menu.

Adds basic node test for the `server_events_dispatch`.
Also adds a new help center documentation article for this
organization setting that is linked to in the UI.

Fixes #21692.
2022-04-26 16:29:12 -07:00
Lauryn Menard d2207d4ad5 backend: Add `org_type` to realm settings updates and events.
`org_type` already exists as a field in the Realm model and is
used when organizations are created / updated in Zulip Cloud,
via the `/analytics/support` view.

Extends the `PATCH /realm` view to be able update `org_type` as
other realm / organization settings are updated, but using the
special log / action that was created for the analytics view.

Adds a field to the `realm op: update` / `realm op: update_dict`
events, which also means an event is now sent when and if the
`org_type` is updated via the analytics view. This is similar
to how updates to an organization's `plan_type` trigger events.

Adds `realm_org_type` as a realm setting fetched from the
`POST /register` endpoint.
2022-04-26 16:29:12 -07:00
Chris Bobbe 9df0f1433e openapi [nfc]: Give user / cross-realm bot properties a better ordering
Greg noticed, "Also [awkward] that `user_id` is in a random place in the
middle instead of at the top, and that `is_guest` is nowhere near
`is_admin` and `is_owner`."

Discussion:
  https://chat.zulip.org/#narrow/stream/412-api-documentation/topic/ordering.20of.20user.20properties/near/1371370
2022-04-26 13:36:07 -07:00
evykassirer e4c51488c0 populate_db: Add support for creating messages older than 5 days ago.
Fixes #21646.
2022-04-25 16:16:15 -07:00
Sahil Batra 37793434f7 user_groups: Add API endpoint to get subgroups of a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra 1b3c972d9b user_groups: Add API endpoint to get members of a user group.
This commit adds 'GET /user_groups/{user_group_id}/members'
endpoint to get members of a user group. "direct_member_only"
parameter can be passed as True to the endpoint to get only
direct members of the user group and not the members of
subgroup.
2022-04-25 10:24:03 -07:00