Commit Graph

14466 Commits

Author SHA1 Message Date
Anders Kaseorg 5344e86c57 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-02 08:16:20 -08:00
Palash a444560684 onboarding: Replace 'edit profile' with 'profile' in bot command list.
And change the tests accordingly.
2021-12-02 08:12:51 -08:00
Alya Abbott 8070b84785 help center: Remove outdated intro from index page. 2021-12-01 17:05:59 -08:00
Palash 6404b7bf61 onboarding: Make Welcome Bot more interactive.
1. The initial welcome message now contains less detail.
2. The bot now responds to these commands: "apps", "edit profile",
"dark mode", "light mode", "streams", "topics", "message formatting",
"keyboard shortcuts" and "help" - the bot still responds if there are
slight variations in these commands.
3. Tests have been made to check if bot responds to the advertised
commands (with variations) and gives a negative message if it doesn't
understand the message.

With substantial tweaks by tabbott.

Fixes #19900.
2021-12-01 16:30:48 -08:00
Mateusz Mandera a014ef75a3 CVE-2021-43791: Validate confirmation keys in /accounts/register/ codepath.
A confirmation link takes a user to the check_prereg_key_and_redirect
endpoint, before getting redirected to POST to /accounts/register/. The
problem was that validation was happening in the check_prereg_key_and_redirect
part and not in /accounts/register/ - meaning that one could submit an
expired confirmation key and be able to register.

We fix this by moving validation into /accouts/register/.
2021-12-01 23:14:04 +00:00
Mateusz Mandera a1cd660147 confirmation: Use error status codes for confirmation link error pages. 2021-12-01 23:14:04 +00:00
Eeshan Garg f976bf723e links: Rename developer-community -> development-community. 2021-12-01 13:41:46 -08:00
Tim Abbott 223d769680 api docs: Clean up description of legacy reaction events.
* Remove `is_mirror_dummy` copy-paste error.
* Clarify explanation of the user dictionary.
2021-11-28 11:52:51 -08:00
Lauryn Menard 7713b371a5 api: Migrate `/update-subscription-settings` response value.
Migrates the `/update-subscription-settings` api endpoint to the
`ignored_parameters_unsupported` model, which is also currently used
by `/update-settings` and `update-realm-user-settings-defaults`.

This change is a step towards preparing for an eventual migration to
have all endpoints return an `ignored_parameters_unsupported` block.

Previously the `/update-subscription-settings` endpoint returned a
copy of the data object sent in the request.

Fixes #15307.
2021-11-26 22:25:53 -08:00
Sahil Batra 88e21d0387 misc: Replace "night mode" with "dark theme" in comments. 2021-11-26 22:03:29 -08:00
Mateusz Mandera 994ab8115e scim: Order Users by id when queried using filter syntax.
django-scim2 doesn't order the rows when fetching them in reponse to a
query using the filter syntax. We ensure that ORDER BY id is always
appended to the SQL queries.
2021-11-26 16:06:16 -08:00
Steve Howell 946e8064a3 user exports: Add several tables to the tarball.
We add the following tables to the user export:

    AlertWord
    CustomProfileFieldValue
    RealmAuditLog
    Service
    UserActivity
    UserActivityInterval
    UserCount
    UserGroup
    UserHotspot
    UserPresence
    UserTopic

Except for UserCount, we achieve this by sharing
code with the realm export via
add_user_profile_child_configs.

UserCount is handled slightly differently than realm
exports due to which key we trigger off.

It's possible that RealmAuditLog is incomplete for
single users, since we may also want rows where they
are the acting_user. This commit finds rows where
they are the modified_user. For non-admins I believe
it's rarely the case that they are the actor, and
they will tend to be the modified user if the two
fields are different at all.  For admins it's
arguable we want to see both changes they enacted
as well as changes that affected them.
2021-11-25 08:36:43 -08:00
strifel 11211bd9c4 integrations: Switch BigBlueButton over to SHA256.
This commit switches the BigBlueButton integration
to use SHA256 instead of SHA1 as BigBlueButton supports
it and scalelite does now, too.

Fixes #19966.
2021-11-25 07:58:27 -08:00
Steve Howell 939032b45f test flake: Sort users for scim test. 2021-11-25 07:57:20 -08:00
Alya Abbott 74b4640727 portico: Add Rust case study. 2021-11-24 15:17:33 -08:00
Sahil Batra 30c190a120 test: Check json_fetch_raw_message raises error.
This commit adds a check to verify that json_fetch_raw_message
raises error when enable_spectator_access is False.
2021-11-24 10:37:51 -08:00
Tim Abbott d860b8e568 api docs: Update API docs and changelog for new setting.
The commit 2c1a87166c did not properly
document the new feature in the changelog; while I was at it, I wrote
more detail in the field's description.
2021-11-23 14:05:37 -08:00
Sahil Batra 56a8443f05 models: Check enable_spectator_access value in web_public_streams_enabled.
This commit changes web_public_streams_enabled to return False if
realm.enable_spectator_access is False. This is added so that
creating web-public streams is not allowed if enable_spectator_access
is False.
2021-11-23 10:48:20 -08:00
Sahil Batra ad99b4fac9 streams: Allow changing stream to be web-public based on creation setting.
We allow a user to make an existing stream web-public only if user is
allowed to create web-public streams.
2021-11-23 10:48:20 -08:00
Sahil Batra 2c1a87166c register: Pass settings.WEB_PUBLIC_STREAMS_ENABLED to clients.
This commit adds 'server_web_public_streams_enabled' field to
the register response to pass settings.WEB_PUBLIC_STREAMS_ENABLED
to clients.
2021-11-23 10:48:20 -08:00
Alex Vandiver 3efed5f1e6 queue_processors: Shut down background missedmessage_emails thread.
Python's behaviour on `sys.exit` is to wait for all non-daemon threads
to exit.  In the context of the missedmessage_emails worker, if any
work is pending, a non-daemon Timer thread exists, which is waiting
for 5 seconds.  As soon as that thread is serviced, it sets up another
5-second Timer, a process which repeats until all
ScheduledMessageNotificationEmail records have been handled.  This
likely takes two minutes, but may theoretically take up to a week
until the thread exits, and thus sys.exit can complete.

Supervisor only gives the process 30 seconds to shut down, so
something else must prevent this endless Timer.

When `stop` is called, take the lock so we can mutate the timer.
However, since `stop` may have been called from a signal handler, our
thread may _already_ have the lock.  As Python provides no way to know
if our thread is the one which has the lock, make the lock a
re-entrant one, allowing us to always try to take it.

With the lock in hand, cancel any outstanding timers.  A race exists
where the timer may not be able to be canceled because it has
finished, maybe_send_batched_emails has been called, and is itself
blocked on the lock.  Handle this case by timing out the thread join
in `stop()`, and signal the running thread to exit by unsetting the
timer event, which will be detected once it claims the lock.
2021-11-23 10:45:49 -08:00
Shlok Patel b3c58f454f api: Prevent special characters in topics.
Special characters, including `\r`, `\n`, and more esoteric codepoints
like non-characters, can negatively affect rendering and UI behaviour.

Check for, and prevent making new messages with, characters in the
Unicode categories of `Cc` (control characters), `Cs`, (surrogates),
and `Cn` (unassigned, non-characters).

Fixes #20128.
2021-11-22 22:09:06 -08:00
Sahil Batra eef9dd963b zcommand: Replace "mode" with "theme" in the returned message.
This commit replaces "dark mode" and "light mode" with "dark theme"
and "light theme" in the message returned and shown in a little
popup in the UI, when color scheme settings are changed through
slash commands.
2021-11-22 14:59:16 -08:00
Lauryn Menard fee7b10db4 api_documentation: Fix `/update-subscription-settings` api doc.
Adds `wildcard_mentions_notify` as a property that can be updated
by the endpoint and removes mention of potential `null` value in
the return object because it is not possible.

Also cleans up the documentation of `in_home_view` legacy property
and updates the return object description to better reflect what
is actually returned.
2021-11-22 11:50:27 -08:00
Mateusz Mandera 8af7ffd9da rate_limit: Fix logging string when rate limiting email gateway.
realm.name is not the right "name" to log, we should use realm.subdomain
like everywhere else.
2021-11-22 10:28:56 -08:00
Aman Agrawal 2c4af0e111 integration: Fix overflowing display name on open collective page.
Removed extra text to stop display name from flowing out of logo
container.
2021-11-21 21:05:06 -08:00
Aman Agrawal 334f046842 events: Don't send data related to custom profile field to spectator.
Since spectators can't access personal profile settings and
can't view profile for other users. Hence, we don't send realm
custom profile field data and user's profile data to spectators.

Fixes #20301.
2021-11-20 21:01:38 -08:00
Sahil Batra f42b8f2fe1 misc: Replace "night theme" with "dark theme". 2021-11-19 11:49:59 -08:00
Sahil Batra 14b91043d2 misc: Replace "Day mode" with "Light theme". 2021-11-19 11:49:59 -08:00
Sahil Batra b72035abfe misc: Replace "Light mode" with "Light theme". 2021-11-19 11:49:59 -08:00
Sahil Batra 5024ba98cd misc: Rename "dark mode" to "dark theme".
Fixes part of #20228.
2021-11-18 16:57:11 -08:00
Sahil Batra cf4f31291c misc: Rename "Night mode" to "Dark theme".
This commit renames "Night mode" to "Dark theme" in the
settings dropdown, help pages and api documentation.

Fixes part of #20228.
2021-11-18 16:55:35 -08:00
Sahil Batra 08404f2824 help: Rename night-mode.md to dark-theme.md.
Fixes part of #20228.
2021-11-18 16:55:35 -08:00
Aman Agrawal 00d3b88257 realm: Enable spectator access by default in development.
Enable spectator access for test `zulip` realm in developement
setup.

Add option in `do_create_realm` to configure
`enable_spectator_access` field of `Realm`.
2021-11-18 12:55:12 -08:00
Aman Agrawal 2a922409aa realm: Add option to enable/disable spectator login.
We restrict access of messages from web public streams if
anonymous login is disabled via `enable_spectator_access`.

Display of `Anonymous login` button is now controlled by
the value of `enable_spectator_access`.

Admins can toggle `enable_spectator_access` via org settings in UI.
2021-11-18 12:55:12 -08:00
Aman Agrawal 28488d9939 test_decorators: Call mock_home via patched zulip_login_required.
This avoids a weird issue that when mocking both
zulip_login_required and home_real, home_realm doesn't stay
patched.
2021-11-18 12:55:12 -08:00
Lauryn Menard 5300ad8ca0 api_documentation: Display `null` data type if potential value.
If null is a potential value of data type for a return value or
parameter in the API endpoint, then it is rendered as an option.

This currently relies on the 'nullable' setting in the OpenAPI spec
that was removed in the 3.1.0 release. If/when the OpenAPI version
is updated, then how the `data_type` for parameters and return values
is rendered will need to be reworked.

Fixes #20264.
2021-11-17 13:20:24 -08:00
Tim Abbott 864fb7c978 api docs: Sync list of realm settings with reality.
It turns out these were just wrong.  We fix a few things:
* Sort the list of settings so that it's possible to compare with reality.
* Deleted additional fields that don't actually exist.
* Fixed various fields missing past feature level updates.
2021-11-16 15:51:03 -08:00
Tim Abbott 744f96df61 api docs: Delete additional improperly documented update_dict fields.
As with 618d0788f0, these fields will
not appear in events requests.
2021-11-16 15:24:34 -08:00
Tim Abbott 618d0788f0 api docs: Delete docs for nonexistent icon_file_size property.
This appears to have been confused with max_icon_file_size.
2021-11-16 14:57:50 -08:00
Tim Abbott cac8bcbc89 api docs: Fix incorrect realm_ prefixes in update_dict field names.
Neither of these fields use the `realm/update_dict` event type; they
use `realm/update`; we've attempted to clarify that in the previous
commit.

That reality means we don't have automated testing for these values,
and that meant that typos like these could slip through.
2021-11-16 14:57:50 -08:00
Tim Abbott e36f549ec6 api docs: Add more details around realm event types. 2021-11-16 14:57:50 -08:00
Mateusz Mandera 0abbb87155 auth: Include user_id in the params returned at the end of mobile flow.
The user id is a very useful piece of information that the mobile
client should have access to - instead of only getting the email. This
makes it much simpler to impleent clients that might be robust to
changes in email address.
2021-11-16 12:12:59 -08:00
Sahil Batra 9e979e9e66 models: Sort UserBaseSettings.property_types dict by key. 2021-11-16 12:01:04 -08:00
Sahil Batra 375b48f013 models: Sort UserBaseSettings.notification_settings_legacy dict by key. 2021-11-16 12:01:04 -08:00
Sahil Batra cae547cfd0 models: Sort Realm.property_types dict by key. 2021-11-16 12:01:04 -08:00
Alex Vandiver faeffa2466 queue_processors: Set a bounded prefetch size on rabbitmq queues.
RabbitMQ clients have a setting called prefetch[1], which controls how
many un-acknowledged events the server forwards to the local queue in
the client.  The default is 0; this means that when clients first
connect, the server must send them every message in the queue.

This itself may cause unbounded memory usage in the client, but also
has other detrimental effects.  While the client is attempting to
process the head of the queue, it may be unable to read from the TCP
socket at the rate that the server is sending to it -- filling the TCP
buffers, and causing the server's writes to block.  If the server
blocks for more than 30 seconds, it times out the send, and closes the
connection with:

```
closing AMQP connection <0.30902.126> (127.0.0.1:53870 -> 127.0.0.1:5672):
{writer,send_failed,{error,timeout}}
```

This is https://github.com/pika/pika/issues/753#issuecomment-318119222.

Set a prefetch limit of 100 messages, or the batch size, to better
handle queues which start with large numbers of outstanding events.

Setting prefetch=1 causes significant performance degradation in the
no-op queue worker, to 30% of the prefetch=0 performance.  Setting
prefetch=100 achieves 90% of the prefetch=0 performance, and higher
values offer only minor gains above that.  For batch workers, their
performance is not notably degraded by prefetch equal to their batch
size, and they cannot function on smaller prefetches than their batch
size.

We also set a 100-count prefetch on Tornado workers, as they are
potentially susceptible to the same effect.

[1] https://www.rabbitmq.com/confirms.html#channel-qos-prefetch
2021-11-16 11:48:50 -08:00
Alex Vandiver 7c3507feef queue: Allow passing down a prefetch count to pika. 2021-11-16 11:48:50 -08:00
Alex Vandiver 8d239f4984 tests: Move lambda in simulated_queue_client to help typing. 2021-11-16 11:48:50 -08:00
Alex Vandiver d1822b5630 tests: Move simulated_queue_client to the only test it is used in. 2021-11-16 11:48:50 -08:00