Commit Graph

14552 Commits

Author SHA1 Message Date
Alya Abbott 74b4640727 portico: Add Rust case study. 2021-11-24 15:17:33 -08:00
Sahil Batra 30c190a120 test: Check json_fetch_raw_message raises error.
This commit adds a check to verify that json_fetch_raw_message
raises error when enable_spectator_access is False.
2021-11-24 10:37:51 -08:00
Tim Abbott d860b8e568 api docs: Update API docs and changelog for new setting.
The commit 2c1a87166c did not properly
document the new feature in the changelog; while I was at it, I wrote
more detail in the field's description.
2021-11-23 14:05:37 -08:00
Sahil Batra 56a8443f05 models: Check enable_spectator_access value in web_public_streams_enabled.
This commit changes web_public_streams_enabled to return False if
realm.enable_spectator_access is False. This is added so that
creating web-public streams is not allowed if enable_spectator_access
is False.
2021-11-23 10:48:20 -08:00
Sahil Batra ad99b4fac9 streams: Allow changing stream to be web-public based on creation setting.
We allow a user to make an existing stream web-public only if user is
allowed to create web-public streams.
2021-11-23 10:48:20 -08:00
Sahil Batra 2c1a87166c register: Pass settings.WEB_PUBLIC_STREAMS_ENABLED to clients.
This commit adds 'server_web_public_streams_enabled' field to
the register response to pass settings.WEB_PUBLIC_STREAMS_ENABLED
to clients.
2021-11-23 10:48:20 -08:00
Alex Vandiver 3efed5f1e6 queue_processors: Shut down background missedmessage_emails thread.
Python's behaviour on `sys.exit` is to wait for all non-daemon threads
to exit.  In the context of the missedmessage_emails worker, if any
work is pending, a non-daemon Timer thread exists, which is waiting
for 5 seconds.  As soon as that thread is serviced, it sets up another
5-second Timer, a process which repeats until all
ScheduledMessageNotificationEmail records have been handled.  This
likely takes two minutes, but may theoretically take up to a week
until the thread exits, and thus sys.exit can complete.

Supervisor only gives the process 30 seconds to shut down, so
something else must prevent this endless Timer.

When `stop` is called, take the lock so we can mutate the timer.
However, since `stop` may have been called from a signal handler, our
thread may _already_ have the lock.  As Python provides no way to know
if our thread is the one which has the lock, make the lock a
re-entrant one, allowing us to always try to take it.

With the lock in hand, cancel any outstanding timers.  A race exists
where the timer may not be able to be canceled because it has
finished, maybe_send_batched_emails has been called, and is itself
blocked on the lock.  Handle this case by timing out the thread join
in `stop()`, and signal the running thread to exit by unsetting the
timer event, which will be detected once it claims the lock.
2021-11-23 10:45:49 -08:00
Shlok Patel b3c58f454f api: Prevent special characters in topics.
Special characters, including `\r`, `\n`, and more esoteric codepoints
like non-characters, can negatively affect rendering and UI behaviour.

Check for, and prevent making new messages with, characters in the
Unicode categories of `Cc` (control characters), `Cs`, (surrogates),
and `Cn` (unassigned, non-characters).

Fixes #20128.
2021-11-22 22:09:06 -08:00
Sahil Batra eef9dd963b zcommand: Replace "mode" with "theme" in the returned message.
This commit replaces "dark mode" and "light mode" with "dark theme"
and "light theme" in the message returned and shown in a little
popup in the UI, when color scheme settings are changed through
slash commands.
2021-11-22 14:59:16 -08:00
Lauryn Menard fee7b10db4 api_documentation: Fix `/update-subscription-settings` api doc.
Adds `wildcard_mentions_notify` as a property that can be updated
by the endpoint and removes mention of potential `null` value in
the return object because it is not possible.

Also cleans up the documentation of `in_home_view` legacy property
and updates the return object description to better reflect what
is actually returned.
2021-11-22 11:50:27 -08:00
Mateusz Mandera 8af7ffd9da rate_limit: Fix logging string when rate limiting email gateway.
realm.name is not the right "name" to log, we should use realm.subdomain
like everywhere else.
2021-11-22 10:28:56 -08:00
Aman Agrawal 2c4af0e111 integration: Fix overflowing display name on open collective page.
Removed extra text to stop display name from flowing out of logo
container.
2021-11-21 21:05:06 -08:00
Aman Agrawal 334f046842 events: Don't send data related to custom profile field to spectator.
Since spectators can't access personal profile settings and
can't view profile for other users. Hence, we don't send realm
custom profile field data and user's profile data to spectators.

Fixes #20301.
2021-11-20 21:01:38 -08:00
Sahil Batra f42b8f2fe1 misc: Replace "night theme" with "dark theme". 2021-11-19 11:49:59 -08:00
Sahil Batra 14b91043d2 misc: Replace "Day mode" with "Light theme". 2021-11-19 11:49:59 -08:00
Sahil Batra b72035abfe misc: Replace "Light mode" with "Light theme". 2021-11-19 11:49:59 -08:00
Sahil Batra 5024ba98cd misc: Rename "dark mode" to "dark theme".
Fixes part of #20228.
2021-11-18 16:57:11 -08:00
Sahil Batra cf4f31291c misc: Rename "Night mode" to "Dark theme".
This commit renames "Night mode" to "Dark theme" in the
settings dropdown, help pages and api documentation.

Fixes part of #20228.
2021-11-18 16:55:35 -08:00
Sahil Batra 08404f2824 help: Rename night-mode.md to dark-theme.md.
Fixes part of #20228.
2021-11-18 16:55:35 -08:00
Aman Agrawal 00d3b88257 realm: Enable spectator access by default in development.
Enable spectator access for test `zulip` realm in developement
setup.

Add option in `do_create_realm` to configure
`enable_spectator_access` field of `Realm`.
2021-11-18 12:55:12 -08:00
Aman Agrawal 2a922409aa realm: Add option to enable/disable spectator login.
We restrict access of messages from web public streams if
anonymous login is disabled via `enable_spectator_access`.

Display of `Anonymous login` button is now controlled by
the value of `enable_spectator_access`.

Admins can toggle `enable_spectator_access` via org settings in UI.
2021-11-18 12:55:12 -08:00
Aman Agrawal 28488d9939 test_decorators: Call mock_home via patched zulip_login_required.
This avoids a weird issue that when mocking both
zulip_login_required and home_real, home_realm doesn't stay
patched.
2021-11-18 12:55:12 -08:00
Lauryn Menard 5300ad8ca0 api_documentation: Display `null` data type if potential value.
If null is a potential value of data type for a return value or
parameter in the API endpoint, then it is rendered as an option.

This currently relies on the 'nullable' setting in the OpenAPI spec
that was removed in the 3.1.0 release. If/when the OpenAPI version
is updated, then how the `data_type` for parameters and return values
is rendered will need to be reworked.

Fixes #20264.
2021-11-17 13:20:24 -08:00
Tim Abbott 864fb7c978 api docs: Sync list of realm settings with reality.
It turns out these were just wrong.  We fix a few things:
* Sort the list of settings so that it's possible to compare with reality.
* Deleted additional fields that don't actually exist.
* Fixed various fields missing past feature level updates.
2021-11-16 15:51:03 -08:00
Tim Abbott 744f96df61 api docs: Delete additional improperly documented update_dict fields.
As with 618d0788f0, these fields will
not appear in events requests.
2021-11-16 15:24:34 -08:00
Tim Abbott 618d0788f0 api docs: Delete docs for nonexistent icon_file_size property.
This appears to have been confused with max_icon_file_size.
2021-11-16 14:57:50 -08:00
Tim Abbott cac8bcbc89 api docs: Fix incorrect realm_ prefixes in update_dict field names.
Neither of these fields use the `realm/update_dict` event type; they
use `realm/update`; we've attempted to clarify that in the previous
commit.

That reality means we don't have automated testing for these values,
and that meant that typos like these could slip through.
2021-11-16 14:57:50 -08:00
Tim Abbott e36f549ec6 api docs: Add more details around realm event types. 2021-11-16 14:57:50 -08:00
Mateusz Mandera 0abbb87155 auth: Include user_id in the params returned at the end of mobile flow.
The user id is a very useful piece of information that the mobile
client should have access to - instead of only getting the email. This
makes it much simpler to impleent clients that might be robust to
changes in email address.
2021-11-16 12:12:59 -08:00
Sahil Batra 9e979e9e66 models: Sort UserBaseSettings.property_types dict by key. 2021-11-16 12:01:04 -08:00
Sahil Batra 375b48f013 models: Sort UserBaseSettings.notification_settings_legacy dict by key. 2021-11-16 12:01:04 -08:00
Sahil Batra cae547cfd0 models: Sort Realm.property_types dict by key. 2021-11-16 12:01:04 -08:00
Alex Vandiver faeffa2466 queue_processors: Set a bounded prefetch size on rabbitmq queues.
RabbitMQ clients have a setting called prefetch[1], which controls how
many un-acknowledged events the server forwards to the local queue in
the client.  The default is 0; this means that when clients first
connect, the server must send them every message in the queue.

This itself may cause unbounded memory usage in the client, but also
has other detrimental effects.  While the client is attempting to
process the head of the queue, it may be unable to read from the TCP
socket at the rate that the server is sending to it -- filling the TCP
buffers, and causing the server's writes to block.  If the server
blocks for more than 30 seconds, it times out the send, and closes the
connection with:

```
closing AMQP connection <0.30902.126> (127.0.0.1:53870 -> 127.0.0.1:5672):
{writer,send_failed,{error,timeout}}
```

This is https://github.com/pika/pika/issues/753#issuecomment-318119222.

Set a prefetch limit of 100 messages, or the batch size, to better
handle queues which start with large numbers of outstanding events.

Setting prefetch=1 causes significant performance degradation in the
no-op queue worker, to 30% of the prefetch=0 performance.  Setting
prefetch=100 achieves 90% of the prefetch=0 performance, and higher
values offer only minor gains above that.  For batch workers, their
performance is not notably degraded by prefetch equal to their batch
size, and they cannot function on smaller prefetches than their batch
size.

We also set a 100-count prefetch on Tornado workers, as they are
potentially susceptible to the same effect.

[1] https://www.rabbitmq.com/confirms.html#channel-qos-prefetch
2021-11-16 11:48:50 -08:00
Alex Vandiver 7c3507feef queue: Allow passing down a prefetch count to pika. 2021-11-16 11:48:50 -08:00
Alex Vandiver 8d239f4984 tests: Move lambda in simulated_queue_client to help typing. 2021-11-16 11:48:50 -08:00
Alex Vandiver d1822b5630 tests: Move simulated_queue_client to the only test it is used in. 2021-11-16 11:48:50 -08:00
Alex Vandiver 64268f47e8 queue_processors: Drop unused current_queue_size, which was local size.
The `current_queue_size` key in the queue monitoring stats file was
the local queue size, not the global queue size -- d5a6b0f99a
renamed the function, but did not adjust the queue monitoring JSON,
despite the last use of it having been removed in cd9b194d88.

The function is still used to mark "we emptied our queue," and it
remains a reasonable metric for that.
2021-11-16 11:48:50 -08:00
Alex Vandiver 800e38016a queue_rate: Output to CSV, and run multiple prefetch values. 2021-11-16 11:48:50 -08:00
Alex Vandiver 49ad188449 rate_limit: Add a flag to lump all TOR exit node IPs together.
TOR users are legitimate users of the system; however, that system can
also be used for abuse -- specifically, by evading IP-based
rate-limiting.

For the purposes of IP-based rate-limiting, add a
RATE_LIMIT_TOR_TOGETHER flag, defaulting to false, which lumps all
requests from TOR exit nodes into the same bucket.  This may allow a
TOR user to deny other TOR users access to the find-my-account and
new-realm endpoints, but this is a low cost for cutting off a
significant potential abuse vector.

If enabled, the list of TOR exit nodes is fetched from their public
endpoint once per hour, via a cron job, and cached on disk.  Django
processes load this data from disk, and cache it in memcached.
Requests are spared from the burden of checking disk on failure via a
circuitbreaker, which trips of there are two failures in a row, and
only begins trying again after 10 minutes.
2021-11-16 11:42:00 -08:00
Mateusz Mandera 2d3d0f862a process_queue: Rename Threaded_worker to ThreadedWorker.
Threaded_worker doesn't fit the python naming convention we rely on.
2021-11-16 11:21:05 -08:00
Mateusz Mandera 7cc345d7b1 process_queue: Improve handling of exceptions in process_queue.
Unhandled exceptions propagating to process_queue were not caught there,
causing improper logging - errors didn't land in errors.log as expected.
Exceptions should be caught and explicitly logged by the process_queue
logger. Exceptions occurring during consuming events are caught and
handled inside the worker's logic - however those that happen while
setting up the worker were not addressed at all, and that's the core bug
we mean to address here.

Furthermore, in multi-threaded mode we want the autoreload mechanism to
be working - which it doesn't without catching the exceptions. The
correct approach is to - again - catch the exception, log it and then
send SIGUSR1 signal to trigger exit and autoreload.
2021-11-16 11:21:05 -08:00
Tim Abbott 3531afd754 migrations: Disable atomic flag on migration 0354.
I believe that this migration with the default of atomic=True will
fail when trying to convert the field to PositiveIntegerField if there
were any 0 values present in the database when the migration began.
The fix is to have each of the steps be their own transaction.
2021-11-15 17:24:49 -08:00
Tim Abbott 3e9c2dc03d models: Add comment documenting how email uniqueness is enforced. 2021-11-15 13:46:28 -08:00
Alex Vandiver a2f89f21e3 email_change: Show deactivated page if realm is now deactivated.
We also catch if the user is now deactivated.
2021-11-14 18:55:40 -08:00
Alex Vandiver bc5539d871 tornado: Move SIGTERM shutdown handler into a callback.
A SIGTERM can show up at any point in the ioloop, even in places which
are not prepared to handle it.  This results in the process ignoring
the `sys.exit` which the SIGTERM handler calls, with an uncaught
SystemExit exception:

```
2021-11-09 15:37:49.368 ERR  [tornado.application:9803] Uncaught exception
Traceback (most recent call last):
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/http1connection.py", line 238, in _read_message
    delegate.finish()
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/httpserver.py", line 314, in finish
    self.delegate.finish()
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/routing.py", line 251, in finish
    self.delegate.finish()
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/web.py", line 2097, in finish
    self.execute()
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/web.py", line 2130, in execute
    **self.path_kwargs)
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/gen.py", line 307, in wrapper
    yielded = next(result)
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/tornado/web.py", line 1510, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/zulip/deployments/2021-11-08-05-10-23/zerver/tornado/handlers.py", line 150, in get
    request = self.convert_tornado_request_to_django_request()
  File "/home/zulip/deployments/2021-11-08-05-10-23/zerver/tornado/handlers.py", line 113, in convert_tornado_request_to_django_request
    request = WSGIRequest(environ)
  File "/home/zulip/deployments/2021-11-08-05-10-23/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line 66, in __init__
    script_name = get_script_name(environ)
  File "/home/zulip/deployments/2021-11-08-05-10-23/zerver/tornado/event_queue.py", line 611, in <lambda>
    signal.signal(signal.SIGTERM, lambda signum, stack: sys.exit(1))
SystemExit: 1
```

Supervisor then terminates the process with a SIGKILL, which results
in dropping data held in the tornado process, as it does not dump its
queue.

The only command which is safe to run in the signal handler is
`ioloop.add_callback_from_signal`, which schedules the callback to run
during the course of the normal ioloop.  This callbacks does an
orderly shutdown of the server and the ioloop before exiting.
2021-11-12 09:57:23 -08:00
Alex Vandiver 6c14978cd1 zilencer: Truncate "remove" notifications from remote servers.
This is 4d055a6695, but for notifications which are received from
remote hosts.
2021-11-10 13:39:35 -08:00
akshatdalton f2c7e8e80f markdown: Ensure all markdown fixtures have unique names.
This commit ensures that all markdown fixtures have unique
test names by rewriting the names of some of them and adding
a test in `test_markdown.py`.

Earlier this was over-writing the value for same keys in
`load_markdown_tests` in `test_markdown.py`.
2021-11-10 12:36:47 -08:00
Alex Vandiver 6b6dcf6ce1 soft_deactivate: Handle multiple SUBSCRIPTION_DEACTIVATEDs.
Race conditions in stream unsubscription may lead to multiple
back-to-back SUBSCRIPTION_DEACTIVATED RealmAuditLog entries for the
same stream.  The current logic constructs duplicate UserMessage
entries for such, which then later fail to insert.

Keep a set of message-ids that have been prep'd to be inserted, so
that we don't duplicate them if there is a duplicated
SUBSCRIPTION_DEACTIVATED row.  This also renames the `message` local
variable, which otherwise overrode the `message` argument of a
different type.
2021-11-10 12:19:25 -08:00
Mateusz Mandera 17485e2f4d saml: Refactor get_issuing_idp to rely on class polymorphism. 2021-11-10 12:08:56 -08:00
YashRE42 778b7c0bdb links: Replace all references to Django docs to link to /3.2/ version.
Previously, our codebase contained links to various versions of the
Django docs, eg https://docs.djangoproject.com/en/1.8/ref/
request-response/#django.http.HttpRequest and https://
docs.djangoproject.com/en/2.2/ref/settings/#std:setting-SERVER_EMAIL
opening a link to a doc with an outdated Django version would show a
warning "This document is for an insecure version of Django that is no
longer supported. Please upgrade to a newer release!".

Most of these links are inside comments.

Following the replacement of these links in our docs, this commit uses
a search with the regex "docs.djangoproject.com/en/([0-9].[0-9]*)/"
and replaces all matches with "docs.djangoproject.com/en/3.2/".

All the new links in this commit have been generated by the above
replace and each link has then been manually checked to ensure that
(1) the page still exists and has not been moved to a new location
(and it has been found that no page has been moved like this), (2)
that the anchor that we're linking to has not been changed (and it has
been found that no anchor has been changed like this).

One comment where we mentioned a Django version in text before linking
to a page for that version has also been changed, the comment
mentioned the specific version when a change happened, and the history
is no longer relevant to us.
2021-11-08 18:06:16 -08:00
Alya Abbott e106caa68f portico: Add Lean case study. 2021-11-07 09:53:28 -08:00
Priyansh Garg 42f231c85c data_import: Ignore Rocket.Chat livechat streams/messages.
This resolves the issues reported in #20108, major chunk of which were
due to the incomplete support for importing the livechat streams/messages
in the tool. So, it's best not to import any livechat streams/messages for
now until a complete support for importing the same is developed.
2021-11-07 09:50:55 -08:00
Anders Kaseorg 8df6eccd42 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:34:13 -07:00
Alex Vandiver c8badbd858 reset_password: Show user-facing page on rate-limit. 2021-11-04 20:34:39 -07:00
Alex Vandiver d3ecbf96a8 rate_limit: Rate-limit password reset attempts by IP, as well. 2021-11-04 20:34:39 -07:00
Alex Vandiver e3ec2e398c rate_limit: Pass seconds to freedom to all RateLimited exceptions. 2021-11-04 20:34:39 -07:00
Tim Abbott 1cad29fc3a settings: Add rate limiting for email address changes.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-11-04 20:34:39 -07:00
Alex Vandiver f0532aecc8 tests: Stop adding an api_by_user rate limit always.
The decorator form is clearer by being more explicit; additionally,
the api_by_user rate-limit only currently used in one place, and makes
it difficult to test per-user rate-limits that are more specific.
2021-11-04 20:34:39 -07:00
Tim Abbott 01e2a495fc rate_limit: Fix missing IP rate limiting on confirmation.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-11-04 20:34:39 -07:00
Alex Vandiver 48ba2e7cc6 tests: Hitting a rate-limit on find accounts should not send emails. 2021-11-04 20:34:39 -07:00
Alex Vandiver 0cfb156545 rate_limit: Merge two IP rate limits domains that send emails.
Both `create_realm_by_ip` and `find_account_by_ip` send emails to
arbitrary email addresses, and as such can be used to spam users.
Lump their IP rate limits into the same bucket; most legitimate users
will likely not be using both of these endpoints at similar times.

The rate is set at 5 in 30 minutes, the more quickly-restrictive of
the two previous rates.
2021-11-04 20:34:39 -07:00
Alex Vandiver 5f0897e6f7 tests: Add a test that IP rate-limiting is by IP.
The existing test did no verify that the rate limit only applied to
127.0.0.1, and that other IPs were unaffected.  For safety, add an
explicit test of this.
2021-11-04 20:34:39 -07:00
Alex Vandiver 9495dad850 tests: Simplify by always clearing IP limits in rate_limit_rule.
The only use case of rate_limit_rule which does not clear the
RateLimitedIPAddr history is test_hit_ratelimits_as_remote_server,
which is not made any worse by clearing out the IP history for a
non-existent `api_by_remote_server` domain.
2021-11-04 20:34:39 -07:00
Alex Vandiver 260ccc9620 tests: Factor out common non-api rate-limit assert function.
The same `assert_func` is used in multiple places, for non-API
requests.  Factor it out and make it a flag to do_test_hit_ratelimits.
2021-11-04 20:34:39 -07:00
Alex Vandiver 272e78de8b tests: Split out test_find_account_rate_limiting_multiple.
This lets rate_limit_rule be used consistently as a decorator, and
improves how parallel the code is here, with other tests.
2021-11-04 20:34:39 -07:00
Alex Vandiver 328a28d772 tests: Correct the assert in the post-rate-limit test. 2021-11-04 20:34:39 -07:00
Aman Agrawal 0879e5e0e2 home: Delete prefers_web_public_view key after user is logged in.
Since `prefers_web_public_view` key in session is only
relevant to users without an account, this key should no longer
be present in the user's session object.

Fixes #19907
2021-11-03 16:52:51 -07:00
Priyansh Garg 17409a78be data_import: Fix a few KeyError bugs in Rocket.Chat import tool.
This commit fixes a few bugs in Rocket.Chat import tool as reported on CZO.

Link: https://chat.zulip.org/#narrow/stream/9-issues/topic/Rocketchat.20Import
2021-11-03 16:50:56 -07:00
Shlok Patel 893c9bc896 export: Remove `--delete-after-upload` flag in realm export.
For export realm following changes have been made:
- `./manage.py export --upload` would delete `.tar.gz` and unpacked dir
- `./manage.py export` would only delete `unpacked dir`

Besides, we have removed `--delete-after-upload` as we have set it as
the default.

Fixes #20081
2021-11-03 11:14:02 -07:00
Tim Abbott 4d055a6695 push_notifications: Truncate overly large remove events.
Fixes #19224.
2021-11-02 21:24:14 -07:00
Aman Agrawal 3e689ebae9 users: Allow spectators to view user avatars.
If realm is web_public, spectators can now view avatar of other
users.

There is a special exception we had to introduce in rest model to
allow `/avatar` type of urls for `anonymous` access, because they
don't have the /api/v1 prefix.

Fixes #19838.
2021-11-02 11:26:19 -07:00
Chris Bobbe cce27aa47d openapi: Say message_content_delete_limit_seconds won't be 0 anymore.
I think this is ensured by an assertion in
parse_message_content_delete_limit in zerver/lib/message.py:
  https://github.com/zulip/zulip/blob/b13bfa09c/zerver/lib/message.py#L1482
2021-11-01 17:17:25 -07:00
Priyansh Garg 0c2e4eec20 data_import: Import Rocket.Chat threads as separate topics. 2021-11-01 17:13:35 -07:00
Priyansh Garg 0db9b7287b data_import: Import Rocket.Chat messages from direct discussions.
This commit adds functionality to import messages from the
Discussions having direct channels as their parent. As we don't
have topics in the PMs, the messages are imported in interleaved
form in the imported direct channels/PMs.

This was completely unsupported earlier and would have resulted in
an error.
2021-11-01 17:09:11 -07:00
Priyansh Garg 5f1e246230 data_import: Import wildcard mention data from Rocket.Chat. 2021-11-01 17:06:15 -07:00
Priyansh Garg 26f16b9eec data_import: Separate logic for naming Rocket.Chat streams and topics. 2021-11-01 16:48:25 -07:00
Sahil Batra e6106cb334 invites: Update error message when max limit for the day is reached.
This commit updates the error message returned when the maximum
invite limit for the day. We update the error returned by API to
only mention that the limit is reached and add the suggestion
to use multi-use link or contact support in the message shown
in webapp.
2021-11-01 16:36:26 -07:00
Lauryn Menard 73710e1cf0 user_settings: Add option to disable escape key navigation to default view.
Add `escape_navigates_to_default_view` as a bool setting in
UserBaseSettings model and implement it as a checkbox that toggles
the hotkey implementation of escape to the default view in the
advanced user display settings.

With /help/ documentation edits from Alya Abbott.

Fixes #20043.
2021-10-29 18:15:30 -07:00
Sahil Batra 9a03a78e3d test_users: Use delivery_email instead of email to generate gravatar url.
We always use delivery_email to generate gravatar_url, but in
test_admin_api_hide_emails we were passing email to get_gravatar_url
and matched with the avatar_url field of the fetched user object.

The tests were passing because the email_address_is_realm_public
was using old realm object and thus email field was incorrectly
set to delivery_email even when email_address_visibility was set
to EMAIL_ADDRESS_VISIBILITY_ADMINS.

This commit fixes the test to pass delivery_email to get_gravatar_url.
2021-10-29 16:19:55 -07:00
Sahil Batra 84cd68a5aa realm: Create RealmUserDefault object for internal realm.
We create RealmUserDefault object for internal realm just
for consistency. The code in migration does so but it
was missed to add the code when creating new internal realm.
2021-10-29 16:19:55 -07:00
Suyash Vardhan Mathur d2c5067839 openapi: Document remove-attachment endpoint. 2021-10-29 15:01:50 -07:00
Tim Abbott f8f8ee7d0e api docs: Remove print statement from get_subscribers example.
This was accidentally added by me in
95606a7347.
2021-10-29 15:01:50 -07:00
Tim Abbott 5d5cf371d8 lint: Fix typo in test email address. 2021-10-27 16:49:30 -07:00
Mateusz Mandera 4105ccdb17 saml: Implement IdP-initated logout for Keycloak.
Fixes #13948.
2021-10-27 13:13:55 -07:00
Alex Vandiver 6a40c17ccf markdown: CSS-escape preview links.
This adds `soupsieve` as an explicit dependency, but intentionally
does not adjust the provision version, as it was already an indirect
dependency.
2021-10-26 18:17:23 -07:00
Alex Vandiver 52f74bbd9b markdown: Run URL preview links through camo.
Not proxying these requests through camo is a security concern.
Furthermore, on the desktop client, any embed image which is hosted on
a server with an expired or otherwise invalid certificate will trigger
a blocking modal window with no clear source and a confusing error
message; see zulip/zulip-desktop#1119.

Rewrite all `message_embed_image` URLs through camo, if it is enabled.
2021-10-26 18:17:23 -07:00
Shlok Patel 033e61a055 webhooks/github: Include discussion ID in messages.
We add discussion id and url in the comments and highlighted title to
the body of disscussion message to make it more meaningful and accessible.

Fixes #19938.
2021-10-26 17:52:57 -07:00
RISHABH SIDANA 41b37dab42
typo: Fix models.py grammatical error. 2021-10-22 13:44:26 -07:00
Anders Kaseorg 58920affd4 python: Remove re.UNICODE flag (redundant in Python 3).
https://docs.python.org/3/library/re.html#re.A

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-22 13:42:29 -07:00
Shlok Patel 0678e2610f webhooks: Add support for GitHub discussions messages.
We aim to use Zulip topics thoughtfully in displaying messages from
discussions, as well as linking to the discussion in every message so
that it's easy to view them.

Fixes #19938.
2021-10-22 13:40:05 -07:00
Alex Vandiver 9381a3bd45 linkifiers: Support URL percent-encoded bytes.
Supporting URL percent-encoded bytes is possible using `%%20`, but this
is not necessarily very understandable to end-users, even those that
understand percent encoding.

Allow `%20` in linkifier URL format strings, and transform them into
`%%20` in the pattern just before they are applied in markdown
translation.  Care must be taken here, such that already-escaped `%`s
are not escaped an extra time.

We do this before rendering, and not before storage, as
a simplification; the JS-side linkifier at present only understands
`%(foo)s` and thus needs no changes, and to avoid an un-escaping pass
before showing in the admin UI.
2021-10-22 13:00:20 -07:00
Alex Vandiver d6768814a1 linkifiers: Explicitly only check format strings after verifying as a URL.
This makes the errors less confusing -- they might otherwise be jammed
together in the frontend.
2021-10-22 13:00:20 -07:00
Alex Vandiver 8dd9b4e812 linkifiers: Loosen regex that validates URLs.
User-supplied custom realm filter has had some sort of regex-based
validation of the format URL since their introduction in
d7e1e4a2c0 -- and this has always been
in addition to the URLValidator.  The URLValidator is the one which
does the security-relevant work of validating that the schema is
reasonable, and that the overall shape of the URL is well-formed.  The
regex has served primarily to arbitrary limit the characters that can
appear in the URL, in the mistaken name of safety.

Adjust the regex, such that its only purpose is to verify that the
usages of `%` characters in the URL are reasonable, and leave the URL
validation to the URLValidator, which can do a far better job.  This
includes broadening the support to include `%%` as an escape
character; this is likely such a niche case as to be unnecessary, but
costs little.

Fixes #16013.
2021-10-22 13:00:20 -07:00
Anders Kaseorg 4839b7ed27 url_preview: Interpret og:image relative to full page URL.
og:image is supposed to be an absolute URL, but some sites incorrectly
provide a relative URL.  In this case, it makes more sense to
interpret it relative to the full page URL after redirects, rather
than relative to just the domain part of the page URL before
redirects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-21 12:20:37 -07:00
Mateusz Mandera 4dae22684a audit_log: Reserve a number for STREAM_REACTIVATED.
We don't yet have a do_reactivate_stream function, but we reserve a
number since:
1. It'll likely be added in the future.
2. For now, we can restore archived stream with some manual intervention
in the Django shell, and for that we'll want to create an appropriate
RealmAuditLog entry.
2021-10-21 11:13:55 -07:00
Jonny Tran ddf4053a73 composebox_typeahead: Remove redundant options from the `/` typeahead.
Removes the `/day` and `/night` options from the typeahead menu while
still allowing the commands to be used. Typing `/day` and `/night`
will now suggest `/light` and `/dark`, respectively. Also changes the
`Dark mode` and `Light mode` popups that appear after using the
corresponding command.

Fixes #18318.
2021-10-21 10:50:27 -07:00
Lauryn Menard 0d0b4eb9d4 documentation_api: Deduplicate ignored_parameters_unsupported.
Created a schema for the ignored_parameters_unsupported that is
returned by the /settings and /realm/user_settings_defaults endpoints
and removed the duplicated text in the api documentation.

Also cleaned up some small errors in the /realm/user_settings_default
definition and sidebar link /api/update-realm-user-settings-defaults.

Fixes #19674
2021-10-21 10:32:31 -07:00
Alex Vandiver 75f1070881 queue_processors: Disable timeouts with PushNotificationsWorker.
Since 3853285241, PushNotificationsWorker uses the aioapns library
to send Apple push notifications.  This introduces an asyncio event
loop into this worker process, which, if unlucky, can respond poorly
when a SIGALRM is introduced to it:

```
[asyncio] Task exception was never retrieved
future: <Task finished coro=<send_apple_push_notification.<locals>.attempt_send() done, defined at /path/to/zerver/lib/push_notifications.py:166> exception=WorkerTimeoutException(30, 1)>
Traceback (most recent call last):
  File "/path/to/zerver/lib/push_notifications.py", line 169, in attempt_send
    result = await apns_context.apns.send_notification(request)
  File "/path/to/zulip-py3-venv/lib/python3.6/site-packages/aioapns/client.py", line 57, in send_notification
    response = await self.pool.send_notification(request)
  File "/path/to/zulip-py3-venv/lib/python3.6/site-packages/aioapns/connection.py", line 407, in send_notification
    response = await connection.send_notification(request)
  File "/path/to/zulip-py3-venv/lib/python3.6/site-packages/aioapns/connection.py", line 189, in send_notification
    data = json.dumps(request.message, ensure_ascii=False).encode()
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/path/to/zerver/worker/queue_processors.py", line 353, in timer_expired
    raise WorkerTimeoutException(limit, len(events))
zerver.worker.queue_processors.WorkerTimeoutException: Timed out after 30 seconds processing 1 events
```

...which subsequently leads to the worker failing to make any progress
on the queue.

Remove the timeout on the worker.  This may result in failing to make
forward progress if Apple/Google take overly long handling requests,
but is likely preferable to failing to make forward progress if _one_
request takes too long and gets unlucky with when the signal comes
through.
2021-10-21 08:59:56 -07:00
Alex Vandiver ab985c0066 queue_processors: Add a comment clarifying that timeouts only happen when single-threaded. 2021-10-21 08:59:56 -07:00
RISHABH SIDANA 97b62f16c2
typo: Fix duplicated word in comment. 2021-10-20 12:53:50 -07:00
Anders Kaseorg d1c05467ec push_notifications: Fix RemoteZulipServer annotation to work at runtime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-20 00:48:55 -07:00
Alex Vandiver c0b557fb2e push_notifications: Move zilencer import inside ZILENCER_ENABLED check. 2021-10-20 06:16:10 +00:00
Alex Vandiver 111ee64e36 push_notifications: Pass down the remote server and user-id for logs.
This makes logging more consistent between FCM and APNs codepaths, and
makes clear which user-ids are for local users, and which are opaque
integers namespaced from some remote zulip server.
2021-10-19 22:04:24 -07:00
Alex Vandiver 5bcd3c01cb push_notifications: Add log line with user-id, UUID, and devices.
Being able to determine how many distinct users are getting push
notifications per remote host is useful, as is the distribution of
device counts.  This parallels the log line in
handle_push_notification for push notifications from local realms,
handled via the event queue.
2021-10-19 22:04:24 -07:00
Sahil Batra 2285338b11 tests: Use more selective query in user group tests.
We should use more selective query for UserGroupMembership
objects in tests for checking adding and removing members.
This is done by checking the membership counts for the
particular user group only.

This will help in keeping the tests more understandable
after we add members to the role-based system groups,
since that would create a lot of membership objects.
2021-10-19 17:02:10 -07:00
Sahil Batra 599b3cc58e test_user_groups: Make UserGroup queries more selective.
We make the UserGroup queries in user group creation and
deletion tests more selective by fitering the user groups
which belong to the realm and not the one included in
lear realm, etc.

This will help us to keep the tests more understandable
when the counts of UserGroup increases due to addition of
system groups. There is no need to consider system groups
of other realms in these tests.
2021-10-19 17:02:10 -07:00
Sahil Batra 8b2213a43e test_user_groups: Fix comment in test_user_group_create. 2021-10-19 17:02:10 -07:00
rht bb8504d925 lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
Eeshan Garg b325a4f1be realm: Rename plan type constants to be more descriptive.
It is confusing to have the plan type constants not be namespaced
by the thing they represent. We already have a namespacing
convention in place for constants, so we should use it for
Realm.plan_type as well.
2021-10-19 12:20:39 -07:00
Alex Vandiver cbbd4b128d push_notifications: Provide a hint when the server is not registered. 2021-10-19 12:17:30 -07:00
Gaurav Pandey 1c1a1e2cad api: Remove encoding of string in update_user endpoint.
* Remove unnecessary json_validator for full_name parameter.
* Update frontend to pass the right parameter.
* Update documentation and note the change.

Fixes #18409.
2021-10-18 19:03:43 -07:00
Alex Vandiver b02754adec html_diff: Handle empty differences between empty strings.
`rendered_content` in historical messages may be empty; examining the
history of them may thus require diff'ing two empty strings, which
itself produces an empty string.

Use `lxml.html.fragment_fromstring` to be able to successfully parse
these, rather than 500.

Part of #19559.
2021-10-18 18:27:40 -07:00
Mateusz Mandera f5a65846a8 scim: Override django-scim2 logic of exception handling in views.
As detailed in the comments, the default behavior is undesirable for us
because we can't really predict all possibilities of exceptions that may
be raised - and thus putting str(e) in the http response is potentially
insecure as it may leak some unexpected sensitive information that was
in the exception.

As a hypothetical example - KeyError resulting from some buggy
some_dict[secret_string] call would leak information. Though of course
we aim to never write code like that.
2021-10-17 21:33:03 -07:00
Mateusz Mandera 9b5a27813a scim: Test the /scim/v2/Users/.search endpoint.
This endpoint has some basic support in our and django-scim2
implementation and we should be testing it.

https://datatracker.ietf.org/doc/html/rfc7644#section-3.4.3
for details on what this endpoint is.
2021-10-17 21:33:03 -07:00
Vishnu KS aff52722a7 billing: Add PLUS as a plan_type to Realm. 2021-10-15 17:27:50 -07:00
Alya Abbott 60535adfc7 onboarding: Update Welcome Bot.
The most important thing is linking to the new user guide.
2021-10-15 12:59:20 -07:00
Sahil Batra 6be20a8c6d users: Allow admins to get subscription status in private stream.
We pass allow_realm_admin as True to access_stream_by_id for
`GET users/{user_id}/subscriptions{stream_id}` endpoint
because we want to allow non-subscribed admins to get
subscription status in private streams.

Fixes #19077.
2021-10-15 10:58:56 -07:00
Anders Kaseorg 3bd3173b1f avatar: Remove ?x=x kludge.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Anders Kaseorg 58771d6e66 avatar: Use append_url_query_string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Anders Kaseorg f42e191776 url_encoding: Use proper parsing for query string appending.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Anders Kaseorg cf7e8e3947 backends: Fix URL encoding in redirect_deactivated_user_to_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Mateusz Mandera 73a6f2a1a7 auth: Add support for using SCIM for account management. 2021-10-14 12:29:10 -07:00
Mateusz Mandera 7c0995b14b actions: Support acting_user=None in check_change_full_name.
do_change_full_name supports it already, so this is a trivial change.
2021-10-14 12:28:35 -07:00
Anders Kaseorg 1e5157b66c user_groups: Add a recursive group membership model.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-13 10:08:06 -07:00
Sahil Batra 6ac9386a29 models: Add related_name to UserGroup and UserGroupMembership fields.
This commit adds related_name parameter to UserGroup.direct_members
such that we can use direct_groups instead of the default
usergroupmembership_set for getting all the groups of which the
user is direct member.

This commit also sets related_name of UserGroupMembership.user_group
and UserGroupMembership.user_profile to "+" which means that we will
not be having backward relations for these. This change is correct
since we would need to use the recursive queries to get all the
groups of a user and all the members of a group after we add the
subgroups concept in next commit. This leads to us using direct_members
field of UserGroup instead of usergroupmembership_set in mention code,
but this will soon be replaced with the recursive query function to
include subgroup's members as well.

Extracted this commit from #19866.

Authored-by : Anders Kaseorg <anders@zulip.com>
2021-10-13 10:08:06 -07:00
Sahil Batra 6f31b959ac user_groups: Fix return type of get_user_group_direct_members.
This commit fixes return type of get_user_group_direct_members
function which returns a list of user profile ids and not UserProfile
objects.
2021-10-13 16:57:04 +05:30
Sahil Batra b65aadae4b user_groups: Make get_user_group_direct_members query efficient.
This commit makes the query in get_user_group_direct_members
efficient by directly fetching user-profile ids instead of
first fetching user profile object and then id.
2021-10-13 16:56:44 +05:30
Tim Abbott f6c95a9fc5 migrations: Add missing noop migration.
The previous commit apparently didn't have its migration attached properly.

Since this is just a reference to a ManyToManyField, this migration
doesn't actually modify the database, but it is needed for CI to pass.
2021-10-12 14:12:53 -07:00
Sahil Batra 2d39b14a38 user_groups: Rename members field of UserGroup to direct_members.
This commit renames members field of UserGroup to direct_members
for better readability because in the new permissions model, a
user group can be a sub-group of another group and thus technically
members of sub-group will also be members of that group.
This is a prep commit for new permissions model.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:53 -07:00
Sahil Batra e47662aa19 user_groups: Rename get_memberships_of_users.
This is a prep commit for new permissions model in
which a user group would be able to have a subgroup.

This commit renames get_memberships_of_users to
get_direct_memberships_of_users to specify that
the function is used only to fetch the direct
memberships and not memberships of subgroups of
the direct group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:47 -07:00
Sahil Batra 07e2ecf221 user_groups: Rename get_user_groups to get_direct_user_groups.
This is a prep commit for new permissions model in which a user
group would be able to have a subgroup.

This commit renames get_user_groups to get_direct_user_groups
to specify that the function is used only to fetch the direct
groups that user is part of and not subgroups of the direct
group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:32 -07:00
Sahil Batra 82af2d3db5 user_groups: Rename get_user_group_members to get_user_group_direct_members.
This is a prep commit for new permissions model in which a user group would
be able to have a subgroup.

This commit renames get_user_group_members to get_user_group_direct_members
to specify that the function is used only to fetch direct members of group
and excludes the subgroup's members.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:26 -07:00
Dinesh e2df0d171f user_settings: Add send_read_receipts setting.
This will be useful to let users enable/disable
sharing read receipts once we add that feature.

Note: Added "I've" to IGNORED_PHRASES in
tools/lib/capitalization.py to avoid capitalization
errors for the label text of this setting.
2021-10-07 17:46:05 -07:00
Dinesh 9cea88c1a5 typing_notifications: Make privacy settings functional.
Changes the view code to skip sending typing events
when these settings are disabled. Returns a json success
response with a msg.
2021-10-07 17:39:22 -07:00
Dinesh 6d2b8f5ca9 user_settings: Add settings to configure sending typing notifications.
Note: These are not functional in enabling/disabling sending of
typing notifications with this commit.

Refactored the privacy settings update to keep the code less
duplicated along with making the addition of new settings easier.
2021-10-07 17:39:21 -07:00
Dinesh 0b68f264c6 apply_event: Do not include new user settings in the top level.
From 430c5cb, in `fetch_initial_state_data`,
we only include legacy settings in the top level of
`state` and the newer ones are stored in `state['user_settings']`.
That should've had a corresponding change in apply_event().

Also, fixed a test related to this logic.
2021-10-07 16:39:51 -07:00
Dinesh 47da353366 Fix change_user_settings_test to consider sending only one event.
From 430c5cb, we do not send events for settings added after
introduction of `user_settings` event.

This test wasn't considering that and started failing on
adding new user_settings.
2021-10-07 16:39:51 -07:00
Eeshan Garg 29b354346b update_realm: Allow demo orgs to be converted to regular orgs.
This commit adds support to the `PATCH /realm` endpoint for
converting a demo organization to a regular organization.

This is a part of #19523.
2021-10-07 14:43:03 -07:00
Sahil Batra e575f83351 test_subs: Remove _test_user_settings_for_adding_streams.
This commit removes _test_user_settings_for_adding_streams
and its callers for testing public and private streams
because it uses excessive mocking and we also test the same
thing in _test_user_settings_for_creating_streams without
mocking, so this test doesn't add anything.
2021-10-07 14:22:10 -07:00
Eeshan Garg f25230c7d4 analytics/support: Add ability to edit realm org type. 2021-10-07 14:08:11 -07:00
rht 58b19761b8 slack import: Fix requests.get usage of get_slack_api_data.
We also rewrite the tests using the `responses` module to avoid the
problematic mocking that made this bug possible.

Fixes #19833.
2021-10-07 11:46:23 -07:00
rht d8e1409fe5 Slack import: Use Python ZipFile to unzip.
This should handle the case when non-ASCII Unicode folder names are
created on Windows.

Fixes #19899.
2021-10-07 09:24:19 -07:00
Alya Abbott 8045355eee user docs: Document public vs. private stream creation permissions. 2021-10-06 17:33:19 -07:00
Aman Agrawal 3b29c61409 auth: Add login buttons for web public access.
For users who are not logged in and for those who don't have
'prefers_web_public_view' set in session, we redirect them
to the default login page where they can choose to login
as spectator or authenticated user.
2021-10-05 16:30:40 -07:00
Sahil Batra ab0d852fd5 register: Add can_create_web_public_streams to register response. 2021-10-05 10:10:32 -07:00
Sahil Batra 2c2c19c4d4 streams: Restrict creating web public streams based on new setting.
This commit restricts creating web public streams based on the
recently added create_web_public_stream_policy setting.
2021-10-05 09:56:00 -07:00
Sahil Batra 3916181770 models: Add can_create_web_public_streams helper.
This commit adds can_create_web_public_streams helper
in models.py which will be used to validate whether
user is allowed to create a web-public stream or not.

This commit also adds the checks for Realm.POLICY_OWNERS_ONLY
in check_has_permission_policies.
2021-10-05 09:48:50 -07:00
Sahil Batra 5f950e3efd models: Add create_web_public_stream_policy setting.
This commit adds create_web_public_stream_policy
field to Realm table which controls the roles that
can create web-public streams and by default its
value is set to POLICY_OWNERS_ONLY.
2021-10-05 09:28:28 -07:00
Sahil Batra be0387b189 test_subs: Enforce invite_only argument to be named.
This commit enforces invite_only argument to be named
in _test_user_settings_for_creating_streams. This will
help in improving readability especially when we will
add is_web_public argument in further commits.
2021-10-05 09:12:56 -07:00
Sahil Batra a63385f47f tests: Check all cases in check_has_permission_policies.
This commit adds tests for POLICY_EVERYONE and POLICY_NOBODY
in check_has_permission_policies test. The original code
used these values but these were not covered in test.
2021-10-05 09:12:56 -07:00