Commit Graph

15476 Commits

Author SHA1 Message Date
Mateusz Mandera a671ae9749 get_old_unclaimed_attachments: Add docstring explaining the logic. 2022-06-02 17:32:23 -07:00
Mateusz Mandera 09dc166b45 do_delete_old_unclaimed_attachments: Consider ArchivedAttachment rows.
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
2022-06-02 17:32:23 -07:00
Mateusz Mandera 5ff4754090 test_upload: Fix some URLs to uploaded files.
Using http://localhost:9991 is incorrect - e.g. messages sent with file
urls constructed trigger do_claim_attachments to be called with empty
list in potential_path_ids.

realm.host should be used in all these places, like in the other tests
in the file.
2022-06-02 17:32:23 -07:00
Aman Agrawal 085443c722 home: Get default_language info from cookie for spectator.
This allows us to set default_language as cookie in frontend which
can be used on server load to send the relevant
translation data to the client.
2022-06-01 17:08:00 -07:00
Adam Sah ba5cf331a2 testing: 100% coverage for zerver/tests/test_digest.py. 2022-06-01 16:09:13 -07:00
Anders Kaseorg 9ee636e920 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 16:01:54 -07:00
Lauryn Menard 2047ca8f16 api-docs: Add changelog entry for user/stream ID narrow options.
Adds a 2.1 release changelog entry for adding support for user
and stream IDs in search/narrow options. Also, adds a Changes
note in the narrow parameter in the OpenAPI `get-messages`
endpoint definition.

Both link to the api documentation for constructing a narrow,
where the 2.1 release update is already mentioned.

Fixes #9474.
2022-06-01 15:11:26 -07:00
Lauryn Menard 82c9c6f599 api-docs: Expand use of `SimpleSuccess` response schema.
Use `SimpleSuccess` response schema for all endpoints that were
already returning a success (200) response without any data beyond
the `response` and `msg` fields, which are standard for all
endpoint responses.

Prep commit for adding `ignored_parameters_unsupported` to
`json_success` responses.
2022-06-01 15:09:39 -07:00
Zixuan James Li 4c6f2ae7be typing: Add assertions for authentication.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li c572d9be5a typing: Add none-checks for db queries.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li 4a5043dd6e typing: Add none-checks for miscellaneous cases.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li e338ada66c typing: Add none-checks for Recipient objects.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li bb6a934c8d typing: Add appropriate none-checks for LOCAL_UPLOADS_DIR.
This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li d5517932cd typing: Use BaseDatabaseSchemaEditor in place of DatabaseSchemaEditor.
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 14:18:53 -07:00
Zixuan James Li 63e9ae8389 typing: Apply trivial fixes to adjust edge cases in typing.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.

This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 12:03:51 -07:00
Zixuan James Li c34ac1fcd4 typing: Access url via key "Location" instead of attribute "url".
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 11:59:47 -07:00
Sahil Batra 9273c314cf invites: Capitalize "ID" in the error raised for invalid stream ids.
This commit changes "id" to "ID" in the error message returned when
invalid stream IDs are passed to multiuse invite endpoint.
2022-05-27 17:06:03 -07:00
Sahil Batra 708204290e streams: Capitalize "ID" in invalid stream errors in API.
This commit changes the error message from "Invalid stream id"
to "Invalid stream ID" for cases where invalid stream IDs are
passed to API endpoints to make it consistent with other similar
error messages.
2022-05-27 17:06:03 -07:00
Zixuan James Li 44ecd66eae types: Better types for API fields.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:43:00 -07:00
Zixuan James Li f1ef27cb85 password_reset: Soft reactivate upon password reset.
Fixes #22066

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:30:34 -07:00
Zixuan James Li 3e95b59f2e soft_deactivation: Add a helper for queuing soft_reactivation.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:28:52 -07:00
Anders Kaseorg bb6bd900cd response: Replace response.asynchronous attribute with new class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-27 14:27:34 -07:00
Alex Vandiver 134977b590 email: Attempt to suppress email auto-replies.
This applies a commonly-used, though non-RFC, header which suppresses
auto-replies to the message.  There is a small chance that this will
result in bad filters thinking the messages *from Zulip* are
themselves auto-replies, but this seems a small risk.

Fixes: #13193.
2022-05-26 18:04:12 -07:00
Aman Agrawal 3991e89f19 models: Add missing database index for open communities directory.
Enable database indexing of `want_advertise_in_communities_directory`
field of `Realm` model.
2022-05-26 11:47:24 -07:00
Chris Bobbe abfbacbdf6
openapi: Clarify custom_profile_field.value can be null in update-user event.
This has always been true, but wasn't documented correctly.
2022-05-26 10:52:13 -07:00
Lauryn Menard aa5536fe30 api-docs: Update `/set-typing-status` endpoint documentation.
Adds Changes notes for feature level 58 where support was added
for stream messages for the `/set-typing-status` endpoint
parameters.

Updates formatting for references to the `type`
parameter in the descriptions of other endpoint parameters.

Improves readability of and updates links in the endpoint's main
description.
2022-05-24 16:02:30 -07:00
Mateusz Mandera 4e1befa209 sessions: Rename get_session_user to get_session_user_id.
The function returns a user id, not a user - we generally try to reflect
that in function names accurately.
2022-05-23 15:57:10 -07:00
Mateusz Mandera 66d8c2ae15 sessions: Don't filter by expire_date in delete_realm_user_sessions.
The other delete_... functions just do .all() - we should be consistent
one way or another.
2022-05-23 15:57:10 -07:00
Mateusz Mandera 003d086f2e sessions: Only fetch user ids in delete_realm_user_sessions.
No need to fetch entire UserProfiles, especially since we're dealing
with all UserProfiles in a Realm here, which would be quite large.
2022-05-23 15:57:09 -07:00
Tim Abbott 0275358da8 openapi: Improve documentation of new_email field in events. 2022-05-23 15:53:05 -07:00
Anders Kaseorg 8b1e7a8b3e github: Accept null for issue and pull request body.
https://github.com/octokit/webhooks/blob/v5.6.0/payload-schemas/api.github.com/common/issue.schema.json#L84
https://github.com/octokit/webhooks/blob/v5.6.0/payload-schemas/api.github.com/common/pull-request.schema.json#L78

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-23 15:52:24 -07:00
Lauryn Menard 7ceb5f6311 api-docs: Add changelog for `stream_id` param to `mute-topic`.
Adds a changelog 2.0 entry for adding support for `stream_id`
parameter to the `mute-topic` endpoint. Also, adds Changes note
to the endpoint parameter description, and reorders/clarifies
that at least one (and only one) stream parameter must be provided
by the client and that the `string_id` parameter is preferred.

Fixes #11136.
2022-05-23 15:48:06 -07:00
Sahil Batra ce34b585a5 streams: Add endpoint "GET /streams/{stream_id}" to get stream by id.
Fixes #22082.
2022-05-23 15:14:04 -07:00
Alex Vandiver 3d66dd9eeb fill_memcached_caches: Document possible arguments to --cache. 2022-05-22 14:52:38 -07:00
Lauryn Menard 2d85612ff0 api-docs: Add `create_web_public_stream_policy` to realm update.
Adds `create_web_public_stream_policy` to the `get-events` API
documentation for the `realm op:update` event.

Also, fixes changelog entries for feature levels 103 and 104,
which are related to the API documentation changes or fix an
error in references to the undocumented endpoint `PATCH /realm`.
2022-05-19 13:37:40 -07:00
Sahil Batra 95a2b580a0 streams: Save one DB query in bulk_remove_susbcriptions.
We remove one call to get_occupied_streams to get occupied
streams before unsubscribing because we already know which
streams can become vacant, i.e. the one from which users are
being unsubscribed, and we can directly use the list of streams
from which users are being unsubscribed and get vacant streams
by checking which of these streams are not in get_occupied_streams
called after unsubscribing users.
2022-05-17 15:09:15 -07:00
Alex Vandiver 21ec1f07d4 name_restrictions: Add your-org.zulipchat.com as a reserved name.
This is used as a placeholder in the mobile app.
2022-05-17 14:58:31 -07:00
Sahil Batra 6b82cbe0a6 user_groups: Rename existing_subgroups variable to existing_direct_subgroup_ids.
This commit renames existing_subgroups variable to existing_direct_subgroup_ids
in add_subgroups_to_group_backend and remove_subgroups_from_group_backend functions
for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra 8edf30caf2 user_groups: Rename subgroups parameter to direct_subgroup_ids.
This commit renames subgroups parameter of do_send_create_user_group_event
to direct_subgroup_ids for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra dfd7902c77 user_groups: Rename subgroups fields to direct_subgroup_ids.
This commit renames subgroups and subgroup_ids field sent in user
group objects to direct_subgroup_ids for better readability.
2022-05-17 14:51:45 -07:00
Adam Sah 04ccd8c6d8 testing: 100% code coverage for zerver/lib/outgoing_webhook.py. 2022-05-17 13:51:00 -07:00
Alex Vandiver 814841c9ec puppet: Remove typo'd cron job.
54b6a83412 fixed the typo introduced in 49ad188449, but that does
not clean up existing installs which had the file with the wrong name
already.

Remove the file with the typo'd name, so two jobs do not race, and fix
the typo in the comment.
2022-05-16 14:57:21 -07:00
Alex Vandiver 513fcb7bd5 Revert "settings: Add web-public streams beta subdomain list."
This reverts commit 20368a936c.  It is
no longer in beta, and this configuration is no longer needed.
2022-05-16 14:49:14 -07:00
Alex Vandiver a6d1c41827 email_mirror: Handle invalid `charset="..."` values. 2022-05-16 12:04:38 -07:00
Anders Kaseorg 0043c0b6b2 django: Use HttpRequest.headers.
Fixes #14769.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d98e3ecb0d tests: Don’t reuse HttpRequest objects.
Django caches some information on HttpRequest objects, including the
headers dict, under the assumption that requests won’t be reused.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d3c6ca8def development: Redirect rather than mutate in register_development_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg 110f7a379a beanstalk: Move %40 kludge into authenticated_rest_api_view.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Alex Vandiver 803982e872 message_flags: Short-circuit if no messages changed.
Omit sending an event, and updating the database, if there are no
matching messages.
2022-05-12 21:57:55 -07:00
Sahil Batra 0bfe973b65 streams: Mark messages as read only for unsubscribed streams.
Previously, we were marking messages of all the streams passed
to bulk_remove_subscriptions even if user was not subscribed
to some of them and those streams would ideally not have
any unread messages. This code was added in 766511e519.

This commit changes the code to only mark messages of actually
unsubscribed streams as read.
2022-05-12 18:53:45 -07:00
Anders Kaseorg 1e453633b0 gitlab: Remove function signature inspection nonsense.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg f05fa9937b gci: Fix get_body_based_on_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg 4c772989a1 front: Fix get_body_based_on_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg 6d1b68c61b freshdesk: Remove unsafe TicketDict class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:56:46 -07:00
Anders Kaseorg cce142c61a middleware: Fix URL encoding of next parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:51:51 -07:00
Tim Abbott 86137dc02f users: clarify confusing check for last owner.
Conceptually, we're clearly intending to check whether the user we're
mutating is the last realm owner. The preexisting code was safe
because we've already checked that the target user is an owner, and
thus if we're the last owner, we're the target user.
2022-05-10 14:09:57 -07:00
Somesh Ranjan 0b1f8c05e3 org_settings: Add backend support to change bot role.
This commit attempts to add the backend support by extending the
/json/bots/{bot_id}/ url support to accept the role field as a
parameter. This was previously already possible via
`/json/users/{user_id}`, so this change just simplifies client
implementation.
2022-05-10 14:09:57 -07:00
Christopher Chong 94457732c1 github, gitlab: Replace the usage of dict.get(key) with dict[key]. 2022-05-10 13:50:20 -07:00
Christopher Chong 27a498e1e7 gitlab: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 76b1a682b7 github: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 2f097c524e gitea, gogs: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 83178a474f gci: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 087d4dff9d front: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 3cd41be3d0 freshstatus: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong f349da218c freshping: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong ec1e958e90 freshdesk: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 6e0e175183 flock: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 5833c05fa6 errbit: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 66fd6b1e55 dialogflow: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 990a046fc7 delighted: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Lauryn Menard 0acf955265
openapi: Fix inconsistencies in OpenAPI error documentation.
Fixes a few small inconsistences / mistakes in the OpenAPI
documentation related to error documentation. Does not change
the rendered API documentation, which is likely why these were
not noticed sooner.
2022-05-06 15:07:08 -07:00
Alya Abbott 86b61d46eb portico: Add Recurse Center case study. 2022-05-05 16:43:00 -07:00
Tim Abbott 24f527cb59 home: Don't send /register response for spectators.
In very large communities, computing page_params can be quite
expensive. Because we've moved the homepage for communities with web
public streams enabled to be the Zulip app, and it's common for
automation to frequently poll the homepage of a Zulip organization,
we'd like to keep those homepages cheap (as the login pages are).

We address this by prototyping something we may end up wanting to do
anyway -- having the web application do a `POST /register` API call in
order to fetch most page_params, and merging those with the mostly
webapp configuration page_params that we leave in the / response for
convenience.

This exact implementation is messy in a few ways:

* We rely on the assumption that ui_init.initialize_everything happens
  before all code that needs to inspect the page_params properties we
  are fetching via /register. This is likely mostly true, but nothing
  in the implementation enforces it.
* The bundle of ~25 keys that are in page_params ideally would be
  considered individually, with some moved to the /register API
  response and perhaps others eliminated or namespaced inside a
  webapp_settings object.
* It's weird to have the spectators network sequence different that
  from logged-in users, and potentially a maintainability risk.
* We might be able to arrange that the initial `/` response be
  cacheable, now that we're no longer embedding our metadata inside
  it. We've made no effort to do that as of yet.

Despite those issues, this commit solves an immediate problem and will
give us helpful experience with a model closer to the one we'll want
in order to happily support a web client that can be run locally
against a production Zulip server's data.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2022-05-05 15:20:46 -07:00
Tim Abbott 2e86ea6540 events: Add support for spectator access to /register.
This is necessary for the mobile/terminal clients to build spectator
support down the line. We'll also be using it for the web application,
in an upcoming commit.
2022-05-05 15:20:44 -07:00
Tim Abbott 53518e8a24 events: Pass an explicit realm to do_events_register. 2022-05-05 15:17:07 -07:00
Tim Abbott d180f7ccfc right sidebar: Clean up rendered description logic.
Previously, we were masking the realm_description raw Markdown with
rendered Markdown, which was a type error.

When we switch to calling /register explicitly in a few commits, this
results in a bug, since the raw Markdown ends up taking priority.

Fix this by just using a different name for this different concept.
2022-05-05 15:17:07 -07:00
yogesh sirsat 08e8a21da0 custom_profile_fields: Clarify an external account error message.
This error message is for a very precise situation -- the pattern not
having the desired format. We should say that, rather than a generic
"Malformed".
2022-05-04 17:57:44 -07:00
NerdyLucifer 6a5d646739 settings (admin/org): Show error for same choices in "list of options".
Currently an user can create multiple options with same text/label in
the select/"list of options" custom profile field type.

Fix this issue by extending the validator to throw an error if there
are duplicate choices in the "list of options" in custom profile
field.

Tweaked by tabbott to use a simpler check.

Fixes: #21880
2022-05-04 17:55:28 -07:00
Adam Sah a9ce02348b testing: Improve backend code coverage for semaphore webhook. 2022-05-04 16:13:24 -07:00
Alex Vandiver 21e80e47bd message_edit: Filter UserMessage better, now that mentions are unneeded.
The previous commit removed the need for `render_incoming_message` to
take a list of mentioned users; simplify the callsites accordingly.
2022-05-04 14:45:18 -07:00
Alex Vandiver cd9c69cd12 message_send: Remove unnecessary user_ids argument.
cfcbf58cd1 rightly removed the use of `user_ids` in
`render_markdown`, which in turn makes it unnecessary in
`render_incoming_message`.

Remove the unnecessary parameter from `render_incoming_message`.
2022-05-04 14:45:18 -07:00
Alex Vandiver 74e9b086f9 embed_links: Check that the message still exists before proceeding. 2022-05-04 14:45:18 -07:00
Alex Vandiver de63000db6 embed_links: Take a lock on the message object while editing.
We leave the fetching of links outside of the lock, as they could take
seconds, which is an unreasonable amount of time to hold a lock on the
message row.  This may result in unnecessary work, in the case that
the message was since edited, but the unnecessary work is preferable
to blocking other work on the message row for the duration.
2022-05-04 14:45:18 -07:00
Sahil Batra 9a6886f630 users: Always pass delivery_email in user's own object.
This commit changes the code to always pass delivery_email
field in the user's own object in 'realm_users'.

This commit also fixes the events sent by notify_created_user.
In the "realm_user/add" event sent when creating the user,
the delivery_email field was set according to the access
for the created user itself as the created user was passed as
acting_user to format_user_row. But now since we have changed
the code to always allow the user themselves to have access
to the email, this bug was caught in tests and we fix the person
object in the event to have delivery_email field based on whether
the user receiving the event has access to email or not.
2022-05-04 12:52:43 -07:00
Sahil Batra aa98b39429 bulk_create: Copy realm-level defaults in bulk_create_users.
This commit adds code to copy the realm-level default of
settings while creating users through bulk_create_users.

We do not directly call 'copy_default_settings' as it
calls ".save()" but here we want to bulk_create the objects
for efficiency.

We also add the code to set realm-default of enter_sends as
True for the Zulip dev server as done in 754b547e8 and thus
we remove enter_sends argument from create_user_profile as
it is of no use now.
2022-05-04 12:52:43 -07:00
Sahil Batra 8ff0cec2ed users: Pass email_address_visibility as parameter to can_access_delivery_email.
This is a prep commit for adding user-level email visibility setting.
2022-05-04 12:52:43 -07:00
Lauryn Menard 44c9b788f9 settings: Add realm setting for Zulip communities directory.
Adds `want_advertise_in_communities_directory` to the realm model
to track organizations that give permission to be listed on such
a site / directory on zulip.com.

Adds a checkbox to the organization profile admin for
organizations to give permission to be advertised in the
Zulip communities directory.

Adds a help center article about the Zulip communities directory
and uses a shared intro documentation file to create sections in
the articles on creating an organization profile and moderating
open organizations.

Co-authored-by: Alya Abbott <alya@zulip.com>
2022-05-04 11:13:28 -07:00
Mateusz Mandera 163910b765 auth: Don't leak info about accounts in other realms in the login form.
The "wrong subdomain" error leaks information that we shouldn't reveal.
We want to return a generic failure message instead.
2022-05-03 17:06:51 -07:00
Mateusz Mandera 1faebecb7a auth: Change the logger for authing to the wrong subdomain situations.
It's better to use zulip.auth logger here instead of the root logger.
Also the level is adjusted to INFO, since WARNING is excessive.
2022-05-03 17:06:51 -07:00
Mateusz Mandera 71f6f2310c auth: Don't log email when user tried authing to the wrong subdomain.
We want to avoid logging this kind of potentially sensitive information.
Instead, it's more useful to log ids of the matching accounts on
different subdomains.
2022-05-03 17:06:51 -07:00
Anders Kaseorg f29553d809 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 10:10:06 -07:00
Adam Sah dbd03b5054 email_backends: Fix concurrent backend testing for generate_emails.
Previously, this command would reliably fail:

```
tools/test-backend --skip-provision-check --parallel=3
  zerver.tests.test_email_log.EmailLogTest.test_forward_address_details
  zerver.tests.test_email_log.EmailLogTest.test_generate_and_clear_email_log
  zerver.tests.test_example.TestDevelopmentEmailsLog
```

and now it reliably succeeds. :-)

After hours of fiddling/googling/hair-tearing, I found that
mocking-away Django Connection.send_messages() was the best:

- We're testing Zulip and not Django.
- Mocking at this lower level exercises more of our code.
- EmailLogBackEnd._do_send_messages() helper method added to simplify mocking.

Fixes #21925.
2022-05-03 08:45:51 -07:00
Anders Kaseorg 7acb642fa5 requirements: Upgrade to Tornado 6.
Fixes #8913.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 6fd1a558b7 runtornado: Switch to asyncio event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg c263bfdb41 queue: Use a thread-local Pika connection.
According to the documentation: “Pika does not have any notion of
threading in the code. If you want to use Pika with threading, make
sure you have a Pika connection per thread, created in that thread. It
is not safe to share one Pika connection across threads, with one
exception: you may call the connection method add_callback_threadsafe
from another thread to schedule a callback within an active pika
connection.”

https://pika.readthedocs.io/en/stable/faq.html

This also means that synchronous Django code running in Tornado will
use its own synchronous SimpleQueueClient rather than sharing the
asynchronous TornadoQueueClient, which is unfortunate but necessary as
they’re about to be on different threads.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg c9faefd50e cache: Instantiate only one BMemcached cache backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 5d69dafddb runtornado: Avoid deprecated IOLoop debugging methods.
IOLoop.set_blocking_log_threshold and IOLoop.handle_callback_exception
are removed in Tornado 6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg e4bf7066f3 tornado: Remove instrument_tornado_ioloop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg bded7180f7 tornado: Unfork tornado.autoreload.
We previously forked tornado.autoreload to work around a problem where
it would crash if you introduce a syntax error and not recover if you
fix it (https://github.com/tornadoweb/tornado/issues/2398).

A much more maintainable workaround for that issue, at least in
current Tornado, is to use tornado.autoreload as the main module.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Lauryn Menard a0d483a9a5 openapi: Fix reference link in `/register-queue` documentation.
Fixes a reference link in `realm_enable_spectator_access`
description in `/register-queue` endpoint documentation.
2022-04-29 08:35:25 -07:00
Alex Vandiver bcd179be0e import: Default the S3 metadata user of emoji to the first owner.
This matches the metadata that we store in the database, and means
that the S3 metadatata invariant of always having a `user_profile_id`
in the metadata.

This does not fix existing imports, which may still have missing
`user_profile_id`s.
2022-04-28 16:51:58 -07:00
Tim Abbott 627144b0c7 settings: Simplify setting organization default language.
The changes in the last few commits changed the semantics of the
organization default language to no longer be the primary source of
information for a user's language when creating a new account.

Here, we change the settings UI and /help/ documentation to reflect
this.
2022-04-28 15:03:26 -07:00
Sahil Batra 47c6100da9 onboarding: Override the language of welcome bot PMs with user setting. 2022-04-28 15:03:26 -07:00
Siddharth Asthana 9954db4b59 users: Use browser locale to set the default language of new user.
This commit reads the browser locale during user registration, and
sets it as default language of user if it is supported by Zulip.
Otherwise, it is set to realm's default language.
2022-04-28 15:03:26 -07:00
Siddharth Asthana 807c73ef17 i18n: Add function to get default language for new users.
It returns realm's default language if the browser's locale
is not available or is unsupported.
2022-04-28 15:03:26 -07:00
Sahil Batra 476a217087 i18n: Add get_browser_language_code function.
This commit adds get_browser_language_code function
which returns None if there is no Accept-language
header in the request or Accept-languge header contains
only unsupported languages or all languages (meaning
header having value of '*'). Otherwise it returns the
language with highest weight/quality-value.
2022-04-28 15:03:26 -07:00
Christopher Chong b6f6f6db8d slack_incoming: Process request payload directly in view.py.
slack_incoming webhook previously used has_request_variables to
extract payload from HttpRequest object first, before trying to
access HttpRequest.body again in view.py. This caused an error
when one sends a request without payload - it is forbidden to
read from request data stream twice.

Instead of relying on has_request_variables, this PR extracts
payload depending on content type in view.py directly to avoid
reading request data stream twice.

Fixes #19056.
2022-04-28 14:45:38 -07:00
Aman Agrawal 4e08c737ca home: For web public realms, skip login for spectators.
To provide a smoother experience of accessing a web public stream,
we don't ask user to login unless user directly requests a
`/login` URL.

Fixes #21690.
2022-04-28 12:34:29 -07:00
Tim Abbott bd2dc7358b help: Add redirect to new public access page. 2022-04-28 12:27:13 -07:00
Aman Agrawal 00fffd1681 help: Rename `web-public-streams` page to `public-access-option`. 2022-04-28 12:09:08 -07:00
Alex Vandiver 127108c7d1 workers: Log the exception if the export fails.
We previously just swallowed the exception entirely.
2022-04-28 11:52:47 -07:00
Anders Kaseorg 6331a314d4 Correctly hyphenate “non-”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Anders Kaseorg d58fece832 Correctly hyphenate “web-public”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Chris Bobbe 8ed605de13 openapi: Mention that `profile_data` will be missing for bots.
For why we put this in the schema, instead of at the places where
the schema is consumed, Tim says:

> The code to not include it for bots is present in the common code
> path for formatting user objects for the API, so it should apply
> to all places we have users in the API documentation.

Discussion:
  https://chat.zulip.org/#narrow/stream/412-api-documentation/topic/.60profile_data.60.20in.20.60.2Fregister.60.20response/near/1374737
2022-04-27 22:09:55 -07:00
Aman Agrawal 5ee4f71701 avatar: Add rate limit similar to attachments on medium avatars.
Followup on #20136
2022-04-27 16:51:18 -07:00
Zixuan James Li b4feb673f1 push_notifications: Soft reactivate mentioned users.
Fixes #19861

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Zixuan James Li a8fd9eb701 email_notifications: Soft reactivate mentioned users.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Zixuan James Li d8101de34d notifications: Move user group mentions helpers together.
This refactored `get_mentioned_user_group_name` from
`zerver/lib/email_notifications.py` to
`zerver/lib/notification_data.py` just after
`get_user_group_mentions_data` to indicate the logical
similarity between them.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Sahil Batra 0c7828aeab user_groups: Add get_user_group_direct_members function.
This commit adds get_user_group_direct_members function
which returns a QuerySet of UserProfile and is used in
is_user_in_group function.
2022-04-27 14:17:52 -07:00
Sahil Batra d9d27b459d user_groups: Rename get_user_group_direct_members function.
This commit renames get_user_group_direct_members function to
get_user_group_direct_member_ids as it returns a list of ids
and to avoid it being parallel to get_recursive_group_members,
which returns a QuerySet.
2022-04-27 14:17:52 -07:00
Sahil Batra 0f01e3d327 tests: Add checks for user groups in different realms. 2022-04-27 14:17:52 -07:00
Sahil Batra 4d319ff34c tests: Use high enough number for invalid user group and user ids.
We use 1111 for testing invalid user and user group id cases instead
of 25, since we may have a user group in 25 in test data in future.
2022-04-27 14:17:52 -07:00
Anders Kaseorg 7c247a11a9 test_openapi: Use Python 3.8 typing.{get_origin,get_args}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 098a514599 python: Use Python 3.8 shlex.join function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 4d903dcc94 i18n: Remove size limit on lru_cache for get_language_list.
Really just for consistency, since the size will be at most 1 anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 2afe82ba2e test_link_embed: Convert "".format to f-string.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg b572b18e70 test_digest: Modernize set literal syntax.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 1c228de049 models: Excise NULL from pointlessly nullable fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Anders Kaseorg fd16f97d6b python: Excise None from pointlessly nullable booleans.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Anders Kaseorg af30538a65 forms: Fix type of domain_override parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Mateusz Mandera ee0021de08 models: Change Stream.history_public_to_subscribers value to True.
The default of a Stream is to be public - having
history_public_to_subscribers default to False is inconsistent with
that. The defaults on the model should generally be consistent.
2022-04-27 12:08:01 -07:00
Mateusz Mandera 04fdf3e4d9 import_utils: Fix history_public_to_subscribers being set incorrectly.
history_public_to_subscribers wasn't explicitly set when creating
streams via build_stream, thus relying on the model's default of False.
This lead to public streams being created with that value set to False,
which doesn't make sense.

We can solve this by inferring the correct value based on invite_only in
the build_stream funtion itself - rather than needing to add a flag
argument to it.

This commit also includes a migration to fix public stream with the
wrong history_public_to_subscribers value.

Fixes #21784.
2022-04-27 12:08:01 -07:00
Anders Kaseorg a543dcc8e3 Remove Debian 10 support.
As a consequence:

• Bump minimum supported Python version to 3.8.
• Move Vagrant environment to Ubuntu 20.04, which has Python 3.8.
• Move CI frontend tests to Ubuntu 20.04.
• Move production build test to Ubuntu 20.04.
• Move 3.4 upgrade test to Ubuntu 20.04.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 16:32:02 -07:00
Lauryn Menard 1292338537 frontend: Add `org_type` to realm settings updates and events.
Adds a drop-down menu for updating the organization type in the
`organization_profile_admin` page. Implements front end for
this setting to work / update like other organization profile,
notification and permissions settings.

One special note about this dropdown is that the listed options
should change once an organization has successfully set a type
other than 'unspecified' in the database. To accomplish this
the initial settings overlay build checks the realm_org_type
value in the page_params to select the correct options list,
and when the dropdown value is reset, either for update events
or for discarding changes, the page_params value is again used
to check for whether the 'unspecified' value should be present
as an option in the dropdown menu.

Adds basic node test for the `server_events_dispatch`.
Also adds a new help center documentation article for this
organization setting that is linked to in the UI.

Fixes #21692.
2022-04-26 16:29:12 -07:00
Lauryn Menard d2207d4ad5 backend: Add `org_type` to realm settings updates and events.
`org_type` already exists as a field in the Realm model and is
used when organizations are created / updated in Zulip Cloud,
via the `/analytics/support` view.

Extends the `PATCH /realm` view to be able update `org_type` as
other realm / organization settings are updated, but using the
special log / action that was created for the analytics view.

Adds a field to the `realm op: update` / `realm op: update_dict`
events, which also means an event is now sent when and if the
`org_type` is updated via the analytics view. This is similar
to how updates to an organization's `plan_type` trigger events.

Adds `realm_org_type` as a realm setting fetched from the
`POST /register` endpoint.
2022-04-26 16:29:12 -07:00
Chris Bobbe 9df0f1433e openapi [nfc]: Give user / cross-realm bot properties a better ordering
Greg noticed, "Also [awkward] that `user_id` is in a random place in the
middle instead of at the top, and that `is_guest` is nowhere near
`is_admin` and `is_owner`."

Discussion:
  https://chat.zulip.org/#narrow/stream/412-api-documentation/topic/ordering.20of.20user.20properties/near/1371370
2022-04-26 13:36:07 -07:00
evykassirer e4c51488c0 populate_db: Add support for creating messages older than 5 days ago.
Fixes #21646.
2022-04-25 16:16:15 -07:00
Sahil Batra 37793434f7 user_groups: Add API endpoint to get subgroups of a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra 1b3c972d9b user_groups: Add API endpoint to get members of a user group.
This commit adds 'GET /user_groups/{user_group_id}/members'
endpoint to get members of a user group. "direct_member_only"
parameter can be passed as True to the endpoint to get only
direct members of the user group and not the members of
subgroup.
2022-04-25 10:24:03 -07:00
Sahil Batra 374d2a66df user_groups: Add endpoint to check whether a user is member of a group.
This commit adds 'GET /user_groups/{id}/members/{id}' endpoint to check
whether a user is member of a group.

This commit also adds for_read parameter to access_user_group_by_id,
which if passed as True will provide access to read user group even
if it a system group or if non-admin acting user is not part of the
group.
2022-04-25 10:24:03 -07:00
Sahil Batra b71067908a user_groups: Add is_user_in_group function.
This commits adds is_user_in_group function
which can be used to check whether a user
is part of a user group or not. It also
supports recursive parameter for including
the members of all the subgroups as well.
2022-04-25 10:24:03 -07:00
Sahil Batra 6f0a7656ac user_groups: Add API endpoint for updating subgroups of a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra b4a9311ef2 actions: Add function to add and remove subgroups from a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra da0b087962 user_groups: Add "subgroups" field to user group objects.
This commit also adds 'subgroups' field to the user_group present
in the event sent on creating a user group. We do not allow passing
the subgroups while creating a user group as of this commit, but added
the field in the event object to pass tests.
2022-04-25 10:24:03 -07:00
Sahil Batra 61365fbe21 invites: Use expiration time in minutes instead of days.
This commit changes the invite API to accept invitation
expiration time in minutes since we are going to add a
custom option in further commits which would allow a user
to set expiration time in minutes, hours and weeks as well.
2022-04-20 13:31:37 -07:00
Tim Abbott 1cf25d783c openapi: Fix Zulip version for feature level 125.
This fixes a bug introduced by
22a5d008c1 having been rebased past the
Zulip 5.0 release.
2022-04-20 12:41:45 -07:00
Lauryn Menard 2e9adad123 update_message: Use `rendering_only` flag for notifications logic.
When `update_message` events were updated to have a consistent
format for both normal message updates/edits and special
rendering preview updates, the logic used in the tornado event
queue processor to identify the special events for sending
notifications no longer applied.

Updates that logic to use the `rendering_only` flag (if present)
that was added to the `update_message` event format to identify
if the event processor should potentially send notifications to
users.

For upgrade compatibility, if `rendering_only` flag is not present,
uses previous event structure and checks for the absence of the
`user_id` property, which indicated the special rendering preview
updates.

Fixes #16022.
2022-04-20 08:24:12 -07:00
Raghav Luthra 22a5d008c1 user_settings: Add a setting to display names of users who reacted.
Added a setting to the bottom of Settings > Display settings > Theme section
to display the reacting users on a message when numnber of reactions are
small.

This is a preparatory commit for #20980.
2022-04-19 17:30:16 -07:00
Mateusz Mandera c8fee06521 sync_ldap_user_data: Show a proper error if no relevant users exist.
It doesn't make sense to run sync_ldap_user_data if user_profiles list
is empty. Otherwise this misleading exception gets raised:

```
raise Exception(
    "LDAP sync would have deactivated all users. This is most likely due "
    "to a misconfiguration of LDAP settings. Rolling back...\n"
    "Use the --force option if the mass deactivation is intended."
)
```

With some work by tabbott to manage the type of user_profiles and
provide a special error message for the empty server case.
2022-04-18 20:32:30 +02:00
Hannah Salameh 02305e5927 tests: Add test coverage for followup_email_delay. 2022-04-17 16:19:28 -07:00
Mateusz Mandera 6a6fd38d03 models: Reserve a RealmAuditLog.USER_DELETED_PRESERVING_MESSAGES number.
The relevant function is waiting to be merged in #21299 - but we have
already used it on Zulip Cloud, creating RealmAuditLog entries with the
number 107 and thus should reserve it before another PR takes
it for another purpose, creating confusion in the logs.
2022-04-17 16:17:58 -07:00
Tim Abbott b63757621a realm_audit_log: Set acting_user=None explicitly, with comments.
Based on an audit, this closes out the last core instances in which
acting_user was not being passed explicitly when creating
RealmAuditLog instances.

There are some outstanding issues in the billing system, which we plan
to extract as a separate issue.

Fixes #14808.
2022-04-15 15:55:21 -07:00
Tim Abbott f446560607 do_delete_user: Clean up acting_user logic.
This now uses our standard acting_user convention for functions called
only from management commands.
2022-04-15 15:55:21 -07:00
Alex Vandiver 9030d53acb test_link_embed: Remove unnecessary TEST_CACHES.
The only purpose of this seems to be to not have to reset the cache;
fae59502ab added it without any explanation for why it is necessary.

Remove it, and explicitly flush the cache in the one place where it is
necessary.
2022-04-15 14:48:12 -07:00
Alex Vandiver 56058f3316 caches: Remove unnecessary "in-memory" cache.
This cache was added in da33b72848 to serve as a replacement for the
durable database cache, in development; the previous commit has
switched that to be the non-durable memcached backend.

The special-case for "in-memory" in development is mostly-unnecessary
in contrast to memcached -- `./tools/run-dev.py` flushes memcached on
every startup.  This differs in behaviour slightly, in that if the
codepath is changed and `run-dev` restarts Django, the cache is not
cleared.  This seems an unlikely occurrence, however, and the code
cleanup from its removal is worth it.
2022-04-15 14:48:12 -07:00
Alex Vandiver 04ca2e92f7 caches: Cache link preview data in memcached, not in PostgreSQL.
The choice to cache these in the database dates back to c93f1d4eda,
with the comment added in da33b72848 while working around the
durability of the "database" cache in local development.

The values were stored in a durable cache, as they needed to be
ensured to persist between when they were inserted in
`get_link_embed_data` and when they were used in
`render_incoming_message` via `link_embed_data_from_cache`.

However, database accesses are not fast compared to memcached, and we
wish to avoid the overhead of the database connection from the
`embed_links` worker.  Specifically, making the connection may not be
thread-safe -- and in low-memory (and Docker) configurations, all
workers run as separate threads in a single process.  This can lead to
stalled database connections in `embed_links` workers, and failed
previews.

Since the previous commit made the durability of the cache no longer
necessary, this will have minimal effect; at worst, posting the same
URL twice, on either side of an upgrade, will result in two preview
fetches of it.
2022-04-15 14:48:12 -07:00
Alex Vandiver 351bdfaf78 preview: Use cache only as a non-durable cache, not an IPC.
The `get_link_embed_data` / `link_embed_data_from_cache` pair as
introduced in c93f1d4eda uses the cache
as a temporary store inside of the `embed_links` worker; this means
that it must be durable storage, or the worker will stall and re-fetch
the same links to preview them.

Switch to plumbing through the fetched URL embed data as an parameter
to the Markdown evaluation which uses them, rather than using the
cache as an intermediary.  This frees up the cache to be merely a
non-durable cache.

As a side-effect, this removes get_cache_with_key, and
link_embed_data_from_cache which was its only callsite.
2022-04-15 14:48:12 -07:00
Alex Vandiver 327ff9ea0f preview: Use a dataclass for the embed data.
This is significantly cleaner than passing around `Dict[str, Any]` all
of the time.
2022-04-15 14:48:12 -07:00
Alex Vandiver ede4a88b49 populate_db: Remove unnecessary pre-population of URL cache.
76deb30312 changed this to not just be the URL, but rather a
prefixed hash of the URL, but failed to update this location which
wrote to it.  This meant that this pre-population step was writing to
the wrong keys in the durable cache, and thus ineffective.

Then, da33b72848 switched the cache to be in-memory, making this
write to the wrong keys in an in-process memory store.  There is no
way to pre-fill this sort of cache, except at server start-up.

Finally, and most fundamentally, 8c0c9ca7a4 then disabled
`inline_url_embed_preview` by default, making the code entirely moot.

Remove the triply-unnecessary code.
2022-04-15 14:48:12 -07:00
Alex Vandiver aaa58a49db cache: Make the cache_name=None behaviour clearer.
`django.core.cache.cache` is equal to
`django.core.cache.caches["default"]`; the latter is more
understandable in context.
2022-04-15 14:48:12 -07:00
Alex Vandiver 661c333377 markdown: Use named parameters to add_a helper.
This has enough parameters that it benefits from making which is which
explicit.
2022-04-15 14:48:12 -07:00
Alex Vandiver 452a30305d markdown: Clarify url parameter of "add_a" helper. 2022-04-15 14:48:12 -07:00
Mateusz Mandera fcf82bf047 digest: Don't send emails to deactivated users, even if queued. 2022-04-15 14:32:55 -07:00
Mateusz Mandera 7a13836d26 test_digest: Fix typo in a comment. 2022-04-15 14:32:55 -07:00
Anders Kaseorg cc30ed8ec7 actions: Delete zerver.lib.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:38 -07:00
Anders Kaseorg 729019acdd actions: Move part into zerver.lib.test_classes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg e01faebd7e actions: Split out zerver.actions.create_realm.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 53f4a395bc actions: Split out zerver.actions.realm_domains.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 59f6b090c7 actions: Split out zerver.actions.realm_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 12de8d797e actions: Move part into zerver.forms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg eda000899b actions: Split out zerver.actions.message_edit.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg 5d1a5a3877 actions: Split out zerver.actions.muted_users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg ec174dfb47 actions: Split out zerver.actions.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg eb4e9fe1e7 actions: Split out zerver.actions.message_flags.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg e5500a2226 actions: Split out zerver.actions.reactions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg cbad5739ab actions: Split out zerver.actions.create_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 5fcbc412cf actions: Split out zerver.actions.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg ec6355389a actions: Split out zerver.actions.user_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg d7981dad62 actions: Split out zerver.actions.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg bbce879c81 actions: Split out zerver.actions.custom_profile_fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg f6a06ba6e3 actions: Move part into zerver.lib.bulk_create.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg c041b68578 actions: Move part into zerver.lib.message.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg 9dd7e34ab3 actions: Move part into zerver.lib.subscription_info.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg b7adfb02f6 actions: Split out zerver.actions.presence.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg ab04068294 actions: Move part into zerver.lib.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg e230ea2598 actions: Split out zerver.actions.uploads.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg a29f1b39da actions: Move part into zerver.lib.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg 6168c0110a actions: Split out zerver.actions.user_activity.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg df4849bb15 actions: Split out zerver.actions.user_topics.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg 385616f27f actions: Split out zerver.actions.realm_emoji.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 8fc5922ebd actions: Split out zerver.actions.realm_export.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 3d7aa98c45 actions: Split out zerver.actions.realm_icon.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 7f088f3403 actions: Split out zerver.actions.realm_logo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg ca8d374e21 actions: Split out zerver.actions.invites.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 241463e215 actions: Split out zerver.actions.alert_words.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 1ac7496855 actions: Split out zerver.actions.default_streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 12130da339 actions: Split out zerver.actions.hotspots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 975f5a3c2d actions: Split out zerver.actions.realm_linkifiers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg e887abcf41 actions: Split out zerver.actions.realm_playgrounds.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 3a135b04d9 actions: Split out zerver.actions.submessage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 62d3b5bfd5 actions: Split out zerver.actions.typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 372c10f5f3 actions: Split out zerver.actions.user_groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 90cae59ea6 actions: Split out zerver.actions.video_calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg c136eebb33 actions: Split out zerver.lib.recipient_users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 703186c339 actions: Split out zerver.lib.user_counts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 05195c02c1 actions: Split out zerver.lib.user_message.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:30 -07:00
Anders Kaseorg 7f00aa078e actions: Move part into zerver.models.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:29 -07:00
Anders Kaseorg 6a70f75587 actions: Split out zerver.lib.sounds.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 14:26:40 -07:00
Anders Kaseorg dd8b1aaba6 actions: Add zerver/actions directory.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 13:48:35 -07:00
Zixuan James Li 8e98bbae85 invite: Bookkeep UserProfile with PreregistrationUser.
Fixes #21267

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-14 12:47:51 -07:00
Zixuan James Li d6f5099285 invite: Unwrap revoke_preregistration_users.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-14 12:47:51 -07:00
Zixuan James Li 532f827a7f profile: Strengthen decorator types using ParamSpec.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-14 12:44:35 -07:00
Zixuan James Li e632a4ced2 decorator: Strengthen decorator types using ParamSpec.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-14 12:44:35 -07:00
Zixuan James Li 9d448e73d2 decorator: Remove cachify in favor of lru_cache.
`cachify` is essentially caching the return value of a function using only
the non-keyword-only arguments as the key.

The use case of the function in the backend can be sufficiently covered by
`functools.lru_cache` as an unbound cache. There is no signficant difference
apart from `cachify` overlooking keyword-only arguments, and
`functools.lru_cache` being conveniently typed.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-14 12:44:35 -07:00
Zixuan James Li f21746ba0b cache: Strength types of cache decorators with ParamSpec.
This demonstrates a way to resolve the long-standing issue
of typing higher-order identity functions without using
`cast` and in a type-safe manner for decorators in `cache.py`.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-14 12:44:35 -07:00
Mateusz Mandera 6ebab1685b scim: Temporarily stop running SCIM change operations atomically.
do_deactivate_user can't be run in an atomic block due to concerns
around revoking session in a transaction. See
62ba8e455d for more details.

Without the change in this commit, the process of deactivating a user
via SCIM is broken.
2022-04-13 16:01:51 -07:00
Lauryn Menard 3f9ccf3225 api_docs: Revise emoji documentation for statuses and reactions.
Adds and updates changelog documentation for
`POST /users/me/status` feature level 86 addition
of new emoji parameters.

Makes description text for emoji `reaction_type` consistent
throughout API documentation and also adds better description
of the `unicode_emoji` namespace.

Redirects emoji field links in `user_status` event to go to
the parameters in `/update-status` endpoint, which was not a
documented endpoint when the event documentation was created.
2022-04-12 16:50:49 -07:00
Alex Vandiver ee04f42897 send_email: Only warn if EMAIL_HOST_PASSWORD is unset, not "".
Some email hosts actually do want an empty password; since the default
is `None`, we should key on that, and not just being false-y.
2022-04-12 16:41:27 -07:00
Lauryn Menard 9f8022de5e help_docs: Add 'About Zulip' to relative help links.
Adds the 'About Zulip' gear menu option to the available
relative link patterns used in help center documentation.
2022-04-12 15:22:58 -07:00
Alex Vandiver a26d109e7a auth: Reset failed authentication attempts on password reset.
It's natural that someone might try a wrong password 5 times, and then
go through a successful password reset; forcing such users to wait
half an hour before typing in the password they just changed the
account to seems unnecessarily punitive.

Clear the rate-limit upon successful password change.
2022-04-11 16:25:32 -07:00
Lauryn Menard 2615eacca5 tests: Remove ignored parameters from push notifications tests.
Removes `token_kind` parameter being passed to
`remove_apns_device_token` and `remove_android_reg_id` code
paths / endpoints. Possibly missed in a refactor of this
function as the tests for adding these tokens do not pass
a `token_kind` parameter.

Removes `zulip_org_id` and `zulip_org_kay` from code testing
`deactivate_remote_server`. These parameters are passed when
a remote server is added, so possibly a copy and paste error
when these tests were written / last refactored.
2022-04-08 11:39:06 -07:00
Lauryn Menard 195a86f242 tests: Remove ignored parameters from custom profile field tests.
`update_realm_custom_profile_field` does not take `field_type`
as a parameter, so this removes it from any related tests.
Possibly these test parameters were missed in a refactor of this
endpoint / code.
2022-04-08 11:39:06 -07:00
Lauryn Menard c8d8e82118 tests: Remove ignored parameters from bots tests.
`service_interface` is not a parameter of `add_bot_backend`, but
`interface_type` is, and that has the same default value as what
was being provided by the test, so updated for the parameter name
change, which was possibly missed in a previous code refactor.
2022-04-08 11:39:06 -07:00
Lauryn Menard 260012a87e tests: Remove ignored parameters from stream groups tests.
`update_default_stream_group_info` was being passed `op` and
`group_name` in various tests, which are not implemented as
parameters for that endpoint / code path. So this removes those
from the existing tests. This is not a documented API endpoint,
so perhaps these were just overlooked when these tests were
written / last refactored.
2022-04-08 11:39:06 -07:00
Lauryn Menard 8154b4a9af middleware: Add `client` REQ parameter to `parse_client`.
If an API request specified a `client` parameter, we were
already prioritizing that value over parsing the UserAgent.

In order to have these parameters logged in the `RequestNotes`
as processed parameters instead of ignored parameters, we add
the `has_request_variables` decorator to `parse_client` and
then process the potential `client` parameter through the REQ
framework.

Co-authored by: Tim Abbott <tabbott@zulip.com>
2022-04-08 11:29:33 -07:00
Alex Vandiver e6e4b7b3ef timeout: Add test coverage. 2022-04-07 17:26:01 -07:00
Alex Vandiver 04159a674c timeout: Minor comment cleanups.
We remove the StackOverflow link because it is now so dated as to be
irrelevant -- it does not use `self.ident`, and cargo-cults the return
value of PyThreadState_SetAsyncExc.
2022-04-07 17:26:01 -07:00
Alex Vandiver 3af2c8d9a3 timeout: Warn if the thread did not exit.
As noted in the docstring for this function, the timeout is
best-effort only -- if the thread is blocked in a syscall, it will not
service the exception until it returns.  It can also choose to catch
and ignore the TimeoutExpired; in either case it will still be running
even after the `timeout()` function returns.

Raising a vare TimeoutExpired it still somewhat accurate, but obscures
that the backend thread may still be running along merrily.  Notice
such cases, and log a warning about them.
2022-04-07 17:26:01 -07:00
Alex Vandiver e714264756 timeout: Re-raise from where the TimeoutExpired hit the thread.
Having just thrown an exception into the thread, it is often useful to
know _what_ was the slow code that we interrupted.  Raising a bare
TimeoutExpired here obscures that information, as any `exc_info` will
end there.

Examine the thread for any exception information, and use that to
re-raise.  This exception information is not guaranteed to exist -- if
the thread didn't respond to the exception in time, or caught it, for
instance.
2022-04-07 17:26:01 -07:00
Alex Vandiver 85eeaf5f18 timeout: Remove cargo-culted and impossible-to-reach code block.
The quote in question originates in python/cpython@b8b6d0c2c6, when
the code was added.  However, the code stopped having that comment,
and was no longer able to return anything but 1 or 0, starting in
python/cpython@4643c2fda1 -- Python 2.5.

Remove the block.
2022-04-07 17:26:01 -07:00
Lauryn Menard 450245a455 api_docs: Clean uses of shared emoji schemas in OpenAPI.
Reformats two events (`reaction op: add` and `reaction op:remove`)
to follow the general format of events in the OpenAPI that are
returned by the `/get-events` endpoint.

Removes unneeded reference to `EmojiBase` schema in `user_status`
return value for the `/register-queue` endpoint. Also, clarifies
the text about the `user_status` object and fields being returned.
2022-04-07 12:44:28 -07:00
Lauryn Menard 460968179c api_docs: Add roles and permissions page to API documentation.
Adds a non-endpoint specific page to the API documentation about
organization-level roles and permissions for users in order to
highlight important and useful information for clients and API
users.

Also, adds links to new documentation page in related areas
of the API documentation.
2022-04-04 17:09:05 -07:00
Aman Agrawal dcdf071751 message_edit: Fix false sub/unsub bookend on using a `near` link.
We were not setting the `historical` flag correctly for
messages fetched via `json_fetch_raw_message` when used didn't
have any UserMessage.

Extended relevant tests to fetch check message flags too.
2022-04-04 11:51:12 -07:00
Lauryn Menard 0b03275329 message_send: Move `sender` parameter to REQ.
Instead of using request.POST to get any potential `sender`
parameters for `send_message_backend`, moves it to the REQ
framework parameters as `req_sender`.

Also, updates `create_mirrored_message_users` to take specific
parameters instead of an HTTP request parameter, which then
accessed parameters via request.POST. And updates existing tests
for those changes.
2022-04-04 11:47:52 -07:00
Alex Vandiver 35e27aef4a migrations: Remove the possibly-duplicated emoji re-uploading.
In 85e531e377, we duplicated this block
of migration code to fix a bug, but moving it (aka deleting the
original copy) is a cleaner solution.
2022-04-01 17:51:00 -07:00
Tim Abbott 85e531e377 migrations: Repeat part of migration 0376.
The blockquote explains the motivation for this change in detail.

Fixes #21608.
2022-04-01 15:20:43 -07:00
Alex Vandiver 2e50ead9d1 data_import: Fix bot email address de-duplication.
4815f6e28b tried to de-duplicate bot
email addresses, but instead caused duplicates to crash:

```
Traceback (most recent call last):
  File "./manage.py", line 157, in <module>
    execute_from_command_line(sys.argv)
  File "./manage.py", line 122, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/management/commands/convert_slack_data.py", line 59, in handle
    do_convert_data(path, output_dir, token, threads=num_threads)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 1320, in do_convert_data
    ) = slack_workspace_to_realm(
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 141, in slack_workspace_to_realm
    ) = users_to_zerver_userprofile(slack_data_dir, user_list, realm_id, int(NOW), domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 248, in users_to_zerver_userprofile
    email = get_user_email(user, domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 406, in get_user_email
    return SlackBotEmail.get_email(user["profile"], domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 85, in get_email
    email_prefix += cls.duplicate_email_count[email]
TypeError: can only concatenate str (not "int") to str
```

Fix the stringification, make it case-insensitive, append with a dash
for readability, and add tests for all of the above.
2022-03-31 11:10:18 -07:00
Lauryn Menard 0008a76703 tests: Remove ignored `stream_name` parameter from test.
Removes unnecessary `stream_name` parameter from
`test_stream_permission_changes_updates_updates_attachments`.
2022-03-30 11:30:31 -07:00
jai2201 20e6315705 streams: Fix empty description in stream change notifications.
Send 'No description' text when either the old or new stream
description is empty, to avoid odd looking notifications.
2022-03-29 01:39:55 -07:00
Tim Abbott c789097184 streams: Fix empty description handling in new stream notifications.
"No description." is conveniniently an existing translated string, in
addition to being the right content for this setting.
2022-03-29 01:39:55 -07:00
jai2201 52cf557d2b openapi: Allow empty value of stream's description.
This is important because otherwise tests that change a stream
description to the empty string don't work.
2022-03-29 01:39:55 -07:00
Alya Abbott f04fb51ecc help: Rewrite pages on inviting users and related permissions.
Fixes #21520.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2022-03-28 17:00:10 -07:00
Lauryn Menard cbfe2707f4 help_docs: Clarify relative link text for Subscribed streams tab. 2022-03-28 15:13:39 -07:00
Alex Vandiver ca506f71dc push_notifications: Increase severity of APNs ConnectionError.
This has only happened when our APNs certificate expired; logging at
the error level ensures that this shows up in Sentry.
2022-03-25 18:12:14 -07:00
Alya Abbott 2a1e08759b portico: Add Asciidoctor case study. 2022-03-25 17:51:15 -07:00
Lauryn Menard 8d242f3467 help_docs: Update edit history documentation for messages.
`disable-message-edit-history`: Remove text about EDITED label
and link to `view-a-messages-edit-history` instead.

`edit-or-delete-a-message`: Reformat 'EDITED' and '(deleted)'
to be bold instead of using backticks. Make link to view
edit history clearer.

Note that the text used in the `OpenGraphTest` in
`test_middleware.py` had to be updated for the changes to
`disable-message-edit-history`.
2022-03-25 17:03:53 -07:00
Anders Kaseorg 55882fb343 python: Use modern set comprehension syntax.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-25 10:45:12 -07:00
Alex Vandiver eae4643cb4 message_edit: Ignore duplicates when re-muting new topic name.
This avoids an error when a user has already muted the new topic name.
We do this by ignoring duplicates, rather than catching the
IntegrityError, because this edit happens in a transaction, and that
would abort the transaction.
2022-03-24 21:27:11 -07:00