Commit Graph

16214 Commits

Author SHA1 Message Date
Sahil Batra c3759814be streams: Allow changing can_remove_subscribers_group through API.
This commit adds API support to change can_remove_subscribers_group
setting for a stream.
2023-02-05 14:46:36 -08:00
Aman Agrawal 9965ad2ea3 registration: Track create organization page in GA.
This will help us track if users actually clicked on the
email confirmation link while creating a new organization.

Replaced all the `reder` calls in `accounts_register` with
`TemplateResponse` to comply with `add_google_analytics`
decorator.
2023-02-05 10:24:32 -08:00
Anders Kaseorg b91788b945 markdown: Replace deprecated UnescapePostprocessor.
See https://github.com/Python-Markdown/markdown/pull/1272.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:36:47 -08:00
Anders Kaseorg b08c81b312 ruff: Fix G201 Logging `.error(..., exc_info=True)` → `.exception(...)`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:36:20 -08:00
Anders Kaseorg 59eca10a43 ruff: Fix G004 Logging statement uses f-string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:36:20 -08:00
Anders Kaseorg da3cf5ea7a ruff: Fix RSE102 Unnecessary parentheses on raised exception.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:34:55 -08:00
Prakhar Pratyush 9997131df3 message_edit: Fix resolve topic notifications.
When 'resolve|unresolve' and 'move stream' actions occurs in
the same api call, 'This topic was marked as resolved|unresolved'
notification is not sent.

Both 'topic moved' and 'topic resolved' notification should be generated.

This commit updates the logic of when and where to send
'topic resolve|unresolve' notification. Unlike previous logic, notification
may be sent even in the case 'new_stream' is not None.

In general, 'topic resolved|unresolved' notification is sent to
'stream_being_edited'. In this particular case ('new_stream' is not None),
notification is sent to the 'new_stream' after check.

Test case is included.

Fixes: #22973
2023-02-03 18:20:39 -08:00
Anders Kaseorg 81a7c7502f requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-03 16:36:54 -08:00
Anders Kaseorg 5b7c4206d7 ruff: Fix SIM300 Yoda conditions are discouraged.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-03 16:36:54 -08:00
Anders Kaseorg b85cb1e700 exceptions: Accept lazy translation as JsonableError argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-03 16:36:54 -08:00
Alessandro Toppi ff89590558 auth: Add JWT-based user API key fetch.
This adds a new endpoint /jwt/fetch_api_key that accepts a JWT and can
be used to fetch API keys for a certain user. The target realm is
inferred from the request and the user email is part of the JWT.

A JSON containing an user API key, delivery email and (optionally)
raw user profile data is returned in response.
The profile data in the response is optional and can be retrieved by
setting the POST param "include_profile" to "true" (default=false).

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-02-03 15:23:35 -08:00
Tim Abbott 7c7ca61e9f auth: Remove now unnecessary return_data parameter. 2023-02-03 15:23:35 -08:00
Mateusz Mandera 75b44337a8 auth: Extract get_api_key_fetch_authenticate_failure. 2023-02-03 15:23:35 -08:00
Mateusz Mandera 3a1d974cee auth: Extract process_api_key_fetch_authenticate_result function.
This will be useful for re-use in the implementation of another JWT
endpoint in the upcoming commits.
2023-02-03 15:23:35 -08:00
Mateusz Mandera 100f4a7152 auth: Add @require_post to remote_user_jwt. 2023-02-03 15:23:35 -08:00
Mateusz Mandera d466da1064 auth: Use REQ for getting token value and rename param in jwt paths.
This makes us use REQ properly instead of fetching from request.POST
manually - also renaming the param to "token" which is more standard.
2023-02-03 15:23:35 -08:00
Mateusz Mandera c9c255b3a8 auth: Improve JsonableError in get_..._jwt_authentication_request. 2023-02-03 15:23:35 -08:00
Mateusz Mandera 6c638a1057 auth: Extract token-check logic of remote_user_jwt.
This will be useful for re-use for implementation of an endpoint for
obtaining the API by submitting a JWT in the next commits.

It's not a pure refactor, as it requires some tweaks to remote_user_jwt
behavior:
1. The expected format of the request is changed a bit. It used to
   expect "user" and "realm" keys, from which the intended email was
   just generated by joining with @. Now it just expects "email"
   straight-up. The prior design was a bt strange to begin with, so this
   might be an improvement actually.
2. In the case of the codepath of new user signup, this will no longer
   pre-populate the Full Name in the registration form with the value
   from the "user" key. This should be a very minor lost of
   functionality, because the "user" value was not going to be a proper
   Full Name anyway. This functionality can be restored in a future
   commit if desired.

This is an API change, but this endpoint is nearly unused as far as
we're aware.
2023-02-03 15:23:35 -08:00
Tim Abbott 8d27d14354 models: Improve documentation for FOLLOWED UserTopics. 2023-02-03 11:25:05 -08:00
Aman Agrawal c3f0d5898a index: Remove `Loading...` text from the app loading overlay.
The new loading spinner animation conveys the message well
that the app is loading and this text is no longer required.
2023-02-03 10:51:39 -08:00
Lauryn Menard dbacc00f0f api-docs: Move markdown files to top level directory.
- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
  markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
  hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.
2023-02-02 17:25:40 -08:00
Lauryn Menard fc54ffd778 documentation: Move check for `api-doc-template.md`.
Moves the check for calling the `api-doc-template.md` directly,
so that we don't return a 500 error from the server, to happen
earlier with other checks for returning a 404 / missing page.

Also adds a specific test to `zerver/tests/test_urls` for this
template.

Prep commit for moving API documentation directory to be a top
level directory.
2023-02-02 17:23:31 -08:00
Alya Abbott 27b703f617
emails: Improve followup_day1 (registration confirmation) email.
- Clean up the language.
- Add a prominent "Go to organization" button.
- Link to guides for new users and admins.
- Fix duplication bug in text email version.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-02-02 17:16:43 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Lauryn Menard b64d8e8a52 templates: Remove `base_path` argument from api_arguments_table_generator.
Removes `base_path` argument when making the markdown extension for
parameters in documentation for API endpoints.

This seems to have been originally included for API parameters that
were documented in JSON files, which is no longer in use. Now all
API endpoints in the documentation are documented in
`zerver/openapi/zulip.yaml`.
2023-02-01 13:12:53 -08:00
Lauryn Menard f38d5a6a26 templates: Remove `base_path` argument from api_return_values_generator.
Removes `base_path` argument when making the markdown extension for
return values in documentation for API endpoints.

This seems to have been a copy and paste error in commit d2ee99a2fd
when `zerver/lib/markdown/api_return_values_generator.py` was created.
2023-02-01 13:12:53 -08:00
Lauryn Menard 20fb9af974 help: Add redirect for `help/configure-default-view`.
Adds URL redirect that was missed in commit 73710e1cf0 when
`change-default-view.md` was renamed to `configure-default-view.md`.
2023-02-01 06:10:14 -08:00
Lalit 4ab824dc4c
emoji: Disallow `.` in custom emoji names.
Until now, custom emojis with "periods" in their name were allowed, even though
they don't really fit the pattern of how we name them, and in fact the Markdown
processor would not render such custom emoji. Fix this by just disallowing the
character.

Also update the error strings accordingly.

Note that this does not include a migration to eliminate any existing custom emoji with this 
character in their name.

Fixes #24066.
2023-01-31 17:28:33 -08:00
Alex Vandiver 92c8c17190 import: Add the UTF-8 flag on file entries in zipfiles from Slack.
Fixes: #22533.
2023-01-31 16:07:48 -08:00
N-Shar-ma 282bf680f2 emoji: Replace `built-in` with `default` in emoji overriding API error. 2023-01-31 15:52:33 -08:00
Tim Abbott 22712f634d fix_unreads: Remove dead duplicate build_topic_unmute_checker.
The only caller for this function, which is part of a very old
migration, was removed in b77da6b834.
2023-01-31 11:18:44 -08:00
Sahil Batra e14b1f03fa register: Refactor template for new account registration.
This commit refactors the template code for source-realm
select element to have same structure as other inputs
and select element in the page. Thus this change also
makes the styling of source-realm select element consistent
with other select element in the page.
2023-01-26 12:36:31 -08:00
Sahil Batra 851d68e0fc create_user: Use transaction.atomic decorator for do_create_user.
We change the do_create_user function to use transaction.atomic
decorator instead of using with block. Due to this change, all
send_event calls are made inside transaction.on_commit.

Some other changes -
- Remove transaction.atomic decorator from send_inital_realm_messages
since it is now called inside a transaction.
- Made changes in tests which tests message events and notifications
to make sure on_commit callbacks are executed.
2023-01-26 10:49:19 -08:00
Sahil Batra 6cc468f6d4 create_user: Use transaction.atomic as decorator with do_reactivate_user.
This commit changes the do_reactivate_user such that the complete function
is called inside an atomic transaction and events are called after the
transaction is commited using on_commit helper. This is a prep commit
for unsubscribing the bots of unaccessible private streams when reactivating
them.
2023-01-26 10:49:19 -08:00
Sahil Batra 5635881664 realm_export: Add transaction.atomic to export_realm. 2023-01-26 10:49:19 -08:00
David Rosa 8d1db6482f help center: Update relative gear menu macro for organization settings.
Renames "Manage organization" to "Organization settings" to reflect
changes in the previous commit.
2023-01-26 10:17:45 -08:00
Anders Kaseorg cb8c7f2a17 ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 4eda29bd86 ruff: Fix RUF005 Consider spread instead of concatenation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 7e3a681f80 ruff: Fix S108 Probable insecure usage of temporary file.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:14:56 -08:00
Lauryn Menard dedea23745
help-docs: Move help center documentation to top level directory.
These files are not Jinja2 templates, so there's no reason that they needed
to be inside `templates/zerver`. Moving them to the top level reflects their
importance and also makes it feel nicer to work on editing the help center content, 
without it being unnecessary buried deep in the codebase.
2023-01-25 14:08:29 -08:00
Alex Vandiver 23f4cde91c email_mirror: Ensure that attachments get space to be included.
The content of a message is truncated to `MAX_MESSAGE_LENGTH`, which
is 1000 characters.  Since the email gateway places attachments at the
very end of the extracted body, that means that they are the first
thing to get truncated off.

That is, if an incoming email message contains 1000 `a`s and an image
attachment, the link that attaches the attachment to the message will
get truncated off, leaving it dangling in the database.

Truncate the message body content separately from the attachment links
which are included at the end of the body.
2023-01-24 13:22:13 -08:00
Lauryn Menard e7cecc989f documentation: Clean up `MarkdownDirectoryView.get_context_data`.
Changes the check for whether the documentation page is a policy
center page to be the `self.policies_view` boolean instead of the
`path_template` value as it reads much more clearly.

Moves a comment in the code to be contextually relevant.
2023-01-24 13:04:04 -08:00
Lauryn Menard 8859da911b documentation: Rename `article_path` variable when getting context.
Because of the overlap with the `DocumentationArticle` dataclass
field `article_path`, we rename the `article_path` variable used
in `MarkdownDirectoryView.get_context_data` for the absolute path
to be `article_absolute_path`.
2023-01-24 13:04:04 -08:00
Lauryn Menard f7d5cd5690 documentation: Remove "not_index_page" from context.
In commit bbecd41, we added "not_index_page" to the context for
some documentation articles, but use of that context key/value was
removed when the help documentation was removed in commit 1cf7ee9.

Changes `not_index_page` to be a boolean value that's used to set
the page title, but is not then passed on as a context key/value.

Also removes an irrelevant comment about disabling "Back to home"
on the homepage.
2023-01-24 13:04:04 -08:00
Aman Agrawal 37431cf0b5 urls: Provide `email` as a GET parameter.
Since we want to use `accounts/new/send_confirm` to know how many
users actually register after visiting the register page, we
added it to Google Tag Manager, but GTM tracks every user
registration separately due <email> in the URL
making it harder to track.

To solve this, we want to pass <email> as a GET parameter which
can be easily filtered inside GTM using a RegEx and all the
registrations can be tracked as one.
2023-01-24 11:29:50 -08:00
Aman Agrawal a51bf96c70 accounts_send_confirm: Show email to which the link was sent.
This can be useful for the user in case user is worried if they
typed the correct email.
2023-01-24 11:29:50 -08:00
Prakhar Pratyush 1a400b21e7 notifications: Fix missed message email notifications of welcome bot.
A missed message email notification, where the message is the welcome
message sent by the welcome bot on account creation, get sent when
the user somehow not focuses the browser tab during account creation.

No missed message email or push notifications should be sent for the
messages generated by the welcome bot.

'internal_send_private_message' accepts a parameter
'disable_external_notifications' and is set to 'True' when the sender
is 'welcome bot'.

A check is introduced in `trivially_should_not_notify`, not to notify
if `disable_external_notifications` is true.

TestCases are updated to include the `disable_external_notifications`
check in the early (False) return patterns of `is_push_notifiable` and
`is_email_notifiable`.

One query reduced for both `test_create_user_with_multiple_streams`
and `test_register`.
Reason: When welcome bot sends message after user creation
`do_send_messages` calls `get_active_presence_idle_user_ids`,
`user_ids` in `get_active_presence_idle_user_ids` remains empty if
`disable_external_notifications` is true because `is_notifiable` returns
false.
`get_active_presence_idle_user_ids` calls `filter_presence_idle_user_ids`
and since the `user_ids` is empty, the query inside the function doesn't
get executed.

MissedMessageHookTest updated.

Fixes: #22884
2023-01-24 11:16:21 -08:00
Prakhar Pratyush b40bbd6ca8 message_send: Refactor internal_send_*, internal_prep_* & _internal_prep_*.
This commit makes all the parameters after 'content' in
'internal_send_*', 'internal_prep_*' and '_internal_prep_*'
a mandatory keyword argument to increase code readability.
2023-01-24 11:16:21 -08:00
Prakhar Pratyush 4595b5d132 notifications: Add separate function for `is_notifiable` trivial checks.
A separate function named `trivially_should_not_notify` is added which
extracts the common checks from `get_push_notification_trigger` and
`get_email_notification_trigger` which are users' notification settings
independent and thus don't depend on what type of notification (email/push)
it is.
2023-01-24 11:16:14 -08:00
Alex Vandiver 994806c505 migrations: Fix ownership, and re-attach, mis-owned email attachments.
608c787c52 fixed a bug where messages sent by the email gateway "as"
a user failed to properly attribute ownership of their attachments,
leaving the attachments orphaned and thus with nobody with permissions
to view them.

These orphaned attachments only remain longer than a few weeks if the
`delete_unclaimed_attachments` script has not been run reliably.
Since there is currently no shipped cron job for this, that is most
likely all deployments.

Add a migration to find such orphaned attachments, and re-attach them
to their original message.  While theoretically the attachments
could have been later referenced in other messages -- which would be
very difficult to find and determine if they had access to the
attachment -- we only fix the original message.

In order to make this somewhat performant, we assume that the Message
rows associated with an Attachment made by the email gateway happened
within 5 minutes, since they must have been made during one HTTP
request.

This is complicated by the message potentially having been deleted; in
this case, the Attachment is moved into ArchivedAttachment, so it can
relate to the ArchivedMessage.  The many-to-many
`zerver_archivedattachment_messages` relationship table cannot use its
own `id` sequence for the value, since the `id` is re-used when the
row is inserted into the `zerver_attachment_messages` table -- we
instead consume a value from the `id` sequence of the
`zerver_attachment_messages` table.
2023-01-24 10:49:46 -08:00