Commit Graph

6890 Commits

Author SHA1 Message Date
Sahil Batra ea0b2286e0 settings: Add email_address_visbility user setting. 2023-02-10 17:35:49 -08:00
Sahil Batra 2396e6858f users: Always pass "delivery_email" field in user objects.
Previously, user objects contained delivery_email field
only when user had access to real email. Also, delivery_email
was not present if visibility setting is set to "everyone"
as email field was itself set to real email.

This commit changes the code to pass "delivery_email" field
always in the user objects with its value being "None" if
user does not have access to real email and real email otherwise.
The "delivery_email" field value is None for logged-out users.
For bots, the "delivery_email" is always set to real email
irrespective of email_address_visibility setting.

Also, since user has access to real email if visibility is set
to "everyone", "delivery_email" field is passed in that case
too.

There is no change in email field and it is same as before.

This commit also adds code to send event to update delivery_email
field when email_address_visibility setting changes to all the
users whose access to emails changes and also changes the code to
send event on changing delivery_email to users who have access
to email.
2023-02-10 17:35:49 -08:00
Anders Kaseorg 41aadc8c63 message_send: Convert RecipientInfoResult from TypedDict to dataclass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-10 15:51:22 -08:00
Abhijeet Prasad Bodas 7d02b20bdb user topics: Rename test file.
This makes the file name in line with the model name and other files,
for better discoverability.

This probably should have been a part of
48883f1580.
2023-02-10 15:39:57 -08:00
Abhijeet Prasad Bodas 9d1c131dc6 muted users: Make file naming consistent.
This makes the names of the relevant files consistant with the database
model name as well as the frontend JS files.
2023-02-10 15:39:57 -08:00
Anders Kaseorg 0a1904a6a7 markdown: Rewrite YouTube URL parser without regex spaghetti.
This also adds support for the new YouTube Shorts URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-09 22:34:51 -08:00
Sahil Batra 891f83601d message_edit: Use move_messages_between_streams_limit_seconds setting.
This commit adds time restriction on moving messages between streams
using the move_messages_between_streams_limit_seconds setting in the
backend. There is no time limit for admins and moderators.
2023-02-08 12:46:05 -08:00
Sahil Batra 2c4e076fef message_edit: Use move_messages_within_stream_limit_seconds setting.
We now use the newly added move_messages_within_stream_limit_seconds
setting to check for how long the user can edit the topic replacing
the previously used 3-day limit. As it was previously, there is no
time limit for admins and moderators.
2023-02-08 12:46:05 -08:00
Sahil Batra b919dfd489 realm: Add time limit setting for moving messages between streams.
This commit adds "move_messages_between_streams_limit_seconds"
setting which would be used to set a time limit to move messages
between streams.
2023-02-08 12:46:05 -08:00
Sahil Batra 73f0eae394 realm: Add time limit setting for moving message within stream.
This commit adds "move_messages_within_streams_limit_seconds"
setting which would be used to set a time limit to move messages
within stream.
2023-02-08 12:46:05 -08:00
Alex Vandiver e3c976fed9 missed_messages: Include messages from topic which differ by case.
In Zulip, message topics are case-insensitive but case-preserving.
The `get_context_for_message` function erroneously did a
case-sensitive search, and thus only messages whose topic matched
exactly were pulled in as context.

Make the missed-message pipeline aware that message topics are not
case-sensitive.  This means that, when collapsing adjacent messages,
we merge messages with topic headers which are "different"; create a
separate explicit "grouping" to know which to collapse.
2023-02-07 11:04:21 -08:00
Prakhar Pratyush 906ff9243a user_mutes: Rename 'muting.py' to 'user_mutes.py'.
Rename 'muting.py' to 'user_mutes.py' because it, now
, contains only user-mute related functions.

Includes minor refactoring needed after renaming the file.
2023-02-07 00:23:47 +05:30
Prakhar Pratyush 49577bbdcd user_topics: Move topic muting functions to user_topics.py.
This commit moves topic related stuff i.e. topic muting functions
to a separate file 'views/user_topics.py'.

'views/muting.py' contains functions related to user-mutes only.
2023-02-07 00:23:47 +05:30
Kartik Srivastava 48883f1580 tests: Rename test_muting_topics to test_topic_visibility_policies.
As we'll be adding support for more visibility policies for topics,
this renamed file will contain the necessary tests.
2023-02-07 00:23:47 +05:30
Sahil Batra 73f11853ec streams: Allow setting can_remove_subscribers_group_id while creating streams.
This commit adds API support to set can_remove_subscribers_group setting
when creating streams.
2023-02-05 14:46:36 -08:00
Sahil Batra c3759814be streams: Allow changing can_remove_subscribers_group through API.
This commit adds API support to change can_remove_subscribers_group
setting for a stream.
2023-02-05 14:46:36 -08:00
Anders Kaseorg da3cf5ea7a ruff: Fix RSE102 Unnecessary parentheses on raised exception.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:34:55 -08:00
Prakhar Pratyush 9997131df3 message_edit: Fix resolve topic notifications.
When 'resolve|unresolve' and 'move stream' actions occurs in
the same api call, 'This topic was marked as resolved|unresolved'
notification is not sent.

Both 'topic moved' and 'topic resolved' notification should be generated.

This commit updates the logic of when and where to send
'topic resolve|unresolve' notification. Unlike previous logic, notification
may be sent even in the case 'new_stream' is not None.

In general, 'topic resolved|unresolved' notification is sent to
'stream_being_edited'. In this particular case ('new_stream' is not None),
notification is sent to the 'new_stream' after check.

Test case is included.

Fixes: #22973
2023-02-03 18:20:39 -08:00
Alessandro Toppi ff89590558 auth: Add JWT-based user API key fetch.
This adds a new endpoint /jwt/fetch_api_key that accepts a JWT and can
be used to fetch API keys for a certain user. The target realm is
inferred from the request and the user email is part of the JWT.

A JSON containing an user API key, delivery email and (optionally)
raw user profile data is returned in response.
The profile data in the response is optional and can be retrieved by
setting the POST param "include_profile" to "true" (default=false).

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-02-03 15:23:35 -08:00
Mateusz Mandera 100f4a7152 auth: Add @require_post to remote_user_jwt. 2023-02-03 15:23:35 -08:00
Mateusz Mandera d466da1064 auth: Use REQ for getting token value and rename param in jwt paths.
This makes us use REQ properly instead of fetching from request.POST
manually - also renaming the param to "token" which is more standard.
2023-02-03 15:23:35 -08:00
Mateusz Mandera c9c255b3a8 auth: Improve JsonableError in get_..._jwt_authentication_request. 2023-02-03 15:23:35 -08:00
Mateusz Mandera 6c638a1057 auth: Extract token-check logic of remote_user_jwt.
This will be useful for re-use for implementation of an endpoint for
obtaining the API by submitting a JWT in the next commits.

It's not a pure refactor, as it requires some tweaks to remote_user_jwt
behavior:
1. The expected format of the request is changed a bit. It used to
   expect "user" and "realm" keys, from which the intended email was
   just generated by joining with @. Now it just expects "email"
   straight-up. The prior design was a bt strange to begin with, so this
   might be an improvement actually.
2. In the case of the codepath of new user signup, this will no longer
   pre-populate the Full Name in the registration form with the value
   from the "user" key. This should be a very minor lost of
   functionality, because the "user" value was not going to be a proper
   Full Name anyway. This functionality can be restored in a future
   commit if desired.

This is an API change, but this endpoint is nearly unused as far as
we're aware.
2023-02-03 15:23:35 -08:00
Aman Agrawal c3f0d5898a index: Remove `Loading...` text from the app loading overlay.
The new loading spinner animation conveys the message well
that the app is loading and this text is no longer required.
2023-02-03 10:51:39 -08:00
Lauryn Menard fc54ffd778 documentation: Move check for `api-doc-template.md`.
Moves the check for calling the `api-doc-template.md` directly,
so that we don't return a 500 error from the server, to happen
earlier with other checks for returning a 404 / missing page.

Also adds a specific test to `zerver/tests/test_urls` for this
template.

Prep commit for moving API documentation directory to be a top
level directory.
2023-02-02 17:23:31 -08:00
Alya Abbott 27b703f617
emails: Improve followup_day1 (registration confirmation) email.
- Clean up the language.
- Add a prominent "Go to organization" button.
- Link to guides for new users and admins.
- Fix duplication bug in text email version.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-02-02 17:16:43 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Lalit 4ab824dc4c
emoji: Disallow `.` in custom emoji names.
Until now, custom emojis with "periods" in their name were allowed, even though
they don't really fit the pattern of how we name them, and in fact the Markdown
processor would not render such custom emoji. Fix this by just disallowing the
character.

Also update the error strings accordingly.

Note that this does not include a migration to eliminate any existing custom emoji with this 
character in their name.

Fixes #24066.
2023-01-31 17:28:33 -08:00
Alex Vandiver 92c8c17190 import: Add the UTF-8 flag on file entries in zipfiles from Slack.
Fixes: #22533.
2023-01-31 16:07:48 -08:00
N-Shar-ma 282bf680f2 emoji: Replace `built-in` with `default` in emoji overriding API error. 2023-01-31 15:52:33 -08:00
Sahil Batra e14b1f03fa register: Refactor template for new account registration.
This commit refactors the template code for source-realm
select element to have same structure as other inputs
and select element in the page. Thus this change also
makes the styling of source-realm select element consistent
with other select element in the page.
2023-01-26 12:36:31 -08:00
Sahil Batra 851d68e0fc create_user: Use transaction.atomic decorator for do_create_user.
We change the do_create_user function to use transaction.atomic
decorator instead of using with block. Due to this change, all
send_event calls are made inside transaction.on_commit.

Some other changes -
- Remove transaction.atomic decorator from send_inital_realm_messages
since it is now called inside a transaction.
- Made changes in tests which tests message events and notifications
to make sure on_commit callbacks are executed.
2023-01-26 10:49:19 -08:00
Sahil Batra 6cc468f6d4 create_user: Use transaction.atomic as decorator with do_reactivate_user.
This commit changes the do_reactivate_user such that the complete function
is called inside an atomic transaction and events are called after the
transaction is commited using on_commit helper. This is a prep commit
for unsubscribing the bots of unaccessible private streams when reactivating
them.
2023-01-26 10:49:19 -08:00
Sahil Batra 5635881664 realm_export: Add transaction.atomic to export_realm. 2023-01-26 10:49:19 -08:00
Anders Kaseorg cb8c7f2a17 ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 4eda29bd86 ruff: Fix RUF005 Consider spread instead of concatenation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 7e3a681f80 ruff: Fix S108 Probable insecure usage of temporary file.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:14:56 -08:00
Lauryn Menard dedea23745
help-docs: Move help center documentation to top level directory.
These files are not Jinja2 templates, so there's no reason that they needed
to be inside `templates/zerver`. Moving them to the top level reflects their
importance and also makes it feel nicer to work on editing the help center content, 
without it being unnecessary buried deep in the codebase.
2023-01-25 14:08:29 -08:00
Alex Vandiver 23f4cde91c email_mirror: Ensure that attachments get space to be included.
The content of a message is truncated to `MAX_MESSAGE_LENGTH`, which
is 1000 characters.  Since the email gateway places attachments at the
very end of the extracted body, that means that they are the first
thing to get truncated off.

That is, if an incoming email message contains 1000 `a`s and an image
attachment, the link that attaches the attachment to the message will
get truncated off, leaving it dangling in the database.

Truncate the message body content separately from the attachment links
which are included at the end of the body.
2023-01-24 13:22:13 -08:00
Aman Agrawal 37431cf0b5 urls: Provide `email` as a GET parameter.
Since we want to use `accounts/new/send_confirm` to know how many
users actually register after visiting the register page, we
added it to Google Tag Manager, but GTM tracks every user
registration separately due <email> in the URL
making it harder to track.

To solve this, we want to pass <email> as a GET parameter which
can be easily filtered inside GTM using a RegEx and all the
registrations can be tracked as one.
2023-01-24 11:29:50 -08:00
Aman Agrawal a51bf96c70 accounts_send_confirm: Show email to which the link was sent.
This can be useful for the user in case user is worried if they
typed the correct email.
2023-01-24 11:29:50 -08:00
Prakhar Pratyush 1a400b21e7 notifications: Fix missed message email notifications of welcome bot.
A missed message email notification, where the message is the welcome
message sent by the welcome bot on account creation, get sent when
the user somehow not focuses the browser tab during account creation.

No missed message email or push notifications should be sent for the
messages generated by the welcome bot.

'internal_send_private_message' accepts a parameter
'disable_external_notifications' and is set to 'True' when the sender
is 'welcome bot'.

A check is introduced in `trivially_should_not_notify`, not to notify
if `disable_external_notifications` is true.

TestCases are updated to include the `disable_external_notifications`
check in the early (False) return patterns of `is_push_notifiable` and
`is_email_notifiable`.

One query reduced for both `test_create_user_with_multiple_streams`
and `test_register`.
Reason: When welcome bot sends message after user creation
`do_send_messages` calls `get_active_presence_idle_user_ids`,
`user_ids` in `get_active_presence_idle_user_ids` remains empty if
`disable_external_notifications` is true because `is_notifiable` returns
false.
`get_active_presence_idle_user_ids` calls `filter_presence_idle_user_ids`
and since the `user_ids` is empty, the query inside the function doesn't
get executed.

MissedMessageHookTest updated.

Fixes: #22884
2023-01-24 11:16:21 -08:00
Anders Kaseorg d3164016f5 ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg 25346bde98 ruff: Fix SIM118 Use `k in d` instead of `k in d.keys()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg ff1971f5ad ruff: Fix SIM105 Use `contextlib.suppress` instead of try-except-pass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Alex Vandiver 608c787c52 email_mirror: Create attachments as the message sender.
When the email mirror gateway is sending messages "as" a user (as
triggered by having access to the missed-message email address),
attachments were still created as the Email Gateway bot.  Since the
sender (the end-user) was not the owner of those attachments (the
gateway bot), nor were they referenced yet anywhere, this resulted in
the attachments being "orphaned" and not allowed to be accessed by
anyone -- despite the attachment links being embedded in the message.
This was accompanied by the error:

```
WARN [] User 12345 tried to share upload 123/3LkSA4OcoG6OpAknS2I0SFAQ/example.jpf in message 123456, but lacks permission
INFO [zerver.lib.email_mirror] Successfully processed email from user 12345 to example-stream
```

We solve this by creating attachment objects as the users the message
will be sent from.
2023-01-18 15:42:40 -08:00
Anders Kaseorg 8f7a7877fe python: Clean up janky URL matching code with urlsplit.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-18 17:25:46 -05:00
Trident Pancake c6ea673cc9 markdown: Update max inline preview from 10 to 24.
The max inline preview limit was previously increased to 10 by #20789.
However, as issue #23624 shows, it's still causing confusion for users
when they include more than 10 links.

Bump this limit up to 24, which is a multiple of the 4 image preview
per line logic.
2023-01-18 14:58:00 -05:00
Lauryn Menard ba443cac03 templates: Set robots noindex for attribution corporate template.
Overrides the default context `allow_search_engine_indexing` to
always be `False` for `templates/corporate/attribution.html` so
that it does not appear in Google / search engine indexes.

Updates test of documentation pages in `test_docs.py` to have an
option for corporate pages to set this value in the template and
verifies that the meta tag for robots noindex, nofollow is
always in the response.
2023-01-17 15:00:39 -08:00
Mateusz Mandera 89d1f1f385 messages: Eliminate redundant realm fetch in has_message_access.
Accessing .realm will cause a fetch query from the database if the
attribute hasn't been fetched already earlier in the codepath. That's
completely redundant if we're just comparing realms, and we should only
access .realm_id attribute. This seems to eliminate a query in some
codepaths, which is nice in this performance-sensitive function.
2023-01-10 15:27:55 -08:00