SHA1PasswordHasher will be removed in Django 5.1. MD5PasswordHasher
will remain for the purpose of speeding up tests.
Followup to commit ac5161f439 (#29620).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Imports (including conversions from Slack, Hipchat, etc) do not write
the `.original` of the file (nor the still versions, for animations),
only the thumbnailed (and possibly animated) version. Looking for
`.original` would thus fail, and overwrite the (working) emoji with
the "?" and disable it.
In the event that we do not find the `.original`, fall back to the
bytes from the non-`.original`, to handle these import cases.
This commit improves the assert statements to verify
that the table name is not "usermessage' instead of
verifying that table name doesn't include a substring
"usermessage".
This prep commit will help to avoid assertion error when
importing "onboardingusermessage" table.
Earlier, the export tool was logging a warning:
"??? NO DATA EXPORTED FOR TABLE zerver_onboardingusermessage!!!"
This bug was due to not configuring a Config object for
'OnboardingUserMessage' in 'get_realm_config()'.
This commit fixes the bug to export the table properly.
This is needed for the command to work in docker-zulip, where the zulip
user may not have write permissions to the parent directory of the
secrets file. By default, crudini creates a temporary file, which makes
the command fail under those conditions.
With --inplace, the secret file gets written to directly without
creation of temporary files.
Adds server and web app support for processing the new `with`
search operator.
Fixes part of #21505.
Co-authored-by: roanster007 <rohan.gudimetla07@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
bulk fetch query of UserPfrofile against which
user_ids are validated, instead of looping
over user_ids and fetchingeach UserPfrofile resulting
in O(n) queries.
We use a truncated SHA256 of the id and a server-side secret to make
emoji have non-guessable filenames, while also making collisions
unlikely.
We also adjust the Slack import to use the same SHA-based name,
instead of taking the same name as it had in Slack.
We thumbnail and serve emoji with the same format as they were
uploaded. However, we preserved the original extension, which might
mismatch with the provided content-type.
Limit the content-type to a subset which is both (a) an image format
we can thumbnail, and (b) a media format which is widely-enough
supported that we are willing to provide it to all browsers. This
prevents uploading a `.tiff` emoji, for instance.
Based on this limited content-type, we then reverse to find the
reasonable extension to use when storing it. This is particularly
important because the local file storage uses the file extension to
choose what content-type to re-serve the emoji as.
This does nothing for existing emoji, which may have odd or missing
file extensions.
Aside from updating the doc structure, this commit
also rephrases step 4 to make it read better. The
list of supported events in step 5 is also removed.
Part of #29592.
We use the already existing server level setting to only allow
settings to be set to system groups, not a named user defined
group as well, in production. But we allow to settings to be set
to any named or anonymous user group in tests and development server.
"can_mention_group" setting can be set to user defined groups
because some of the realms already do that in production.
The existing server level setting is also renamed to make it clear
that both user defined groups and anonymous groups are not allowed
if that setting is set to False.
This commit also changes the error message to be consistent for the
case when a setting cannot be set to user defined groups as per
server level and setting and when a particular setting cannot be set
to user defined groups due to the configuration of that particular
setting. For this we add a new class SystemGroupRequiredError in
exceptions.py so that we need not re-write the error message in
multiple places.
Besides "RE:" and "FWD:", "AW:" (from German "Antwort" for answer) is
another common prefix in email subjects. Let email_mirror automatically
remove it as well.
When using the sub-expression purely for filtering, and not for
accessing the value in the resultset, .alias() is potentially faster
since it does not pull the value in as well.
This commit adds an 'transaction.atomic' decorator to the
'add_subscriptions_backend' view thus making the db operations
within the view atomic and helps to avoid race between events sent.
In tests where we make POST requests to this view, we have
wrapped the API calls with a transaction.atomic() context
manager. It helps us with NOT rolling back the entire test
transaction due to error responses.
In 'test_subs' we were making POST request to add
subscrption using 'self.api_post'.
This commit updates the code to use the test helper
function 'common_subscribe_to_streams' instead.
This prep commit will also help us to avoid adding
'transaction.atomic' context manager to these API calls
individually in the case of error response in the next commit.
Previously the bot sent bot commands whenever an undefined message
was sent by the user. This commit intends to fix the problem so that
the bot will only respond to the first message it does not understand
and not reply to any future undefined messages.
Fixes part of #30049.
The reply given by welcome bot when it does not understand a message,
sent by the user is changed, so that the bot is more clear to the user.
Fixes part of #30049.
The previous message that welcome bot sent with
the bot commands has been removed in order to be more
clear and concise to the user. Previously weclome bot
also sent bot commands in intial message, this commit
intends to remove those commands.
Fixes part of #30049.
migrate the following endpoints from @has_request_variables
to @typed_endpoint :
- upgrade()
- remote_realm_upgrade()
- upgrade_page()
- remote_server_upgrade()
Updates the instructions for `{settings_tab|invitations}` to work
like the deactivated users link, and updates the one article that
currently uses it: `help/invite-new-users.md`.
Using an if block when {!event-filtering-additional-feature.md!}
is used implied that we plan to have this in all webhook doc .md
files.
But, it actually makes sense to only use this macro when the
webhook integration actually supports event filtering.
In a2ef1d7e93, we made changes so
that when you send a message, your view jumps to the conversation
where you sent it.
For some users it was an improvement, few reported that it
disrupts their workflows.
This prep commit adds a setting which will be used to allow users
to decide whether to automatically go to conversation where they
sent a message.
This commit updates the Welcome Bot's initial
direct message content.
We inform about the tracked onboarding messages
via direct message only if it exists.
Fixes#30051.
This commit renames the "Huddle" Django model class to
"DirectMessageGroup", while maintaining the same table --
"zerver_huddle".
Fixes part of #28640.
Hash the salt, user-id, and now avatar version into the filename.
This allows the URL contents to be immutable, and thus to be marked as
immutable and cacheable. Since avatars are served unauthenticated,
hashing with a server-side salt makes the current and past avatars not
enumerable.
This requires plumbing the current (or future) avatar version through
various parts of the upload process.
Since this already requires a full migration of current avatars, also
take the opportunity to fix the missing `.png` on S3 uploads (#12852).
We switch from SHA-1 to SHA-256, but truncate it such that avatar URL
data does not substantially increase in size.
Fixes: #12852.
This commit fixes the bug where the "topic unresolved" notification
is wrongly triggered when moving a message between a resolved and
unresolved topic, except for when the topics have the same name.
To resolve this issue, the commit ensures that resolved/unresolved
notifications are not sent if a message has been moved to a new
topic. This is achieved by comparing the names of the old and new
topics without considering the "resolved prefix".
The commit also accounts for the scenario where `new_topic_name`
has been truncated, indicating that it was resolved and the name
had to change to accommodate the "resolved prefix".
This solution does not try to specially handle the possible case that
a stream has two topics with the same name, even if one is resolved
and another unresolved.
Fixes#29007.
This commit updates the code so that only the moved notification is
triggered when moving a message between a resolved and unresolved
topic in different streams or when moving a topic itself.
This change takes place even when both stream change and resolve
or unresolve a topic takes place in the same API request, as we
now consider it only a move operation.
This fixes a case where a message is moved between topics that
have the same name, but one resolved and another unresolved and
in different streams. Previously a resolved or unresolved
notification would also be sent. Now, this will not happen,
ensuring only the move operation is notified.
Fixes part of #29007.
Previously, when a referrer's invitation to Zulip was accepted,
they got a notification from notification-bot indicating
their invitation has been accepted.
This commit adds an option for referrer to decide
whether he wants to receive the direct notification
from the notification-bot.
Fixes: #20398
Reformatted Grafana doc into 3 part sections. One for
creating the Zulip bot and webhook URL. One for setting
up Grafana version 8.3 and above and the last one for
setting up Grafana version 8.2 and below.
Part of #29592.
Co-authored-by: Lauryn Menard <lauryn@zulip.com>
To improve onboarding experience following onboarding
messages are marked as starred:
* First message in each onboarding topic.
* Initial DM sent by Welcome bot
Note: The onboarding topic messages needs to be tracked
in 'OnboardingUserMessage' model to get starred.
Fixes#29298.
We give the user some messages in their feed, so that they can
learn how to use the home view in a realistic way.
For realms having older onboarding messages, we mark the very
most recent messages as unread.
This commit updates the logic to ONLY mark the tracked onboarding
messages (if present) i.e. messages tracked in 'OnboardingUserMessage'
as unread.
Fixes part of #29298.
This prep commit reorders code blocks in 'do_create_user'
to call the function 'send_initial_realm_messages' before
'process_new_human_user'.
This is required because for the first user (who creates realm)
we need to send the onboarding messages before adding message
history (performed by process_new_human_user) otherwise messages
won't be starred as required.
This commit also takes care of NOT marking the onboarding
messages as 'historical' for the realm creator.
This commit updates the 'send_initial_realm_messages' function
to create OnboardingUserMessage rows corresponding to the
initial realm messages we send.
It helps to distinguish new onboarding messages from the
onboarding messages in the old realms.
This prep commit adds a new OnboardingUserMessage model
that will be used to mark the new onboarding messages
for new users as unread and the first message of each
onboarding topic as starred.
This table won't include the old onboarding messages.
This commit adds the missing historical flag to recent
messages added in a new user's feed.
Reason:
* User didn't receive when they were sent, so semantically
the 'historical' flag should be present.
* It helps to avoid the "You (un)subscribed to.." bookend
while reading older messages.
The bookend appears whenever the historical flag flips
between adjacent messages. Earlier, the bookend was visible
between 'recent messages' and 'older messages'. This makes
sure that the bookend is visible only at the moment new
message is sent by the user after account creation.
This commit renames the constants:
MAX_NUM_ONBOARDING_MESSAGES to MAX_NUM_RECENT_MESSAGES
MAX_NUM_ONBOARDING_UNREAD_MESSAGES to MAX_NUM_RECENT_UNREAD_MESSAGES
ONBOARDING_RECENT_TIMEDELTA to RECENT_MESSAGES_TIMEDELTA
The term 'onboarding' is preffered to be used for the
new messages sent during realm creation or new user creation.
These constants are related to already present recent messages.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.
It also renames variables and methods of type -
"huddle_message" to "group_direct_message".
This is a part of #28640
Due to recent refactoring in 9fb03cb2c7, a user could not
upload avatar if the server uses local upload backend and there
was already an avatar file for that user.
This commit fixes it to just check if there exists a file only
when importing and not when the user is actually trying to
change the avatar.
Fixes#30676.
Revises doc to have three instructions sections. One for creating
the Zulip bot and webhook URL. One for adding notifications for
new Freshdesk tickets. And one for getting notifications for
existing Freshdesk tickets.
Part of #29592.
As a follow up for f49a11c810, this
commit standardizes the naming of the day and night themes to light
and dark, respectively in the backend. This makes the backend
consistent with the naming used in the frontend and UI.
This also solves a regression introduced in
f49a11c810, where the frontend was sending
"/light" and "/dark" commands to the backend, but the backend was
expecting "/day" and "/night" commands.
Earlier, a one-time 'visibility_policy_banner' was displayed to
existing as well as new users to inform them about the new
"follow/unmute topics" feature.
It makes sense to educate only the existing Zulip users about
the new feature using this banner. New users don't need to know
about following topics right away.
This commit makes changes to NOT show the banner to new users.
This will also help to avoid banner overload in the new user
experience.
Fixes#30615.
Revises descriptive text for this endpoint to be clearer and more
robust.
Fixes issues with the rendering of the modern and legacy formats
for the user presence data.
Updates the description for `slim_presence` parameter as it relates
to the `presences` object that's returned in the response.
Adds the modern format for the `presences` object in the response
as that is what will be returned when the parameter above is True.
Because this might impact how past RealmAuditLog-reading migrations
are processed, I think it's better to this one as an actual Django
migration, even though it has no actual SQL to run.
Creates a new "realm_deactivated" email that can be sent to realm
owners as part of `do_deactivate_realm`, via a boolean flag,
`email_owners`.
This flag is set to `False` when `do_deactivate_realm` is used for
realm exports or changing a realm's subdomain, so that the active
organization owners are not emailed in those cases.
This flag is optional for the `deactivate_realm` management command,
but as there is no active user passed in that case, then the email
is sent without referencing who deactivated the realm.
It is passed as `True` for the support analytics view, but the email
that is generated does not include information about the support
admin user who completed the request for organization deactivation.
When an active organization owner deactivates the organization, then
the flag is `True` and an email is sent to them as well as any other
active organization owners, with a slight variation in the email text
for those two cases.
Adds specific tests for when `email_owners` is passed as `True`. All
existing tests for other functionality of `do_deactivate_user` pass
the flag as `False`.
Adds `localize` from django.util.formats as a jinja env filter so
that the dates in these emails are internationlized for the owner's
default language setting in the "realm_deactivated" email templates.
Fixes#24685.
This is done in as much of a drop-in fashion as possible. Note that
libvips does not support animated PNGs[^1], and as such this
conversion removes support for them as emoji; however, libvips
includes support for webp images, which future commits will take
advantage of.
This removes the MAX_EMOJI_GIF_SIZE limit, since that existed to work
around bugs in Pillow. MAX_EMOJI_GIF_FILE_SIZE_BYTES is fixed to
actually be 128KiB (not 128MiB, as it actually was), and is counted
_after_ resizing, since the point is to limit the amount of data
transfer to clients.
[^1]: https://github.com/libvips/libvips/discussions/2000
Multiple events may be created with the same timestamp; we add a
default ordering by "id" such that these have a consistent ordering in
tests (to solve for flaky tests), as well as in the internal API.
This commit updates the message contents to use triple quotes.
It helps to maintain a consistent pattern for multi-line strings
in 'zerver/lib/onboarding.py' file and it'll be a lot more
convenient to edit in that form in the future.
Clarifies that the ID of the edited message will always be included
in the `message_ids` list, even if there are no topic or channel
updates to be made.
Currently, for computing fields like can_create_public_streams
and can_create_private_steams fields, is_user_in_group is called
to check whether the user is part of the group which has the
permission. This means that there will be one DB query for each
field.
To optimize this, we now first fetch all the groups that the
user is member of, including the anonymous groups which are
used for settings, such that we can then just check whether
the user is part of the group which has the permission meaning
we would need only one query to compute all the fields.
This would be helpful when settings for other similar fields
will also be migrated to groups framework.
The test asserts entries about the zephyr realm anyway. The reason the
filter hasn't been limiting the query to zephyr is that we might
simultaneously want to ensure no other realms received changes - but
that doesn't seem quite right, given that the test doesn't dilligently
set up the initial conditions for all realms to have control over what
exactly happens with them. That makes this logic pretty fragile since if
some new (potentially unrelated) changes to Realm/RemoteRealm initial
state make it so some updates to other realms occur during the early
analytics upload calls in the test, the remote_audit_logs asserts here
will break in a very annoying to debug way.
If we want this query to be general, without limiting to the zephyr
realm, we could alter the test a bit to set up initial conditions
precisely.
Adds assert statements to validate error response for "400" error code. The
current validation using `validate_against_openapi_schema` doesn't work for
"400" responses.
Adds separate functions to validate "200" and "400" responses and removes `validate_response_result`.
This commit removes create_private_stream_policy setting as
we now use new group based setting.
The "/register" response includes realm_create_private_stream_policy
field to return a value representing superset of users who have the
permission to create private channels, as older clients still expect
this field.
This commit helps in using the realm object which has the
prefetched group settings so that we can avoid extra queries
when calculating fields like can_create_public_streams.
There is no need to prefetch group settings using select_related
in get_realm as we only need to save queries when computing the
setting values in fetch_initial_state_data and we already refetch
the realm using get_realm_with_settings for that.
There is no need to call settings_user.can_create_public_streams
and similar functions for private and web-public streams twice,
once to compute the field for a single stream type and one to
compute can_create_streams.
The value for each stream type can be used to compute value of
can_create_streams field.
Earlier we were using the type `OptionalNarrowListT` for all functions
that required "narrow" as a parameter.
This commit changes all the functions accepting a "narrow"
to use a list of the new `NarrowParameter`
instead of `OptionalNarrowListT` which is a list of dicts.
Removed the old `narrow_parameter` as we have shifted to
the new `NarrowParameter` Pydantic Object.
This new object provides better error messages for data validation,
hence changed the error messages in `test_message_fetch`.
Earlier `num_before` and `num_after` wasn't being passed while
testing.
Moving to typed_endpoint requires all the "required" parameters
to be passed to the function explicitly in tests.
Convert `custom_profile_fields.py` to use `typed_endpoint`.
Use `TypedDict` from `typing_extensions` instead of `typing`,
to support Pydantic's type checking.
Updates the main description of the API error handling article so
that the `code` key in the error reponse is more accurate.
Adds a bullet point to the API changelog for feature level 76 about
the `code` key being consistently in all error responses. This is
based on commit 8b9011dff8, which completed the migration to
JsonableError, at which point the API feature level was already
set to 75 and not yet set to 76.
Also, revises the other error related feature level 76 entries in
the API changelog.
Adds a header for the common, shared error responses that are
documented on the error handling page, and updates those error
descriptions to include information about any extra fields and to
end with any changes notes.
We believe this to already be obsolete and dead code and is about to be
removed with the migration to the FCM HTTP v1 API, where the concept
doesn't exist anymore.
Create the is:followed search operator.
Fetch all messages that are from followed topics
using exists.
Update API documentation and changelog.
Co-authored-by: Kenneth Rodrigues <kenneth.nrk123@gmail.com>
Fixes#27309.
Previously, there were three different sections for managing active
users, deactivated users and invitations.
This commit combines users section has into a single tabbed panel.
Fixes: #26949.
Co-authored-by: shashank-23002 <21bec103@iiitdmj.ac.in>
When there was a race during bulk insertion of UserTopic
rows, it resulted in Integrity error.
We update the 'last_updated' and 'visibility_policy'
columns for conflicting rows.
We also removed the separate update query to update
visibility_policy because now the new SQL query can
handle the updates too. This leads to have fewer round
trips to the database.
In the 'bulk_set_user_topic_visibility_policy_in_database' function,
the 'duplicate_request' variable wasn't improving any readability.
This commit cleans up that variable.
This commit replaces the local 'is_same_server_message_link'
function used in 'get_mobile_push_content' with the
'is_same_server_message_link' lib function.
The lib function is the same logically but uses urllib instead
of regex for parsing and is backed by tests, hence more robust.
This prep commit adds a lib function 'is_same_server_message_link'.
This will be currently used while compressing quote and reply
in push notifications and later can be used at other places.
This is a prep commit to extract out the test cases
for 'test_is_same_server_message_link' in hash_util.test.js
as a test fixture.
This will help in reusing the same test cases in the node and
backend tests.
Also, it will help in making sure that the logic for
'is_same_server_message_link' is in sync, in both the
web client and server code.
This was discussed in the review of #29999:
https://github.com/zulip/zulip/pull/29999#discussion_r1620818568
The previous way of handling wasn't entirely correct, as unnecessary
events were omitted, with a bad guarantee of even being in the correct
order.
This is an improvement as now the function detects that it ended up
doing nothing and can skip sending an event.
The race condition is hard to make up in an automated test, but we can
hackily simulate it by injecting a side_effect which will create a
conflicting UserPresence row when the function requests a cursor. Aside
of that, the actual race was simulated in manual testing to verify the
expected behavior.
Before this commit, 3 DB queries were actually needed for "realm"
event type but the test mentioned 1 because, the related stream
setting fields were already fetched when testing the case for all
event types above it.
So, when testing the query count for only "realm" event type the
cached realm object was used and no extra queries were made for
the stream settings.
This commit updates the test to refetch the realm before testing
each event type separately, so that we test the actual query count.
This commit updates code to prefetch realm group settings like
"can_create_public_channel_group" only when computing settings
for "/register" response by refetching the realm object with
select_related instead of fetching those settings in UserProfile
query.
This change is done because we do not need to prefetch these
settings for every UserProfile object and for most of the cases
where these settings are actually accessed, we can afford extra
query like when checking permission to create streams. But we
cannot afford one query extra for each setting when computing
these settings for "/register" response, so we re-fetch the
realm object with select_related leading to only one extra
query.
The query count changes in tests are -
- Query count increases by 1 when calling fetch_initial_state_data
for computing can_create_public_streams because Realm object from
UserProfile does not have prefetched setting fields.
- Query count increases by one in test_subs where streams are
created which is as expected due to the setting not being prefetched.
- Query count increases by 2 in tests in test_home.py where one
query is to refetch the realm object and one for computing
can_create_public_streams as mentioned above.
This commit makes passing realm mandatory to fetch_initial_state_data.
This is a prep commit for refetching the realm object with
select_related for group setting fields so that extra queries
can be avoided when computing "/registe" response.
To make better use of the limited characters in mobile push
notifications for messages quoting another message, we compress
the blockquotes and "user said" paragraphs to make space for the
actual message.
Fixes#28951.
Alex Vandiver
Anders Kaseorg
Niloth P
Prakhar Pratyush
Kenneth Rodrigues
Mateusz Mandera
Vector73
Tim Abbott
Aman Agrawal
bedo
Alya Abbott
PieterCK
Sahil Batra
Niklas Fiekas
Lauryn Menard
codewithnick
adnan-td
roanster007
Pedro Almeida
Shashank Singh
Sayam Samal
tnmkr
Shubham Padia
nimish
Shubham Padia