As a follow up for f49a11c810, this
commit standardizes the naming of the day and night themes to light
and dark, respectively in the backend. This makes the backend
consistent with the naming used in the frontend and UI.
This also solves a regression introduced in
f49a11c810, where the frontend was sending
"/light" and "/dark" commands to the backend, but the backend was
expecting "/day" and "/night" commands.
Earlier, a one-time 'visibility_policy_banner' was displayed to
existing as well as new users to inform them about the new
"follow/unmute topics" feature.
It makes sense to educate only the existing Zulip users about
the new feature using this banner. New users don't need to know
about following topics right away.
This commit makes changes to NOT show the banner to new users.
This will also help to avoid banner overload in the new user
experience.
Fixes#30615.
Creates a new "realm_deactivated" email that can be sent to realm
owners as part of `do_deactivate_realm`, via a boolean flag,
`email_owners`.
This flag is set to `False` when `do_deactivate_realm` is used for
realm exports or changing a realm's subdomain, so that the active
organization owners are not emailed in those cases.
This flag is optional for the `deactivate_realm` management command,
but as there is no active user passed in that case, then the email
is sent without referencing who deactivated the realm.
It is passed as `True` for the support analytics view, but the email
that is generated does not include information about the support
admin user who completed the request for organization deactivation.
When an active organization owner deactivates the organization, then
the flag is `True` and an email is sent to them as well as any other
active organization owners, with a slight variation in the email text
for those two cases.
Adds specific tests for when `email_owners` is passed as `True`. All
existing tests for other functionality of `do_deactivate_user` pass
the flag as `False`.
Adds `localize` from django.util.formats as a jinja env filter so
that the dates in these emails are internationlized for the owner's
default language setting in the "realm_deactivated" email templates.
Fixes#24685.
This is done in as much of a drop-in fashion as possible. Note that
libvips does not support animated PNGs[^1], and as such this
conversion removes support for them as emoji; however, libvips
includes support for webp images, which future commits will take
advantage of.
This removes the MAX_EMOJI_GIF_SIZE limit, since that existed to work
around bugs in Pillow. MAX_EMOJI_GIF_FILE_SIZE_BYTES is fixed to
actually be 128KiB (not 128MiB, as it actually was), and is counted
_after_ resizing, since the point is to limit the amount of data
transfer to clients.
[^1]: https://github.com/libvips/libvips/discussions/2000
Currently, for computing fields like can_create_public_streams
and can_create_private_steams fields, is_user_in_group is called
to check whether the user is part of the group which has the
permission. This means that there will be one DB query for each
field.
To optimize this, we now first fetch all the groups that the
user is member of, including the anonymous groups which are
used for settings, such that we can then just check whether
the user is part of the group which has the permission meaning
we would need only one query to compute all the fields.
This would be helpful when settings for other similar fields
will also be migrated to groups framework.
The test asserts entries about the zephyr realm anyway. The reason the
filter hasn't been limiting the query to zephyr is that we might
simultaneously want to ensure no other realms received changes - but
that doesn't seem quite right, given that the test doesn't dilligently
set up the initial conditions for all realms to have control over what
exactly happens with them. That makes this logic pretty fragile since if
some new (potentially unrelated) changes to Realm/RemoteRealm initial
state make it so some updates to other realms occur during the early
analytics upload calls in the test, the remote_audit_logs asserts here
will break in a very annoying to debug way.
If we want this query to be general, without limiting to the zephyr
realm, we could alter the test a bit to set up initial conditions
precisely.
This commit removes create_private_stream_policy setting as
we now use new group based setting.
The "/register" response includes realm_create_private_stream_policy
field to return a value representing superset of users who have the
permission to create private channels, as older clients still expect
this field.
This commit helps in using the realm object which has the
prefetched group settings so that we can avoid extra queries
when calculating fields like can_create_public_streams.
Earlier we were using the type `OptionalNarrowListT` for all functions
that required "narrow" as a parameter.
This commit changes all the functions accepting a "narrow"
to use a list of the new `NarrowParameter`
instead of `OptionalNarrowListT` which is a list of dicts.
Removed the old `narrow_parameter` as we have shifted to
the new `NarrowParameter` Pydantic Object.
This new object provides better error messages for data validation,
hence changed the error messages in `test_message_fetch`.
Earlier `num_before` and `num_after` wasn't being passed while
testing.
Moving to typed_endpoint requires all the "required" parameters
to be passed to the function explicitly in tests.
Convert `custom_profile_fields.py` to use `typed_endpoint`.
Use `TypedDict` from `typing_extensions` instead of `typing`,
to support Pydantic's type checking.
We believe this to already be obsolete and dead code and is about to be
removed with the migration to the FCM HTTP v1 API, where the concept
doesn't exist anymore.
Create the is:followed search operator.
Fetch all messages that are from followed topics
using exists.
Update API documentation and changelog.
Co-authored-by: Kenneth Rodrigues <kenneth.nrk123@gmail.com>
Fixes#27309.
This prep commit adds a lib function 'is_same_server_message_link'.
This will be currently used while compressing quote and reply
in push notifications and later can be used at other places.
This is a prep commit to extract out the test cases
for 'test_is_same_server_message_link' in hash_util.test.js
as a test fixture.
This will help in reusing the same test cases in the node and
backend tests.
Also, it will help in making sure that the logic for
'is_same_server_message_link' is in sync, in both the
web client and server code.
This was discussed in the review of #29999:
https://github.com/zulip/zulip/pull/29999#discussion_r1620818568
The previous way of handling wasn't entirely correct, as unnecessary
events were omitted, with a bad guarantee of even being in the correct
order.
This is an improvement as now the function detects that it ended up
doing nothing and can skip sending an event.
The race condition is hard to make up in an automated test, but we can
hackily simulate it by injecting a side_effect which will create a
conflicting UserPresence row when the function requests a cursor. Aside
of that, the actual race was simulated in manual testing to verify the
expected behavior.
Before this commit, 3 DB queries were actually needed for "realm"
event type but the test mentioned 1 because, the related stream
setting fields were already fetched when testing the case for all
event types above it.
So, when testing the query count for only "realm" event type the
cached realm object was used and no extra queries were made for
the stream settings.
This commit updates the test to refetch the realm before testing
each event type separately, so that we test the actual query count.
This commit updates code to prefetch realm group settings like
"can_create_public_channel_group" only when computing settings
for "/register" response by refetching the realm object with
select_related instead of fetching those settings in UserProfile
query.
This change is done because we do not need to prefetch these
settings for every UserProfile object and for most of the cases
where these settings are actually accessed, we can afford extra
query like when checking permission to create streams. But we
cannot afford one query extra for each setting when computing
these settings for "/register" response, so we re-fetch the
realm object with select_related leading to only one extra
query.
The query count changes in tests are -
- Query count increases by 1 when calling fetch_initial_state_data
for computing can_create_public_streams because Realm object from
UserProfile does not have prefetched setting fields.
- Query count increases by one in test_subs where streams are
created which is as expected due to the setting not being prefetched.
- Query count increases by 2 in tests in test_home.py where one
query is to refetch the realm object and one for computing
can_create_public_streams as mentioned above.
This commit makes passing realm mandatory to fetch_initial_state_data.
This is a prep commit for refetching the realm object with
select_related for group setting fields so that extra queries
can be avoided when computing "/registe" response.
To make better use of the limited characters in mobile push
notifications for messages quoting another message, we compress
the blockquotes and "user said" paragraphs to make space for the
actual message.
Fixes#28951.
Currently we send a notification to the topic if it has been resolved
or unresolved even if there is an immediate event of resolving and
then unresolving or vice-versa. This adds a setting of
RESOLVE_TOPIC_UNDO_GRACE_PERIOD_SECONDS under which if a topic has
been unresolved after being resolved immediately and the last message
was the notification of resolving, then delete the last message and
don't send a new notification and vice-versa.
We use the new message.type field to precisely identify relevant
messages.
Fixes#19181.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This commit removes create_public_stream_policy setting
since public channel creation permissions are now handled
by group-based setting.
We still pass "realm_create_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate
enum corresponding to the group setting, then we set
it to "Members only" considering that server will not
allow the users without permissions to create public
channels but the client can make sure that UI is
available to the users who have permission.
This was a bug, where in the realm.presence_disabled (synonymous to
being a zephyr mirror realm) case we would return None. We have decided
on the convention of using only integers here, and -1 representing lack
of data.
We immediately navigate the user to the conversation they just
sent a message to if they are not already in the appropriate
conversation view.
This commit adds a first-time banner to explain the same.
Fixes#29575.
This commit adds a server level setting which controls whether the setting
can be set to anonymous user groups. We only allow it in the tests for
now because the UI can only handle named user groups.
This commit fixes the event sent for updating first_message_id
when a message is deleted, to include the name field as it is
required for all "stream/update" events.
This commit also adds a test in test_events for the case when
first_message_id of a stream is updated on deleting a message.
Fixes#26369.
There are two important fixes to make to the dicts in edit_history:
1. Update the user_id so that it points to the imported sender.
2. Apply fix_message_rendered_content to the prev_rendered_content data
to fix up mentions and other such syntax.
Previously, when the operand of id operator was more than
2147483647, it was raising server error. This is because the
maximum permissible PostgreSQL integers value is 2147483647.
This is fixed by raising a BadNarrowOperatorError in case the
id operand is larger than 2147483647.
Earlier, the 'remove_single_newlines' function wasn't working
correctly when '\n' was followed by asterix, hyphen, or number.
Specifically, they were not added as a syntax for ordered or
unordered list in markdown.
For example, see the workaround fix in 330439a83b.
This commit updates the function to replace '\n' with ' ', when
"\n" is not preceded by "\n" and not followed by:
* Another newline (\n)
* A hyphen followed by a space
* An asterisk followed by a space
* A number followed by a dot and a space
We won't have to do fixes like 330439a83b in the future.
Earlier, for new realm with zulip_update_announcements_stream
set to None, an assertion error was raised in
'is_group_direct_message_sent_to_admins_within_days' because
no 'None' to new level change took place for such realm.
A new realm is on the latest level and a default stream is set.
This commit updates the logic to simply skip sending
update messages as the stream was manually set to None
in such cases.
This commit adds a new helper function to create or update
a UserGroup object for a setting. We could have used existing
update_or_create_user_group_for_setting but that also validates
user IDs and subgroup IDs which we can skip in tests.
This commit fixes the code store correct old value in audit
log data when changing can_mention_group setting from a
anonymous group to another anonymous group. The bug was
because the old value was being computed after updating
the UserGroup object with new members and subgroups and
is fixed by computing the old value for all the cases
and passing it to do_change_user_group_permission_setting.
This commit replaces occurrences of realm_uri with realm_url in email templates
and other related backend files.
Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
Earlier, we were using 'send_event' in 'do_update_message'
which can lead to a situation where we enqueue events but
the transaction fails at a later stage.
Events should not be sent until we know we're not rolling back.
The documentation Creates a shared UserStatus schema that's used for
the return value of this new endpoint and for the existing user_status
objects returned by the register queue endpoint.
Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>
Fixes#19079.
This commit updates code, majorly in tests, to use
setting values from enums instead of directly using
the constants defined in Realm.
We still have those constants defined Realm as they
are used in a couple of places where the same code
is used for different settings. These will be
handled later.
This commits fixes the code which checks group mention permission
to handle anonymous user groups correctly. Basically we were
not checking whether the UserGroup is linked to a NamedUserGroup
and directly accessing named_user_group which results in an
error.
We also update the error messages to include the group name
which has permission to mention the groups since now there
might be a comnbination of groups and users who has permission
to mention the group.
This commit also adds tests to check sending and editing messages
when can_mention_group is set to a anonymous user group.
Updates the API doc for GET /users/me/subscriptions endpoint to
use channel in descriptions instead of stream.
Part of the stream to channel rename project.
Updates the API documentation for the /get_stream_id endpoint to
use channel in descriptions instead of stream.
Part of the stream to channel rename project.
Earlier, we were using 'send_event' & 'queue_json_publish' in
'do_send_messages' which can lead to a situation where we enqueue
events but the transaction fails at a later stage.
Events should not be sent until we know we're not rolling back.
It's going to be helpful in the future to record the reason for realm
deactivation.
- For information tracking
- For making a distinction between cases where we can allow realm owners
to reactivate their realm via a self-serve flow (e.g.
"owner_request") vs where we can't (ToS abuse).
If an invalid timezone (such as +32h) was provided, the
timestamp.astimezone call would throw an exception, causing the
message send to fail. Replace that with a user-facing error.
Fixes#19658.
This is a follow-up to #29425. It assigns creator to three streams
that are used in tests and another two that aren't. For tests,
we assign a guest, an admin and an owner as creator.
To help users focus on the onboarding experience, we no longer
send the introductory "Zulip updates" message as a part of
onboarding.
Now, we send the introductory message just before the first
update message.
Fixes#30053.
This commit removes set_up_db, which called realm.save to
update the settings, and its calls are replaced with either
self.assertEqual (which is enough to verify the initial value)
or a "do_.." function call to set the setting to initial value.
It is better to avoid calling "realm.save" and instead use the
"do_.." functions to set the settings.
There are places where it is easy to just call "do_.." function
as default value is not the first value in the list of test values.
This commit improves how we handle video_chat_provider
and jitsi_server_url settings in do_test_realm_update_api.
For video_chat_provider, there is no use for calling
update_with_api_multiple_value and it can be handled
similar to other tests, so did that change and also
added one more value to test, so that it actually
tests changing the setting.
We need to handle jitsi_server_url separately as even
though value is a string, we need to serialize to
JSON. We do not use a separate function for it as
this is the only setting to handle differently and
update_with_api_multiple_value anyways was added in
bbff797a93 to update multiple settings in a request
which we are not doing here.
As a result of above changes, update_with_api_multiple_value
function is also removed.
This commit relocates all the scripts in the tools directory which
are used for auto-generating screenshots to the new
tools/screenshots directory to avoid cluttering the tools/ root.
This commit adds include_realm_default_subscriptions parameter
to the invite endpoints and the corresponding field in
PreregistrationUser and MultiuseInvite objects. This field will
be used to subscribe the new users to the default streams at the
time of account creation and not to the streams that were default
when sending the invite.
This commit changes the code to subscribe the invited user to default
streams even if the user who invited the new user was not allowed to
subscribe others to streams.
This commit adds support to pass object containing both old and new
values of the can_mention_group setting, as well as detailed API
documentation for this part of the API system.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Co-authored-by: Greg PRice <greg@zulip.com>
When importing a Stream, UserProfiles don't yet exist. so trying to
import stream.creator fails with something like:
```
psycopg2.errors.ForeignKeyViolation: insert or update on table
"zerver_stream" violates foreign key constraint
"zerver_stream_creator_id_65aeba7e_fk_zerver_userprofile_id"
DETAIL: Key (creator_id)=(5) is not present in table "zerver_userprofile".
```
- Tighten up the content.
- Use pretty quotes.
- Reorder "experiments" topic to appear after "start a conversation"
in Inbox.
- Drop reference to messages being starred (not implemented yet).
This commit adds API documentation for "/users/me/android_gcm_reg_id:post"
and "/users/me/android_gcm_reg_id:delete" endpoints.
Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>
This commit adds API documentation for "/users/me/apns_device_token:post"
and "/users/me/apns_device_token:delete" endpoints.
openapi: Document android_gcm_reg_id endpoints.
Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>
Mark the channel name of the initial channel created during
realm creation for translation.
It doesn't mark the topic names and description for translation
because we are planning to remove these topics and update the
description as a part of improving the onboarding experience.
We no longer create the 'core team' private channel when
a realm is created.
Earlier, "New user announcements" channel was set to the
"core team" channel. Now it is disabled by default.
populate_db still creates the 'core team' channel to
represent a private channel.
The reactions added to messages in the test database are
random in nature, so there is a possibility of any number
of reaction records in the exported user data.
The test now verifies the reaction which was added in the test
itself.
Earlier, the test was assuming the reaction added in the test
is the only record in the exported data.
That would result in a flaky behavior. This commit fixes the
flaky behavior.
The messages populated in the test database are random in nature,
so there is a possibility of any number of messages in
'new_stream_announcements_stream' in 'test_merge_streams'.
Earlier, the test was assuming of exactly one message in that
stream.
That would result in a flaky behavior. This commit fixes the
flaky behavior.
The messages populated in the test database are random in nature,
so there is a possibility of more than one resolved topic
messages returned by 'self.get_and_check_messages()' query
in 'test_get_messages_for_resolved_topics'.
That would result in a flaky behavior. This commit fixes the
flaky behavior.
Fixes#29632.
The issue description explains this well:
We currently recalculate `currently_used_upload_space_bytes` every file
upload, by dint of calling `flush_used_upload_space_cache` on
save/delete, and then immediately calling
`user_profile.realm.currently_used_upload_space_bytes()` in
`notify_attachment_update`. Since this walks the Attachments table,
recalculating this can take seconds in large realms.
Switch this to using a CountStat, so we don't need to walk significant
chunks of the Attachment table when we upload an attachment. This will
also give us a historical daily graph of usage.
The naming `uri` is deprecated while `url` should be used in order to
satisfy URL standards. For this reason, four endpoints are affected:
* The response content of three endpoints `/server_settings`,
`/register` and `/realm` that contain a field `realm_uri` is
changed to `realm_url`.
* In one of the common fields for all mobile push notifications payloads,
`realm_url` field is now added as an alias to `realm_uri`.
For backwards compatibility, we keep the field `realm_uri` and add
an alias `realm_url`.
Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
When the export is NOT generated by another zulip server,
while importing:
* Set the 'zulip_update_announcements_level' to the latest level
as we don't want to send all the older update messages to them.
* Send a group DM to admins, suggesting them to configure the
stream in order to avoid missing future update messages.
Fixes#29041.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.
This commit changes just the model property, which has no API
compatibility concerns.
The previous implementation using Django's `get_or_create` for
`do_increment_logging_stat` involved two separate database queries,
potentially leading to race conditions.
Use an `ON CONFLICT ... DO UPDATE` (aka "upsert") query, which
eliminates race conditions and improves performance. This is mildly
complicated due to the different unique indexes across the various
tables, and the need for bug-for-bug compatibility with the previous
implementation.
Fixes#28947.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email. Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation. That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.
Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites. The volume of
invites is small enough that this will not affect other uses of that
worker.
Fixes: #21306Fixes: #24275
This prevents users from hammering the invitation endpoint, causing
races, and inviting more users than they should otherwise be allowed
to.
Doing this requires that we not raise InvitationError when we have
partially succeeded; that behaviour is left to the one callsite of
do_invite_users.
Reported by Lakshit Agarwal (@chiekosec).
Gitter broke their older API as part of being integrated
into Matrix.
Their announcement blog says:
"Anything left using the Gitter APIs will need to be
updated to use the Matrix API"
This commit drops the legacy Gitter import tool and
we plan to build a new one for Matrix in future.
Updates the base hash for the streams setting overlay to be
"channels" instead of "streams".
Because there are Welcome Bot and Notification Bot messages that
would have been sent with the "/#streams" hash, we will need to
support parsing those overlay hashes as an alias for "/#channels"
permanently.
Part of the stream to channels rename project.
We now "first_message_id" of the stream on the deletion of the first
message that was sent to it. This results in 1 extra query when any
stream message is deleted and 3 extra queries when the first message
sent to any stream is deleted.
Fixes#28877.
Creates an IncompatibleParametersError to be used in cases where
there are two (or more) optional parameters for an endpoint that
are incompatible with each other, e.g. there's a parameter for a
user name and a user ID but only one should be sent in the request
to identify the user.
Documents the error on the /api/rest-error-handling article.
Updates the PATCH users/me/subscriptions/muted_topics endpoint to
use this error when both the stream and stream_id parameters are
passed (note this endpoint is currently deprecated).
This commit removes name, description, is_system_group and
can_mention_group fields from UserGroup model and rename
them in NamedUserGroup model.
Fixes#29554.
This commit updates code to access name from named_user_group
field which points to the "NamedUserGroup" instead of directly
accessing name from "UserGroup", since name field will only
be present on NamedUserGroup objects in further commits.
Vector73
Sayam Samal
Prakhar Pratyush
Anders Kaseorg
Lauryn Menard
Alex Vandiver
Sahil Batra
Mateusz Mandera
Kenneth Rodrigues
Shubham Padia
nimish
Gaurav Pandey
Prakhar Pratyush
Aman Agrawal
roanster007
Tomas Fonseca
tnmkr
Alya Abbott
Mahhheshh