This is done in as much of a drop-in fashion as possible. Note that
libvips does not support animated PNGs[^1], and as such this
conversion removes support for them as emoji; however, libvips
includes support for webp images, which future commits will take
advantage of.
This removes the MAX_EMOJI_GIF_SIZE limit, since that existed to work
around bugs in Pillow. MAX_EMOJI_GIF_FILE_SIZE_BYTES is fixed to
actually be 128KiB (not 128MiB, as it actually was), and is counted
_after_ resizing, since the point is to limit the amount of data
transfer to clients.
[^1]: https://github.com/libvips/libvips/discussions/2000
Multiple events may be created with the same timestamp; we add a
default ordering by "id" such that these have a consistent ordering in
tests (to solve for flaky tests), as well as in the internal API.
This commit updates the message contents to use triple quotes.
It helps to maintain a consistent pattern for multi-line strings
in 'zerver/lib/onboarding.py' file and it'll be a lot more
convenient to edit in that form in the future.
Clarifies that the ID of the edited message will always be included
in the `message_ids` list, even if there are no topic or channel
updates to be made.
Currently, for computing fields like can_create_public_streams
and can_create_private_steams fields, is_user_in_group is called
to check whether the user is part of the group which has the
permission. This means that there will be one DB query for each
field.
To optimize this, we now first fetch all the groups that the
user is member of, including the anonymous groups which are
used for settings, such that we can then just check whether
the user is part of the group which has the permission meaning
we would need only one query to compute all the fields.
This would be helpful when settings for other similar fields
will also be migrated to groups framework.
The test asserts entries about the zephyr realm anyway. The reason the
filter hasn't been limiting the query to zephyr is that we might
simultaneously want to ensure no other realms received changes - but
that doesn't seem quite right, given that the test doesn't dilligently
set up the initial conditions for all realms to have control over what
exactly happens with them. That makes this logic pretty fragile since if
some new (potentially unrelated) changes to Realm/RemoteRealm initial
state make it so some updates to other realms occur during the early
analytics upload calls in the test, the remote_audit_logs asserts here
will break in a very annoying to debug way.
If we want this query to be general, without limiting to the zephyr
realm, we could alter the test a bit to set up initial conditions
precisely.
Adds assert statements to validate error response for "400" error code. The
current validation using `validate_against_openapi_schema` doesn't work for
"400" responses.
Adds separate functions to validate "200" and "400" responses and removes `validate_response_result`.
This commit removes create_private_stream_policy setting as
we now use new group based setting.
The "/register" response includes realm_create_private_stream_policy
field to return a value representing superset of users who have the
permission to create private channels, as older clients still expect
this field.
This commit helps in using the realm object which has the
prefetched group settings so that we can avoid extra queries
when calculating fields like can_create_public_streams.
There is no need to prefetch group settings using select_related
in get_realm as we only need to save queries when computing the
setting values in fetch_initial_state_data and we already refetch
the realm using get_realm_with_settings for that.
There is no need to call settings_user.can_create_public_streams
and similar functions for private and web-public streams twice,
once to compute the field for a single stream type and one to
compute can_create_streams.
The value for each stream type can be used to compute value of
can_create_streams field.
Earlier we were using the type `OptionalNarrowListT` for all functions
that required "narrow" as a parameter.
This commit changes all the functions accepting a "narrow"
to use a list of the new `NarrowParameter`
instead of `OptionalNarrowListT` which is a list of dicts.
Removed the old `narrow_parameter` as we have shifted to
the new `NarrowParameter` Pydantic Object.
This new object provides better error messages for data validation,
hence changed the error messages in `test_message_fetch`.
Earlier `num_before` and `num_after` wasn't being passed while
testing.
Moving to typed_endpoint requires all the "required" parameters
to be passed to the function explicitly in tests.
Convert `custom_profile_fields.py` to use `typed_endpoint`.
Use `TypedDict` from `typing_extensions` instead of `typing`,
to support Pydantic's type checking.
Updates the main description of the API error handling article so
that the `code` key in the error reponse is more accurate.
Adds a bullet point to the API changelog for feature level 76 about
the `code` key being consistently in all error responses. This is
based on commit 8b9011dff8, which completed the migration to
JsonableError, at which point the API feature level was already
set to 75 and not yet set to 76.
Also, revises the other error related feature level 76 entries in
the API changelog.
Adds a header for the common, shared error responses that are
documented on the error handling page, and updates those error
descriptions to include information about any extra fields and to
end with any changes notes.
We believe this to already be obsolete and dead code and is about to be
removed with the migration to the FCM HTTP v1 API, where the concept
doesn't exist anymore.
Create the is:followed search operator.
Fetch all messages that are from followed topics
using exists.
Update API documentation and changelog.
Co-authored-by: Kenneth Rodrigues <kenneth.nrk123@gmail.com>
Fixes#27309.
Previously, there were three different sections for managing active
users, deactivated users and invitations.
This commit combines users section has into a single tabbed panel.
Fixes: #26949.
Co-authored-by: shashank-23002 <21bec103@iiitdmj.ac.in>
When there was a race during bulk insertion of UserTopic
rows, it resulted in Integrity error.
We update the 'last_updated' and 'visibility_policy'
columns for conflicting rows.
We also removed the separate update query to update
visibility_policy because now the new SQL query can
handle the updates too. This leads to have fewer round
trips to the database.
In the 'bulk_set_user_topic_visibility_policy_in_database' function,
the 'duplicate_request' variable wasn't improving any readability.
This commit cleans up that variable.
This commit replaces the local 'is_same_server_message_link'
function used in 'get_mobile_push_content' with the
'is_same_server_message_link' lib function.
The lib function is the same logically but uses urllib instead
of regex for parsing and is backed by tests, hence more robust.
This prep commit adds a lib function 'is_same_server_message_link'.
This will be currently used while compressing quote and reply
in push notifications and later can be used at other places.
This is a prep commit to extract out the test cases
for 'test_is_same_server_message_link' in hash_util.test.js
as a test fixture.
This will help in reusing the same test cases in the node and
backend tests.
Also, it will help in making sure that the logic for
'is_same_server_message_link' is in sync, in both the
web client and server code.
This was discussed in the review of #29999:
https://github.com/zulip/zulip/pull/29999#discussion_r1620818568
The previous way of handling wasn't entirely correct, as unnecessary
events were omitted, with a bad guarantee of even being in the correct
order.
This is an improvement as now the function detects that it ended up
doing nothing and can skip sending an event.
The race condition is hard to make up in an automated test, but we can
hackily simulate it by injecting a side_effect which will create a
conflicting UserPresence row when the function requests a cursor. Aside
of that, the actual race was simulated in manual testing to verify the
expected behavior.
Before this commit, 3 DB queries were actually needed for "realm"
event type but the test mentioned 1 because, the related stream
setting fields were already fetched when testing the case for all
event types above it.
So, when testing the query count for only "realm" event type the
cached realm object was used and no extra queries were made for
the stream settings.
This commit updates the test to refetch the realm before testing
each event type separately, so that we test the actual query count.
This commit updates code to prefetch realm group settings like
"can_create_public_channel_group" only when computing settings
for "/register" response by refetching the realm object with
select_related instead of fetching those settings in UserProfile
query.
This change is done because we do not need to prefetch these
settings for every UserProfile object and for most of the cases
where these settings are actually accessed, we can afford extra
query like when checking permission to create streams. But we
cannot afford one query extra for each setting when computing
these settings for "/register" response, so we re-fetch the
realm object with select_related leading to only one extra
query.
The query count changes in tests are -
- Query count increases by 1 when calling fetch_initial_state_data
for computing can_create_public_streams because Realm object from
UserProfile does not have prefetched setting fields.
- Query count increases by one in test_subs where streams are
created which is as expected due to the setting not being prefetched.
- Query count increases by 2 in tests in test_home.py where one
query is to refetch the realm object and one for computing
can_create_public_streams as mentioned above.
This commit makes passing realm mandatory to fetch_initial_state_data.
This is a prep commit for refetching the realm object with
select_related for group setting fields so that extra queries
can be avoided when computing "/registe" response.
To make better use of the limited characters in mobile push
notifications for messages quoting another message, we compress
the blockquotes and "user said" paragraphs to make space for the
actual message.
Fixes#28951.
Extracts code for generating the events table strings to a separate
function and uses templates so that the HTML is clearer.
Updates events table classes for CSS to start with "api-" for
clarity.
Creates a dataclass for getting the data needed for an individual
event's documentation and uses that dataclass in a separate
function for generating the strings for each event's documentation.
Uses templates for parts of event documentation with HTML and CSS,
and updates CSS classes to have clearer names.
Updates the self-hosted and Zulip Cloud billing articles for features
that have been added to billing management such as paying by invoice,
adding billing contact information, viewing past invoices, etc.
Revises text to be cleaner and clearer in some places.
More consistently uses bold (when not linked) for plan names.
Fixes a few errors or omissions in existing instructions.
Co-authored-by: Alya Abbott <alya@zulip.com>
The general format for our bold "Changes:" notes are to have the
most recent one start with the "Changes:" text and the older notes
following that with paragraph breaks for each note.
Sending to a topic based on the number of firing alerts makes no
sense, and leads to conversations and alerts scattered randomly across
topics based on how on fire the alerting is.
Send a separate message for each alert in the Grafana webhook payload,
with the alert's name as its topic; if no alert name can be found,
fall back to the alert's fingerprint. Also include all alert values
in the body of the message, along with links to the alert generator,
silence, and image, if available.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
Currently we send a notification to the topic if it has been resolved
or unresolved even if there is an immediate event of resolving and
then unresolving or vice-versa. This adds a setting of
RESOLVE_TOPIC_UNDO_GRACE_PERIOD_SECONDS under which if a topic has
been unresolved after being resolved immediately and the last message
was the notification of resolving, then delete the last message and
don't send a new notification and vice-versa.
We use the new message.type field to precisely identify relevant
messages.
Fixes#19181.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This commit removes create_public_stream_policy setting
since public channel creation permissions are now handled
by group-based setting.
We still pass "realm_create_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate
enum corresponding to the group setting, then we set
it to "Members only" considering that server will not
allow the users without permissions to create public
channels but the client can make sure that UI is
available to the users who have permission.
This was a bug, where in the realm.presence_disabled (synonymous to
being a zephyr mirror realm) case we would return None. We have decided
on the convention of using only integers here, and -1 representing lack
of data.
Migrate all `ids` of anything which does not have a foreign key from
the Message or UserMessage table (and would thus require walking
those) to be `bigint`. This is done by removing explicit
`BigAutoField`s, trading them for explicit `AutoField`s on the tables
to not be migrated, while updating `DEFAULT_AUTO_FIELD` to the new
default.
In general, the tables adjusted in this commit are small tables -- at
least compared to Messages and UserMessages.
Many-to-many tables without their own model class are adjusted by a
custom Operation, since they do not automatically pick up migrations
when `DEFAULT_AUTO_FIELD` changes[^1].
Note that this does multiple scans over tables to update foreign
keys[^2]. Large installs may wish to hand-optimize this using the
output of `./manage.py sqlmigrate` to join multiple `ALTER TABLE`
statements into one, to speed up the migration. This is unfortunately
not possible to do generically, as constraint names may differ between
installations.
This leaves the following primary keys as non-`bigint`:
- `auth_group.id`
- `auth_group_permissions.id`
- `auth_permission.id`
- `django_content_type.id`
- `django_migrations.id`
- `otp_static_staticdevice.id`
- `otp_static_statictoken.id`
- `otp_totp_totpdevice.id`
- `two_factor_phonedevice.id`
- `zerver_archivedmessage.id`
- `zerver_client.id`
- `zerver_message.id`
- `zerver_realm.id`
- `zerver_recipient.id`
- `zerver_userprofile.id`
[^1]: https://code.djangoproject.com/ticket/32674
[^2]: https://code.djangoproject.com/ticket/24203
These models have no external references, and are among the larger
tables. Migrate them to bigints, starting with `useractivityinterval`
which is less likely to be being touched.
This table, while it does not contain a large number of rows, consumes
a primary key for every message send and update. Since it is not
referenced by any other table, the migration is simple; and since it
does not contain many rows at any time, it should be fast.
We immediately navigate the user to the conversation they just
sent a message to if they are not already in the appropriate
conversation view.
This commit adds a first-time banner to explain the same.
Fixes#29575.
4430ab9cbe changed this, assuming that all servers would send
`realm_url` -- however, only servers running that commit do. Update
to accept either `realm_url` or `realm_uri` payload properties.
This index is used by `active_users_audit:is_bot:day`, and provides
roughly a 2x speedup. The existing
`zerver_realmauditlog_realm__event_type__event_time` is used if there
is a realm limit, but the standard statistics fill runs for all realms
at once, and thus cannot use it.
Creates a helper function `get_subscribed_stream_ids` to fetch
subscribed streams' ids of the client.
Updates various functions to remove hardcoded values in request
body and used available API calls to fetch them.
This commit adds a server level setting which controls whether the setting
can be set to anonymous user groups. We only allow it in the tests for
now because the UI can only handle named user groups.
This commit fixes the event sent for updating first_message_id
when a message is deleted, to include the name field as it is
required for all "stream/update" events.
This commit also adds a test in test_events for the case when
first_message_id of a stream is updated on deleting a message.
This commit has the mechanism for setting the .last_update_id on the
backend for UserPresence objects, without introducing any API changes.
Therefore this should invisible to clients and able to be deployed
safely without impact to users.
Migration plan:
1. Add NULLable .last_update_id column to UserPresence with default 0
for new objects.
2. Backfill the value to 0 for old UserPresences, can be done in the
background while server is running.
3. Make the column non-NULL.
4. Add new model PresenceSequence and create its rows for old realms.
Fixes#26369.
There are two important fixes to make to the dicts in edit_history:
1. Update the user_id so that it points to the imported sender.
2. Apply fix_message_rendered_content to the prev_rendered_content data
to fix up mentions and other such syntax.
The docstring was misleading talking about the import from non-Zulip
platforms, when this function is also very much applicable for
Zulip-to-Zulip imports.
Previously, when the operand of id operator was more than
2147483647, it was raising server error. This is because the
maximum permissible PostgreSQL integers value is 2147483647.
This is fixed by raising a BadNarrowOperatorError in case the
id operand is larger than 2147483647.
Previously, the "Changes" sections for narrows in the api_docs
used to mention the entire set of changes for the narrow. This
commit replaces it to rather refer to the changelog at
[Construct a narrow](/api/construct-narrow).
This avoids the risk of the details getting out of sync between
the different descriptions, and reduces the burden for updating
or editing them.
Earlier, the 'remove_single_newlines' function wasn't working
correctly when '\n' was followed by asterix, hyphen, or number.
Specifically, they were not added as a syntax for ordered or
unordered list in markdown.
For example, see the workaround fix in 330439a83b.
This commit updates the function to replace '\n' with ' ', when
"\n" is not preceded by "\n" and not followed by:
* Another newline (\n)
* A hyphen followed by a space
* An asterisk followed by a space
* A number followed by a dot and a space
We won't have to do fixes like 330439a83b in the future.
Earlier, for new realm with zulip_update_announcements_stream
set to None, an assertion error was raised in
'is_group_direct_message_sent_to_admins_within_days' because
no 'None' to new level change took place for such realm.
A new realm is on the latest level and a default stream is set.
This commit updates the logic to simply skip sending
update messages as the stream was manually set to None
in such cases.
Removes the 'onboarding_steps' field on 'UserProfile'
model which is no longer used.
It was introduced back in 2013 (b5e22bf), and is no
longer used.
Fixes part of #30043.
Alex Vandiver
Lauryn Menard
Vector73
Kenneth Rodrigues
Prakhar Pratyush
Sahil Batra
Mateusz Mandera
Anders Kaseorg
tnmkr
Alya Abbott
Shubham Padia
nimish
Shubham Padia
Ryan Crisanti
Gaurav Pandey
Tim Abbott
Prakhar Pratyush
Aman Agrawal
Kislay Verma
roanster007