We used 'savepoint=False' in #31169 which was prior to our discussion
in CZO to use 'durable=True' instead.
This commit makes changes to use 'durable=True' in the outermost
transaction.atomic block.
This commit removes the 'prev_rendered_content_version'
field from:
* the 'edit_history' object within message objects in the
API response of `GET /messages`, `GET /messages/{message_id}`
and `POST /zulip-outgoing-webhook`.
* the 'update_message' event type
as it is an internal server implementation detail not used
by any client.
Note: The field is still stored in the 'edit_history' column
of the 'Message' table as it will be helpful when making
major changes to the markup rendering process.
Thumbnails are usually enqueued in the worker when the image is
uploaded. However, for images which were uploaded before the
existence of the thumbnailing worker, and whose metadata was
backfilled (see previous commit) this leaves a permanent spinner,
since nothing triggers the thumbnail worker for them.
Enqueue a thumbnail worker for every spinner which we render into
Markdown. This ensures that _something_ is attempting to resolve the
spinner which the user sees. In the case of freshly-uploaded images
which are still in the queue, this results in a duplicate entry in the
thumbnailing queue -- this is harmless, since the worker determines
that all of the thumbnails we need have already been generated, and it
does no further work. However, in the case of historical uploads, it
properly kicks off the thumbnailing process and results in a
subsequent message update to include the freshly-generated thumbnail.
While specifically useful for backfilled uploads, this is also
generally a good safety step for a good user experience, as it also
prevents dropped events in the queue from unknown causes from leaving
perpetual spinners in the message feed.
Because `get_user_upload_previews` is potentially called twice for
every message with spinners (see 6f20c15ae9), we add an additional
flag to `get_user_upload_previews` to suppress a _second_ event from
being enqueued for every spinner generated.
We previously used the file extension to determine if we should
attempt to inline an image. After b42863be4b, we rely on the
existence of ImageAttachment rows to determine if something is an
image which can be viewed inline. This means that messages
containing files uploaded before that commit, when (re-)rendered, will
be judged as not having inline'able images.
Backfill all of the ImageAttachment rows for image-like file
extensions. We are careful to only download the bytes that we need in
the image headers, to minimize bandwidth from S3 in the event that the
S3 backend is in use. We do _not_ produce thumbnails for the images
during this migration; see the subsequent commit.
Because this migration will be backported to 9.x, it is marked as only
depending on the last migration in `9.x`, with a subsequent merge
migration into the tip of `main`.
Since each loop may add more than one file to the `storage_paths`
list, this may result in more than 1000 files being sent to
delete_message_attachments. Since the S3 backend only supports 1000
elements being deleted at once, we must partition the list into chunks
which are no more than 1000 elements long.
Fixes the URLRedirects for "help/about-streams-and-topics" and
"help/streams-and-topics" to go to "/help/introduction-to-topics"
since "help/channels-and-topics" has been redirected to that URL.
There are a few places where we want to set the max invites for a
realm to the default for a realm's plan type, so this creates a
helper function that can be used consistently to get that default
value.
This has the impact of making rebuilding the database in a Zulip
development environment, or initializing a new production database,
dramatically faster.
This was generated by merging the output of `manage.py makemigrations`
with an empty migration.
Tested using `tools/rebuild-test-database` before and after this
change, and comparing the output of `pg_dump -d zulip_test` using
Git's diff comparison algorithm. Differences in that SQL dump include:
- The actual generated table contents, due to timestamps and the like;
this is expected and unrelated to schema.
- Orders of fields within tables, which is not significant in SQL.
- IDs assigned to tables in the ContentType table, which is expected
and not a problem with how that Django table is designed.
- Names of generated indexes and constraints; modern Django seems to
abbreviate long names differently for these, and it's not obviously
possible to configure those used by the `db_index` property. If
necessarily, likely this can be converged via a migration filled
with `IF EXISTS` rename operations like the one done in
zerver/migrations/0246_message_date_sent_finalize_part2.py.
- Names of the ~3 sequences related to renamed tables:
usertopic/mutedtopic, botconfigdata/botuserconfigdata,
realmdomain/realmalias. Probably there's no action required here,
but we could do rename operations if desired.
Generated using manage.py squashmigrations, with minimal manual
surgery to replace the original migration.
This should in theory produce the exact same database state as
previously.
This has no functional changes, but it helps the squashmigrations tool
realize some squashing opportunities to not have models declared
before other models that they will gain a foreign key to.
Since it's the initial migration, this can't have any useful effect.
I'm pretty sure the backstory is we did a manual squash of migrations
during the process of open-sourcing Zulip, and incorrectly didn't
remove this code.
It's not possible to directly upgrade from pre-5.3 versions to main,
since they do not have any supported OSes in common. Thus, this
dependency on the confirmation model, which risks creating a circular
dependency if we squash migrations, can be removed.
Adds some validation for changing the realm's max invites via the
support view so that it is not set below the default max for the
realm's plan type, and so that if it's currently set to the default
max it's not reset to that same value.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.
Fixes part of #25713
Earlier, we were using 'send_event' in 'do_change_is_billing_admin'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in
'do_update_outgoing_webhook_service' which can lead to a
situation, if any db operation is added after the 'send_event'
in future, where we enqueue events but the action function fails
at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in
'do_set_push_notifications_enabled_end_timestamp' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in 'do_set_realm_stream' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in
'do_set_realm_authentication_methods' which can lead to a situation
where we enqueue events but there's an error at a later stage in
the codepath using this function.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in do_set_realm_user_default_setting
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in 'do_regenerate_api_key'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in 'do_change_full_name'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
The Slack API when returning the emoji records, returns the record for
its thumbsup_all emoji with the url ending with .png, even though the
file is a gif.
For that reason, we have to make that code correct file extensions based
on the response content-type. Emojis are the smallest set of images to
download, so for simplicity of implementation, we remove the
parallelization of the downloads in favor of just processing them
serially.
Ideally this would besplit up into two commits, but it's hard to split
into self-contained, atomic chunks now that this segment of the
import/export system is generally kind of broken after thumbnailing
system changes.
1. 3rd party export converters don't make .original image files.
Insteadthey provide a single file, which the import should treat as if
it's .original.
2. 3rd party converters create all the records with is_animated=False.
That's an issue, because without setting that correctly on the
RealmEmoji objects, Zulip doesn't know that it should use the "still"
thumbnail when the emoji is being used in a user's status. Which leads
to incorrectly displaying the user status with the distracting
animation.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.
Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
Convert `message_send.py` use `typed endpoint`.
Disable `message_send` endpoint `to` parameter in the `openapi`
`validate_json_schema` check, because it is a special case where the
content type of the parameter is application/json but the
parameter may or may not be JSON encoded since previously we also
accepted a raw string and some ad-hoc bot might still depend on sending
a raw string.
Remove unused validators from `validator.py`.
This commit updates the db transaction to be durable for
do_update_user_custom_profile_data_if_changed to avoid
addition of any outer atomic block.
While adding any outer atomic block this will raise a runtime error
and we can replace the durable argument with 'savepoint=False'
otherwise we'll have to manually track down the action functions
getting called in that outer atomic block and set the savepoint=False
otherwise it'll lead to creation of savepoints which we don't want.
We can't set savepoint=False before hand to the outermost action
function because it leads to rollback of transaction in tests when
an error is raised in action function.
Earlier, we were using 'send_event' in
check_remove_custom_profile_field_value which can lead to a
situation where we enqueue events but the function fails at a
later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
There was a bug here that would trigger an exception inside
`sync_user_profile_custom_fields`, causing it to get logged with
logging.warning, when an attribute configured for SAML custom profile
field sync was missing from a SAMLResponse or had an empty value.
`sync_user_profile_custom_fields` expects valid values, and None is not
valid.
We could consider a slightly different behavior here instead - when an
attribute is sent with no value in the SAMLResponse, that means the attr
has no value in the IdP's user directory - so perhaps a better behavior
would be to also remove the custom profile field value in Zulip. However
there are two issues with that:
1. It's not necessarily the best behavior, because an organization might
want the "user doesn't have this attribute set at the IdP level" state
to just mean that the user should be free to set the value manually in
Zulip if they wish. And having that value get reset on every login would
then be an issue. The implementation in this commit is consistent with
this philosophy.
2. There's some implementation difficulty - upstream
`self.get_attr(...)`, which we use for reading the attr value from the
SAMLResponse, doesn't distinguish between an attribute being sent with
no value and the attribute not being sent at all - in both cases it
returns None. So we'd need some extra work here with parsing the
SAMLResponse properly, to be able to know when the custom profile field
should get cleared.
Replace the SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT with
SOCIAL_AUTH_SYNC_ATTRS_DICT, designed to support also regular user attrs
like role or full name (in the future).
Custom attributes can stay configured as they were and will get merged
into SOCIAL_AUTH_SYNC_ATTRS_DICT in computed_settings, or can be
specified in SOCIAL_AUTH_SYNC_ATTRS_DICT directly with "custom__"
prefix.
The role sync is plumbed through to user creation, so users can
immediately be created with their intended role as provided by the IdP
when they're creating their account, even when doing this flow without
an invitiation.
This commit make changes in code to include can_manage_group
field to UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.
Prakhar Pratyush
Alya Abbott
bedo
Alex Vandiver
Lauryn Menard
Kenneth Rodrigues
Anders Kaseorg
Tim Abbott
afeefuddin
roanster007
Mateusz Mandera
Ujjawal Modi