Migrate all `ids` of anything which does not have a foreign key from
the Message or UserMessage table (and would thus require walking
those) to be `bigint`. This is done by removing explicit
`BigAutoField`s, trading them for explicit `AutoField`s on the tables
to not be migrated, while updating `DEFAULT_AUTO_FIELD` to the new
default.
In general, the tables adjusted in this commit are small tables -- at
least compared to Messages and UserMessages.
Many-to-many tables without their own model class are adjusted by a
custom Operation, since they do not automatically pick up migrations
when `DEFAULT_AUTO_FIELD` changes[^1].
Note that this does multiple scans over tables to update foreign
keys[^2]. Large installs may wish to hand-optimize this using the
output of `./manage.py sqlmigrate` to join multiple `ALTER TABLE`
statements into one, to speed up the migration. This is unfortunately
not possible to do generically, as constraint names may differ between
installations.
This leaves the following primary keys as non-`bigint`:
- `auth_group.id`
- `auth_group_permissions.id`
- `auth_permission.id`
- `django_content_type.id`
- `django_migrations.id`
- `otp_static_staticdevice.id`
- `otp_static_statictoken.id`
- `otp_totp_totpdevice.id`
- `two_factor_phonedevice.id`
- `zerver_archivedmessage.id`
- `zerver_client.id`
- `zerver_message.id`
- `zerver_realm.id`
- `zerver_recipient.id`
- `zerver_userprofile.id`
[^1]: https://code.djangoproject.com/ticket/32674
[^2]: https://code.djangoproject.com/ticket/24203
These models have no external references, and are among the larger
tables. Migrate them to bigints, starting with `useractivityinterval`
which is less likely to be being touched.
This index is used by `active_users_audit:is_bot:day`, and provides
roughly a 2x speedup. The existing
`zerver_realmauditlog_realm__event_type__event_time` is used if there
is a realm limit, but the standard statistics fill runs for all realms
at once, and thus cannot use it.
Migration plan:
1. Add NULLable .last_update_id column to UserPresence with default 0
for new objects.
2. Backfill the value to 0 for old UserPresences, can be done in the
background while server is running.
3. Make the column non-NULL.
4. Add new model PresenceSequence and create its rows for old realms.
Previously, when the operand of id operator was more than
2147483647, it was raising server error. This is because the
maximum permissible PostgreSQL integers value is 2147483647.
This is fixed by raising a BadNarrowOperatorError in case the
id operand is larger than 2147483647.
Removes the 'onboarding_steps' field on 'UserProfile'
model which is no longer used.
It was introduced back in 2013 (b5e22bf), and is no
longer used.
Fixes part of #30043.
This commit updates code, majorly in tests, to use
setting values from enums instead of directly using
the constants defined in Realm.
We still have those constants defined Realm as they
are used in a couple of places where the same code
is used for different settings. These will be
handled later.
This commit adds include_realm_default_subscriptions parameter
to the invite endpoints and the corresponding field in
PreregistrationUser and MultiuseInvite objects. This field will
be used to subscribe the new users to the default streams at the
time of account creation and not to the streams that were default
when sending the invite.
Mark the channel name of the initial channel created during
realm creation for translation.
It doesn't mark the topic names and description for translation
because we are planning to remove these topics and update the
description as a part of improving the onboarding experience.
We no longer create the 'core team' private channel when
a realm is created.
Earlier, "New user announcements" channel was set to the
"core team" channel. Now it is disabled by default.
populate_db still creates the 'core team' channel to
represent a private channel.
Fixes#29632.
The issue description explains this well:
We currently recalculate `currently_used_upload_space_bytes` every file
upload, by dint of calling `flush_used_upload_space_cache` on
save/delete, and then immediately calling
`user_profile.realm.currently_used_upload_space_bytes()` in
`notify_attachment_update`. Since this walks the Attachments table,
recalculating this can take seconds in large realms.
Switch this to using a CountStat, so we don't need to walk significant
chunks of the Attachment table when we upload an attachment. This will
also give us a historical daily graph of usage.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.
This commit changes just the model property, which has no API
compatibility concerns.
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email. Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation. That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.
Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites. The volume of
invites is small enough that this will not affect other uses of that
worker.
Fixes: #21306Fixes: #24275
This commit removes name, description, is_system_group and
can_mention_group fields from UserGroup model and rename
them in NamedUserGroup model.
Fixes#29554.
This commit updates code to access name from named_user_group
field which points to the "NamedUserGroup" instead of directly
accessing name from "UserGroup", since name field will only
be present on NamedUserGroup objects in further commits.
In zerver/models/streams.py, updates translated string for when everyone
can post to a stream/channel to use channel instead of stream.
Part of stream to channel rename project.
In zerver/models/realms.py, updates the topic for stream/channel updates
to be "channel events" instead of "stream events".
Part of stream to channel rename project.
Adds nullable creator field, containing a reference to the user who
created the stream. When creating a stream, acting user is set as
the creator of the stream. Since API calls to create streams always
have an acting user, this field should always be set when streams
are created using the API.
Because streams can be created with no acting user, this field is
nullable. We try to backfill existing streams using RealmAuditLog table,
but not all streams are guaranteed to have a recorded create log. Thus
this new field is left null when it cannot be backfilled. We also set
this field to null when the creator user is deleted.
Adds an index on RealmAuditLog for the realm, event_type, and
event_time in order to improve database queries on these audit logs.
tabbott verified using EXPLAIN ANALYZE that this also considerably
speeds up queries that order by ID rather than event_time, but
event_time is how these should be ordered given the possibility of
backfills.
This commit adds an option to the advanced section of
Preferences settings, that would allow users to choose
whether to receive typing notifications from other
users.
Fixes#29642
For simiplicty's sake, we can avoid trying to do cache invalidation in
the variety of events that can cause the seat count to change - since
having an up to 1 day delay between users being added and the upload
limit going up is quite reasonable.
Fixes#28621
Till now, this was actually a flat 50GB despite what the /plans/ page
says and was adjusted flexibly when somebody asked for a higher limit.
This actually implements the advertised formula, but changing it to
5GB/user since that's a more reasonable limit.
Keeps the 50GB limit for sponsored Standard Free organizations and also
places it as the floor for the quota for paid orgs, to not lower this
for tiny orgs with less than 5 users.
Most importantly, fixes a bug where a realm with a custom
.upload_quota_gb value (set by changing it in the database via e.g.
manage.py shell) would end up having it lowered while upgrading their
plan via the do_change_realm_plan_type function, which used to just set
it to the value implied by the new plan without caring about whether
that isn't lower than the original limit.
The new approach is cleaner since we don't do db queries by
upload_quota_gb so it's nicer to just generate these dynamically, making
changes to our limit-per-plan rules much easier - skipping the need for
migrations.
For endpoints with a type parameter to indicate whether a message is
a direct or stream message, adds support for passing "channel" as a
value for stream messages.
Part of stream to channel rename project.
Previously, users were allowed to signup or change their names to
those which already existed in the realm.
This commit adds an Organization Permission, that shall enforce
users to use unique names while signing up or changing their
names. If a same or normalized full name is found in realm,
then a validation error is thrown.
Fixes#7830.
Calling `.select_related()` with no arguments joins through every
possible table, recursively. In this case, this currently produces a
query which joins through forty-three tables.
This is rather inefficient, particularly for what is a very common
call which should be very fast.
No callsite depends on having prefetched any joined table on the
object; drop all of the joins.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.
Fixes part of #28640.
Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
This commit adds a management command that will run regularly
as a cron job to send zulip updates to realms based on their
current and latest zulip_update_announcements_level.
For realms with:
* level = None: Send a group DM to admins notifying them about
this new feature & suggestion to set the stream accordingly.
* level = 0:
* If stream is still not configured, wait for a week
before setting their level to latest level. They will
miss updates until their configure the stream.
* If stream is configured, send updates.
* level > 0: Send one message/update per level & increase
the level by 1 till the latest level.
Fixes#28604.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.
The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
restricted due to requiring server-side configuration to be done by us
This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
This commit updates default for delete_own_message_policy
setting to "Everyone" as it is helpful to allow everyone
to delete their own messages in a new organization where
users might be using Zulip for the first time.
Mateusz Mandera
Alex Vandiver
Kislay Verma
roanster007
Anders Kaseorg
Prakhar Pratyush
Sahil Batra
Prakhar Pratyush
Vector73
Lauryn Menard
tnmkr
Aman Agrawal
Karl Stolley
John Lu