Commit Graph

2515 Commits

Author SHA1 Message Date
Alex Vandiver d121a80b78 upload: Serve thumbnailed images. 2024-07-16 13:22:15 -07:00
Anders Kaseorg 8843f9f62a tests: Remove deprecated SHA1PasswordHasher.
SHA1PasswordHasher will be removed in Django 5.1.  MD5PasswordHasher
will remain for the purpose of speeding up tests.

Followup to commit ac5161f439 (#29620).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 13:06:31 -07:00
Alex Vandiver bf9539386a katex: Default the server on.
The memory costs are low (~60MB), and likely worth the stability.
2024-07-15 13:16:31 -07:00
Anders Kaseorg 0fa5e7f629 ruff: Fix UP035 Import from `collections.abc`, `typing` instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 531b34cb4c ruff: Fix UP007 Use `X | Y` for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg e08a24e47f ruff: Fix UP006 Use `list` instead of `List` for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Alex Vandiver 49c0f7306e thumbnail: Increase logging level to ERROR and above.
Warnings are quite common when dealing with bogus image data.
2024-07-12 13:26:47 -07:00
Sahil Batra ffb7744974 settings: Allow settings to be set to system groups only in production.
We use the already existing server level setting to only allow
settings to be set to system groups, not a named user defined
group as well, in production. But we allow to settings to be set
to any named or anonymous user group in tests and development server.

"can_mention_group" setting can be set to user defined groups
because some of the realms already do that in production.

The existing server level setting is also renamed to make it clear
that both user defined groups and anonymous groups are not allowed
if that setting is set to False.

This commit also changes the error message to be consistent for the
case when a setting cannot be set to user defined groups as per
server level and setting and when a particular setting cannot be set
to user defined groups due to the configuration of that particular
setting. For this we add a new class SystemGroupRequiredError in
exceptions.py so that we need not re-write the error message in
multiple places.
2024-07-12 09:51:49 -07:00
Lauryn Menard 673a01ea0c realm-deactivation: Send email to owners as part of deactivation.
Creates a new "realm_deactivated" email that can be sent to realm
owners as part of `do_deactivate_realm`, via a boolean flag,
`email_owners`.

This flag is set to `False` when `do_deactivate_realm` is used for
realm exports or changing a realm's subdomain, so that the active
organization owners are not emailed in those cases.

This flag is optional for the `deactivate_realm` management command,
but as there is no active user passed in that case, then the email
is sent without referencing who deactivated the realm.

It is passed as `True` for the support analytics view, but the email
that is generated does not include information about the support
admin user who completed the request for organization deactivation.

When an active organization owner deactivates the organization, then
the flag is `True` and an email is sent to them as well as any other
active organization owners, with a slight variation in the email text
for those two cases.

Adds specific tests for when `email_owners` is passed as `True`. All
existing tests for other functionality of `do_deactivate_user` pass
the flag as `False`.

Adds `localize` from django.util.formats as a jinja env filter so
that the dates in these emails are internationlized for the owner's
default language setting in the "realm_deactivated" email templates.

Fixes #24685.
2024-06-26 16:48:18 -07:00
Alex Vandiver 5cd10ce51d s3: Allow setting a CloudFront URL prefix for avatar and emoji images. 2024-06-26 16:43:11 -07:00
Alex Vandiver 08b24484d1 upload: Remove redundant acting_user_profile argument.
This argument, effectively added in 9eb47f108c, was never actually
used.
2024-06-26 16:43:11 -07:00
Alex Vandiver 475d4800f9 thumbnailing: Add pyvips dependency. 2024-06-20 23:06:08 -04:00
Alex Vandiver b36ad31f0e backends: Early return in sync_avatar_from_ldap. 2024-06-20 23:06:08 -04:00
Prakhar Pratyush bfc8cdb8f6 urls: Update stale file name 'hotspots' to 'onboarding_steps'.
In 643998fcfa, we renamed the file name
so it was stale in the comment.
2024-06-18 09:39:50 -07:00
Mateusz Mandera 4917e01ffb push_notifications: Migrate to FCM HTTP v1 API.
The legacy API we use via python-gcm is deprecated and about to be
disabled.

Fixes #29768.
2024-06-17 18:26:26 -07:00
Gaurav Pandey 1c5007461a topic: Add resolve topic undo grace period.
Currently we send a notification to the topic if it has been resolved
or unresolved even if there is an immediate event of resolving and
then unresolving or vice-versa. This adds a setting of
RESOLVE_TOPIC_UNDO_GRACE_PERIOD_SECONDS under which if a topic has
been unresolved after being resolved immediately and the last message
was the notification of resolving, then delete the last message and
don't send a new notification and vice-versa.

We use the new message.type field to precisely identify relevant
messages.

Fixes #19181.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2024-06-10 13:34:23 -07:00
Vector73 faa06497ed api_docs: Document "/invites/{invite_id}/resend" endpoint. 2024-06-10 11:48:39 -07:00
Alex Vandiver 50c3dd88e6 models: Migrate ids of all non-Message-related tables to bigint.
Migrate all `ids` of anything which does not have a foreign key from
the Message or UserMessage table (and would thus require walking
those) to be `bigint`.  This is done by removing explicit
`BigAutoField`s, trading them for explicit `AutoField`s on the tables
to not be migrated, while updating `DEFAULT_AUTO_FIELD` to the new
default.

In general, the tables adjusted in this commit are small tables -- at
least compared to Messages and UserMessages.

Many-to-many tables without their own model class are adjusted by a
custom Operation, since they do not automatically pick up migrations
when `DEFAULT_AUTO_FIELD` changes[^1].

Note that this does multiple scans over tables to update foreign
keys[^2].  Large installs may wish to hand-optimize this using the
output of `./manage.py sqlmigrate` to join multiple `ALTER TABLE`
statements into one, to speed up the migration.  This is unfortunately
not possible to do generically, as constraint names may differ between
installations.

This leaves the following primary keys as non-`bigint`:
- `auth_group.id`
- `auth_group_permissions.id`
- `auth_permission.id`
- `django_content_type.id`
- `django_migrations.id`
- `otp_static_staticdevice.id`
- `otp_static_statictoken.id`
- `otp_totp_totpdevice.id`
- `two_factor_phonedevice.id`
- `zerver_archivedmessage.id`
- `zerver_client.id`
- `zerver_message.id`
- `zerver_realm.id`
- `zerver_recipient.id`
- `zerver_userprofile.id`

[^1]: https://code.djangoproject.com/ticket/32674
[^2]: https://code.djangoproject.com/ticket/24203
2024-06-05 11:48:27 -07:00
Sahil Batra d3ea6520dc user_groups: Add server level setting disallow anonymous groups for settings.
This commit adds a server level setting which controls whether the setting
can be set to anonymous user groups. We only allow it in the tests for
now because the UI can only handle named user groups.
2024-06-03 09:45:26 -07:00
Alex Vandiver 5814583694 rabbitmq: Support non-/ vhosts.
Fixes: #30255.
2024-05-31 15:41:19 -07:00
Alex Vandiver f246b82f67 puppet: Factor out pattern of writing a nagios state file atomically. 2024-05-24 11:31:25 -07:00
Alex Vandiver 5c2fd1de5a docs: Update Django links to our current version. 2024-05-24 10:18:37 -07:00
Alex Vandiver 5f3d6ac983 sentry: More gracefully handle a value of 0.
If the value at runtime is actually an int, not a float, we should not
try to treat it as a dict.
2024-05-23 14:09:19 -07:00
Vector73 62dfd93a83 api: Add "users/<int:user_id>/status" endpoint.
The documentation Creates a shared UserStatus schema that's used for
the return value of this new endpoint and for the existing user_status
objects returned by the register queue endpoint.

Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>

Fixes #19079.
2024-05-22 18:07:22 -07:00
Prakhar Pratyush 643998fcfa onboarding_steps: Rename 'hotspots' files to 'onboarding_steps'. 2024-05-10 12:30:22 -07:00
Prakhar Pratyush bf2360bcf2 onboarding_steps: Remove hotspot as an onboarding_step.
Earlier, hotspots and one-time notices were the valid
type of onboarding step.

Now, one-time notice is the only valid type.

Fixes #29296.
2024-05-10 12:30:22 -07:00
Alex Vandiver 16a53d08fd default_settings: Set NAGIOS_BOT_HOST to most likely correct hostname.
Using only EXTERNAL_HOST results in:
```
CRITICAL: Error subscribing to Zulips: Account is not associated with this subdomain
```

Since system bots like the nagios bot are created in
`SYSTEM_BOT_REALM`, default to that subdomain of `EXTERNAL_HOST`.
2024-05-08 15:51:20 -07:00
Vector73 8ab526a25a models: Replace realm.uri with realm.url.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.

This commit changes just the model property, which has no API
compatibility concerns.
2024-05-08 11:12:43 -07:00
Tim Abbott 0a756c652c push_notifications: Shard mobile push notifications. 2024-05-02 14:25:10 -07:00
Alex Vandiver 4dd801eb71 settings: Fix a typo in a comment. 2024-04-30 12:37:02 -07:00
Sahil Batra 27558315a2 settings: Use named_user_group field to access name.
This commit updates code to access name from named_user_group
field which points to the "NamedUserGroup" instead of directly
accessing name from "UserGroup", since name field will only
be present on NamedUserGroup objects in further commits.
2024-04-26 17:03:09 -07:00
Sahil Batra 0ff9aacb96 audit_logs: Set modified_user_group field to NamedUserGroup. 2024-04-26 17:03:09 -07:00
Prakhar Pratyush c727b36e9c soft_reactivate: Soft reactivate if group mention has < 12 members.
Earlier, we didn't soft-reactivate users for group mentions
at all because it wasn't easy to calculate group size.

Now, we will soft reactivate if the user group mentions has
less than 12 members.

We don't reactivate all users because a user group can have a
very large size, which can lead to large backlogs in the
deferred-work queue.

Fixes part of #27586.
2024-04-24 17:40:37 -07:00
Alex Vandiver 11dd6791c4 management: Provide a common lockfile dir, and a decorator for it.
Factor out the repeated pattern of taking a lock, or immediately
aborting with a message if it cannot be acquired.  The exit code in
that situation is changed to be exit code 1, rather than the successful
0; we are likely missing new work since that process started.

We move the lockfiles to a common directory under `/srv/zulip-locks`
rather than muddy up `/home/zulip/deployments`.
2024-04-24 14:40:28 -07:00
Alex Vandiver 62cf6ab35c sentry: Support setting project_dsn in zulip.conf. 2024-04-24 14:40:28 -07:00
Vector73 0f97733687 api_documentation: Document "/invites/{prereg_id}" endpoint. 2024-04-20 18:12:04 -07:00
Tim Abbott 342a9bd5cd zulip_updates: Prevent racing with self using a lock. 2024-04-18 15:14:02 -07:00
Anders Kaseorg 3b4527843d requirements: Upgrade Django to 5.0.5.dev.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 12:39:47 -07:00
Anders Kaseorg 166c233b2a default_settings: Weaken CUSTOM_AUTHENTICATION_WRAPPER_FUNCTION type.
Fixes an import cycle that breaks mypy inference with django-stubs:

zproject.settings → zproject.configured_settings →
zproject.default_settings → zerver.models.users → django.conf →
zproject.settings

(The Django settings system is really not an appropriate place to put
functions.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-17 16:49:03 -07:00
Anders Kaseorg 73116f78a5 settings: Hide django_stubs_ext from mypy.
Fixes an import cycle that breaks mypy inference with django-stubs:

zproject.settings → django_stubs_ext → django_stubs_ext.patch →
django.contrib.admin.options → django.contrib.contenttypes.models →
confirmation.models → django.conf → zproject.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-17 16:49:03 -07:00
Alex Vandiver 5654d051f7 worker: Split into separate files.
This makes each worker faster to start up.
2024-04-16 23:00:02 -07:00
Alex Vandiver 1424a2e748 documentation: Move OpenGraph description updating out of middleware.
This middleware was highly-specific to a set of URLs, and pulled in a
beautifulsoup dependency for Tornado.  Move it closer to where it is
used, minimizing action at a distance, as well as trimming out a
dependency.
2024-04-16 10:41:45 -07:00
Mateusz Mandera 066de96a86 uploads: Implement 5GB/user quota for paid orgs on Zulip Cloud.
Fixes #28621

Till now, this was actually a flat 50GB despite what the /plans/ page
says and was adjusted flexibly when somebody asked for a higher limit.

This actually implements the advertised formula, but changing it to
5GB/user since that's a more reasonable limit.

Keeps the 50GB limit for sponsored Standard Free organizations and also
places it as the floor for the quota for paid orgs, to not lower this
for tiny orgs with less than 5 users.
2024-04-15 15:08:56 -07:00
Mateusz Mandera c9ca4e68e5 scim: Add config option to disable initial streams for guests.
When an organization (without open ability for anyone to join) invites a
guest user, the invitation prompts allows them to choose whether the
guest should be added to default streams or not. This is useful, because
since we don't have per-role default streams configs, they may want
default streams to be for full Members.

SCIM provisioning doesn't have this control, since a newly provisioned
user gets created via a direct do_create_user call, thus adding them to
the organization's default streams, with no workaround possible aside of
just getting rid of default streams in the organization.

To make provisioning guests in such an organization usable, we add a
simple config option to create them with no streams. It's configured by
adding
```
"create_guests_without_streams": True
```

to the config dict in settings.SCIM_CONFIG.
2024-04-11 12:28:26 -07:00
Alex Vandiver bd82c6edf9 settings: Support multiple database replicas in REMOTE_POSTGRES_HOST.
The libpq client library, used under the hood by psycopg2, supports
passing a list of hosts; they are tried sequentially until one of them
works[^1].

In cases where this is used, it is often the case that the other
servers are read-only hot spare replicas.  Since Zulip does not expect
to be in a read-only transaction, we require that the server that we
connect to be writable, by passing `target_session_attrs`[^2].

To limit how long we may block connecting to a potentially bad host
before moving on, we set `connection_timeout` from null (meaning
forever) to 2 (the lowest supported value)[^3], so we move on quickly
in the case that the server is running but unable to handle new
connections.

[^1]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-MULTIPLE-HOSTS
[^2]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS
[^3]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-CONNECT-TIMEOUT
2024-04-04 16:48:58 -07:00
Anders Kaseorg d8ebb2db95 auth: Avoid deprecated django.contrib.auth.views.logout_then_login.
It’s removed in Django 5.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 16:27:58 -07:00
Anders Kaseorg ac5161f439 tests: Downgrade deprecated SHA1PasswordHasher to MD5PasswordHasher.
SHA1PasswordHasher will be removed in Django 5.1.  MD5PasswordHasher
will remain for exactly this purpose of speeding up tests.

Use MD5PasswordHasher by default, but leave SHA1PasswordHasher in the
list for compatibility with test databases that have already been
generated.  Once some other change forces test databases to be
rebuilt, we can remove SHA1PasswordHasher.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 12:43:54 -07:00
roanster007 c7a08f3b77 settings: Add permission to enforce unique names in realm.
Previously, users were allowed to signup or change their names to
those which already existed in the realm.

This commit adds an Organization Permission, that shall enforce
users to use unique names while signing up or changing their
names. If a same or normalized full name is found in realm,
then a validation error is thrown.

Fixes #7830.
2024-04-02 14:55:59 -07:00
Anders Kaseorg 807a4428f6 compose_validate: Remove autosubscribe feature.
This was only used in the undocumented narrow_stream mode, and relied
on a deprecated synchronous XHR request.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-28 09:06:48 -07:00
Alex Vandiver 9451d08bb9 worker: Split out worker sampling rate, and add Sentry transactions. 2024-03-21 12:46:13 -07:00