Commit Graph

2500 Commits

Author SHA1 Message Date
Gaurav Pandey 1c5007461a topic: Add resolve topic undo grace period.
Currently we send a notification to the topic if it has been resolved
or unresolved even if there is an immediate event of resolving and
then unresolving or vice-versa. This adds a setting of
RESOLVE_TOPIC_UNDO_GRACE_PERIOD_SECONDS under which if a topic has
been unresolved after being resolved immediately and the last message
was the notification of resolving, then delete the last message and
don't send a new notification and vice-versa.

We use the new message.type field to precisely identify relevant
messages.

Fixes #19181.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2024-06-10 13:34:23 -07:00
Vector73 faa06497ed api_docs: Document "/invites/{invite_id}/resend" endpoint. 2024-06-10 11:48:39 -07:00
Alex Vandiver 50c3dd88e6 models: Migrate ids of all non-Message-related tables to bigint.
Migrate all `ids` of anything which does not have a foreign key from
the Message or UserMessage table (and would thus require walking
those) to be `bigint`.  This is done by removing explicit
`BigAutoField`s, trading them for explicit `AutoField`s on the tables
to not be migrated, while updating `DEFAULT_AUTO_FIELD` to the new
default.

In general, the tables adjusted in this commit are small tables -- at
least compared to Messages and UserMessages.

Many-to-many tables without their own model class are adjusted by a
custom Operation, since they do not automatically pick up migrations
when `DEFAULT_AUTO_FIELD` changes[^1].

Note that this does multiple scans over tables to update foreign
keys[^2].  Large installs may wish to hand-optimize this using the
output of `./manage.py sqlmigrate` to join multiple `ALTER TABLE`
statements into one, to speed up the migration.  This is unfortunately
not possible to do generically, as constraint names may differ between
installations.

This leaves the following primary keys as non-`bigint`:
- `auth_group.id`
- `auth_group_permissions.id`
- `auth_permission.id`
- `django_content_type.id`
- `django_migrations.id`
- `otp_static_staticdevice.id`
- `otp_static_statictoken.id`
- `otp_totp_totpdevice.id`
- `two_factor_phonedevice.id`
- `zerver_archivedmessage.id`
- `zerver_client.id`
- `zerver_message.id`
- `zerver_realm.id`
- `zerver_recipient.id`
- `zerver_userprofile.id`

[^1]: https://code.djangoproject.com/ticket/32674
[^2]: https://code.djangoproject.com/ticket/24203
2024-06-05 11:48:27 -07:00
Sahil Batra d3ea6520dc user_groups: Add server level setting disallow anonymous groups for settings.
This commit adds a server level setting which controls whether the setting
can be set to anonymous user groups. We only allow it in the tests for
now because the UI can only handle named user groups.
2024-06-03 09:45:26 -07:00
Alex Vandiver 5814583694 rabbitmq: Support non-/ vhosts.
Fixes: #30255.
2024-05-31 15:41:19 -07:00
Alex Vandiver f246b82f67 puppet: Factor out pattern of writing a nagios state file atomically. 2024-05-24 11:31:25 -07:00
Alex Vandiver 5c2fd1de5a docs: Update Django links to our current version. 2024-05-24 10:18:37 -07:00
Alex Vandiver 5f3d6ac983 sentry: More gracefully handle a value of 0.
If the value at runtime is actually an int, not a float, we should not
try to treat it as a dict.
2024-05-23 14:09:19 -07:00
Vector73 62dfd93a83 api: Add "users/<int:user_id>/status" endpoint.
The documentation Creates a shared UserStatus schema that's used for
the return value of this new endpoint and for the existing user_status
objects returned by the register queue endpoint.

Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>

Fixes #19079.
2024-05-22 18:07:22 -07:00
Prakhar Pratyush 643998fcfa onboarding_steps: Rename 'hotspots' files to 'onboarding_steps'. 2024-05-10 12:30:22 -07:00
Prakhar Pratyush bf2360bcf2 onboarding_steps: Remove hotspot as an onboarding_step.
Earlier, hotspots and one-time notices were the valid
type of onboarding step.

Now, one-time notice is the only valid type.

Fixes #29296.
2024-05-10 12:30:22 -07:00
Alex Vandiver 16a53d08fd default_settings: Set NAGIOS_BOT_HOST to most likely correct hostname.
Using only EXTERNAL_HOST results in:
```
CRITICAL: Error subscribing to Zulips: Account is not associated with this subdomain
```

Since system bots like the nagios bot are created in
`SYSTEM_BOT_REALM`, default to that subdomain of `EXTERNAL_HOST`.
2024-05-08 15:51:20 -07:00
Vector73 8ab526a25a models: Replace realm.uri with realm.url.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.

This commit changes just the model property, which has no API
compatibility concerns.
2024-05-08 11:12:43 -07:00
Tim Abbott 0a756c652c push_notifications: Shard mobile push notifications. 2024-05-02 14:25:10 -07:00
Alex Vandiver 4dd801eb71 settings: Fix a typo in a comment. 2024-04-30 12:37:02 -07:00
Sahil Batra 27558315a2 settings: Use named_user_group field to access name.
This commit updates code to access name from named_user_group
field which points to the "NamedUserGroup" instead of directly
accessing name from "UserGroup", since name field will only
be present on NamedUserGroup objects in further commits.
2024-04-26 17:03:09 -07:00
Sahil Batra 0ff9aacb96 audit_logs: Set modified_user_group field to NamedUserGroup. 2024-04-26 17:03:09 -07:00
Prakhar Pratyush c727b36e9c soft_reactivate: Soft reactivate if group mention has < 12 members.
Earlier, we didn't soft-reactivate users for group mentions
at all because it wasn't easy to calculate group size.

Now, we will soft reactivate if the user group mentions has
less than 12 members.

We don't reactivate all users because a user group can have a
very large size, which can lead to large backlogs in the
deferred-work queue.

Fixes part of #27586.
2024-04-24 17:40:37 -07:00
Alex Vandiver 11dd6791c4 management: Provide a common lockfile dir, and a decorator for it.
Factor out the repeated pattern of taking a lock, or immediately
aborting with a message if it cannot be acquired.  The exit code in
that situation is changed to be exit code 1, rather than the successful
0; we are likely missing new work since that process started.

We move the lockfiles to a common directory under `/srv/zulip-locks`
rather than muddy up `/home/zulip/deployments`.
2024-04-24 14:40:28 -07:00
Alex Vandiver 62cf6ab35c sentry: Support setting project_dsn in zulip.conf. 2024-04-24 14:40:28 -07:00
Vector73 0f97733687 api_documentation: Document "/invites/{prereg_id}" endpoint. 2024-04-20 18:12:04 -07:00
Tim Abbott 342a9bd5cd zulip_updates: Prevent racing with self using a lock. 2024-04-18 15:14:02 -07:00
Anders Kaseorg 3b4527843d requirements: Upgrade Django to 5.0.5.dev.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 12:39:47 -07:00
Anders Kaseorg 166c233b2a default_settings: Weaken CUSTOM_AUTHENTICATION_WRAPPER_FUNCTION type.
Fixes an import cycle that breaks mypy inference with django-stubs:

zproject.settings → zproject.configured_settings →
zproject.default_settings → zerver.models.users → django.conf →
zproject.settings

(The Django settings system is really not an appropriate place to put
functions.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-17 16:49:03 -07:00
Anders Kaseorg 73116f78a5 settings: Hide django_stubs_ext from mypy.
Fixes an import cycle that breaks mypy inference with django-stubs:

zproject.settings → django_stubs_ext → django_stubs_ext.patch →
django.contrib.admin.options → django.contrib.contenttypes.models →
confirmation.models → django.conf → zproject.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-17 16:49:03 -07:00
Alex Vandiver 5654d051f7 worker: Split into separate files.
This makes each worker faster to start up.
2024-04-16 23:00:02 -07:00
Alex Vandiver 1424a2e748 documentation: Move OpenGraph description updating out of middleware.
This middleware was highly-specific to a set of URLs, and pulled in a
beautifulsoup dependency for Tornado.  Move it closer to where it is
used, minimizing action at a distance, as well as trimming out a
dependency.
2024-04-16 10:41:45 -07:00
Mateusz Mandera 066de96a86 uploads: Implement 5GB/user quota for paid orgs on Zulip Cloud.
Fixes #28621

Till now, this was actually a flat 50GB despite what the /plans/ page
says and was adjusted flexibly when somebody asked for a higher limit.

This actually implements the advertised formula, but changing it to
5GB/user since that's a more reasonable limit.

Keeps the 50GB limit for sponsored Standard Free organizations and also
places it as the floor for the quota for paid orgs, to not lower this
for tiny orgs with less than 5 users.
2024-04-15 15:08:56 -07:00
Mateusz Mandera c9ca4e68e5 scim: Add config option to disable initial streams for guests.
When an organization (without open ability for anyone to join) invites a
guest user, the invitation prompts allows them to choose whether the
guest should be added to default streams or not. This is useful, because
since we don't have per-role default streams configs, they may want
default streams to be for full Members.

SCIM provisioning doesn't have this control, since a newly provisioned
user gets created via a direct do_create_user call, thus adding them to
the organization's default streams, with no workaround possible aside of
just getting rid of default streams in the organization.

To make provisioning guests in such an organization usable, we add a
simple config option to create them with no streams. It's configured by
adding
```
"create_guests_without_streams": True
```

to the config dict in settings.SCIM_CONFIG.
2024-04-11 12:28:26 -07:00
Alex Vandiver bd82c6edf9 settings: Support multiple database replicas in REMOTE_POSTGRES_HOST.
The libpq client library, used under the hood by psycopg2, supports
passing a list of hosts; they are tried sequentially until one of them
works[^1].

In cases where this is used, it is often the case that the other
servers are read-only hot spare replicas.  Since Zulip does not expect
to be in a read-only transaction, we require that the server that we
connect to be writable, by passing `target_session_attrs`[^2].

To limit how long we may block connecting to a potentially bad host
before moving on, we set `connection_timeout` from null (meaning
forever) to 2 (the lowest supported value)[^3], so we move on quickly
in the case that the server is running but unable to handle new
connections.

[^1]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-MULTIPLE-HOSTS
[^2]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS
[^3]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-CONNECT-TIMEOUT
2024-04-04 16:48:58 -07:00
Anders Kaseorg d8ebb2db95 auth: Avoid deprecated django.contrib.auth.views.logout_then_login.
It’s removed in Django 5.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 16:27:58 -07:00
Anders Kaseorg ac5161f439 tests: Downgrade deprecated SHA1PasswordHasher to MD5PasswordHasher.
SHA1PasswordHasher will be removed in Django 5.1.  MD5PasswordHasher
will remain for exactly this purpose of speeding up tests.

Use MD5PasswordHasher by default, but leave SHA1PasswordHasher in the
list for compatibility with test databases that have already been
generated.  Once some other change forces test databases to be
rebuilt, we can remove SHA1PasswordHasher.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 12:43:54 -07:00
roanster007 c7a08f3b77 settings: Add permission to enforce unique names in realm.
Previously, users were allowed to signup or change their names to
those which already existed in the realm.

This commit adds an Organization Permission, that shall enforce
users to use unique names while signing up or changing their
names. If a same or normalized full name is found in realm,
then a validation error is thrown.

Fixes #7830.
2024-04-02 14:55:59 -07:00
Anders Kaseorg 807a4428f6 compose_validate: Remove autosubscribe feature.
This was only used in the undocumented narrow_stream mode, and relied
on a deprecated synchronous XHR request.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-28 09:06:48 -07:00
Alex Vandiver 9451d08bb9 worker: Split out worker sampling rate, and add Sentry transactions. 2024-03-21 12:46:13 -07:00
Alex Vandiver f64b9475c1 sentry: Optionally enable tracing, with transactions and spans. 2024-03-21 12:46:13 -07:00
Alex Vandiver 6ad777c86f dev_urls: Make requests to /docs redirect to the index.
This makes it easier to browse to the docs in development mode, and
more closely matches ReadTheDocs production.
2024-03-21 11:27:14 -07:00
Anders Kaseorg 71e1b3c91b sentry: Fix type error in add_context.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver 1cba83f9c5 settings: Use bool-ish get_config on postgresql.missing_dictionaries. 2024-03-15 15:34:12 -07:00
Alex Vandiver 0f70ab7679 config: Use logic from zulip_tools, to support bools. 2024-03-15 15:34:12 -07:00
Mateusz Mandera e39f400f94 remote_billing: Make "plan management" always available.
Just shows a config error page if the bouncer is not enabled. Uses a new
endpoint for this so that it can work nicely for both browser and
desktop app clients.
It's necessary, because the desktop app expects to get a json response
with either an error or billing_access_url to redirect to. Showing a
nice config error page can't be done via the json error mechanism, so
instead we just serve a redirect to the new error page, which the app
will open in the browser in a new window or tab.
2024-03-05 11:53:51 -08:00
Mateusz Mandera da9e4e6e54 backends: Implementation of restricting certain backends by plan.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.

The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
  restricted due to requiring server-side configuration to be done by us

This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
2024-03-05 11:48:58 -08:00
Mateusz Mandera fdbdf8c620 backends: Add ZulipAuthMixin inheritance to ZulipRemoteUserBackend.
By convention, all our backends inherit form ZulipAuthMixin. This is
helpful to have a single place if we want to add some class attributes
that all the backends should have.
2024-03-05 11:48:58 -08:00
Mateusz Mandera 634015411a update_analytics_count: Use a correct lock mechanism.
Adds a re-usable lockfile_nonblocking helper to context_managers.

Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.

We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.
2024-03-05 10:21:14 -08:00
Anders Kaseorg 869d9d9a79 ruff: Fix RUF025 Unnecessary dict comprehension for iterable.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Alex Vandiver da6b0b1cc6 tornado: Add a web_reload_clients endpoint to notify web clients. 2024-02-15 15:42:50 -08:00
Alex Vandiver e6fa1f5436 email_mirror_message: Move endpoint under /api/internal/.
This also has the side effect that it saves the endpint from being
improperly redirected by the HostDomainMiddleware's canonicalization
code.
2024-02-15 15:42:50 -08:00
Alex Vandiver 84fa9be73a tornado: Move the /notify-tornado endpoint, and document it better. 2024-02-15 15:42:50 -08:00
Alex Vandiver b23d90ed62 puppet: Rename puppet/zulip_ops to puppet/kandra.
This makes for easier tab-completion, and also is a bit more explicit
about the expected consumer.
2024-02-06 17:56:27 -08:00