Commit Graph

47531 Commits

Author SHA1 Message Date
Tim Abbott 1c90c50ec8 docs: Document Transifex steps for new major releases. 2022-05-27 15:22:17 -07:00
Alya Abbott 15ae18fffa portico: Make minor updates to Senior Mobile Engineer job listing. 2022-05-27 14:56:24 -07:00
Zixuan James Li 44ecd66eae types: Better types for API fields.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:43:00 -07:00
Aman Agrawal e6e975b470 recent_topics: Don't restore filters for spectators.
This fixes the bug where spectators can have filters selected
in recent topics if a logged in user has selected filters in the
same browser.

Log in, select a filter and log out to reproduce this.
2022-05-27 14:40:11 -07:00
Zixuan James Li f1ef27cb85 password_reset: Soft reactivate upon password reset.
Fixes #22066

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:30:34 -07:00
Zixuan James Li 3e95b59f2e soft_deactivation: Add a helper for queuing soft_reactivation.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:28:52 -07:00
Anders Kaseorg bb6bd900cd response: Replace response.asynchronous attribute with new class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-27 14:27:34 -07:00
Alex Vandiver 134977b590 email: Attempt to suppress email auto-replies.
This applies a commonly-used, though non-RFC, header which suppresses
auto-replies to the message.  There is a small chance that this will
result in bad filters thinking the messages *from Zulip* are
themselves auto-replies, but this seems a small risk.

Fixes: #13193.
2022-05-26 18:04:12 -07:00
Alex Vandiver 1ea3f83c1e help: Clarify potentially confusing wording. 2022-05-26 15:55:55 -07:00
Aman Agrawal 3991e89f19 models: Add missing database index for open communities directory.
Enable database indexing of `want_advertise_in_communities_directory`
field of `Realm` model.
2022-05-26 11:47:24 -07:00
Tim Abbott 1b09e0895f i18n: Update translation data from Transifex.
This includes the new Mongolian translation.
2022-05-26 11:31:11 -07:00
Chris Bobbe abfbacbdf6
openapi: Clarify custom_profile_field.value can be null in update-user event.
This has always been true, but wasn't documented correctly.
2022-05-26 10:52:13 -07:00
Anders Kaseorg 98ed6248e3 apt-repos: Remove now-unneeded Ubuntu 21.10 repository on 22.04.
Followup to commit f8957863a2 (#22055).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-25 17:25:23 -07:00
Lauryn Menard aa5536fe30 api-docs: Update `/set-typing-status` endpoint documentation.
Adds Changes notes for feature level 58 where support was added
for stream messages for the `/set-typing-status` endpoint
parameters.

Updates formatting for references to the `type`
parameter in the descriptions of other endpoint parameters.

Improves readability of and updates links in the endpoint's main
description.
2022-05-24 16:02:30 -07:00
Mateusz Mandera 4e1befa209 sessions: Rename get_session_user to get_session_user_id.
The function returns a user id, not a user - we generally try to reflect
that in function names accurately.
2022-05-23 15:57:10 -07:00
Mateusz Mandera 66d8c2ae15 sessions: Don't filter by expire_date in delete_realm_user_sessions.
The other delete_... functions just do .all() - we should be consistent
one way or another.
2022-05-23 15:57:10 -07:00
Mateusz Mandera 003d086f2e sessions: Only fetch user ids in delete_realm_user_sessions.
No need to fetch entire UserProfiles, especially since we're dealing
with all UserProfiles in a Realm here, which would be quite large.
2022-05-23 15:57:09 -07:00
Tim Abbott 0275358da8 openapi: Improve documentation of new_email field in events. 2022-05-23 15:53:05 -07:00
Anders Kaseorg 8b1e7a8b3e github: Accept null for issue and pull request body.
https://github.com/octokit/webhooks/blob/v5.6.0/payload-schemas/api.github.com/common/issue.schema.json#L84
https://github.com/octokit/webhooks/blob/v5.6.0/payload-schemas/api.github.com/common/pull-request.schema.json#L78

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-23 15:52:24 -07:00
Lauryn Menard 7ceb5f6311 api-docs: Add changelog for `stream_id` param to `mute-topic`.
Adds a changelog 2.0 entry for adding support for `stream_id`
parameter to the `mute-topic` endpoint. Also, adds Changes note
to the endpoint parameter description, and reorders/clarifies
that at least one (and only one) stream parameter must be provided
by the client and that the `string_id` parameter is preferred.

Fixes #11136.
2022-05-23 15:48:06 -07:00
Sahil Batra ce34b585a5 streams: Add endpoint "GET /streams/{stream_id}" to get stream by id.
Fixes #22082.
2022-05-23 15:14:04 -07:00
Sahil Batra 1c0ece73f2 version: Update API_FEATURE_LEVEL.
This was missed in the original commit dfd7902c7.
2022-05-23 15:08:07 -07:00
Alex Vandiver 6337f17923 upgrade: Add --skip-restart which preps but does not restart.
This adds a --skip-restart which makes `deployments/next` in a state
where it can be restarted into, but holds off on conducting that
restart.

This requires many of the same guarantees as `--skip-tornado`, in
terms of there being no Puppet or database schema changes between the
versions.  Enforce those with `--skip-restart`, and also broaden both
flags to prevent other, less common changes which nonetheless
potentially might affect the other deploy.
2022-05-22 15:07:37 -07:00
Alex Vandiver 86a4e64726 upgrade: Enforce that --skip-tornado does not have Puppet or DB changes. 2022-05-22 15:07:18 -07:00
Alex Vandiver ef7c2ea0ea upgrade: Copy cache prefix with --skip-tornado.
Because Tornado and Django use memcached as a shared cache for
checking session information, they must agree on the prefix used to
store those values.

Subsequent commits will work to ensure that it is always _safe_ to
share that cache.
2022-05-22 14:52:38 -07:00
Alex Vandiver fa77be6e6c upgrade: Only run Django system checks once, explicitly.
These are expensive, and moving them to one explicit call early has
considerable time savings in the critical period:

```
$ hyperfine './manage.py fill_memcached_caches' './manage.py fill_memcached_caches --skip-checks'
Benchmark #1: ./manage.py fill_memcached_caches
  Time (mean ± σ):      5.264 s ±  0.146 s    [User: 4.885 s, System: 0.344 s]
  Range (min … max):    5.119 s …  5.569 s    10 runs

Benchmark #2: ./manage.py fill_memcached_caches --skip-checks
  Time (mean ± σ):      3.090 s ±  0.089 s    [User: 2.853 s, System: 0.214 s]
  Range (min … max):    2.950 s …  3.204 s    10 runs

Summary
  './manage.py fill_memcached_caches --skip-checks' ran
    1.70 ± 0.07 times faster than './manage.py fill_memcached_caches'
```
2022-05-22 14:52:38 -07:00
Alex Vandiver 3928606886 restart-server: Treat as a start if nothing is running.
Treating the restart as a start is important in reducing the critical
period during upgrades -- we call restart even when we suspect the
services are stopped, because puppet has a small possibility of
placing them in indeterminate state.  However, restart orders the
workers first, then tornado/django, which prolongs the outage.

Recognize when no services are currently started, and switch to acting
like a start, not a restart, which places tornado/django first.
2022-05-22 14:52:38 -07:00
Alex Vandiver 3717c329b8 stop-server: Only stop services if they exist and are running.
This hides ugly output if the services were already stopped:

```
2022-03-25 23:26:04,165 upgrade-zulip-stage-2: Stopping Zulip...
process-fts-updates: ERROR (not running)
zulip-django: ERROR (not running)
zulip_deliver_scheduled_emails: ERROR (not running)
zulip_deliver_scheduled_messages: ERROR (not running)

Zulip stopped successfully!
```

Being able to skip having to shell out to `supervisorctl`, if all
services are already stopped is also a significant performance
improvement.
2022-05-22 14:52:38 -07:00
Alex Vandiver 2e5a079ef4 upgrade: Check with zulip-puppet-apply to see if we can skip it. 2022-05-22 14:52:38 -07:00
Alex Vandiver ecfc23bd0b zulip-puppet-apply: Make --force --noop have an exit code. 2022-05-22 14:52:38 -07:00
Alex Vandiver c91725bfb5 zulip-puppet-apply: Factor out the --noop returncode logic. 2022-05-22 14:52:38 -07:00
Alex Vandiver b15d8e0118 upgrade: Skip the pre-work if the server is already stopped.
This optimization makes sense if the server is already running, but if
it is already stopped, it is just prolonging the downtime.
2022-05-22 14:52:38 -07:00
Alex Vandiver 05af4b0a11 upgrade: Fill caches before the critical period, if possible. 2022-05-22 14:52:38 -07:00
Alex Vandiver 3d66dd9eeb fill_memcached_caches: Document possible arguments to --cache. 2022-05-22 14:52:38 -07:00
Alex Vandiver 2f7068ffbb upgrade: Move puppet class renames earlier.
These do not need to happen during the critical period when the server
is stopped.
2022-05-22 14:52:38 -07:00
Alex Vandiver a201e3b25b puppet: Upgrade wal-g to 2.0.0. 2022-05-22 14:51:18 -07:00
Alex Vandiver c8ee53619d puppet: Upgrade go and smokescreen. 2022-05-22 14:51:18 -07:00
Alex Vandiver 4a5e530743 puppet: Upgrade Grafana to 8.5.3, for CVE-2022-29170. 2022-05-22 14:51:18 -07:00
Anders Kaseorg b0592ade63 requirements: Remove importlib-resources.
It’s only used by jsonschema >= 4.2.0, but current semgrep holds
jsonschema ~= 3.2:
https://github.com/returntocorp/semgrep/issues/4739

Not bothering to bump PROVISION_VERSION because it’s not important
whether this backport is installed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-19 20:16:44 -07:00
Lauryn Menard 2d85612ff0 api-docs: Add `create_web_public_stream_policy` to realm update.
Adds `create_web_public_stream_policy` to the `get-events` API
documentation for the `realm op:update` event.

Also, fixes changelog entries for feature levels 103 and 104,
which are related to the API documentation changes or fix an
error in references to the undocumented endpoint `PATCH /realm`.
2022-05-19 13:37:40 -07:00
Alex Vandiver fb338f22d7 ci: Stop rabbitmq when making CI images.
The production CI image starts `rabbitmq-server` but does not stop it,
which leaves a stale `/var/run/rabbitmq/pid` file in the image.

`rabbitmqctl wait --timeout 600 /var/run/rabbitmq/pid`, which is run
after starting the rabbitmq node, reads the PID file and waits for the
PID to be running, and for rabbitmq's port to be responding to pings.

If it reads an old PID file before the new PID is written, it
aborts (all but the first and last lines are output from `rabbitmqctl
wait` that is hidden by `/etc/init.d/rabbitmq-server`):

```
 * Starting RabbitMQ Messaging Server rabbitmq-server
Waiting for pid file '/var/run/rabbitmq/pid' to appear
pid is 341
Waiting for erlang distribution on node 'rabbit@fc8f64d6acdb' while OS process '341' is running
Error:
process_not_running
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
 ```

If it failed, the `production-upgrade` script tried to start
`rabbitmq` again -- despite it already still starting in the
background.  These two attempts conflicted, and often one or both
failed.

Stop `rabbitmq-server` when building the image, which removes the
stale PID file.
2022-05-19 13:30:59 -07:00
Sahil Batra 95a2b580a0 streams: Save one DB query in bulk_remove_susbcriptions.
We remove one call to get_occupied_streams to get occupied
streams before unsubscribing because we already know which
streams can become vacant, i.e. the one from which users are
being unsubscribed, and we can directly use the list of streams
from which users are being unsubscribed and get vacant streams
by checking which of these streams are not in get_occupied_streams
called after unsubscribing users.
2022-05-17 15:09:15 -07:00
Anders Kaseorg f8957863a2 Revert "apt-repos: Downgrade PostgreSQL to dodge PGroonga regression."
This reverts commit 9c8d2b7be3 (#21115).

The PostgreSQL fix was released 2022-05-12.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-17 15:07:37 -07:00
Alex Vandiver baed1214f2 puppet: Only fix certbot certificates if https is enabled.
This is a reprise of c97162e485, but for the case where certbot
certs are no longer in use by way of enabling `http_only` and letting
another server handle TLS termination.

Fixes: #22034.
2022-05-17 15:03:44 -07:00
Alex Vandiver 62f234328d puppet: Include the OS-enabled nginx module configurations.
This allows system-level configuration to be done by `apt-get install`
of nginx modules, which place their load statements in this directory.

The initial import in ed0cb0a5f8 of the stock nginx config omitted
this include -- one potential explanation was in an effort to reduce
the memory footprint of the server.

The default nginx install enables:

    50-mod-http-auth-pam.conf
    50-mod-http-dav-ext.conf
    50-mod-http-echo.conf
    50-mod-http-geoip2.conf
    50-mod-http-geoip.conf
    50-mod-http-image-filter.conf
    50-mod-http-subs-filter.conf
    50-mod-http-upstream-fair.conf
    50-mod-http-xslt-filter.conf
    50-mod-mail.conf
    50-mod-stream.conf

While Zulip doesn't actively use any of these, they likely don't do
any harm to simply be loaded -- they are loaded into every nginx by
default.

Having the `modules-enabled` include allows easier extension of the
server, as neither of the existing wildcard
includes (`/etc/nginx/conf.d/*.conf` and
`/etc/nginx/zulip-include/app.d/*.conf`) are in the top context, and
thus able to load modules.
2022-05-17 15:03:07 -07:00
Alex Vandiver 21ec1f07d4 name_restrictions: Add your-org.zulipchat.com as a reserved name.
This is used as a placeholder in the mobile app.
2022-05-17 14:58:31 -07:00
Sahil Batra f38ab85f02 user_groups: Pass user group object instead of id to get_recursive_subgroups.
We directly pass the user group object to get_recursive_subgroups as we
already have the object in the caller. We can add separate function which
will accept id as parameter in the future if required.
2022-05-17 14:51:55 -07:00
Sahil Batra 6b82cbe0a6 user_groups: Rename existing_subgroups variable to existing_direct_subgroup_ids.
This commit renames existing_subgroups variable to existing_direct_subgroup_ids
in add_subgroups_to_group_backend and remove_subgroups_from_group_backend functions
for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra 8edf30caf2 user_groups: Rename subgroups parameter to direct_subgroup_ids.
This commit renames subgroups parameter of do_send_create_user_group_event
to direct_subgroup_ids for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra dfd7902c77 user_groups: Rename subgroups fields to direct_subgroup_ids.
This commit renames subgroups and subgroup_ids field sent in user
group objects to direct_subgroup_ids for better readability.
2022-05-17 14:51:45 -07:00