Commit Graph

47249 Commits

Author SHA1 Message Date
Alex Vandiver 05af4b0a11 upgrade: Fill caches before the critical period, if possible. 2022-05-22 14:52:38 -07:00
Alex Vandiver 3d66dd9eeb fill_memcached_caches: Document possible arguments to --cache. 2022-05-22 14:52:38 -07:00
Alex Vandiver 2f7068ffbb upgrade: Move puppet class renames earlier.
These do not need to happen during the critical period when the server
is stopped.
2022-05-22 14:52:38 -07:00
Alex Vandiver a201e3b25b puppet: Upgrade wal-g to 2.0.0. 2022-05-22 14:51:18 -07:00
Alex Vandiver c8ee53619d puppet: Upgrade go and smokescreen. 2022-05-22 14:51:18 -07:00
Alex Vandiver 4a5e530743 puppet: Upgrade Grafana to 8.5.3, for CVE-2022-29170. 2022-05-22 14:51:18 -07:00
Anders Kaseorg b0592ade63 requirements: Remove importlib-resources.
It’s only used by jsonschema >= 4.2.0, but current semgrep holds
jsonschema ~= 3.2:
https://github.com/returntocorp/semgrep/issues/4739

Not bothering to bump PROVISION_VERSION because it’s not important
whether this backport is installed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-19 20:16:44 -07:00
Lauryn Menard 2d85612ff0 api-docs: Add `create_web_public_stream_policy` to realm update.
Adds `create_web_public_stream_policy` to the `get-events` API
documentation for the `realm op:update` event.

Also, fixes changelog entries for feature levels 103 and 104,
which are related to the API documentation changes or fix an
error in references to the undocumented endpoint `PATCH /realm`.
2022-05-19 13:37:40 -07:00
Alex Vandiver fb338f22d7 ci: Stop rabbitmq when making CI images.
The production CI image starts `rabbitmq-server` but does not stop it,
which leaves a stale `/var/run/rabbitmq/pid` file in the image.

`rabbitmqctl wait --timeout 600 /var/run/rabbitmq/pid`, which is run
after starting the rabbitmq node, reads the PID file and waits for the
PID to be running, and for rabbitmq's port to be responding to pings.

If it reads an old PID file before the new PID is written, it
aborts (all but the first and last lines are output from `rabbitmqctl
wait` that is hidden by `/etc/init.d/rabbitmq-server`):

```
 * Starting RabbitMQ Messaging Server rabbitmq-server
Waiting for pid file '/var/run/rabbitmq/pid' to appear
pid is 341
Waiting for erlang distribution on node 'rabbit@fc8f64d6acdb' while OS process '341' is running
Error:
process_not_running
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
 ```

If it failed, the `production-upgrade` script tried to start
`rabbitmq` again -- despite it already still starting in the
background.  These two attempts conflicted, and often one or both
failed.

Stop `rabbitmq-server` when building the image, which removes the
stale PID file.
2022-05-19 13:30:59 -07:00
Sahil Batra 95a2b580a0 streams: Save one DB query in bulk_remove_susbcriptions.
We remove one call to get_occupied_streams to get occupied
streams before unsubscribing because we already know which
streams can become vacant, i.e. the one from which users are
being unsubscribed, and we can directly use the list of streams
from which users are being unsubscribed and get vacant streams
by checking which of these streams are not in get_occupied_streams
called after unsubscribing users.
2022-05-17 15:09:15 -07:00
Anders Kaseorg f8957863a2 Revert "apt-repos: Downgrade PostgreSQL to dodge PGroonga regression."
This reverts commit 9c8d2b7be3 (#21115).

The PostgreSQL fix was released 2022-05-12.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-17 15:07:37 -07:00
Alex Vandiver baed1214f2 puppet: Only fix certbot certificates if https is enabled.
This is a reprise of c97162e485, but for the case where certbot
certs are no longer in use by way of enabling `http_only` and letting
another server handle TLS termination.

Fixes: #22034.
2022-05-17 15:03:44 -07:00
Alex Vandiver 62f234328d puppet: Include the OS-enabled nginx module configurations.
This allows system-level configuration to be done by `apt-get install`
of nginx modules, which place their load statements in this directory.

The initial import in ed0cb0a5f8 of the stock nginx config omitted
this include -- one potential explanation was in an effort to reduce
the memory footprint of the server.

The default nginx install enables:

    50-mod-http-auth-pam.conf
    50-mod-http-dav-ext.conf
    50-mod-http-echo.conf
    50-mod-http-geoip2.conf
    50-mod-http-geoip.conf
    50-mod-http-image-filter.conf
    50-mod-http-subs-filter.conf
    50-mod-http-upstream-fair.conf
    50-mod-http-xslt-filter.conf
    50-mod-mail.conf
    50-mod-stream.conf

While Zulip doesn't actively use any of these, they likely don't do
any harm to simply be loaded -- they are loaded into every nginx by
default.

Having the `modules-enabled` include allows easier extension of the
server, as neither of the existing wildcard
includes (`/etc/nginx/conf.d/*.conf` and
`/etc/nginx/zulip-include/app.d/*.conf`) are in the top context, and
thus able to load modules.
2022-05-17 15:03:07 -07:00
Alex Vandiver 21ec1f07d4 name_restrictions: Add your-org.zulipchat.com as a reserved name.
This is used as a placeholder in the mobile app.
2022-05-17 14:58:31 -07:00
Sahil Batra f38ab85f02 user_groups: Pass user group object instead of id to get_recursive_subgroups.
We directly pass the user group object to get_recursive_subgroups as we
already have the object in the caller. We can add separate function which
will accept id as parameter in the future if required.
2022-05-17 14:51:55 -07:00
Sahil Batra 6b82cbe0a6 user_groups: Rename existing_subgroups variable to existing_direct_subgroup_ids.
This commit renames existing_subgroups variable to existing_direct_subgroup_ids
in add_subgroups_to_group_backend and remove_subgroups_from_group_backend functions
for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra 8edf30caf2 user_groups: Rename subgroups parameter to direct_subgroup_ids.
This commit renames subgroups parameter of do_send_create_user_group_event
to direct_subgroup_ids for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra dfd7902c77 user_groups: Rename subgroups fields to direct_subgroup_ids.
This commit renames subgroups and subgroup_ids field sent in user
group objects to direct_subgroup_ids for better readability.
2022-05-17 14:51:45 -07:00
Adam Sah 04ccd8c6d8 testing: 100% code coverage for zerver/lib/outgoing_webhook.py. 2022-05-17 13:51:00 -07:00
Alex Vandiver c93024cd5b oneclick: Fail if the fab command fails. 2022-05-17 13:41:12 -07:00
Alex Vandiver ff647dff03 oneclick: Do not use a stale Zulip client.
Initializing the Zulip client opens a long-lived TCP connection due to
connection pooling in urllib3.  In Github Actions, the network kills
such requests after ~270s, making the later `send_message` call fail.

Use a singular call to `zulip.Client()` early on to verify the
credentials, and do not cache the resulting client object.  Instead,
re-create it during the final step when it is needed, so we do not run
afoul of bad TCP connection state.

This would ideally be fixed via connection keepalive or retry at the
level of the Zulip module.
2022-05-17 13:41:12 -07:00
Chris Bobbe c341414c60 help docs: Fix a wrong link in create-a-stream doc
Discussion:
  https://chat.zulip.org/#narrow/stream/19-documentation/topic/Wrong.20link.20.28stream.20creation.29/near/1380786
2022-05-16 16:44:19 -07:00
Adam Sah 534754442a test-backend: Improve performance by disabling XML report.
We don't use the XML report ourselves. We add options to make this
easy to control if specific circumstances indicate doing so.
2022-05-16 16:38:02 -07:00
Adam Sah 492272d597 test-backend: Display test function in HTML coverage reports. 2022-05-16 16:38:02 -07:00
Alex Vandiver 814841c9ec puppet: Remove typo'd cron job.
54b6a83412 fixed the typo introduced in 49ad188449, but that does
not clean up existing installs which had the file with the wrong name
already.

Remove the file with the typo'd name, so two jobs do not race, and fix
the typo in the comment.
2022-05-16 14:57:21 -07:00
Alex Vandiver 513fcb7bd5 Revert "settings: Add web-public streams beta subdomain list."
This reverts commit 20368a936c.  It is
no longer in beta, and this configuration is no longer needed.
2022-05-16 14:49:14 -07:00
Alex Vandiver a6d1c41827 email_mirror: Handle invalid `charset="..."` values. 2022-05-16 12:04:38 -07:00
Anders Kaseorg 0043c0b6b2 django: Use HttpRequest.headers.
Fixes #14769.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d98e3ecb0d tests: Don’t reuse HttpRequest objects.
Django caches some information on HttpRequest objects, including the
headers dict, under the assumption that requests won’t be reused.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d3c6ca8def development: Redirect rather than mutate in register_development_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg 110f7a379a beanstalk: Move %40 kludge into authenticated_rest_api_view.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Alex Vandiver 803982e872 message_flags: Short-circuit if no messages changed.
Omit sending an event, and updating the database, if there are no
matching messages.
2022-05-12 21:57:55 -07:00
Alex Vandiver 20b7a2d450 puppet: Each worker should chdir after forking.
The top-level `chdir` setting only does the chdir once, at initial
`uwsgi` startup time.  Rolling restarts, however, however, require
that `uwsgi` pick up the _new_ value of the `current` directory, and
start new workers in that directory -- as currently implemented,
rolling restarts cannot restart into newer versions of the code, only
the same one in which they were started.

Use [configurable hooks][1] to execute the `chdir` after every fork.
This causes the following behaviour:

```
Thu May 12 18:56:55 2022 - chain reload starting...
Thu May 12 18:56:55 2022 - chain next victim is worker 1
Gracefully killing worker 1 (pid: 1757689)...
worker 1 killed successfully (pid: 1757689)
Respawned uWSGI worker 1 (new pid: 1757969)
Thu May 12 18:56:56 2022 - chain is still waiting for worker 1...
running "chdir:/home/zulip/deployments/current" (post-fork)...
Thu May 12 18:56:57 2022 - chain is still waiting for worker 1...
Thu May 12 18:56:58 2022 - chain is still waiting for worker 1...
Thu May 12 18:56:59 2022 - chain is still waiting for worker 1...
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55dfca409170 pid: 1757969 (default app)
Thu May 12 18:57:00 2022 - chain next victim is worker 2
[...]
```

..and so forth down the line of processes.  Each process is correctly
started in the _current_ value of `current`, and thus picks up the
correct code.

[1]: https://uwsgi-docs.readthedocs.io/en/latest/Hooks.html
2022-05-12 21:54:02 -07:00
Sahil Batra 0bfe973b65 streams: Mark messages as read only for unsubscribed streams.
Previously, we were marking messages of all the streams passed
to bulk_remove_subscriptions even if user was not subscribed
to some of them and those streams would ideally not have
any unread messages. This code was added in 766511e519.

This commit changes the code to only mark messages of actually
unsubscribed streams as read.
2022-05-12 18:53:45 -07:00
Anders Kaseorg 1e453633b0 gitlab: Remove function signature inspection nonsense.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg f05fa9937b gci: Fix get_body_based_on_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg 4c772989a1 front: Fix get_body_based_on_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg 6d1b68c61b freshdesk: Remove unsafe TicketDict class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:56:46 -07:00
Anders Kaseorg cce142c61a middleware: Fix URL encoding of next parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:51:51 -07:00
Alex Vandiver 6bd7aac152 docs: Document the ./scripts/log-search tool. 2022-05-10 14:31:54 -07:00
Alex Vandiver 04d4ae9862 docs: Clarify nginx extension points. 2022-05-10 14:31:01 -07:00
Tim Abbott 86137dc02f users: clarify confusing check for last owner.
Conceptually, we're clearly intending to check whether the user we're
mutating is the last realm owner. The preexisting code was safe
because we've already checked that the target user is an owner, and
thus if we're the last owner, we're the target user.
2022-05-10 14:09:57 -07:00
somesh202 6ef8cdb380 org_settings: Add main feature to change bot role.
This commit attempts to refactor the `handle_bot_form` by adding new
field for `Role` in the `Manage bot` modal.

It uses the `/json/users/` url for passing the role of a bot and allow
changing it as in case of a normal user.

Fixes: #21105
2022-05-10 14:09:57 -07:00
Somesh Ranjan 0b1f8c05e3 org_settings: Add backend support to change bot role.
This commit attempts to add the backend support by extending the
/json/bots/{bot_id}/ url support to accept the role field as a
parameter. This was previously already possible via
`/json/users/{user_id}`, so this change just simplifies client
implementation.
2022-05-10 14:09:57 -07:00
somesh202 0b92e4c22f org_settings: Fix formatting and field ordering for bot modal.
This commit fixes the formatting of the "Manage bot" modal and re-orders
the field labels as per the bot-list table.
2022-05-10 14:09:57 -07:00
somesh202 b1cafb1280 org_settings: Add "Role" column in bot-list table.
This commit adds a new "Role" column for the bot-list table in the
org-settings, and removes the user_id column from the same.
The role of a bot is fetched using the `get_user_type` function inside
bot_info().
This also adds the `sort_role` in the sort_fields which sorts the role
column in the bot-list table.
2022-05-10 14:09:57 -07:00
Christopher Chong 94457732c1 github, gitlab: Replace the usage of dict.get(key) with dict[key]. 2022-05-10 13:50:20 -07:00
Christopher Chong 27a498e1e7 gitlab: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 76b1a682b7 github: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 2f097c524e gitea, gogs: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00