Commit Graph

1244 Commits

Author SHA1 Message Date
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
manavdesai27 572cef9a0f provision: Add support for Fedora 34. 2021-07-20 12:10:41 -07:00
Alex Vandiver 91282ab490 reindex-textual-data: Provide a tool to reindex all text indices.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic.  It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
2021-07-19 16:34:23 -07:00
Anders Kaseorg 47897c76a2 scripts: Use curl -f (--fail).
This makes curl exit with nonzero status on HTTP 4xx/5xx errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 16:47:49 -07:00
Alex Vandiver 16691110a6 scripts: Only stop/restart zulip_deliver_scheduled_* processes if known.
Running `supervisorctl stop` or `supervisorctl restart` on a process
name which is not known is an error:

```
$ supervisorctl stop nonexistent-process
nonexistent-process: ERROR (no such process)
$ echo $?
1
```

ef6d0ec5ca moved
zulip_deliver_scheduled_* out of the `workers:` group.  Since upgrades
run `stop-server` before applying puppet, the list of processes at
that time is from the previous version of Zulip, so may not have the
new `zulip_deliver_scheduled_*` names -- and the `stop-server` will
hence fail.

If the upgrade is not applying puppet, it will `restart-server`. At
that point, the old names will still be in the configuration, so
relying on the current `superisorctl status` is the best gauge of what
exists to restart.

In short, only ever stop/start/restart the `zulip_deliver_scheduled_*`
processes if `supervisorctl status` knows about them already.
2021-07-09 10:04:53 -07:00
Alex Vandiver c94bdd8534 zulip_tools: Find missing processes/groups in list_supervisor_processes.
Nonexistent processes and groups passed to `supervisortctl status` are
printed to STDOUT as follows:

```
$ supervisorctl status zulip-django nonexistent-process nonexistent-group:*
nonexistent-process: ERROR (no such process)
nonexistent-group: ERROR (no such group)
zulip-django                     RUNNING   pid 16043, uptime 17:31:31
```

On supervisor 4 and above, this exits with an exit code of 4;
previously, it returned exit code 0.  Ubuntu 18.04 has version 3.3.1,
and Ubuntu 20.04 has version 4.1.0.

Skip any lines with `ERROR (no such ...)`, and accept exit code 4 from
`supervisorctl status`.
2021-07-09 10:04:53 -07:00
Alex Vandiver 85a9c0982a zulip_tools: Extract out `list_supervisor_processes`. 2021-07-09 10:04:53 -07:00
Anders Kaseorg d83c91526b install-node: Upgrade Node.js from 14.17.0 to 14.17.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 14:51:24 -07:00
Anders Kaseorg 684dad8145 tools: Use root-based absolute import for tools.lib, etc.
Mypy can’t follow absolute imports based on directories other than the
root.  This was hiding some type errors due to ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:21:52 -07:00
Anders Kaseorg 7d71a1a31a setup: Add missing __init__.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:20:39 -07:00
Alex Vandiver 88c19bf65a puppet: Catch when a comma is left out of puppet_classes.
With two space-separated classes in `puppet_classes`, the second one
is silently ignored.  With three of more, puppet generates the
following very opaque error message:

```
Error: Could not parse for environment production: This
Name has no effect. A value was produced and then forgotten (one or
more preceding expressions may have the wrong form)
```

Catch when this has happened, and give an error message to the user.

Fixes #18992.
2021-06-28 20:58:56 -04:00
Anders Kaseorg 0ba9114c22 install-yarn: Rewrite Yarn installer.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-27 16:58:17 -07:00
Gaurav Pandey af08bcdb3f management: Delete send_stats command.
This command is part of a statsd infrastructure that we stopped
supporting years ago. Its only purpose for some time has been to
provide sample code for how the restart script might trigger a
notification to a graphing system, which doesn't justify maintaining
it.

Fixes part of #18898.
2021-06-25 09:13:48 -07:00
Anders Kaseorg 91bfebca7d install: Replace wget with curl.
curl uses Happy Eyeballs to avoid long timeouts on systems with broken
IPv6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-25 09:05:07 -07:00
Anders Kaseorg 3b60b25446 ci: Remove bullseye hack.
base-files 11.1 marked bullseye as Debian 11 in /etc/os-release.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Anders Kaseorg bf361e9951 ci: Remove uses of VERSION_CODENAME.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Tim Abbott 83738f7e6d install: Use a period at end of root error message. 2021-06-23 08:42:46 -07:00
Gaurav Pandey faae845366 upgrade: Modify upgrade scripts to handle failure.
The current `upgrade-zulip` and `upgrade-zulip-from-git`
bash scripts exit with a zero status even if the
upgrade commands exit with a non-zero status.
Hence add `set -e` command which exits the script with
the same status as the non-zero command.

For pipe commands however, the net status of a command
is the status of the last command, hence if the other parts
fail, the net status is only determined by the last command.
This is the case with our main /lib/upgrade-zulip* command
in the scripts whose status is determined by the `tee` command
instead. Hence add a small condition to get the status of the
actual upgrade command and exit the script if it fails with
a non-zero command.

We also check whether the script is being run as root, matching the
install script logic.
2021-06-23 08:42:20 -07:00
Tim Abbott 28d49edee3 script: Add --no-headings option to purge-old-deployments.
This parameter is somewhat useful, and adding this also fixes a
regression where purge-old-deployments would crash since the changes
around c5580607a7 because of
inconsistent supported args lists.
2021-06-17 15:49:23 -07:00
Mateusz Mandera 06c0a29e47 email-mirror-postfix: Choose scheme based on http_only config.
Fixes #16659.
If the server is behind a reverse proxy with http_only=True, the
requests made by email-mirror-postfix need to use http, as https
doesn't work.
2021-06-17 09:06:09 -07:00
Alex Vandiver d51272cc3d puppet: Remove zulip_deliver_scheduled_* from zulip-workers:*.
Staging and other hosts that are `zulip::app_frontend_base` but not
`zulip::app_frontend_once` do not have a
/etc/supervisor/conf.d/zulip/zulip-once.conf and as such do not have
`zulip_deliver_scheduled_emails` or `zulip_deliver_scheduled_messages`
and thus supervisor will fail to reload.

Making the contents of `zulip-workers` contingent on if the server is
_also_ a `-once` server is complicated, and would involve using Concat
fragments, which severely limit readability.

Instead, expel those two from `zulip-workers`; this is somewhat
reasonable, since they are use an entirely different codepath from
zulip_events_*, using the database rather than RabbitMQ for their
queuing.
2021-06-14 17:12:59 -07:00
Riken Shah c5580607a7 purge-old-deployments: Use the `clean_unused_caches.main` function.
We currently run the `clean_unused_caches.py` as a
script to clean the unused caches.

This commit replaces that with
`clean_unused_caches.main` function as it would be
faster.
2021-06-12 07:28:16 -07:00
Riken Shah 45af71e33b clean_unused_caches: Allow the main function to accept `Namespace` args.
This commit will allow us to pass the arguments in the
'clean...' functions when calling the `main` function (in
`provision`). It also changes args parsing
function location to `if __name__ == "__main__"` block as
we wouldn't need it to parse args when we call the
function.
2021-06-12 07:28:16 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Anders Kaseorg d8cb418586 zulip_tools: Flush ‘set -x’-style messages in run.
Otherwise they often get buffered until after the command actually
runs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg 342834ee9c python: Simplify stdio flushing using print(…, flush=True).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg bc169d63a7 install-node: Upgrade Node.js from 14.16.1 to 14.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:02:12 -07:00
Anders Kaseorg 61e1e38a00 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-07 17:57:51 -07:00
Alex Vandiver 1cdf14d195 puppet: Add a teleport server.
See https://goteleport.com/docs/architecture/overview/ for the general
architecture of a Teleport cluster.  This commit adds a Teleport auth[1]
and proxy[2] server.  The auth server serves as a CA for granting
time-bounded access to users and authenticating nodes on the cluster;
the proxy provides access and a management UI.

[1] https://goteleport.com/docs/architecture/authentication/
[2] https://goteleport.com/docs/architecture/proxy/
2021-06-02 18:38:38 -07:00
Alex Vandiver e080a05b05 node_cache: Serialize to structured data before hashing.
Appending data back-to-back without serializing it loses the
information about where the breaks between them lie, which can lead to
different inputs having the same hash.
2021-05-27 22:47:56 -07:00
Alex Vandiver 87a109e3e0 puppet: Pull in pinned puppet modules.
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
2021-05-27 21:14:48 -07:00
Anders Kaseorg cb8d9a1f8a create-db: Default dbuser and dbname to zulip.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-26 17:19:11 -07:00
Alex Vandiver f3eea72c2a setup: Merge multiple setup-apt-repo scripts into one.
This moves the `.asc` files into subdirectories, and writes out the
according `.list` files into them.  It moves from templates to
written-out `.list` files for clarity and ease of
implementation (Debian and Ubuntu need different templates for
`zulip`), and as a way of making explicit which releases are supported
for each list.  For the special-case of the PGroonga signing key, we
source an additional file within the directory.

This simplifies the process for adding another class of `.list` file.
2021-05-26 14:42:29 -07:00
Adam Birds 4539899cae installer: Add support for custom database user and dbname.
Add support for custom database names and database users, which can be
set with the `--postgresql-database-name` and
`--postgresql-database-user` install script options. If these
parameters aren't provided, then the defaults remain "zulip".

Fixes #17662.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-05-25 13:56:05 -07:00
Alex Vandiver 7ff3c9f966 upgrade-zulip: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Alex Vandiver 1d59330cbc postgresql-init-db: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:04 -07:00
Alex Vandiver 54c222d3f8 settings: Support arbitrary database user and dbname.
This adds basic support for `postgresql.database_user` and
`postgresql.database_name` settings in `zulip.conf`; the defaults if
unspecified are left as `zulip`.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Adam Birds 21cc186105 installer: Add run_psql_as_postgres function zulip_tools.py.
Add a helper `run_psql_as_postgres` function in
`scripts/lib/zulip_tools.py`.  This is preparatory refactoring for the
work to add custom database and user names.
2021-05-24 16:58:11 -07:00
Alex Vandiver 81644f110e install: $ZULIP_ADMINISTRATOR may be unset for non-frontend hosts. 2021-05-23 13:29:23 -07:00
Anders Kaseorg 09f6ba1971 install: Run git config commands from a known readable cwd.
Fixes this error when running the installer from a directory that
isn’t world-readable:

+ su zulip -c 'git config --global user.email anders@zulip.com'
fatal: cannot come back to cwd: Permission denied

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Anders Kaseorg bc45525369 postgresql-init-db: Fix installation from world-unreadable directory.
This reverts part of commit 476524c0c1
(#18215), to fix this error when running the installer from a
directory that isn’t world-readable:

+ '[' -e /var/run/supervisor.sock ']'
+++ dirname /root/zulip-server-4.1/scripts/setup/postgresql-init-db
++ dirname /root/zulip-server-4.1/scripts/setup
+ su zulip -c /root/zulip-server-4.1/scripts/stop-server
bash: /root/zulip-server-4.1/scripts/stop-server: Permission denied

Zulip installation failed (exit code 126)!

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Anders Kaseorg 6766a3f780 purge-old-deployments: Check /srv/zulip.git existence before pruning it.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 20:36:27 -07:00
Tim Abbott de47feab43 scripts: Fix check for services running when upgrading.
When upgrading from a pre-4.0 release, scripts/stop-server logic would
check whether supervisord configuration files were present to
determine what it needed to restart, but only considered paths to
those files that are introduced in Zulip 4.0.

Fixed #18493.
2021-05-13 18:57:19 -07:00
Anders Kaseorg 3f83b843c2 upgrade-zulip-from-git: Create deployment directories with git worktree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 13:42:05 -07:00
Tim Abbott 03420831b0 upgrade-zulip-from-git: Fetch tags from upstream repository.
This ensures that the `git describe` queries that we run for caching
Zulip's Git version are guaranteed to include recent releases.

This change ensures that we have accurate output even if we're pointed
at a fork of Zulip that never updates its tags.

Additionally, it will make it possible to record the
`git merge-base upstream/master` in future commits.

Note that because we run this code before unpacking the new version,
the pre-upgrade version of this code runs.

As a result, we cannot assume that the upstream repository exists.
2021-05-13 11:17:25 -07:00
Alex Vandiver 3ccb77da74 install: Tell NVM to not change $PATH earlier.
This removes a possible window where an installer error could leave
`nvm` in a state where it had prepended the full path to the
newly-installed `npm` to `$PATH`; we would like to avoid `nvm`
fiddling with path whenever possible (ref ebe930ab2c).
2021-05-11 11:25:34 -10:00
Anders Kaseorg 9ba48c4ed3 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Robert Imschweiler 534d78232c scripts: Add {start,stop,restart}-server support for postgresql role.
During the upgrade process of a postgresql-only Zulip installation,
(`puppet_classes = zulip::profile::postgresql` in
`/etc/zulip/zulip.conf`) either `scripts/start-server` or
`scripts/stop-server` fail because they try to handle supervisor
services that are not available (e.g. Tornado) since only
`/etc/supervisor/conf.d/zulip/zulip_db.conf` is present and not
`/etc/supervisor/conf.d/zulip/zulip.conf`.

While this wasn't previously supported, it's a pretty reasonable thing
to do, and can be readily supported by just adding a few conditionals.
2021-05-07 09:41:05 -07:00
Anders Kaseorg 9d57fa9759 puppet: Use pgrep -x to avoid accidental matches.
Matching the full process name (-x without -f) or full command
line (-xf) is less prone to mistakes like matching a random substring
of some other command line or pgrep matching itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 08:54:41 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Alex Vandiver eda9ce2364 locale: Use `C.UTF-8` rather than `en_US.UTF-8`.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.

Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.

Fixes #15819.
2021-05-04 08:51:46 -07:00
Mateusz Mandera dd7f3a1dce upgrade: Use restart-server unless --skip-puppet is used.
In some cases, puppet can end up restarting supervisord services - which
will use code from the old deployment, because when puppet runs,
/home/zulip/deployments/current still points there. Thus restart-server
needs to be used in favor of start-server, unless we know that puppet
has been skipped.
2021-05-03 08:12:54 -07:00
Alex Vandiver ebe930ab2c upgrade: Set an explicit value for PATH.
Previous versions of zulip used `nvm alias default ...` to have `nvm`
prepend the full path to the latest `node` install to the `PATH` in
root's shell.  Unfortunately, this means that `update-prod-static`,
when called from `upgrade-zulip-stage-2` after an upgrade of node in
`install-node`, would still have the full path to the _old_ `node` at
the start of its PATH, because the PATH of `upgrade-zulip-stage-2`
would still be unchanged.

Bootstrap out of this by setting a known-reasonable PATH during
upgrade, and remove the problematic `nvm alias default` behaviour.

Fixes #18258.
2021-05-01 07:16:45 -07:00
Alex Vandiver 49144247dd install: Set explicit value for PATH.
In Debian, becoming root as `su` does not alter the `$PATH`; this can
lead to the root user not having `/usr/sbin` in its path, and thus
the `useradd zulip` step of the installer fails.

Fixes #17441.
2021-05-01 07:16:45 -07:00
Alex Vandiver daabc52a78 restart-server: Reorder supervisorctl calls for less downtime.
Instead of taking the "onion" approach, where all services are
stopped, and then started back up again, default to a rolling restart
across all processes.  This draws out how long the overall "restart"
takes, but minimizes the time that any of the services are down.  This
minimizes user-visible impact and queue buildup.

In cases where speed is more important than minimal impact (for
example, there is already a current outage), a --less-graceful flag is
provided, which brings the services down more suddenly, and back up in
a still-correct order.
2021-04-30 16:47:15 -07:00
Alex Vandiver 4c88da8ed9 scripts: Tool to find the diff to an original settings.py prod template.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour.  This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
2021-04-27 21:50:33 -07:00
Alex Vandiver ae2c377d13 postgresql: Switch to defaulting to PostgreSQL 13. 2021-04-27 16:55:04 -07:00
Robert Imschweiler ba25580b19 clean-unused-caches: Handle non-existent yarn cache. 2021-04-27 10:02:49 -07:00
Riken Shah 1288dcbaaf clean-unused-caches: Add script to remove redundant yarn cache.
This commit removes redundant yarn cache by removing the old
version directories, i.e. All the directory under `~/.cache/yarn`
except `~/.cache/yarn/v6` (current version directory).

Fixes #15964.
2021-04-26 16:28:08 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Alex Vandiver 6db454b252 upgrade: Call start-server rather than restart-server if we stopped it.
This saves a little time, and thus causes a shorter outage window,
since we will not try to stop the services; we know they are already
down.
2021-04-21 10:28:30 -07:00
Alex Vandiver 16650ba239 upgrade: Call ./scripts/stop-server rather than duplicate the logic. 2021-04-21 10:28:30 -07:00
Alex Vandiver ec12a6128a scripts: Add a start-server as well.
In general, `./scripts/restart-server` will already work in any
circumstance where the server is already stopped and needs to be
started.  However, it will output a couple minor warnings, and it is
not readily obvious that it *will* work correctly.

Add an alias for `restart-server` named `start-server`, for
parallelism with `stop-server`, which omits the steps of
`restart-server` which would stop the server first.
2021-04-21 10:24:08 -07:00
Alex Vandiver 476524c0c1 scripts: Add a script to stop the server.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.

Add an explicit tool which only stops the relevant services.  It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.

Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.

Fixes #14959.
2021-04-21 10:24:08 -07:00
Alex Vandiver 31169526ec scripts: Say "Zulip" rather than "Application". 2021-04-21 10:24:08 -07:00
Alex Vandiver 0de8357820 scripts: Fix path to additional Zulip supervisor files.
The path which contains all of the Zulip supervisor files changed in
3ab9b31d2f to make it easier to purge
now-unwanted supervisor configuration files.  However, the paths that
the zulip upgrade process, and restart-server, look at were not
adjusted.

Fix the supervisor configuration file paths.
2021-04-21 10:24:08 -07:00
Alex Vandiver de41a10d38
upgrade: Install python3-yaml as needed.
3314fefaec started needing `python3-yaml`, but incorrectly claimed
that it was always an indirect dependency; it is a dependency of
`ubuntu-minimal` on 20.04, but not required on 18.04 or Debian.  We
cannot install it in puppet because then is definitionally too late;
it is needed at load time by `zulip-puppet-apply`.

Install `python3-yaml`, but guarded by a simple check so as to not
further slow most installs.

Fixes #18179.
2021-04-21 09:52:56 -07:00
Alex Vandiver 4c8502f7fd upgrade: Show fewer stacktraces.
The stacktraces here are seldom useful -- for the calls to
upgrade-stage-2, we know precisely what was run.  For the `run`
wrapper, the output contains the command that failed, which is
sufficient to identify where in the upgrade process it was.  Showing
more stacktrace below the actual error merely confuses users and
scrolls the real error off of the screen.
2021-04-21 09:51:40 -07:00
Siddharth Asthana d2706fa246 install: Create a .gitconfig file for the zulip user.
For installs which use the `upgrade-zulip-from-git` process, the
deployment directory is a git checkout. This means that an
administrator can, as an emergency tool, run `git revert` and similar
commands -- assuming there is a `~/.gitconfig` set up for the zulip
user.

Add commands to `scripts/lib/install` to create a `~/.gitconfig` file
at installation time.  The `user.name` and `user.email` fields are set
to the hostname and passed-in `--email` value, respectively.

Fixes #18039.
2021-04-20 22:47:20 -07:00
Gaurav Pandey feb720b463 install: Add beta support for debian bullseye for production.
This won't work on a real bullseye system until Bullseye actually
officially releases.

Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Gaurav Pandey 78524d4f87 provision: Add support for debian bullseye.
Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Anders Kaseorg b6b117274c install-node: Upgrade Node.js to 14.16.1 and nvm to 0.38.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Ganesh Pawar c1628e7605 provision: Upgrade support for Fedora to version 33.
Note that the `overwrite_symlink` changes fix a bug introduced in
5c20ee998c, that we need root
permissions to do those operations.
2021-03-22 19:34:18 -07:00
Ganesh Pawar 666ab59b03 pgroonga: Bump pgroonga version to 2.2.8 when building from source. 2021-03-22 19:33:48 -07:00
Ganesh Pawar 7cdb26108c minor: Avoid verbose tar output.
It isn't much helpful and clutters the logs.
2021-03-22 19:33:48 -07:00
Anders Kaseorg 6364e1b5f3 requirements: Upgrade talon fork to 1.4.8.
https://github.com/mailgun/talon/pull/200

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-18 17:10:18 -07:00
Alex Vandiver 3314fefaec puppet: Do not require a venv for zulip-puppet-apply.
0663b23d54 changed zulip-puppet-apply to
use the venv, because it began using `yaml` to parse the output of
puppet to determine if changes would happen.

However, not every install ends with a venv; notably, non-frontend
servers do not have one.  Attempting to run zulip-puppet-apply on them
hence now fails.

Remove this dependency on the venv, by installing a system
python3-yaml package -- though in reality, this package is already an
indirect dependency of the system.  Especially since pyyaml is quite
stable, we're not using it in any interesting way, and it does not
actually add to the dependencies, it is preferable to parsing the YAML
by hand in this instance.
2021-03-14 17:50:57 -07:00
Alex Vandiver 52f155873f puppet: Ensure that all `scripts/lib/install` packages are installed.
These have all been required packages for some time, but this helps
keep the install-time list more clearly a subset of the upgrade-time
list.
2021-03-14 17:50:57 -07:00
Anders Kaseorg d393ac5034 update-prod-static: Remove unused --prev-deploy option.
It’s unused since commit 079ddae4c8
(#12676).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-04 18:04:57 -08:00
Anders Kaseorg 25bb98dcf5 install-node: Upgrade Node.js from 14.15.1 to 14.16.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg ccad00b7e9 provision: Suppress exception chaining for CalledProcessError retries.
When exception is raised inside an exception handler, Python 3
helpfully prints both tracebacks separated by “During handling of the
above exception, another exception occurred:”.  But when we’re using
an exception handler to retry the same operation, multiple tracebacks
are just noise.  Suppress the earlier one using PEP 409 syntax.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 16:25:03 -08:00
Alex Vandiver 32149c6a1c puppet: Add ksplice uptrack for kernel hotpatches. 2021-02-25 18:05:47 -08:00
Alex Vandiver 0663b23d54 puppet: Only prompt to apply if there are changes to apply.
Since `yaml` is not a module in the standard library, this requires
makings `zulip-puppet-apply` use the venv.
2021-02-23 18:16:02 -08:00
Alex Vandiver d15e6990e5 puppet: Only execute setup-apt-repo if necessary.
This means that in steady-state, `zulip-puppet-apply` is expected to
produce no changes or commands to execute.  The verification step of
`setup-apt-repo` is quite fast, so this cleans up the output for very
little cost.
2021-02-23 18:16:02 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 1a4f70f1bc lint: Convert sudo exclusion to double quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 08:34:43 -08:00
Ganesh Pawar 7eeca9da46 provision: Add provision support for Ubuntu 20.10(Groovy).
PostgreSQL 13 is used when os_version is 20.10.
2021-02-05 09:30:34 -08:00
Anders Kaseorg 948f2ee2ad manage: Quote commands correctly in log_management_command.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-26 13:26:57 -08:00
Alex Vandiver c2526844e9 worker: Remove SignupWorker and friends.
ZULIP_FRIENDS_LIST_ID and MAILCHIMP_API_KEY are not currently used in
production.

This removes the unused 'signups' queue and worker.
2021-01-17 11:16:35 -08:00
Sutou Kouhei 0d3f9fc855 install: Use PGroonga packages built for PostgreSQL packages by PGDG
Because we always use PostgreSQL packages by PGDG since Zulip 3.0.

Fixes #16058.
2020-12-18 15:38:21 -08:00
Anders Kaseorg 77fdac3579 install-node: Upgrade Node.js to 14.15.1 and nvm to 0.37.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Vishnu KS eb008fc864 emails: Use macros for email tags in invitation email. 2020-10-30 11:50:30 -07:00
Anders Kaseorg aaa7b766d8 python: Use universal_newlines to get str from subprocess.
We can replace ‘universal_newlines’ with ‘text’ when we bump our
minimum Python version to 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 86e8d81c7f python: Skip unnecessary decode before JSON parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Tim Abbott c537912a77 puppet: Migrate postgres_backups puppet manifest name. 2020-10-29 11:29:44 -07:00
Alex Vandiver 2332113c97 upgrade: Adjust puppet class names even with --skip-puppet.
The class names need to be renamed even if we are not about to run
puppet ourselves; otherwise, deployments which rely on running puppet
themselves will still have the wrong class names.
2020-10-28 17:49:14 -07:00
Alex Vandiver 6b9d7000b5 puppet: Set proxy environment variables.
These are respected by `urllib`, and thus also `requests`.  We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
2020-10-28 12:17:35 -07:00
Alex Vandiver 97745688ca docs: Link to the new doc home of the email gateway. 2020-10-28 12:13:04 -07:00
Alex Vandiver f1cf730c5b restore-backup: Rename variables to postgresql. 2020-10-28 11:57:03 -07:00
Alex Vandiver 5ee3379ce0 upgrade: Rename variables to postgresql. 2020-10-28 11:57:03 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Anders Kaseorg 23a289ecd5 install-node: Upgrade Node.js to 12.19.0 and Yarn to 1.22.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Anders Kaseorg de5282d2cf install-node: Install npm and npx symlinks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Alex Vandiver 5f3765b872 upgrade: Adjust puppet classes to new names. 2020-10-27 13:29:19 -07:00
Alex Vandiver 16d9dd84b8 upgrade: Switch to using crudini to update zulip.conf contents.
Using `config_file.write()` only writes out what python stored of the
file; as such, it strips all comments and whitespace.

Use `crudini --set`, which only modifies the line whose contents are
changed.
2020-10-27 13:29:19 -07:00
Alex Vandiver 5365af544a puppet: Rename zulip::profile::rabbit to ::rabbitmq. 2020-10-27 13:29:19 -07:00
Alex Vandiver 188af57296 puppet: Rename postgres_appdb to postgresql.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name.  This is minor
cleanup, but enabled by the other renames in the previous commit.
2020-10-27 13:29:19 -07:00
Alex Vandiver 0f25acc7b3 puppet: Rename "voyager"/"dockervoyager" to "standalone"/"docker".
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
2020-10-27 13:29:19 -07:00
Alex Vandiver c2185a81d6 puppet: Move top-level zulip deployments into "profile" directory.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes.  Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.

Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.

[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
2020-10-27 13:29:19 -07:00
Alex Vandiver 7cf737988d queue: Be more explicit about test/real queue division. 2020-10-26 12:32:47 -07:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg 16aa48d9b2 configure-rabbitmq: Wait for RabbitMQ to start up.
Fixes an occasional failure in ‘vagrant up --provision’.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 17:01:00 -07:00
Anders Kaseorg f16aa8f264 configure-rabbitmq: Put the command and flags in one array.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 17:01:00 -07:00
Alex Vandiver 1fa4ef0271 upgrade-postgres: Catch failed pg_upgradecluster exit code.
Because the command is part of a pipe sequence, the exitcode defaults
to the last in the sequence, which is not the most important one here.

Set pipefail, which sets the exit status to the exit code of the last
program in the sequence to exit non-zero, or 0 if all succeeded.  This
prevents the upgrade from barreling onward and setting
`postgres.version` improperly if the database upgrade step failed.
2020-10-15 15:21:30 -07:00
Anders Kaseorg dfaea9df65 shfmt: Reformat shell scripts with shfmt.
https://github.com/mvdan/sh

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 15:16:00 -07:00
Anders Kaseorg dd48dbd912 docs: Add spaces to “check out”, “log in”, “set up”, “sign up” as verbs.
“Checkout”, “login”, “setup”, and “signup” are nouns, not verbs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Tim Abbott 5de6f3523c upgrade-postgres: Pass the requested postgres explicitly. 2020-10-01 14:29:24 -07:00
Alex Vandiver 4d65ea256a rabbitmq: Consolidate check_rabbitmq_queue to call rabbitmqctl once.
`rabbitmqctl` tends to be slow; this shaves half a second off the time
to run `check-rabbitmq-consumers` in some cases.
2020-09-29 17:44:44 -07:00
Alex Vandiver c0e240277b tornado: Remove fingerprinting, write out .tmp files always.
Fingerprinting the config is somewhat brittle -- it requires either
custom bootstrapping for old (fingerprint-less) configs, and may have
false-positives.

Since generating the config is lightweight, do so into the .tmp files,
and compare the output to the originals to determine if there are
changes to apply.

In order to both surface errors, as well as notify the user in case a
restart is necessary, we must run it twice.  The `onlyif`
functionality cannot show configuration errors to the user, only
determine if the command runs or not.  We thus run the command once,
judging errors as "interesting" enough to run the actual command,
whose failure will be verbose in Puppet and halt any steps that depend
on it.

Removing the `onlyif` would result in `stage_updated_sharding` showing
up in the output of every Puppet run, which obscures the important
messages it displays when an update to sharding is necessary.
Removing the `command` (e.g. making it an `echo`) would result in
removing the ability to report configuration errors.  We thus have no
choice but to run it twice; this is thankfully low-overhead.
2020-09-25 10:52:40 -07:00
Alex Vandiver 4b3121db0b certbot: Explicitly apt-get update before installing certbot.
There is no guarantee that the apt data is up-to-date, unless we
explicitly update.

Fixes: zulip/docker-zulip#275
2020-09-21 15:26:28 -07:00
Mateusz Mandera e2dcdc2758 queue: Increase allowed expected_time_to_clear_backlog for embed_links.
It's okay for this queue to be a bit slow, and the default limits are
kind of too low for it.
2020-09-21 15:24:04 -07:00
Mateusz Mandera cd9b194d88 queue: Eliminate useless "burst" concept in monitoring.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
2020-09-21 15:24:04 -07:00
Mateusz Mandera 2365a53496 queue: Fix a race condition in monitoring after queue stops being idle.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
2020-09-21 15:22:56 -07:00
Alex Vandiver 2a12fedcf1 tornado: Remove explicit tornado_processes setting; compute it.
We can compute the intended number of processes from the sharding
configuration.  In doing so, also validate that all of the ports are
contiguous.

This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding.  Having behaviour which
changes merely based on if an empty section exists is surprising.

This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.

After this commit, configuring sharding can be done by adding to
`zulip.conf`:

```
[tornado_sharding]
9800 =              # default
9801 = other_realm
```

Followed by running `./scripts/refresh-sharding-and-restart`.
2020-09-18 15:13:40 -07:00
Anders Kaseorg b7874ac82e install-node: Upgrade Node.js to 12.18.4 and Yarn to 1.22.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Alex Vandiver efdaa58c24 supervisor: Use more specific process_name than "port-9800".
Making this include "zulip-tornado" makes it clearer in supervisor
logs.  Without this, one only sees:
```
2020-09-14 03:43:13,788 INFO waiting for port-9807 to stop
2020-09-14 03:43:14,466 INFO stopped: port-9807 (exit status 1)
2020-09-14 03:43:14,469 INFO spawned: 'port-9807' with pid 24289
2020-09-14 03:43:15,470 INFO success: port-9807 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
```
2020-09-14 22:17:51 -07:00
Alex Vandiver 13fb7875e2 nagios: Remove an unnecessary path.append. 2020-09-14 18:20:12 -07:00
Alex Vandiver dd68cc98fd upgrade: Stop in the same order as restart-server.
restart-server explicitly stops the workers first, then the core
services.  Keep that ordering consistently.
2020-09-14 16:27:15 -07:00
Alex Vandiver dc58dec231 restart-server: Start services in opposite order from stop.
`supervisorctl` starts and stops its arguments sequentially, in the
order they are passed[1].  Start them in the opposite order from the
order in which they were stopped -- this puts the dependencies first,
and the most core services (`zulip-django`) last.

While the only "dependency" here is currently thumbor, this sets us up
in case others are added later.

[1] https://github.com/Supervisor/supervisor/blob/master/supervisor/supervisorctl.py#L782
2020-09-14 16:27:15 -07:00
Alex Vandiver 8adf530400 puppet: Generate sharding in puppet, then refresh-sharding-and-restart.
This supports running puppet to pick up new sharding changes, which
will warn of the need to finalize them via
`refresh-sharding-and-restart`, or simply running that directly.
2020-09-14 16:27:15 -07:00
Alex Vandiver bf029d99f1 sharding: Also mark sharding.json 644 for consistency.
There is no reason to limit this to 640; mark it 644 for consistency
with the other file.
2020-09-14 16:27:15 -07:00
Alex Vandiver b5bcff04e5 sharding: Consistent mode for nginx sharding file.
This disagreed between `tornado_sharding.pp` in puppet and
`scripts/refresh-sharding-and-restart`.
2020-09-14 16:27:15 -07:00
Mateusz Mandera aae84197e8 check-rabbitmq-queue: Use list_queues output for current backlog size.
The value in the stats file can get outdated if the queue hasn't done
enough iterations to update the stats file for a while. The queue size
output by rabbitmqctl list_queues is more up to date, and empirically
tends to agree with the value in the stats file (when the stats file is
fresh).
2020-09-11 15:51:07 -07:00
Anders Kaseorg b7b7475672 python: Use standard secrets module to generate random tokens.
There are three functional side effects:

• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.

• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.

• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.

(The base32 alphabet is A-Z 2-7.  We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-09 15:52:57 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bb4fc3c4c7 python: Prefer --flag=option over --flag option.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg 9edcafb7a0 setup_venv: Add missing comma in COMMON_YUM_VENV_DEPENDENCIES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:25:54 -07:00
Anders Kaseorg a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg fbfd4b399d python: Elide action="store" for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 1f2ac1962f python: Elide default=None for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 3c5b39da9c python: Elide nargs for argparse flag arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 1ded51aa9d python: Replace list literal concatenation with * unpacking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Sutou Kouhei ebf4048dd4
create-db.sql: Ensure using en_US.UTF-8 encoding.
PostgreSQL packages for Ubuntu run "initdb" without specifying locale
on installation. It means that the default template
database (template1) is created by the system default locale. If the
system default locale is non UTF-8 compatible encoding such as
en_US.ISO-8859-15, "zulip" database is also created non UTF-8
compatible encoding such as LATIN9.

You can reproduce this case by running the following script:

    apt update

    apt install -y locales
    locale-gen en_US.ISO-8859-15
    update-locale LANG=en_US.ISO-8859-15 LANGUAGE=en_US:

    apt install -y wget
    wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
    tar xf zulip-server-latest.tar.gz
    zulip-server-*/scripts/setup/install \
      --hostname=zulip-test.example.com \
      --email=zulip-test-admin@example.com \
      --self-signed-cert

scripts/setup/install is failed with the following error:

    + ./manage.py migrate --noinput
    Operations to perform:
      Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
    Running migrations:
      Applying contenttypes.0001_initial... OK
      Applying auth.0001_initial... OK
      Applying zerver.0001_initial...Traceback (most recent call last):
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
        return self.cursor.execute(sql)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
        return wrapper_execute(self, super().execute, query, vars)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
        return action(sql, params)
    psycopg2.errors.UntranslatableCharacter: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
    CONTEXT:  line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "./manage.py", line 50, in <module>
        execute_from_command_line(sys.argv)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
        utility.execute()
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
        output = self.handle(*args, **options)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
        res = handle_func(*args, **kwargs)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
        post_migrate_state = executor.migrate(
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
        state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
        state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
        operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 105, in database_forwards
        self._run_sql(schema_editor, self.sql)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 130, in _run_sql
        schema_editor.execute(statement, params=None)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
        cursor.execute(sql, params)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
        return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
        return executor(sql, params, many, context)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
        return self.cursor.execute(sql)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
        return wrapper_execute(self, super().execute, query, vars)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
        return action(sql, params)
    django.db.utils.DataError: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
    CONTEXT:  line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"
2020-08-24 12:24:38 -07:00
Anders Kaseorg 0f608176ad install-node: Upgrade Node.js from 12.18.2 to 12.18.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 18:37:25 -07:00
Aman 7b9fe77bf1 provision: Fix missing <sasl/sasl.h> headers during provision. 2020-08-12 16:19:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 3582183fba setup_venv: Install libyaml-dev.
This will let PyYAML link against LibYAML when PyYAML is next
installed.  Due to virtualenv-clone, that won’t happen until the next
Python package removal anyway, so we don’t bother bumping
PROVISION_VERSION.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 20:58:07 -07:00
Anders Kaseorg dbdf67301b memcached: Switch from pylibmc to python-binary-memcached.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-06 12:51:14 -07:00
Alex Vandiver 519b1e9b4d upgrade: With `skip_puppet`, show what puppet changes are outstanding.
This prevents puppet changes from building up over time.
2020-08-02 12:47:31 -07:00
Alex Vandiver c1923e19b0 puppet: --noop implies --force (i.e. no prompt).
The combination of `--force --noop` is potentially confusing, but
currently `--noop` makes no sense without `--force`, as it will prompt
and then not make changes.

Make `--noop` skip the prompt as well.
2020-08-02 12:47:31 -07:00
Alex Vandiver 38d01cd4db puppet: Generalize install-wal-g to be arbitrary tarballs. 2020-07-24 17:24:57 -07:00
Anders Kaseorg b3da022bdf install-node: Upgrade Node.js to 12.18.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-20 10:56:31 -07:00
Anders Kaseorg c2f9db4602 logo: Update Zulip logo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-16 01:37:08 -07:00
Tim Abbott 525b42cecc setup_venv: Require same Python version for virtualenv-clone.
This prevents us cloning a virtualenv in a way that would cause us to
ignore a newly updated Python version on the system.
2020-07-13 13:06:15 -07:00
Aman Agrawal 685ec2a098 hash_reqs: Include python version when generating hash.
Fixes #12868.
We now also include python version in the format
'major.minor.patchlevel', when generating hash for a
requirement file. This was necessary since packages tend to
break on different versions of python, so it is important to
track the version on which the venv was setup.

WARN: This commit will force all zulip venvs to be recreated.
2020-07-13 13:06:15 -07:00
Aman Agrawal 2668829c93 hash_reqs: Use combined package name and version to generate hash.
We were already using packages names along with their versions
to generate hash for the requirement file, as we were passing
the `.txt` files to the hash_reqs file instead of intended `.in` files
for which the functions in this file was originially designed.

Changed the expand_reqs_helper function to adapt for the `.txt` files.
2020-07-13 13:06:15 -07:00
Alex Vandiver 5ff3025411 upgrade: Add additional comments. 2020-07-13 12:47:49 -07:00
Alex Vandiver 47400cd04b upgrade: Drop unnecessary memcached restart.
The contents in the database are unchanged across the PostgreSQL
restart; as such, there is no reason to invalidate the caches.

This step was inherited from the general operating system upgrade
documentation.  When Python versions change, such as during OS
upgrades, we must ensure that memcached is cleared.  However, the
`do-release-upgrade` process uninstalled and upgraded to a new
memcached, as well as likely restarted the system; a separate step for
OS upgrades to restart memcached is thus unnecessary.
2020-07-13 12:47:04 -07:00
Alex Vandiver 0502b7a8d5 upgrade: Drop the unnecessary step that stops the old cluster.
The initial step in pg_upgradecluster stops the cluster for us; this
removes the somewhat ugly hack we are otherwise forced into.
2020-07-13 12:45:50 -07:00
Alex Vandiver bf0f712c81 upgrade: Use the in-place pg_upgrade, not a full dump/restore.
pg_upgradecluster has two possibilities for `--method`: `dump`, and
`upgrade`.  The former is the default, and does a `pg_dump` of all of
the databases in the old cluster and feeds them into the new cluster.
This is a sure-fire way of getting the same information in both
databases, but may be extremely slow on large databases, and is
guaranteed to fail on servers whose databases take up >50% of their
disk.

The `--method=upgrade` method, by contrast, uses pg_upgrade to copy
the raw database data file over to the new cluster, and then fiddles
with their internal structure as needed by the upgrade to let them be
correct for the new version[1].  This is slightly faster than the
dump/load method, since it skips the serialization step, but still
requires that there be enough space on disk for both old and new
versions at once.  `pg_upgrade` is currently supported for all
versions of PostgreSQL from 8.4 to 12.

Using `pg_upgrade` incurs slightly more risk, but since the it is
widely used by now, using it in the relatively-controlled Zulip server
environment is reasonable.  The expected worst failure is failure to
upgrade, not corruption or data loss.

Additionally passing `--link` uses hardlinks to link the data files
into both the old and new directories simultaneously.  This resolve
both the runtime of the operation, as well as the disk space usage.
The only potential downside to this is that as soon as writes have
occurred on the upgraded cluster, the old cluster can no longer be
started.  Since this tooling intends to remove the old cluster
immediately after the upgrade completes successfully, this is not a
significant drawback.

Switch to using `--method=upgrade --link`.  This technique spits out
two shell scripts which are expected to be run after completion of the
upgrade; one re-analyzes the statistics, the other does an `rm -rf` of
the data where it is still hardlinked in the old cluster.  Extract the
location of these scripts from parsing the `pg_upgradecluster` output;
since the path is not static, we must rely on it being relatively easy
to parse.  The risk of the path changing is lower, and has more
obvious failure modes, than inserting the current contents of these
upgrade steps into the overall `upgrade-postgres`.

[1] https://www.postgresql.org/docs/12/pgupgrade.html
2020-07-13 12:45:50 -07:00
Mateusz Mandera c231d88d9f upgrade: Add management command to fix FTS indexes.
Upgrading the base OS's dictionary files can corrupt our FTS
indexes. We add a command for fixing them.

Fixes #14982.
2020-07-13 12:40:44 -07:00
Anders Kaseorg ff1622afcf zulip_tools: Replace deprecated mktemp call.
Although mktemp is deprecated due to security issues, this is not a
security issue.

The security problems with mktemp happen when you open the resulting
filename (without O_EXCL) in a publicly writable directory, because
then someone else might have predicted the filename and created or
symlinked or hardlinked something there between the mktemp and the
open, causing you to write to a file you didn’t expect.

Here we don’t open the resulting filename, we symlink to it.  symlink
will refuse to clobber an existing file, and we handle the error that
arises from this case.  This is the normal way to atomically create a
symlink.

We should still replace mktemp because it’s deprecated, but we can’t
replace it with a function that creates the temporary file.  Instead
we build a random filename ourselves.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-09 14:32:02 -07:00
Anders Kaseorg 9900298315 zthumbor: Remove Python 2 residue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 18:44:58 -07:00
Alex Vandiver 0d7dbd1b07 puppet: Apply basic PostgreSQL configuration before pg_upgradecluster.
Running `pg-upgradecluster` runs the `CREATE TEXT SEARCH DICTIONARY`
and `CREATE TEXT SEARCH CONFIGURATION` from
`zerver/migrations/0001_initial.py` on the new PostgreSQL cluster;
this requires that the stopwords file and dictionary exist _prior_
to `pg_upgradecluster` being run.

This causes a minor dependency conflict -- we do not wish to duplicate
the functionality from `zulip::postgres_appdb_base` which configures
those files, but installing all of `zulip::postgres_appdb_tuned` will
attempt to restart PostgreSQL -- which has not configured the cluster
for the new version yet.

In order to split out configuration of the prerequisites for the
application database, and the steps required to run it, we need to be
able to apply only part of the puppet configuration.  Use the
newly-added `--config` argument to provide a more limited `zulip.conf`
which only applies `zulip::postgres_appdb_base` to the new version of
Postgres, creating the required tsearch data files.

This also preserves the property that a failure at any point prior to
the `pg_upgradecluster` is easily recoverable, by re-running
`zulip-puppet-apply`.
2020-07-06 18:30:16 -07:00
Alex Vandiver 17002f2a0e puppet: Allow passing an alternate config path to zulip-puppet-apply.
When temporary configuration changes are desired, this lets one set up
an alternate `zulip.conf` to apply while leaving the true one in place.
2020-07-06 18:30:16 -07:00
Alex Vandiver efe2b6e5cd puppet: Switch `zulip-puppet-apply` to argparse.
This allows additional arguments other than `-f` or `--force`.
2020-07-06 18:30:16 -07:00
Aman Agrawal a486872a8e requirements: Upgrade Thumbor to 7.0.0a5 on Python 3.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 16:09:53 -07:00
Vishnu KS 97403a09d0 install: Create zulip user only if required.
Otherwise, the useradd command will fail during the DigitalOcean
1-Click App installation because the install script is called
twice during the whole process. Plus the Zulip install script
is designed to be idempotent and this bug compromises that.
2020-07-02 14:55:04 -07:00
Anders Kaseorg e3835554a7 postgres-init-db: Read terminate-psql-sessions script as root.
Fixes #15646.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 14:54:36 -07:00
Tim Abbott ab1ee1f061 install: Add a comment on crudini deletion. 2020-07-01 15:13:00 -07:00
Alex Vandiver 6df99677d3 installer: Remove unnecessary nginx restart.
Puppet takes care of this.
2020-07-01 15:07:52 -07:00
Alex Vandiver 2d4fae0ffe installer: Remove out-of-date comment. 2020-07-01 15:07:52 -07:00
Alex Vandiver 2de8400a32 installer: Only set `deploy_type = production` in zulip.conf.
The value is a holdover from when it controlled runtime behavior,
which it no longer does.

Stop taking a DEPLOYMENT_TYPE, which is unused; the python code only
care about if the option exists, not its value.
2020-07-01 15:07:52 -07:00
Alex Vandiver 117d32cd8c installer: Switch to checking dockervoyager as a class, not a deployment.
The DEPLOYMENT_TYPE=dockervoyager is otherwise unused; and always
happens in conjunction with a `zulip::dockervoyager` puppet class.
2020-07-01 15:07:52 -07:00
Alex Vandiver 8236cb52d2 installer: Switch has_* variables for has_class checks.
These are more correct to the sense of "is this a service we
configured for Zulip", and removes potential confusion around the 0/1
values being backwards from how binary is usually interpreted.
2020-07-01 15:07:52 -07:00
Alex Vandiver 2c79909a5d installer: Switch other PUPPET_CLASSES check for has_class. 2020-07-01 15:07:52 -07:00
Alex Vandiver ec2383dcde installer: Move missing_dictionaries configuration to with other config.
It already has been made to explicitly conflict with
`--no-overwrite-settings`, so moving it inside the else block is safe.
2020-07-01 15:07:52 -07:00
Alex Vandiver 9c0fd632bb installer: Use `puppet --write-catalog-summary` to determine classes.
Using checks of `,$PUPPET_CLASSES,` is repetitive and error-prone; it
does not properly deal with `zulip_ops::` classes, for instance, which
include the `zulip::` classes.

As alluded to in ca9d27175b, this can be fixed by inspecting the
classes that would be applied, using `puppet --write-catalog-summary`.
We work around the chicken-and-egg problem alluded to therein by
writing out as complete `zulip.conf` as would be necessary, before
running puppet and removing the sections we then know to not be
needed.

Unfortunately, there are two checks for `$PUPPET_CLASSES` which cannot
be switched to this technique, as they concern errors that we wish to
catch quite early, and thus before we have puppet installed.  Since we
expect failures of those to only concern warnings, and only be
mistakenly omitted for internal `zulip_ops::` classes, this seems a
reasonable risk to admit in exchange for catching common errors early.
2020-07-01 15:07:51 -07:00
Alex Vandiver 64b44a12f5 puppet: Add an exec rule to reload the whole supervisor config.
When supervisor is first installed, it is started automatically, and
creates the socket, owned by root.  Subsequent reconfiguration in
puppet only calls `reread + update`, which is insufficient to apply
the `chown = zulip:zulip` line in `supervisord.conf`, leaving the
socket owned by `root` and the last part of the installation unable to
restart `supervisor` services as the `zulip` user.  The `chown` line
in `scripts/lib/install` exists to paper over this.

Add a separate exec target for changes to `supervisord.conf` itself,
which restarts the full service.  This leaves the default `restart`
action on the service for the lightweight `reread + update` action,
which is more common.

We use `systemctl` only on redhat-esque builds, because CI runs
Ubuntu, but init is not systemd in that context.  `systemctl reload`
is sufficient to re-apply the socket ownership, but a full `restart`
and not `reload` is necessary under `/etc/init.d/supervisor`.
2020-07-01 10:40:54 -07:00
Anders Kaseorg 7f46886696 settings: Split hostname from port more carefully.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Anders Kaseorg fa89d1b266 generate-self-signed-cert: Correct subjectAltName for an IP address.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Alex Vandiver 918fcb9f6f upgrade: Make upgrade-postgres work without systemctl.
The only postgres cluster which need be stopped is the one we are
upgrading.
2020-06-29 17:18:47 -07:00
Alex Vandiver cd290c2c66 installer: Be tighter about the search for postgres server packages. 2020-06-29 13:37:16 -07:00
Alex Vandiver eb6802057a upgrade: Don't prompt in the second apt-get upgrade. 2020-06-26 16:16:12 -07:00
Alex Vandiver b7a135f037 upgrade: Add a tool to upgrade PostgreSQL.
This is based on the existing steps in the documentation, with
additional changes now that the PostgreSQL version is stored in
`/etc/zulip/zulip.conf`.
2020-06-26 16:07:39 -07:00
Alex Vandiver 31f1f10501 installer: Halt if wrong version of PostgreSQL is already installed.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository.  On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.

During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found.  This is safer than attempting to automatically uninstall or
reconfigure existing databases.
2020-06-24 12:57:38 -07:00
Alex Vandiver 814198d649 installer: Abstract out version of postgres installed.
This allows for installing from-scratch with a different pinned
version of PostgreSQL, and provides a single place to change when the
default should increase.
2020-06-24 12:57:38 -07:00
Alex Vandiver ca9d27175b installer: Write PostgreSQL version based on puppet classes.
Using `/etc/init.d/postgresql` as the detection of if Postgres is on
the server is incorrect, because this line runs _before_ puppet and
any packages are installed.  Thus, it cannot tell the difference
between a new Ubuntu one-host first-time-install without PostgreSQL
yet, and one which is merely a front-end and will never have
PostgreSQL.  This leads to failures in first-time installs:

```
Error: Evaluation Error: Error while evaluating a Function Call,
  Could not find template 'zulip/postgresql//postgresql.conf.template.erb'
```

The only way to detect if PostgreSQL will be present in the _end_
state of the install is to examine the puppet classes that are
applied.

To do this, we must inspect `PUPPET_CLASSES`.  Unfortunately, this can
be fragile to subclassing (e.g. `zulip_ops::postgres_appdb`).  We
might desire to use `puppet apply --write-catalog-summary` to deduce
the _applied_ classes, which would unroll the inheritance; however,
this causes a chicken-and-egg problem, because `zulip.conf` must be
already written out (including a value for `postgresql.version`, if
necessary!) before such a puppet run could successfully complete.

Switch to predicating the `postgresql.version` key on the puppet
classes that are known to install postgres.
2020-06-24 12:57:38 -07:00
Alex Vandiver 253246185f installer: Update documentation.
Where appropriate, documentation wording is shared with
docs/production/install.md
2020-06-24 12:57:38 -07:00