Commit Graph

914 Commits

Author SHA1 Message Date
Alex Vandiver bd7deed691 upgrade: Show output from (re)starting zulip.
5c450afd2d, in ancient history, switched from `check_call` to
`check_output` and throwing away its result.

Use check_call, so that we show the steps to (re)starting the server.
2022-01-25 01:52:34 +00:00
Alex Vandiver e705883857 CVE-2021-43799: During upgrades, restart rabbitmq if necessary.
Check if it is listening on a public interface on port 25672, and if
so shut it down so it can pick up the new configuration.
2022-01-25 01:51:56 +00:00
Alex Vandiver da5201b986 upgrade: Make calling shutdown_server twice, only try once. 2022-01-25 01:48:05 +00:00
Alex Vandiver 43d63bd5a1 puppet: Always set the RabbitMQ nodename to zulip@localhost.
This is required in order to lock down the RabbitMQ port to only
listen on localhost.  If the nodename is `rabbit@hostname`, in most
circumstances the hostname will resolve to an external IP, which the
rabbitmq port will not be bound to.

Installs which used `rabbit@hostname`, due to RabbitMQ having been
installed before Zulip, would not have functioned if the host or
RabbitMQ service was restarted, as the localhost restrictions in the
RabbitMQ configuration would have made rabbitmqctl (and Zulip cron
jobs that call it) unable to find the rabbitmq server.

The previous commit ensures that configure-rabbitmq is re-run after
the nodename has changed.  However, rabbitmq needs to be stopped
before `rabbitmq-env.conf` is changed; we use an `onlyif` on an `exec`
to print the warning about the node change, and let the subsequent
config change and notify of the service and configure-rabbitmq to
complete the re-configuration.
2022-01-25 01:48:02 +00:00
Alex Vandiver 3bfcfeac24 puppet: Run configure-rabbitmq on nodename change.
`/etc/rabbitmq/rabbitmq-env.conf` sets the nodename; anytime the
nodename changes, the backing database changes, and this requires
re-creating the rabbitmq users and permissions.

Trigger this in puppet by running configure-rabbitmq after the file
changes.
2022-01-25 01:46:51 +00:00
Anders Kaseorg 21548ff7c0 install-node: Upgrade Node.js from 16.13.1 to 16.13.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-24 15:55:38 -08:00
Alex Vandiver a3adaf4aa3 puppet: Fix standalone certbot configurations.
This addresses the problems mentioned in the previous commit, but for
existing installations which have `authenticator = standalone` in
their configurations.

This reconfigures all hostnames in certbot to use the webroot
authenticator, and attempts to force-renew their certificates.
Force-renewal is necessary because certbot contains no way to merely
update the configuration.  Let's Encrypt allows for multiple extra
renewals per week, so this is a reasonable cost.

Because the certbot configuration is `configobj`, and not
`configparser`, we have no way to easily parse to determine if webroot
is in use; additionally, `certbot certificates` does not provide this
information.  We use `grep`, on the assumption that this will catch
nearly all cases.

It is possible that this will find `authenticator = standalone`
certificates which are managed by Certbot, but not Zulip certificates.
These certificates would also fail to renew while Zulip is running, so
switching them to use the Zulip webroot would still be an improvement.

Fixes #20593.
2022-01-24 12:13:44 -08:00
Alex Vandiver 76ce8631c0 setup: Install a temporary certificate, before certbot runs.
Installing certbot with --method=standalone means that the
configuration file will be written to assume that the standalone
method will be used going forward.  Since nginx will be running,
attempts to renew the certificate will fail.

Install a temporary self-signed certificate, just to allow nginx to
start, and then follow up (after applying puppet to start nginx) with
the call to setup-certbot, which will use the webroot authenticator.

The `setup-certbot --method=standalone` option is left intact, for use
in development environments.

Fixes part of #20593; it does not address installs which were
previously improperly configured with `authenticator = standalone`.
2022-01-24 12:13:44 -08:00
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Anders Kaseorg a58a71ef43 Remove Ubuntu 18.04 support.
As a consequence:

• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Alex Vandiver 677467f040 upgrade-zulip-from-git: Fix upstream URL for existing deploys. 2022-01-18 21:10:38 -08:00
Alex Vandiver bad58cdca6 upgrade-zulip-from-git: Fix the upstream URL not be the custom remote. 2022-01-18 21:10:38 -08:00
Anders Kaseorg e2cc554077 zulip_tools: Rename may_be_perform_purging to maybe_perform_purging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-12 13:21:35 -08:00
Alex Vandiver b31658482b upgrade-zulip: Pass any arguments down to upgrade-zulip-stage-2.
This is the equivalent of 93f3da4c05 but
for the tarball codepath.
2022-01-11 14:26:54 -08:00
Alex Vandiver 06e115bb00 zulip_tools: Switch get_deploy_options to use shlex.split.
This makes it honor quoting in the config file.
2022-01-11 14:26:54 -08:00
Alex Vandiver 4aaa250623 zulip_tools: Fix a typo in a comment. 2022-01-05 14:48:52 -08:00
Alex Vandiver 9d85f64e5a upgrade-zulip-stage-2: Pass through --skip-tornado and --less-graceful.
These restart-server arguments are useful to be able to provide to
`upgrade-zulip`.
2021-12-31 11:17:14 -08:00
Alex Vandiver fb3368b482 restart-server: Factor out argparser, to allow reuse. 2021-12-31 11:17:14 -08:00
Alex Vandiver 93f3da4c05 upgrade-from-git: Pass unknown options through to the upgrade process. 2021-12-31 11:17:14 -08:00
Anders Kaseorg 82748d45d8 install-yarn: Use test -ef in case /srv is a symlink.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-30 13:42:07 -08:00
Anders Kaseorg 0b454dda12 install: Try apt-get update if the Ubuntu universe check fails.
On a system where ‘apt-get update’ has never been run, ‘apt-cache
policy’ may show no repositories at all.  Try to correct this with
‘apt-get update’ before giving up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-16 17:56:23 -08:00
Alex Vandiver 01e8f752a8 puppet: Use certbot package timer, not our own cron job.
The certbot package installs its own systemd timer (and cron job,
which disabled itself if systemd is enabled) which updates
certificates.  This process races with the cron job which Zulip
installs -- the only difference being that Zulip respects the
`certbot.auto_renew` setting, and that it passes the deploy hook.
This means that occasionally nginx would not be reloaded, when the
systemd timer caught the expiration first.

Remove the custom cron job and `certbot-maybe-renew` script, and
reconfigure certbot to always reload nginx after deploying, using
certbot directory hooks.

Since `certbot.auto_renew` can't have an effect, remove the setting.
In turn, this removes the need for `--no-zulip-conf` to
`setup-certbot`.  `--deploy-hook` is similarly removed, as running
deploy hooks to restart nginx is now the default; pass
`--no-directory-hooks` in standalone mode to not attempt to reload
nginx.  The other property of `--deploy-hook`, of skipping symlinking
into place, is given its own flog.
2021-12-09 13:47:33 -08:00
Tim Abbott 9aa2e0ad45 upgrade-zulip-from-git: Improve webpack failure error handling.
We've had a number of unhappy reports of upgrades failing due to
webpack requiring too much memory.  While the previous commit will
likely fix this issue for everyone, it's worth improving the error
message for failures here.

We avoid doing the stop+retry ourselves, because that could cause an
outage in a production system if webpack fails for another reason.

Fixes #20105.
2021-12-09 12:26:34 -08:00
Tim Abbott 72b381d749 upgrade-zulip-from-git: Require more memory to run webpack.
Since the upgrade to Webpack 5, we've been seeing occasional reports
that servers with roughly 4GiB of RAM were getting OOM kills while
running webpack.

Since we can't readily optimize the memory requirements for webpack
itself, we should raise the RAM requirements for doing the
lower-downtime upgrade strategy.

Fixes #20231.
2021-12-09 12:23:25 -08:00
Anders Kaseorg 2e5af073b7 install-node: Upgrade Node.js from 16.13.0 to 16.13.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-03 14:33:53 -08:00
Alex Vandiver 544e8c569e install: Switch default to PostgreSQL 14. 2021-11-08 18:21:46 -08:00
Anders Kaseorg f2a443a736 install-node: Upgrade Node.js from 14.18.1 to 16.13.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:34:13 -07:00
Anders Kaseorg 458844a2f5 install-yarn: Verify that the install location is /srv/zulip-yarn.
scripts.lib.node_cache expects Yarn to be in /srv/zulip-yarn, so if
it’s installed somewhere else, even if it’s the right version, we need
to reinstall it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-03 16:49:58 -07:00
rht bb8504d925 lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
Anders Kaseorg 291087d70c install-yarn: Upgrade Yarn from 1.22.11 to 1.22.17.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-17 07:15:09 -07:00
Anders Kaseorg 7df96b78c6 install-node: Upgrade Node.js from 14.17.6 to 14.18.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-17 07:15:09 -07:00
Anders Kaseorg 2f993f1a79 install-node: Stop using NVM.
NVM doesn’t check hashes or signatures and really just adds
complexity we don’t need.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-24 06:58:32 -07:00
Anders Kaseorg 902883d818 setup_venv: Skip virtualenv’s automatic download of setuptools.
It recently started failing on Debian 10 (buster).  We immediately
follow this by replacing these packages with our own versions from
pip.txt, anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-23 14:29:04 -07:00
Anders Kaseorg 08e459b393 zulip_tools: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-22 13:58:46 -07:00
Anders Kaseorg 9bed17e0ab install-node: Upgrade Node.js from 14.17.5 to 14.17.6.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-13 10:12:43 -07:00
Gaurav Pandey 502697d239 docs: Add documentation for bullseye support.
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.

Since now bullseye is released as a stable
version, it's support can be documented.
2021-09-09 11:05:16 -07:00
Anders Kaseorg 02582c6956 upgrade-zulip-from-git: Run git fetch with --prune.
This prevents upgrading to an obsolete version of a branch that has
been deleted or renamed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-01 05:34:57 -07:00
Anders Kaseorg 3cb66d59ac install: Remove /dev/null redirect for zulip-puppet-apply.
The usual output from this command looks like

Notice: Compiled catalog for localhost in environment production in 2.33 seconds
Notice: /Stage[main]/Zulip::Apt_repository/Exec[setup_apt_repo]/returns: current_value 'notrun', should be ['0'] (noop)
Notice: Class[Zulip::Apt_repository]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 1.20 seconds

which doesn’t seem abnormally alarming, and hiding it makes failures
harder to diagnose.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 16:30:53 -07:00
Anders Kaseorg 7b2e585213 install-yarn: Upgrade Yarn from 1.22.10 to 1.22.11.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:33:27 -07:00
Anders Kaseorg ebb8e9109c install-node: Upgrade Node.js from 14.17.3 to 14.17.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:29:04 -07:00
Anders Kaseorg 4206e5f00b python: Remove locally dead code.
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
Anders Kaseorg 5483ebae37 python: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg ad5f0c05b5 python: Remove default "utf8" argument for encode(), decode().
Partially generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 1760897a8c python: Remove default "r" mode for open().
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
manavdesai27 572cef9a0f provision: Add support for Fedora 34. 2021-07-20 12:10:41 -07:00
Anders Kaseorg 47897c76a2 scripts: Use curl -f (--fail).
This makes curl exit with nonzero status on HTTP 4xx/5xx errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 16:47:49 -07:00
Alex Vandiver c94bdd8534 zulip_tools: Find missing processes/groups in list_supervisor_processes.
Nonexistent processes and groups passed to `supervisortctl status` are
printed to STDOUT as follows:

```
$ supervisorctl status zulip-django nonexistent-process nonexistent-group:*
nonexistent-process: ERROR (no such process)
nonexistent-group: ERROR (no such group)
zulip-django                     RUNNING   pid 16043, uptime 17:31:31
```

On supervisor 4 and above, this exits with an exit code of 4;
previously, it returned exit code 0.  Ubuntu 18.04 has version 3.3.1,
and Ubuntu 20.04 has version 4.1.0.

Skip any lines with `ERROR (no such ...)`, and accept exit code 4 from
`supervisorctl status`.
2021-07-09 10:04:53 -07:00
Alex Vandiver 85a9c0982a zulip_tools: Extract out `list_supervisor_processes`. 2021-07-09 10:04:53 -07:00
Anders Kaseorg d83c91526b install-node: Upgrade Node.js from 14.17.0 to 14.17.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 14:51:24 -07:00
Anders Kaseorg 0ba9114c22 install-yarn: Rewrite Yarn installer.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-27 16:58:17 -07:00
Anders Kaseorg 91bfebca7d install: Replace wget with curl.
curl uses Happy Eyeballs to avoid long timeouts on systems with broken
IPv6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-25 09:05:07 -07:00
Anders Kaseorg 3b60b25446 ci: Remove bullseye hack.
base-files 11.1 marked bullseye as Debian 11 in /etc/os-release.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Anders Kaseorg bf361e9951 ci: Remove uses of VERSION_CODENAME.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Tim Abbott 28d49edee3 script: Add --no-headings option to purge-old-deployments.
This parameter is somewhat useful, and adding this also fixes a
regression where purge-old-deployments would crash since the changes
around c5580607a7 because of
inconsistent supported args lists.
2021-06-17 15:49:23 -07:00
Mateusz Mandera 06c0a29e47 email-mirror-postfix: Choose scheme based on http_only config.
Fixes #16659.
If the server is behind a reverse proxy with http_only=True, the
requests made by email-mirror-postfix need to use http, as https
doesn't work.
2021-06-17 09:06:09 -07:00
Alex Vandiver d51272cc3d puppet: Remove zulip_deliver_scheduled_* from zulip-workers:*.
Staging and other hosts that are `zulip::app_frontend_base` but not
`zulip::app_frontend_once` do not have a
/etc/supervisor/conf.d/zulip/zulip-once.conf and as such do not have
`zulip_deliver_scheduled_emails` or `zulip_deliver_scheduled_messages`
and thus supervisor will fail to reload.

Making the contents of `zulip-workers` contingent on if the server is
_also_ a `-once` server is complicated, and would involve using Concat
fragments, which severely limit readability.

Instead, expel those two from `zulip-workers`; this is somewhat
reasonable, since they are use an entirely different codepath from
zulip_events_*, using the database rather than RabbitMQ for their
queuing.
2021-06-14 17:12:59 -07:00
Riken Shah 45af71e33b clean_unused_caches: Allow the main function to accept `Namespace` args.
This commit will allow us to pass the arguments in the
'clean...' functions when calling the `main` function (in
`provision`). It also changes args parsing
function location to `if __name__ == "__main__"` block as
we wouldn't need it to parse args when we call the
function.
2021-06-12 07:28:16 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Anders Kaseorg d8cb418586 zulip_tools: Flush ‘set -x’-style messages in run.
Otherwise they often get buffered until after the command actually
runs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg 342834ee9c python: Simplify stdio flushing using print(…, flush=True).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg bc169d63a7 install-node: Upgrade Node.js from 14.16.1 to 14.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:02:12 -07:00
Alex Vandiver e080a05b05 node_cache: Serialize to structured data before hashing.
Appending data back-to-back without serializing it loses the
information about where the breaks between them lie, which can lead to
different inputs having the same hash.
2021-05-27 22:47:56 -07:00
Alex Vandiver 87a109e3e0 puppet: Pull in pinned puppet modules.
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
2021-05-27 21:14:48 -07:00
Alex Vandiver f3eea72c2a setup: Merge multiple setup-apt-repo scripts into one.
This moves the `.asc` files into subdirectories, and writes out the
according `.list` files into them.  It moves from templates to
written-out `.list` files for clarity and ease of
implementation (Debian and Ubuntu need different templates for
`zulip`), and as a way of making explicit which releases are supported
for each list.  For the special-case of the PGroonga signing key, we
source an additional file within the directory.

This simplifies the process for adding another class of `.list` file.
2021-05-26 14:42:29 -07:00
Adam Birds 4539899cae installer: Add support for custom database user and dbname.
Add support for custom database names and database users, which can be
set with the `--postgresql-database-name` and
`--postgresql-database-user` install script options. If these
parameters aren't provided, then the defaults remain "zulip".

Fixes #17662.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-05-25 13:56:05 -07:00
Alex Vandiver 7ff3c9f966 upgrade-zulip: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Adam Birds 21cc186105 installer: Add run_psql_as_postgres function zulip_tools.py.
Add a helper `run_psql_as_postgres` function in
`scripts/lib/zulip_tools.py`.  This is preparatory refactoring for the
work to add custom database and user names.
2021-05-24 16:58:11 -07:00
Alex Vandiver 81644f110e install: $ZULIP_ADMINISTRATOR may be unset for non-frontend hosts. 2021-05-23 13:29:23 -07:00
Anders Kaseorg 09f6ba1971 install: Run git config commands from a known readable cwd.
Fixes this error when running the installer from a directory that
isn’t world-readable:

+ su zulip -c 'git config --global user.email anders@zulip.com'
fatal: cannot come back to cwd: Permission denied

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Tim Abbott de47feab43 scripts: Fix check for services running when upgrading.
When upgrading from a pre-4.0 release, scripts/stop-server logic would
check whether supervisord configuration files were present to
determine what it needed to restart, but only considered paths to
those files that are introduced in Zulip 4.0.

Fixed #18493.
2021-05-13 18:57:19 -07:00
Anders Kaseorg 3f83b843c2 upgrade-zulip-from-git: Create deployment directories with git worktree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 13:42:05 -07:00
Tim Abbott 03420831b0 upgrade-zulip-from-git: Fetch tags from upstream repository.
This ensures that the `git describe` queries that we run for caching
Zulip's Git version are guaranteed to include recent releases.

This change ensures that we have accurate output even if we're pointed
at a fork of Zulip that never updates its tags.

Additionally, it will make it possible to record the
`git merge-base upstream/master` in future commits.

Note that because we run this code before unpacking the new version,
the pre-upgrade version of this code runs.

As a result, we cannot assume that the upstream repository exists.
2021-05-13 11:17:25 -07:00
Alex Vandiver 3ccb77da74 install: Tell NVM to not change $PATH earlier.
This removes a possible window where an installer error could leave
`nvm` in a state where it had prepended the full path to the
newly-installed `npm` to `$PATH`; we would like to avoid `nvm`
fiddling with path whenever possible (ref ebe930ab2c).
2021-05-11 11:25:34 -10:00
Anders Kaseorg 9ba48c4ed3 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Robert Imschweiler 534d78232c scripts: Add {start,stop,restart}-server support for postgresql role.
During the upgrade process of a postgresql-only Zulip installation,
(`puppet_classes = zulip::profile::postgresql` in
`/etc/zulip/zulip.conf`) either `scripts/start-server` or
`scripts/stop-server` fail because they try to handle supervisor
services that are not available (e.g. Tornado) since only
`/etc/supervisor/conf.d/zulip/zulip_db.conf` is present and not
`/etc/supervisor/conf.d/zulip/zulip.conf`.

While this wasn't previously supported, it's a pretty reasonable thing
to do, and can be readily supported by just adding a few conditionals.
2021-05-07 09:41:05 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Alex Vandiver eda9ce2364 locale: Use `C.UTF-8` rather than `en_US.UTF-8`.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.

Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.

Fixes #15819.
2021-05-04 08:51:46 -07:00
Mateusz Mandera dd7f3a1dce upgrade: Use restart-server unless --skip-puppet is used.
In some cases, puppet can end up restarting supervisord services - which
will use code from the old deployment, because when puppet runs,
/home/zulip/deployments/current still points there. Thus restart-server
needs to be used in favor of start-server, unless we know that puppet
has been skipped.
2021-05-03 08:12:54 -07:00
Alex Vandiver ebe930ab2c upgrade: Set an explicit value for PATH.
Previous versions of zulip used `nvm alias default ...` to have `nvm`
prepend the full path to the latest `node` install to the `PATH` in
root's shell.  Unfortunately, this means that `update-prod-static`,
when called from `upgrade-zulip-stage-2` after an upgrade of node in
`install-node`, would still have the full path to the _old_ `node` at
the start of its PATH, because the PATH of `upgrade-zulip-stage-2`
would still be unchanged.

Bootstrap out of this by setting a known-reasonable PATH during
upgrade, and remove the problematic `nvm alias default` behaviour.

Fixes #18258.
2021-05-01 07:16:45 -07:00
Alex Vandiver 49144247dd install: Set explicit value for PATH.
In Debian, becoming root as `su` does not alter the `$PATH`; this can
lead to the root user not having `/usr/sbin` in its path, and thus
the `useradd zulip` step of the installer fails.

Fixes #17441.
2021-05-01 07:16:45 -07:00
Alex Vandiver ae2c377d13 postgresql: Switch to defaulting to PostgreSQL 13. 2021-04-27 16:55:04 -07:00
Robert Imschweiler ba25580b19 clean-unused-caches: Handle non-existent yarn cache. 2021-04-27 10:02:49 -07:00
Riken Shah 1288dcbaaf clean-unused-caches: Add script to remove redundant yarn cache.
This commit removes redundant yarn cache by removing the old
version directories, i.e. All the directory under `~/.cache/yarn`
except `~/.cache/yarn/v6` (current version directory).

Fixes #15964.
2021-04-26 16:28:08 -07:00
Alex Vandiver 6db454b252 upgrade: Call start-server rather than restart-server if we stopped it.
This saves a little time, and thus causes a shorter outage window,
since we will not try to stop the services; we know they are already
down.
2021-04-21 10:28:30 -07:00
Alex Vandiver 16650ba239 upgrade: Call ./scripts/stop-server rather than duplicate the logic. 2021-04-21 10:28:30 -07:00
Alex Vandiver 0de8357820 scripts: Fix path to additional Zulip supervisor files.
The path which contains all of the Zulip supervisor files changed in
3ab9b31d2f to make it easier to purge
now-unwanted supervisor configuration files.  However, the paths that
the zulip upgrade process, and restart-server, look at were not
adjusted.

Fix the supervisor configuration file paths.
2021-04-21 10:24:08 -07:00
Alex Vandiver de41a10d38
upgrade: Install python3-yaml as needed.
3314fefaec started needing `python3-yaml`, but incorrectly claimed
that it was always an indirect dependency; it is a dependency of
`ubuntu-minimal` on 20.04, but not required on 18.04 or Debian.  We
cannot install it in puppet because then is definitionally too late;
it is needed at load time by `zulip-puppet-apply`.

Install `python3-yaml`, but guarded by a simple check so as to not
further slow most installs.

Fixes #18179.
2021-04-21 09:52:56 -07:00
Alex Vandiver 4c8502f7fd upgrade: Show fewer stacktraces.
The stacktraces here are seldom useful -- for the calls to
upgrade-stage-2, we know precisely what was run.  For the `run`
wrapper, the output contains the command that failed, which is
sufficient to identify where in the upgrade process it was.  Showing
more stacktrace below the actual error merely confuses users and
scrolls the real error off of the screen.
2021-04-21 09:51:40 -07:00
Siddharth Asthana d2706fa246 install: Create a .gitconfig file for the zulip user.
For installs which use the `upgrade-zulip-from-git` process, the
deployment directory is a git checkout. This means that an
administrator can, as an emergency tool, run `git revert` and similar
commands -- assuming there is a `~/.gitconfig` set up for the zulip
user.

Add commands to `scripts/lib/install` to create a `~/.gitconfig` file
at installation time.  The `user.name` and `user.email` fields are set
to the hostname and passed-in `--email` value, respectively.

Fixes #18039.
2021-04-20 22:47:20 -07:00
Gaurav Pandey feb720b463 install: Add beta support for debian bullseye for production.
This won't work on a real bullseye system until Bullseye actually
officially releases.

Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Gaurav Pandey 78524d4f87 provision: Add support for debian bullseye.
Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Anders Kaseorg b6b117274c install-node: Upgrade Node.js to 14.16.1 and nvm to 0.38.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Ganesh Pawar c1628e7605 provision: Upgrade support for Fedora to version 33.
Note that the `overwrite_symlink` changes fix a bug introduced in
5c20ee998c, that we need root
permissions to do those operations.
2021-03-22 19:34:18 -07:00
Ganesh Pawar 666ab59b03 pgroonga: Bump pgroonga version to 2.2.8 when building from source. 2021-03-22 19:33:48 -07:00
Ganesh Pawar 7cdb26108c minor: Avoid verbose tar output.
It isn't much helpful and clutters the logs.
2021-03-22 19:33:48 -07:00
Anders Kaseorg 6364e1b5f3 requirements: Upgrade talon fork to 1.4.8.
https://github.com/mailgun/talon/pull/200

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-18 17:10:18 -07:00
Alex Vandiver 3314fefaec puppet: Do not require a venv for zulip-puppet-apply.
0663b23d54 changed zulip-puppet-apply to
use the venv, because it began using `yaml` to parse the output of
puppet to determine if changes would happen.

However, not every install ends with a venv; notably, non-frontend
servers do not have one.  Attempting to run zulip-puppet-apply on them
hence now fails.

Remove this dependency on the venv, by installing a system
python3-yaml package -- though in reality, this package is already an
indirect dependency of the system.  Especially since pyyaml is quite
stable, we're not using it in any interesting way, and it does not
actually add to the dependencies, it is preferable to parsing the YAML
by hand in this instance.
2021-03-14 17:50:57 -07:00
Alex Vandiver 52f155873f puppet: Ensure that all `scripts/lib/install` packages are installed.
These have all been required packages for some time, but this helps
keep the install-time list more clearly a subset of the upgrade-time
list.
2021-03-14 17:50:57 -07:00
Anders Kaseorg d393ac5034 update-prod-static: Remove unused --prev-deploy option.
It’s unused since commit 079ddae4c8
(#12676).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-04 18:04:57 -08:00
Anders Kaseorg 25bb98dcf5 install-node: Upgrade Node.js from 14.15.1 to 14.16.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg ccad00b7e9 provision: Suppress exception chaining for CalledProcessError retries.
When exception is raised inside an exception handler, Python 3
helpfully prints both tracebacks separated by “During handling of the
above exception, another exception occurred:”.  But when we’re using
an exception handler to retry the same operation, multiple tracebacks
are just noise.  Suppress the earlier one using PEP 409 syntax.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 16:25:03 -08:00
Alex Vandiver 32149c6a1c puppet: Add ksplice uptrack for kernel hotpatches. 2021-02-25 18:05:47 -08:00
Alex Vandiver d15e6990e5 puppet: Only execute setup-apt-repo if necessary.
This means that in steady-state, `zulip-puppet-apply` is expected to
produce no changes or commands to execute.  The verification step of
`setup-apt-repo` is quite fast, so this cleans up the output for very
little cost.
2021-02-23 18:16:02 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 1a4f70f1bc lint: Convert sudo exclusion to double quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 08:34:43 -08:00
Ganesh Pawar 7eeca9da46 provision: Add provision support for Ubuntu 20.10(Groovy).
PostgreSQL 13 is used when os_version is 20.10.
2021-02-05 09:30:34 -08:00
Anders Kaseorg 948f2ee2ad manage: Quote commands correctly in log_management_command.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-26 13:26:57 -08:00
Alex Vandiver c2526844e9 worker: Remove SignupWorker and friends.
ZULIP_FRIENDS_LIST_ID and MAILCHIMP_API_KEY are not currently used in
production.

This removes the unused 'signups' queue and worker.
2021-01-17 11:16:35 -08:00
Sutou Kouhei 0d3f9fc855 install: Use PGroonga packages built for PostgreSQL packages by PGDG
Because we always use PostgreSQL packages by PGDG since Zulip 3.0.

Fixes #16058.
2020-12-18 15:38:21 -08:00
Anders Kaseorg 77fdac3579 install-node: Upgrade Node.js to 14.15.1 and nvm to 0.37.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Anders Kaseorg aaa7b766d8 python: Use universal_newlines to get str from subprocess.
We can replace ‘universal_newlines’ with ‘text’ when we bump our
minimum Python version to 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 86e8d81c7f python: Skip unnecessary decode before JSON parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Tim Abbott c537912a77 puppet: Migrate postgres_backups puppet manifest name. 2020-10-29 11:29:44 -07:00
Alex Vandiver 2332113c97 upgrade: Adjust puppet class names even with --skip-puppet.
The class names need to be renamed even if we are not about to run
puppet ourselves; otherwise, deployments which rely on running puppet
themselves will still have the wrong class names.
2020-10-28 17:49:14 -07:00
Alex Vandiver 6b9d7000b5 puppet: Set proxy environment variables.
These are respected by `urllib`, and thus also `requests`.  We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
2020-10-28 12:17:35 -07:00
Alex Vandiver 97745688ca docs: Link to the new doc home of the email gateway. 2020-10-28 12:13:04 -07:00
Alex Vandiver 5ee3379ce0 upgrade: Rename variables to postgresql. 2020-10-28 11:57:03 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Anders Kaseorg 23a289ecd5 install-node: Upgrade Node.js to 12.19.0 and Yarn to 1.22.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Anders Kaseorg de5282d2cf install-node: Install npm and npx symlinks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Alex Vandiver 5f3765b872 upgrade: Adjust puppet classes to new names. 2020-10-27 13:29:19 -07:00
Alex Vandiver 16d9dd84b8 upgrade: Switch to using crudini to update zulip.conf contents.
Using `config_file.write()` only writes out what python stored of the
file; as such, it strips all comments and whitespace.

Use `crudini --set`, which only modifies the line whose contents are
changed.
2020-10-27 13:29:19 -07:00
Alex Vandiver 5365af544a puppet: Rename zulip::profile::rabbit to ::rabbitmq. 2020-10-27 13:29:19 -07:00
Alex Vandiver 188af57296 puppet: Rename postgres_appdb to postgresql.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name.  This is minor
cleanup, but enabled by the other renames in the previous commit.
2020-10-27 13:29:19 -07:00
Alex Vandiver 0f25acc7b3 puppet: Rename "voyager"/"dockervoyager" to "standalone"/"docker".
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
2020-10-27 13:29:19 -07:00
Alex Vandiver c2185a81d6 puppet: Move top-level zulip deployments into "profile" directory.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes.  Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.

Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.

[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
2020-10-27 13:29:19 -07:00
Alex Vandiver 7cf737988d queue: Be more explicit about test/real queue division. 2020-10-26 12:32:47 -07:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg dfaea9df65 shfmt: Reformat shell scripts with shfmt.
https://github.com/mvdan/sh

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 15:16:00 -07:00
Anders Kaseorg dd48dbd912 docs: Add spaces to “check out”, “log in”, “set up”, “sign up” as verbs.
“Checkout”, “login”, “setup”, and “signup” are nouns, not verbs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Alex Vandiver 4d65ea256a rabbitmq: Consolidate check_rabbitmq_queue to call rabbitmqctl once.
`rabbitmqctl` tends to be slow; this shaves half a second off the time
to run `check-rabbitmq-consumers` in some cases.
2020-09-29 17:44:44 -07:00
Alex Vandiver c0e240277b tornado: Remove fingerprinting, write out .tmp files always.
Fingerprinting the config is somewhat brittle -- it requires either
custom bootstrapping for old (fingerprint-less) configs, and may have
false-positives.

Since generating the config is lightweight, do so into the .tmp files,
and compare the output to the originals to determine if there are
changes to apply.

In order to both surface errors, as well as notify the user in case a
restart is necessary, we must run it twice.  The `onlyif`
functionality cannot show configuration errors to the user, only
determine if the command runs or not.  We thus run the command once,
judging errors as "interesting" enough to run the actual command,
whose failure will be verbose in Puppet and halt any steps that depend
on it.

Removing the `onlyif` would result in `stage_updated_sharding` showing
up in the output of every Puppet run, which obscures the important
messages it displays when an update to sharding is necessary.
Removing the `command` (e.g. making it an `echo`) would result in
removing the ability to report configuration errors.  We thus have no
choice but to run it twice; this is thankfully low-overhead.
2020-09-25 10:52:40 -07:00
Mateusz Mandera e2dcdc2758 queue: Increase allowed expected_time_to_clear_backlog for embed_links.
It's okay for this queue to be a bit slow, and the default limits are
kind of too low for it.
2020-09-21 15:24:04 -07:00
Mateusz Mandera cd9b194d88 queue: Eliminate useless "burst" concept in monitoring.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
2020-09-21 15:24:04 -07:00
Mateusz Mandera 2365a53496 queue: Fix a race condition in monitoring after queue stops being idle.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
2020-09-21 15:22:56 -07:00
Alex Vandiver 2a12fedcf1 tornado: Remove explicit tornado_processes setting; compute it.
We can compute the intended number of processes from the sharding
configuration.  In doing so, also validate that all of the ports are
contiguous.

This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding.  Having behaviour which
changes merely based on if an empty section exists is surprising.

This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.

After this commit, configuring sharding can be done by adding to
`zulip.conf`:

```
[tornado_sharding]
9800 =              # default
9801 = other_realm
```

Followed by running `./scripts/refresh-sharding-and-restart`.
2020-09-18 15:13:40 -07:00
Anders Kaseorg b7874ac82e install-node: Upgrade Node.js to 12.18.4 and Yarn to 1.22.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Alex Vandiver dd68cc98fd upgrade: Stop in the same order as restart-server.
restart-server explicitly stops the workers first, then the core
services.  Keep that ordering consistently.
2020-09-14 16:27:15 -07:00
Alex Vandiver 8adf530400 puppet: Generate sharding in puppet, then refresh-sharding-and-restart.
This supports running puppet to pick up new sharding changes, which
will warn of the need to finalize them via
`refresh-sharding-and-restart`, or simply running that directly.
2020-09-14 16:27:15 -07:00
Mateusz Mandera aae84197e8 check-rabbitmq-queue: Use list_queues output for current backlog size.
The value in the stats file can get outdated if the queue hasn't done
enough iterations to update the stats file for a while. The queue size
output by rabbitmqctl list_queues is more up to date, and empirically
tends to agree with the value in the stats file (when the stats file is
fresh).
2020-09-11 15:51:07 -07:00
Anders Kaseorg bb4fc3c4c7 python: Prefer --flag=option over --flag option.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg 9edcafb7a0 setup_venv: Add missing comma in COMMON_YUM_VENV_DEPENDENCIES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:25:54 -07:00
Anders Kaseorg a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg fbfd4b399d python: Elide action="store" for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 1f2ac1962f python: Elide default=None for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 3c5b39da9c python: Elide nargs for argparse flag arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 1ded51aa9d python: Replace list literal concatenation with * unpacking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Anders Kaseorg 0f608176ad install-node: Upgrade Node.js from 12.18.2 to 12.18.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 18:37:25 -07:00
Aman 7b9fe77bf1 provision: Fix missing <sasl/sasl.h> headers during provision. 2020-08-12 16:19:06 -07:00
Anders Kaseorg 3582183fba setup_venv: Install libyaml-dev.
This will let PyYAML link against LibYAML when PyYAML is next
installed.  Due to virtualenv-clone, that won’t happen until the next
Python package removal anyway, so we don’t bother bumping
PROVISION_VERSION.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 20:58:07 -07:00
Anders Kaseorg dbdf67301b memcached: Switch from pylibmc to python-binary-memcached.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-06 12:51:14 -07:00
Alex Vandiver 519b1e9b4d upgrade: With `skip_puppet`, show what puppet changes are outstanding.
This prevents puppet changes from building up over time.
2020-08-02 12:47:31 -07:00
Alex Vandiver c1923e19b0 puppet: --noop implies --force (i.e. no prompt).
The combination of `--force --noop` is potentially confusing, but
currently `--noop` makes no sense without `--force`, as it will prompt
and then not make changes.

Make `--noop` skip the prompt as well.
2020-08-02 12:47:31 -07:00
Anders Kaseorg b3da022bdf install-node: Upgrade Node.js to 12.18.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-20 10:56:31 -07:00
Anders Kaseorg c2f9db4602 logo: Update Zulip logo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-16 01:37:08 -07:00
Tim Abbott 525b42cecc setup_venv: Require same Python version for virtualenv-clone.
This prevents us cloning a virtualenv in a way that would cause us to
ignore a newly updated Python version on the system.
2020-07-13 13:06:15 -07:00
Aman Agrawal 685ec2a098 hash_reqs: Include python version when generating hash.
Fixes #12868.
We now also include python version in the format
'major.minor.patchlevel', when generating hash for a
requirement file. This was necessary since packages tend to
break on different versions of python, so it is important to
track the version on which the venv was setup.

WARN: This commit will force all zulip venvs to be recreated.
2020-07-13 13:06:15 -07:00
Aman Agrawal 2668829c93 hash_reqs: Use combined package name and version to generate hash.
We were already using packages names along with their versions
to generate hash for the requirement file, as we were passing
the `.txt` files to the hash_reqs file instead of intended `.in` files
for which the functions in this file was originially designed.

Changed the expand_reqs_helper function to adapt for the `.txt` files.
2020-07-13 13:06:15 -07:00
Mateusz Mandera c231d88d9f upgrade: Add management command to fix FTS indexes.
Upgrading the base OS's dictionary files can corrupt our FTS
indexes. We add a command for fixing them.

Fixes #14982.
2020-07-13 12:40:44 -07:00
Anders Kaseorg ff1622afcf zulip_tools: Replace deprecated mktemp call.
Although mktemp is deprecated due to security issues, this is not a
security issue.

The security problems with mktemp happen when you open the resulting
filename (without O_EXCL) in a publicly writable directory, because
then someone else might have predicted the filename and created or
symlinked or hardlinked something there between the mktemp and the
open, causing you to write to a file you didn’t expect.

Here we don’t open the resulting filename, we symlink to it.  symlink
will refuse to clobber an existing file, and we handle the error that
arises from this case.  This is the normal way to atomically create a
symlink.

We should still replace mktemp because it’s deprecated, but we can’t
replace it with a function that creates the temporary file.  Instead
we build a random filename ourselves.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-09 14:32:02 -07:00
Anders Kaseorg 9900298315 zthumbor: Remove Python 2 residue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 18:44:58 -07:00
Aman Agrawal a486872a8e requirements: Upgrade Thumbor to 7.0.0a5 on Python 3.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 16:09:53 -07:00
Vishnu KS 97403a09d0 install: Create zulip user only if required.
Otherwise, the useradd command will fail during the DigitalOcean
1-Click App installation because the install script is called
twice during the whole process. Plus the Zulip install script
is designed to be idempotent and this bug compromises that.
2020-07-02 14:55:04 -07:00
Tim Abbott ab1ee1f061 install: Add a comment on crudini deletion. 2020-07-01 15:13:00 -07:00
Alex Vandiver 6df99677d3 installer: Remove unnecessary nginx restart.
Puppet takes care of this.
2020-07-01 15:07:52 -07:00
Alex Vandiver 2d4fae0ffe installer: Remove out-of-date comment. 2020-07-01 15:07:52 -07:00
Alex Vandiver 2de8400a32 installer: Only set `deploy_type = production` in zulip.conf.
The value is a holdover from when it controlled runtime behavior,
which it no longer does.

Stop taking a DEPLOYMENT_TYPE, which is unused; the python code only
care about if the option exists, not its value.
2020-07-01 15:07:52 -07:00
Alex Vandiver 117d32cd8c installer: Switch to checking dockervoyager as a class, not a deployment.
The DEPLOYMENT_TYPE=dockervoyager is otherwise unused; and always
happens in conjunction with a `zulip::dockervoyager` puppet class.
2020-07-01 15:07:52 -07:00
Alex Vandiver 8236cb52d2 installer: Switch has_* variables for has_class checks.
These are more correct to the sense of "is this a service we
configured for Zulip", and removes potential confusion around the 0/1
values being backwards from how binary is usually interpreted.
2020-07-01 15:07:52 -07:00
Alex Vandiver 2c79909a5d installer: Switch other PUPPET_CLASSES check for has_class. 2020-07-01 15:07:52 -07:00
Alex Vandiver ec2383dcde installer: Move missing_dictionaries configuration to with other config.
It already has been made to explicitly conflict with
`--no-overwrite-settings`, so moving it inside the else block is safe.
2020-07-01 15:07:52 -07:00
Alex Vandiver 9c0fd632bb installer: Use `puppet --write-catalog-summary` to determine classes.
Using checks of `,$PUPPET_CLASSES,` is repetitive and error-prone; it
does not properly deal with `zulip_ops::` classes, for instance, which
include the `zulip::` classes.

As alluded to in ca9d27175b, this can be fixed by inspecting the
classes that would be applied, using `puppet --write-catalog-summary`.
We work around the chicken-and-egg problem alluded to therein by
writing out as complete `zulip.conf` as would be necessary, before
running puppet and removing the sections we then know to not be
needed.

Unfortunately, there are two checks for `$PUPPET_CLASSES` which cannot
be switched to this technique, as they concern errors that we wish to
catch quite early, and thus before we have puppet installed.  Since we
expect failures of those to only concern warnings, and only be
mistakenly omitted for internal `zulip_ops::` classes, this seems a
reasonable risk to admit in exchange for catching common errors early.
2020-07-01 15:07:51 -07:00
Alex Vandiver 64b44a12f5 puppet: Add an exec rule to reload the whole supervisor config.
When supervisor is first installed, it is started automatically, and
creates the socket, owned by root.  Subsequent reconfiguration in
puppet only calls `reread + update`, which is insufficient to apply
the `chown = zulip:zulip` line in `supervisord.conf`, leaving the
socket owned by `root` and the last part of the installation unable to
restart `supervisor` services as the `zulip` user.  The `chown` line
in `scripts/lib/install` exists to paper over this.

Add a separate exec target for changes to `supervisord.conf` itself,
which restarts the full service.  This leaves the default `restart`
action on the service for the lightweight `reread + update` action,
which is more common.

We use `systemctl` only on redhat-esque builds, because CI runs
Ubuntu, but init is not systemd in that context.  `systemctl reload`
is sufficient to re-apply the socket ownership, but a full `restart`
and not `reload` is necessary under `/etc/init.d/supervisor`.
2020-07-01 10:40:54 -07:00
Anders Kaseorg 7f46886696 settings: Split hostname from port more carefully.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Alex Vandiver cd290c2c66 installer: Be tighter about the search for postgres server packages. 2020-06-29 13:37:16 -07:00
Alex Vandiver eb6802057a upgrade: Don't prompt in the second apt-get upgrade. 2020-06-26 16:16:12 -07:00
Alex Vandiver 31f1f10501 installer: Halt if wrong version of PostgreSQL is already installed.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository.  On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.

During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found.  This is safer than attempting to automatically uninstall or
reconfigure existing databases.
2020-06-24 12:57:38 -07:00
Alex Vandiver 814198d649 installer: Abstract out version of postgres installed.
This allows for installing from-scratch with a different pinned
version of PostgreSQL, and provides a single place to change when the
default should increase.
2020-06-24 12:57:38 -07:00
Alex Vandiver ca9d27175b installer: Write PostgreSQL version based on puppet classes.
Using `/etc/init.d/postgresql` as the detection of if Postgres is on
the server is incorrect, because this line runs _before_ puppet and
any packages are installed.  Thus, it cannot tell the difference
between a new Ubuntu one-host first-time-install without PostgreSQL
yet, and one which is merely a front-end and will never have
PostgreSQL.  This leads to failures in first-time installs:

```
Error: Evaluation Error: Error while evaluating a Function Call,
  Could not find template 'zulip/postgresql//postgresql.conf.template.erb'
```

The only way to detect if PostgreSQL will be present in the _end_
state of the install is to examine the puppet classes that are
applied.

To do this, we must inspect `PUPPET_CLASSES`.  Unfortunately, this can
be fragile to subclassing (e.g. `zulip_ops::postgres_appdb`).  We
might desire to use `puppet apply --write-catalog-summary` to deduce
the _applied_ classes, which would unroll the inheritance; however,
this causes a chicken-and-egg problem, because `zulip.conf` must be
already written out (including a value for `postgresql.version`, if
necessary!) before such a puppet run could successfully complete.

Switch to predicating the `postgresql.version` key on the puppet
classes that are known to install postgres.
2020-06-24 12:57:38 -07:00
Alex Vandiver 253246185f installer: Update documentation.
Where appropriate, documentation wording is shared with
docs/production/install.md
2020-06-24 12:57:38 -07:00
Alex Vandiver 85dbb13c56 installer: Abstract out apt/yum divide into a variable.
This check is done in several places, using a somewhat fragile `case`
statement; move it into an explicit variable.
2020-06-24 12:57:38 -07:00
Alex Vandiver 876ee4a8ed installer: Remove code specific to stretch or xenial.
Support for Xenial and Stretch was removed (5154ddafca, 0f4b1076ad,
8944e0ad53, 79acd5ae40, 1219a2e854), but not all codepaths were
updated to remove their conditionals on it.

Remove all code predicated on Xenial or Stretch.  debathena support
was migrated to Bionic, since that appears to be the current state of
existing debathena servers.
2020-06-24 12:57:38 -07:00
Alex Vandiver e4899eae8b installer: Sync the claimed supported distros with the check.
0f4b1076ad removed Ubuntu 16.04 "xenial" and Debian 9 "stretch" from
the printed list of supported operating systems, but left them in the
verification check that controls if that message is printed,
effectively continuing to support them.

Conversely, 439f0d3004 added Ubuntu 20.04 "focal" to the check, but
not to the printed list.

Synchronize to check and print the right supported distributions:
Ubuntu 18.04 "bionic", Ubuntu 20.04 "focal", and Debian 10 "buster".
2020-06-24 12:57:38 -07:00
Alex Vandiver 58cb7cecd8 installer: Remove `--remote-postgres`, redundant with `--no-init-db`.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.

Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
2020-06-24 12:57:38 -07:00
Alex Vandiver 7c6a25a43d installer: Group and unify ordering of installer options.
This also adds the missing `--no-overwrite-settings` option to
`--help`.
2020-06-24 12:57:38 -07:00
Alex Vandiver b165b4144d installer: Prevent flags which conflict with `--no-overwrite-settings`.
Since `--postgres-missing-dictionaries` edits `/etc/zulip/zulip.conf`,
it interferes with the intent of `--no-overwrite-settings`.

Make the two settings conflict, to prevent this unclear state.
2020-06-23 13:40:28 -07:00
Alex Vandiver 7f4a2527c0 installer: Make `--no-overwrite-settings` also preserve `zulip.conf`.
This allows a path through the installer for places that have already
configured `zulip.conf`, by extending the existing flag and behavior.
2020-06-23 13:40:28 -07:00
Alex Vandiver 27100b4507 installer: Fix mis-indentation. 2020-06-23 13:36:26 -07:00
Alex Vandiver 5b7be7ba5d installer: Do not initialize db with --no-init-db.
The `--no-init-db` option previously only controlled if
`initialize-database` was run, which sets up the tables inside the
database.  If PostgreSQL was installed locally, it still attempted to
create the user and empty database.

This fails on hosts which are remote PostgreSQL hosts, and not
application hosts, as:

 - They may already have a local database, and while
 `initialize-datbase` will detect and offer to abort if one is
 found,`--no-init-db` seems like it should be the option to not
 overwrite it

 - `flush-memcached` requires that a local venv be installed, which it
 often is not on non-frontend machines.

Skip the database configuration when run with `--no-init-db`.
2020-06-23 13:36:26 -07:00