Commit Graph

1267 Commits

Author SHA1 Message Date
Anders Kaseorg c8bb98554e setup-apt-repo: Use /etc/os-release instead of lsb_release.
But still install lsb-release for now since Puppet acts funny without
it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-13 19:11:49 -08:00
Anders Kaseorg d1241be496 configure-rabbitmq: Use rabbitmqctl ping.
Our supported distributions now all have RabbitMQ ≥ 3.7.8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-13 19:09:41 -08:00
Tim Abbott 1a7c4a0276 scripts: Fix typo in logging statement. 2022-02-11 13:47:24 -08:00
Alex Vandiver 8da6098631 upgrade: Catch "upgrade" attempts which would downgrade the database.
Attempting to "upgrade" from `main` to 4.x should abort; Django does
not prevent running old code against the new database (though it
likely errors at runtime), and `./manage.py migrate` from the old
version during the "upgrade" does not downgrade the database, since
the migrations are entirely missing in that directory, so don't get
reversed.

Compare the list of applied migrations to the list of on-disk
migrations, and abort if there are applied migrations which are not
found on disk.

Fixes: #19284.
2022-02-10 16:02:49 -08:00
Alex Vandiver 71e02d7893 zulip_tools: Factor out ZULIP_VERSION parsing. 2022-02-10 16:02:49 -08:00
Anders Kaseorg e1f42c1ac5 docs: Add missing space to compound verbs “back up”, “log in”, etc.
Noun: backup, login, logout, lookup, setup.

Verb: back up, log in, log out, look up, set up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 19:20:54 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Alex Vandiver 2066860ab6 start-server: Start auxiliary services, if they exist.
Services like go-camo and smokescreen are not stopped in stop-server,
since they are upgraded and restarted by puppet application.  As such,
they also do not appear in start-server, despite the server relying on
them to be running to function properly.

Ensure those services are started, by starting them in start-server,
if they are configured in supervisor on the host.
2022-01-26 12:39:54 -08:00
Alex Vandiver 88c3f560ae supervisor: Add a filter for only(-not)-running. 2022-01-26 12:39:54 -08:00
Alex Vandiver 7243c3c73d scripts: Re-implement list_supervisor_processes using API. 2022-01-26 12:39:54 -08:00
Alex Vandiver 8e35cdb3da scripts: Add a supervisor package, to use the XMLRPC Supervisor API.
For many uses, shelling out to `supervisorctl` is going to produce
better error messages.  However, for instances where we wish to parse
the output of `supervisorctl`, using the API directly is less brittle.
2022-01-26 12:39:54 -08:00
Anders Kaseorg aec6cd4cdb reindex-textual-data: Find psycopg2 in the virtualenv.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-26 11:56:30 -08:00
Alex Vandiver a5496f4098 CVE-2021-43799: Set a secure Erlang cookie.
The RabbitMQ docs state ([1]):

    RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to
    determine whether they are allowed to communicate with each
    other. [...] The cookie is just a string of alphanumeric
    characters up to 255 characters in size. It is usually stored in a
    local file.

...and goes on to state (emphasis ours):

    If the file does not exist, Erlang VM will try to create one with
    a randomly generated value when the RabbitMQ server starts
    up. Using such generated cookie files are **appropriate in
    development environments only.**

The auto-generated cookie does not use cryptographic sources of
randomness, and generates 20 characters of `[A-Z]`.  Because of a
semi-predictable seed, the entropy of this password is thus less than
the idealized 26^20 = 94 bits of entropy; in actuality, it is 36 bits
of entropy, or potentially as low as 20 if the performance of the
server is known.

These sizes are well within the scope of remote brute-force attacks.

On provision, install, and upgrade, replace the default insecure
20-character Erlang cookie with a cryptographically secure
255-character string (the max length allowed).

[1] https://www.rabbitmq.com/clustering.html#erlang-cookie
2022-01-25 02:13:53 +00:00
Alex Vandiver 93a344fc3c configure-rabbitmq: Set -u, and not -x. 2022-01-25 01:52:36 +00:00
Alex Vandiver ece96c9729 configure-rabbitmq: Factor out sudo, instead of rabbitmqctl. 2022-01-25 01:52:36 +00:00
Alex Vandiver bd7deed691 upgrade: Show output from (re)starting zulip.
5c450afd2d, in ancient history, switched from `check_call` to
`check_output` and throwing away its result.

Use check_call, so that we show the steps to (re)starting the server.
2022-01-25 01:52:34 +00:00
Alex Vandiver e705883857 CVE-2021-43799: During upgrades, restart rabbitmq if necessary.
Check if it is listening on a public interface on port 25672, and if
so shut it down so it can pick up the new configuration.
2022-01-25 01:51:56 +00:00
Alex Vandiver da5201b986 upgrade: Make calling shutdown_server twice, only try once. 2022-01-25 01:48:05 +00:00
Alex Vandiver 43d63bd5a1 puppet: Always set the RabbitMQ nodename to zulip@localhost.
This is required in order to lock down the RabbitMQ port to only
listen on localhost.  If the nodename is `rabbit@hostname`, in most
circumstances the hostname will resolve to an external IP, which the
rabbitmq port will not be bound to.

Installs which used `rabbit@hostname`, due to RabbitMQ having been
installed before Zulip, would not have functioned if the host or
RabbitMQ service was restarted, as the localhost restrictions in the
RabbitMQ configuration would have made rabbitmqctl (and Zulip cron
jobs that call it) unable to find the rabbitmq server.

The previous commit ensures that configure-rabbitmq is re-run after
the nodename has changed.  However, rabbitmq needs to be stopped
before `rabbitmq-env.conf` is changed; we use an `onlyif` on an `exec`
to print the warning about the node change, and let the subsequent
config change and notify of the service and configure-rabbitmq to
complete the re-configuration.
2022-01-25 01:48:02 +00:00
Alex Vandiver 3bfcfeac24 puppet: Run configure-rabbitmq on nodename change.
`/etc/rabbitmq/rabbitmq-env.conf` sets the nodename; anytime the
nodename changes, the backing database changes, and this requires
re-creating the rabbitmq users and permissions.

Trigger this in puppet by running configure-rabbitmq after the file
changes.
2022-01-25 01:46:51 +00:00
Alex Vandiver b6cd89440e setup: Remove unused RABBITMQ_NODE.
This reverts commit 889547ff5e.  It is
unused in the Docker container, as the configurtaion of the `zulip`
user in the rabbitmq node is done via environment variables.  The
Zulip host in that context does not have `rabbitmqctl` installed, and
would have needed to know the Erlang cookie to be able to run these
commands.
2022-01-25 01:46:51 +00:00
Anders Kaseorg 21548ff7c0 install-node: Upgrade Node.js from 16.13.1 to 16.13.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-24 15:55:38 -08:00
Alex Vandiver a3adaf4aa3 puppet: Fix standalone certbot configurations.
This addresses the problems mentioned in the previous commit, but for
existing installations which have `authenticator = standalone` in
their configurations.

This reconfigures all hostnames in certbot to use the webroot
authenticator, and attempts to force-renew their certificates.
Force-renewal is necessary because certbot contains no way to merely
update the configuration.  Let's Encrypt allows for multiple extra
renewals per week, so this is a reasonable cost.

Because the certbot configuration is `configobj`, and not
`configparser`, we have no way to easily parse to determine if webroot
is in use; additionally, `certbot certificates` does not provide this
information.  We use `grep`, on the assumption that this will catch
nearly all cases.

It is possible that this will find `authenticator = standalone`
certificates which are managed by Certbot, but not Zulip certificates.
These certificates would also fail to renew while Zulip is running, so
switching them to use the Zulip webroot would still be an improvement.

Fixes #20593.
2022-01-24 12:13:44 -08:00
Alex Vandiver 76ce8631c0 setup: Install a temporary certificate, before certbot runs.
Installing certbot with --method=standalone means that the
configuration file will be written to assume that the standalone
method will be used going forward.  Since nginx will be running,
attempts to renew the certificate will fail.

Install a temporary self-signed certificate, just to allow nginx to
start, and then follow up (after applying puppet to start nginx) with
the call to setup-certbot, which will use the webroot authenticator.

The `setup-certbot --method=standalone` option is left intact, for use
in development environments.

Fixes part of #20593; it does not address installs which were
previously improperly configured with `authenticator = standalone`.
2022-01-24 12:13:44 -08:00
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Anders Kaseorg a58a71ef43 Remove Ubuntu 18.04 support.
As a consequence:

• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Alex Vandiver 677467f040 upgrade-zulip-from-git: Fix upstream URL for existing deploys. 2022-01-18 21:10:38 -08:00
Alex Vandiver bad58cdca6 upgrade-zulip-from-git: Fix the upstream URL not be the custom remote. 2022-01-18 21:10:38 -08:00
Alex Vandiver 6bc5849ea8 puppet: Remove now-unused debathena apt repository. 2022-01-18 14:13:28 -08:00
Anders Kaseorg e2cc554077 zulip_tools: Rename may_be_perform_purging to maybe_perform_purging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-12 13:21:35 -08:00
Alex Vandiver b31658482b upgrade-zulip: Pass any arguments down to upgrade-zulip-stage-2.
This is the equivalent of 93f3da4c05 but
for the tarball codepath.
2022-01-11 14:26:54 -08:00
Alex Vandiver 06e115bb00 zulip_tools: Switch get_deploy_options to use shlex.split.
This makes it honor quoting in the config file.
2022-01-11 14:26:54 -08:00
Anders Kaseorg 1cc1de82cd reindex-textual-data: Reindex textual functional indexes too.
This catches nine functional indexes that the previous query didn’t:

upper_preregistration_email_idx
upper_stream_name_idx
upper_subject_idx
upper_userprofile_email_idx
zerver_message_recipient_upper_subject
zerver_mutedtopic_stream_topic
zerver_stream_realm_id_name_uniq
zerver_userprofile_realm_id_delivery_email_uniq
zerver_userprofile_realm_id_email_uniq

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-07 10:37:04 -08:00
Alex Vandiver 6218ed91c2 puppet: Use lazy-apps and uwsgi control sockets for rolling reloads.
Restarting the uwsgi processes by way of supervisor opens a window
during which nginx 502's all responses.  uwsgi has a configuration
called "chain reloading" which allows for rolling restart of the uwsgi
processes, such that only one process at once in unavailable; see
uwsgi documentation ([1]).

The tradeoff is that this requires that the uwsgi processes load the
libraries after forking, rather than before ("lazy apps"); in theory
this can lead to larger memory footprints, since they are not shared.
In practice, as Django defers much of the loading, this is not as much
of an issue.  In a very basic test of memory consumption (measured by
total memory - free - caches - buffers; 6 uwsgi workers), both
immediately after restarting Django, and after requesting `/` 60 times
with 6 concurrent requests:

                      |  Non-lazy  |  Lazy app  | Difference
    ------------------+------------+------------+-------------
    Fresh             |  2,827,216 |  2,870,480 |   +43,264
    After 60 requests |  3,332,284 |  3,409,608 |   +77,324
    ..................|............|............|.............
    Difference        |   +505,068 |   +539,128 |   +34,060

That is, "lazy app" loading increased the footprint pre-requests by
43MB, and after 60 requests grew the memory footprint by 539MB, as
opposed to non-lazy loading, which grew it by 505MB.  Using wsgi "lazy
app" loading does increase the memory footprint, but not by a large
percentage.

The other effect is that processes may be served by either old or new
code during the restart window.  This may cause transient failures
when new frontend code talks to old backend code.

Enable chain-reloading during graceful, puppetless restarts, but only
if enabled via a zulip.conf configuration flag.

Fixes #2559.

[1]: https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#chain-reloading-lazy-apps
2022-01-05 14:48:52 -08:00
Alex Vandiver 4aaa250623 zulip_tools: Fix a typo in a comment. 2022-01-05 14:48:52 -08:00
Alex Vandiver 9d85f64e5a upgrade-zulip-stage-2: Pass through --skip-tornado and --less-graceful.
These restart-server arguments are useful to be able to provide to
`upgrade-zulip`.
2021-12-31 11:17:14 -08:00
Alex Vandiver fb3368b482 restart-server: Factor out argparser, to allow reuse. 2021-12-31 11:17:14 -08:00
Alex Vandiver 93f3da4c05 upgrade-from-git: Pass unknown options through to the upgrade process. 2021-12-31 11:17:14 -08:00
Anders Kaseorg 82748d45d8 install-yarn: Use test -ef in case /srv is a symlink.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-30 13:42:07 -08:00
Anders Kaseorg 0b454dda12 install: Try apt-get update if the Ubuntu universe check fails.
On a system where ‘apt-get update’ has never been run, ‘apt-cache
policy’ may show no repositories at all.  Try to correct this with
‘apt-get update’ before giving up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-16 17:56:23 -08:00
Alex Vandiver f6520a97cd setup-certbot: Reinstate nginx reload after installation.
If nginx was already installed, and we're using the webroot method of
initializing certbot, nginx needs to be reloaded.  Hooks in
`/etc/letsencrypt/renewal-hooks/deploy/` do not run during initial
`certbot certonly`, so an explicit reload is required.
2021-12-10 16:43:53 -08:00
Alex Vandiver 01e8f752a8 puppet: Use certbot package timer, not our own cron job.
The certbot package installs its own systemd timer (and cron job,
which disabled itself if systemd is enabled) which updates
certificates.  This process races with the cron job which Zulip
installs -- the only difference being that Zulip respects the
`certbot.auto_renew` setting, and that it passes the deploy hook.
This means that occasionally nginx would not be reloaded, when the
systemd timer caught the expiration first.

Remove the custom cron job and `certbot-maybe-renew` script, and
reconfigure certbot to always reload nginx after deploying, using
certbot directory hooks.

Since `certbot.auto_renew` can't have an effect, remove the setting.
In turn, this removes the need for `--no-zulip-conf` to
`setup-certbot`.  `--deploy-hook` is similarly removed, as running
deploy hooks to restart nginx is now the default; pass
`--no-directory-hooks` in standalone mode to not attempt to reload
nginx.  The other property of `--deploy-hook`, of skipping symlinking
into place, is given its own flog.
2021-12-09 13:47:33 -08:00
Tim Abbott 9aa2e0ad45 upgrade-zulip-from-git: Improve webpack failure error handling.
We've had a number of unhappy reports of upgrades failing due to
webpack requiring too much memory.  While the previous commit will
likely fix this issue for everyone, it's worth improving the error
message for failures here.

We avoid doing the stop+retry ourselves, because that could cause an
outage in a production system if webpack fails for another reason.

Fixes #20105.
2021-12-09 12:26:34 -08:00
Tim Abbott 72b381d749 upgrade-zulip-from-git: Require more memory to run webpack.
Since the upgrade to Webpack 5, we've been seeing occasional reports
that servers with roughly 4GiB of RAM were getting OOM kills while
running webpack.

Since we can't readily optimize the memory requirements for webpack
itself, we should raise the RAM requirements for doing the
lower-downtime upgrade strategy.

Fixes #20231.
2021-12-09 12:23:25 -08:00
Alex Vandiver 939d2e2705 scripts: Only stop/start existing tornado processes.
Stopping both `zulip-tornado` and `zulip-tornado:*` causes errors on
deploys with tornado sharding, as the plain `zulip-tornado` service
does not exist.

Pass `zulip-tornado:*`, which matches both plain `zulip-tornado`, as
well as the sharded `zulip-tornado:zulip-tornado-port-9800` cases.
2021-12-08 14:06:06 -08:00
Tim Abbott 73d503995a scripts: Fix running compare-settings-to-template from any CWD.
This matches the number of dirname() calls for other files in its
directory.

Fixes #20489.
2021-12-07 14:45:53 -08:00
Anders Kaseorg 2e5af073b7 install-node: Upgrade Node.js from 16.13.0 to 16.13.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-03 14:33:53 -08:00
Anders Kaseorg 2e1a8ff632 configure-rabbitmq: Increase startup timeout.
Starting RabbitMQ at boot seems to have gotten slower, which broke
‘vagrant up --provision’.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-03 14:32:23 -08:00
Alex Vandiver 3455fc137a upgrade-postgresql: Check for extension upgrade steps. 2021-11-20 07:13:50 -08:00
Alex Vandiver 544e8c569e install: Switch default to PostgreSQL 14. 2021-11-08 18:21:46 -08:00
Alex Vandiver f77bbd3323 upgrade-postgresql: Switch to vacuumdb --all --analzyze-only --jobs 10.
The `analyze_new_cluster.sh` script output by `pg_upgrade` just runs
`vacuumdb --all --analyze-in-stages`, which runs three passes over the
database, getting better stats each time.  Each of these passes is
independent; the third pass does not require the first two.
`--analyze-in-stages` is only provided to get "something" into the
database, on the theory that it could then be started and used.  Since
we wait for all three passes to complete before starting the database,
the first two passes add no value.

Additionally, PosttgreSQL 14 and up stop writing the
`analyze_new_cluster.sh` script as part of `pg_upgrade`, suggesting
the equivalent `vacuumdb --all --analyze-in-stages` call instead.

Switch to explicitly call `vacuumdb --all --analyze-only`, since we do
not gain any benefit from `--analyze-in-stages`.  We also enable
parallelism, with `--jobs 10`, in order to analyze up to 10 tables in
parallel.  This may increase load, but will accelerate the upgrade
process.
2021-11-08 18:21:46 -08:00
Anders Kaseorg f2a443a736 install-node: Upgrade Node.js from 14.18.1 to 16.13.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:34:13 -07:00
Anders Kaseorg 458844a2f5 install-yarn: Verify that the install location is /srv/zulip-yarn.
scripts.lib.node_cache expects Yarn to be in /srv/zulip-yarn, so if
it’s installed somewhere else, even if it’s the right version, we need
to reinstall it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-03 16:49:58 -07:00
rht bb8504d925 lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
Anders Kaseorg 291087d70c install-yarn: Upgrade Yarn from 1.22.11 to 1.22.17.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-17 07:15:09 -07:00
Anders Kaseorg 7df96b78c6 install-node: Upgrade Node.js from 14.17.6 to 14.18.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-17 07:15:09 -07:00
Anders Kaseorg 2f993f1a79 install-node: Stop using NVM.
NVM doesn’t check hashes or signatures and really just adds
complexity we don’t need.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-24 06:58:32 -07:00
Anders Kaseorg 902883d818 setup_venv: Skip virtualenv’s automatic download of setuptools.
It recently started failing on Debian 10 (buster).  We immediately
follow this by replacing these packages with our own versions from
pip.txt, anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-23 14:29:04 -07:00
Anders Kaseorg 08e459b393 zulip_tools: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-22 13:58:46 -07:00
Anders Kaseorg 9bed17e0ab install-node: Upgrade Node.js from 14.17.5 to 14.17.6.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-13 10:12:43 -07:00
Gaurav Pandey 502697d239 docs: Add documentation for bullseye support.
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.

Since now bullseye is released as a stable
version, it's support can be documented.
2021-09-09 11:05:16 -07:00
Anders Kaseorg 915884bff7 docs: Apply bullet style changes from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 02582c6956 upgrade-zulip-from-git: Run git fetch with --prune.
This prevents upgrading to an obsolete version of a branch that has
been deleted or renamed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-01 05:34:57 -07:00
Anders Kaseorg 3cb66d59ac install: Remove /dev/null redirect for zulip-puppet-apply.
The usual output from this command looks like

Notice: Compiled catalog for localhost in environment production in 2.33 seconds
Notice: /Stage[main]/Zulip::Apt_repository/Exec[setup_apt_repo]/returns: current_value 'notrun', should be ['0'] (noop)
Notice: Class[Zulip::Apt_repository]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 1.20 seconds

which doesn’t seem abnormally alarming, and hiding it makes failures
harder to diagnose.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 16:30:53 -07:00
Alex Vandiver faf71eea41 upgrade-postgresql: Do not remove other supervisor configs.
We previously used `zulip-puppet-apply` with a custom config file,
with an updated PostgreSQL version but more limited set of
`puppet_classes`, to pre-create the basic settings for the new cluster
before running `pg_upgradecluster`.

Unfortunately, the supervisor config uses `purge => true` to remove
all SUPERVISOR configuration files that are not included in the puppet
configuration; this leads to it removing all other supervisor
processes during the upgrade, only to add them back and start them
during the second `zulip-puppet-apply`.

It also leads to `process-fts-updates` not being started after the
upgrade completes; this is the one supervisor config file which was
not removed and re-added, and thus the one that is not re-started due
to having been re-added.  This was not detected in CI because CI added
a `start-server` command which was not in the upgrade documentation.

Set a custom facter fact that prevents the `purge` behaviour of the
supervisor configuration.  We want to preserve that behaviour in
general, and using `zulip-puppet-apply` continues to be the best way
to pre-set-up the PostgreSQL configuration -- but we wish to avoid
that behaviour when we know we are applying a subset of the puppet
classes.

Since supervisor configs are no longer removed and re-added, this
requires an explicit start-server step in the instructions after the
upgrades complete.  This brings the documentation into alignment with
what CI is testing.
2021-08-24 19:00:58 -07:00
Anders Kaseorg 7b2e585213 install-yarn: Upgrade Yarn from 1.22.10 to 1.22.11.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:33:27 -07:00
Anders Kaseorg ebb8e9109c install-node: Upgrade Node.js from 14.17.3 to 14.17.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:29:04 -07:00
Anders Kaseorg 4206e5f00b python: Remove locally dead code.
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
Alex Vandiver c9bb2c16cc restart-server: Add a --skip-tornado.
Tornado restarts are the most user-visible; provide a means to restart
everything but them, for changes which are known to not affect
Tornado.
2021-08-04 10:57:53 -07:00
Tim Abbott d439a2a53e emails: Create wider marketing email base template.
For our marketing emails, we want a width that's more appropriate for
newsletter context, vs. the narrow emails we use for transactional
content.

I haven't figured out a cleaner way to do this than duplicating most
of email_base_default.source.html. But it's not a big deal to
duplicate, since we've been changing that base template only about
once a year.
2021-08-03 11:57:31 -07:00
Anders Kaseorg 5483ebae37 python: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg ad5f0c05b5 python: Remove default "utf8" argument for encode(), decode().
Partially generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 1760897a8c python: Remove default "r" mode for open().
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
manavdesai27 572cef9a0f provision: Add support for Fedora 34. 2021-07-20 12:10:41 -07:00
Alex Vandiver 91282ab490 reindex-textual-data: Provide a tool to reindex all text indices.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic.  It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
2021-07-19 16:34:23 -07:00
Anders Kaseorg 47897c76a2 scripts: Use curl -f (--fail).
This makes curl exit with nonzero status on HTTP 4xx/5xx errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 16:47:49 -07:00
Alex Vandiver 16691110a6 scripts: Only stop/restart zulip_deliver_scheduled_* processes if known.
Running `supervisorctl stop` or `supervisorctl restart` on a process
name which is not known is an error:

```
$ supervisorctl stop nonexistent-process
nonexistent-process: ERROR (no such process)
$ echo $?
1
```

ef6d0ec5ca moved
zulip_deliver_scheduled_* out of the `workers:` group.  Since upgrades
run `stop-server` before applying puppet, the list of processes at
that time is from the previous version of Zulip, so may not have the
new `zulip_deliver_scheduled_*` names -- and the `stop-server` will
hence fail.

If the upgrade is not applying puppet, it will `restart-server`. At
that point, the old names will still be in the configuration, so
relying on the current `superisorctl status` is the best gauge of what
exists to restart.

In short, only ever stop/start/restart the `zulip_deliver_scheduled_*`
processes if `supervisorctl status` knows about them already.
2021-07-09 10:04:53 -07:00
Alex Vandiver c94bdd8534 zulip_tools: Find missing processes/groups in list_supervisor_processes.
Nonexistent processes and groups passed to `supervisortctl status` are
printed to STDOUT as follows:

```
$ supervisorctl status zulip-django nonexistent-process nonexistent-group:*
nonexistent-process: ERROR (no such process)
nonexistent-group: ERROR (no such group)
zulip-django                     RUNNING   pid 16043, uptime 17:31:31
```

On supervisor 4 and above, this exits with an exit code of 4;
previously, it returned exit code 0.  Ubuntu 18.04 has version 3.3.1,
and Ubuntu 20.04 has version 4.1.0.

Skip any lines with `ERROR (no such ...)`, and accept exit code 4 from
`supervisorctl status`.
2021-07-09 10:04:53 -07:00
Alex Vandiver 85a9c0982a zulip_tools: Extract out `list_supervisor_processes`. 2021-07-09 10:04:53 -07:00
Anders Kaseorg d83c91526b install-node: Upgrade Node.js from 14.17.0 to 14.17.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 14:51:24 -07:00
Anders Kaseorg 684dad8145 tools: Use root-based absolute import for tools.lib, etc.
Mypy can’t follow absolute imports based on directories other than the
root.  This was hiding some type errors due to ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:21:52 -07:00
Anders Kaseorg 7d71a1a31a setup: Add missing __init__.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:20:39 -07:00
Alex Vandiver 88c19bf65a puppet: Catch when a comma is left out of puppet_classes.
With two space-separated classes in `puppet_classes`, the second one
is silently ignored.  With three of more, puppet generates the
following very opaque error message:

```
Error: Could not parse for environment production: This
Name has no effect. A value was produced and then forgotten (one or
more preceding expressions may have the wrong form)
```

Catch when this has happened, and give an error message to the user.

Fixes #18992.
2021-06-28 20:58:56 -04:00
Anders Kaseorg 0ba9114c22 install-yarn: Rewrite Yarn installer.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-27 16:58:17 -07:00
Gaurav Pandey af08bcdb3f management: Delete send_stats command.
This command is part of a statsd infrastructure that we stopped
supporting years ago. Its only purpose for some time has been to
provide sample code for how the restart script might trigger a
notification to a graphing system, which doesn't justify maintaining
it.

Fixes part of #18898.
2021-06-25 09:13:48 -07:00
Anders Kaseorg 91bfebca7d install: Replace wget with curl.
curl uses Happy Eyeballs to avoid long timeouts on systems with broken
IPv6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-25 09:05:07 -07:00
Anders Kaseorg 3b60b25446 ci: Remove bullseye hack.
base-files 11.1 marked bullseye as Debian 11 in /etc/os-release.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Anders Kaseorg bf361e9951 ci: Remove uses of VERSION_CODENAME.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Tim Abbott 83738f7e6d install: Use a period at end of root error message. 2021-06-23 08:42:46 -07:00
Gaurav Pandey faae845366 upgrade: Modify upgrade scripts to handle failure.
The current `upgrade-zulip` and `upgrade-zulip-from-git`
bash scripts exit with a zero status even if the
upgrade commands exit with a non-zero status.
Hence add `set -e` command which exits the script with
the same status as the non-zero command.

For pipe commands however, the net status of a command
is the status of the last command, hence if the other parts
fail, the net status is only determined by the last command.
This is the case with our main /lib/upgrade-zulip* command
in the scripts whose status is determined by the `tee` command
instead. Hence add a small condition to get the status of the
actual upgrade command and exit the script if it fails with
a non-zero command.

We also check whether the script is being run as root, matching the
install script logic.
2021-06-23 08:42:20 -07:00
Tim Abbott 28d49edee3 script: Add --no-headings option to purge-old-deployments.
This parameter is somewhat useful, and adding this also fixes a
regression where purge-old-deployments would crash since the changes
around c5580607a7 because of
inconsistent supported args lists.
2021-06-17 15:49:23 -07:00
Mateusz Mandera 06c0a29e47 email-mirror-postfix: Choose scheme based on http_only config.
Fixes #16659.
If the server is behind a reverse proxy with http_only=True, the
requests made by email-mirror-postfix need to use http, as https
doesn't work.
2021-06-17 09:06:09 -07:00
Alex Vandiver d51272cc3d puppet: Remove zulip_deliver_scheduled_* from zulip-workers:*.
Staging and other hosts that are `zulip::app_frontend_base` but not
`zulip::app_frontend_once` do not have a
/etc/supervisor/conf.d/zulip/zulip-once.conf and as such do not have
`zulip_deliver_scheduled_emails` or `zulip_deliver_scheduled_messages`
and thus supervisor will fail to reload.

Making the contents of `zulip-workers` contingent on if the server is
_also_ a `-once` server is complicated, and would involve using Concat
fragments, which severely limit readability.

Instead, expel those two from `zulip-workers`; this is somewhat
reasonable, since they are use an entirely different codepath from
zulip_events_*, using the database rather than RabbitMQ for their
queuing.
2021-06-14 17:12:59 -07:00
Riken Shah c5580607a7 purge-old-deployments: Use the `clean_unused_caches.main` function.
We currently run the `clean_unused_caches.py` as a
script to clean the unused caches.

This commit replaces that with
`clean_unused_caches.main` function as it would be
faster.
2021-06-12 07:28:16 -07:00
Riken Shah 45af71e33b clean_unused_caches: Allow the main function to accept `Namespace` args.
This commit will allow us to pass the arguments in the
'clean...' functions when calling the `main` function (in
`provision`). It also changes args parsing
function location to `if __name__ == "__main__"` block as
we wouldn't need it to parse args when we call the
function.
2021-06-12 07:28:16 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Anders Kaseorg d8cb418586 zulip_tools: Flush ‘set -x’-style messages in run.
Otherwise they often get buffered until after the command actually
runs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg 342834ee9c python: Simplify stdio flushing using print(…, flush=True).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg bc169d63a7 install-node: Upgrade Node.js from 14.16.1 to 14.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:02:12 -07:00
Anders Kaseorg 61e1e38a00 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-07 17:57:51 -07:00
Alex Vandiver 1cdf14d195 puppet: Add a teleport server.
See https://goteleport.com/docs/architecture/overview/ for the general
architecture of a Teleport cluster.  This commit adds a Teleport auth[1]
and proxy[2] server.  The auth server serves as a CA for granting
time-bounded access to users and authenticating nodes on the cluster;
the proxy provides access and a management UI.

[1] https://goteleport.com/docs/architecture/authentication/
[2] https://goteleport.com/docs/architecture/proxy/
2021-06-02 18:38:38 -07:00
Alex Vandiver e080a05b05 node_cache: Serialize to structured data before hashing.
Appending data back-to-back without serializing it loses the
information about where the breaks between them lie, which can lead to
different inputs having the same hash.
2021-05-27 22:47:56 -07:00
Alex Vandiver 87a109e3e0 puppet: Pull in pinned puppet modules.
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
2021-05-27 21:14:48 -07:00
Anders Kaseorg cb8d9a1f8a create-db: Default dbuser and dbname to zulip.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-26 17:19:11 -07:00
Alex Vandiver f3eea72c2a setup: Merge multiple setup-apt-repo scripts into one.
This moves the `.asc` files into subdirectories, and writes out the
according `.list` files into them.  It moves from templates to
written-out `.list` files for clarity and ease of
implementation (Debian and Ubuntu need different templates for
`zulip`), and as a way of making explicit which releases are supported
for each list.  For the special-case of the PGroonga signing key, we
source an additional file within the directory.

This simplifies the process for adding another class of `.list` file.
2021-05-26 14:42:29 -07:00
Adam Birds 4539899cae installer: Add support for custom database user and dbname.
Add support for custom database names and database users, which can be
set with the `--postgresql-database-name` and
`--postgresql-database-user` install script options. If these
parameters aren't provided, then the defaults remain "zulip".

Fixes #17662.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-05-25 13:56:05 -07:00
Alex Vandiver 7ff3c9f966 upgrade-zulip: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Alex Vandiver 1d59330cbc postgresql-init-db: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:04 -07:00
Alex Vandiver 54c222d3f8 settings: Support arbitrary database user and dbname.
This adds basic support for `postgresql.database_user` and
`postgresql.database_name` settings in `zulip.conf`; the defaults if
unspecified are left as `zulip`.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Adam Birds 21cc186105 installer: Add run_psql_as_postgres function zulip_tools.py.
Add a helper `run_psql_as_postgres` function in
`scripts/lib/zulip_tools.py`.  This is preparatory refactoring for the
work to add custom database and user names.
2021-05-24 16:58:11 -07:00
Alex Vandiver 81644f110e install: $ZULIP_ADMINISTRATOR may be unset for non-frontend hosts. 2021-05-23 13:29:23 -07:00
Anders Kaseorg 09f6ba1971 install: Run git config commands from a known readable cwd.
Fixes this error when running the installer from a directory that
isn’t world-readable:

+ su zulip -c 'git config --global user.email anders@zulip.com'
fatal: cannot come back to cwd: Permission denied

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Anders Kaseorg bc45525369 postgresql-init-db: Fix installation from world-unreadable directory.
This reverts part of commit 476524c0c1
(#18215), to fix this error when running the installer from a
directory that isn’t world-readable:

+ '[' -e /var/run/supervisor.sock ']'
+++ dirname /root/zulip-server-4.1/scripts/setup/postgresql-init-db
++ dirname /root/zulip-server-4.1/scripts/setup
+ su zulip -c /root/zulip-server-4.1/scripts/stop-server
bash: /root/zulip-server-4.1/scripts/stop-server: Permission denied

Zulip installation failed (exit code 126)!

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Anders Kaseorg 6766a3f780 purge-old-deployments: Check /srv/zulip.git existence before pruning it.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 20:36:27 -07:00
Tim Abbott de47feab43 scripts: Fix check for services running when upgrading.
When upgrading from a pre-4.0 release, scripts/stop-server logic would
check whether supervisord configuration files were present to
determine what it needed to restart, but only considered paths to
those files that are introduced in Zulip 4.0.

Fixed #18493.
2021-05-13 18:57:19 -07:00
Anders Kaseorg 3f83b843c2 upgrade-zulip-from-git: Create deployment directories with git worktree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 13:42:05 -07:00
Tim Abbott 03420831b0 upgrade-zulip-from-git: Fetch tags from upstream repository.
This ensures that the `git describe` queries that we run for caching
Zulip's Git version are guaranteed to include recent releases.

This change ensures that we have accurate output even if we're pointed
at a fork of Zulip that never updates its tags.

Additionally, it will make it possible to record the
`git merge-base upstream/master` in future commits.

Note that because we run this code before unpacking the new version,
the pre-upgrade version of this code runs.

As a result, we cannot assume that the upstream repository exists.
2021-05-13 11:17:25 -07:00
Alex Vandiver 3ccb77da74 install: Tell NVM to not change $PATH earlier.
This removes a possible window where an installer error could leave
`nvm` in a state where it had prepended the full path to the
newly-installed `npm` to `$PATH`; we would like to avoid `nvm`
fiddling with path whenever possible (ref ebe930ab2c).
2021-05-11 11:25:34 -10:00
Anders Kaseorg 9ba48c4ed3 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Robert Imschweiler 534d78232c scripts: Add {start,stop,restart}-server support for postgresql role.
During the upgrade process of a postgresql-only Zulip installation,
(`puppet_classes = zulip::profile::postgresql` in
`/etc/zulip/zulip.conf`) either `scripts/start-server` or
`scripts/stop-server` fail because they try to handle supervisor
services that are not available (e.g. Tornado) since only
`/etc/supervisor/conf.d/zulip/zulip_db.conf` is present and not
`/etc/supervisor/conf.d/zulip/zulip.conf`.

While this wasn't previously supported, it's a pretty reasonable thing
to do, and can be readily supported by just adding a few conditionals.
2021-05-07 09:41:05 -07:00
Anders Kaseorg 9d57fa9759 puppet: Use pgrep -x to avoid accidental matches.
Matching the full process name (-x without -f) or full command
line (-xf) is less prone to mistakes like matching a random substring
of some other command line or pgrep matching itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 08:54:41 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Alex Vandiver eda9ce2364 locale: Use `C.UTF-8` rather than `en_US.UTF-8`.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.

Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.

Fixes #15819.
2021-05-04 08:51:46 -07:00
Mateusz Mandera dd7f3a1dce upgrade: Use restart-server unless --skip-puppet is used.
In some cases, puppet can end up restarting supervisord services - which
will use code from the old deployment, because when puppet runs,
/home/zulip/deployments/current still points there. Thus restart-server
needs to be used in favor of start-server, unless we know that puppet
has been skipped.
2021-05-03 08:12:54 -07:00
Alex Vandiver ebe930ab2c upgrade: Set an explicit value for PATH.
Previous versions of zulip used `nvm alias default ...` to have `nvm`
prepend the full path to the latest `node` install to the `PATH` in
root's shell.  Unfortunately, this means that `update-prod-static`,
when called from `upgrade-zulip-stage-2` after an upgrade of node in
`install-node`, would still have the full path to the _old_ `node` at
the start of its PATH, because the PATH of `upgrade-zulip-stage-2`
would still be unchanged.

Bootstrap out of this by setting a known-reasonable PATH during
upgrade, and remove the problematic `nvm alias default` behaviour.

Fixes #18258.
2021-05-01 07:16:45 -07:00
Alex Vandiver 49144247dd install: Set explicit value for PATH.
In Debian, becoming root as `su` does not alter the `$PATH`; this can
lead to the root user not having `/usr/sbin` in its path, and thus
the `useradd zulip` step of the installer fails.

Fixes #17441.
2021-05-01 07:16:45 -07:00
Alex Vandiver daabc52a78 restart-server: Reorder supervisorctl calls for less downtime.
Instead of taking the "onion" approach, where all services are
stopped, and then started back up again, default to a rolling restart
across all processes.  This draws out how long the overall "restart"
takes, but minimizes the time that any of the services are down.  This
minimizes user-visible impact and queue buildup.

In cases where speed is more important than minimal impact (for
example, there is already a current outage), a --less-graceful flag is
provided, which brings the services down more suddenly, and back up in
a still-correct order.
2021-04-30 16:47:15 -07:00
Alex Vandiver 4c88da8ed9 scripts: Tool to find the diff to an original settings.py prod template.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour.  This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
2021-04-27 21:50:33 -07:00
Alex Vandiver ae2c377d13 postgresql: Switch to defaulting to PostgreSQL 13. 2021-04-27 16:55:04 -07:00
Robert Imschweiler ba25580b19 clean-unused-caches: Handle non-existent yarn cache. 2021-04-27 10:02:49 -07:00
Riken Shah 1288dcbaaf clean-unused-caches: Add script to remove redundant yarn cache.
This commit removes redundant yarn cache by removing the old
version directories, i.e. All the directory under `~/.cache/yarn`
except `~/.cache/yarn/v6` (current version directory).

Fixes #15964.
2021-04-26 16:28:08 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Alex Vandiver 6db454b252 upgrade: Call start-server rather than restart-server if we stopped it.
This saves a little time, and thus causes a shorter outage window,
since we will not try to stop the services; we know they are already
down.
2021-04-21 10:28:30 -07:00
Alex Vandiver 16650ba239 upgrade: Call ./scripts/stop-server rather than duplicate the logic. 2021-04-21 10:28:30 -07:00
Alex Vandiver ec12a6128a scripts: Add a start-server as well.
In general, `./scripts/restart-server` will already work in any
circumstance where the server is already stopped and needs to be
started.  However, it will output a couple minor warnings, and it is
not readily obvious that it *will* work correctly.

Add an alias for `restart-server` named `start-server`, for
parallelism with `stop-server`, which omits the steps of
`restart-server` which would stop the server first.
2021-04-21 10:24:08 -07:00
Alex Vandiver 476524c0c1 scripts: Add a script to stop the server.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.

Add an explicit tool which only stops the relevant services.  It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.

Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.

Fixes #14959.
2021-04-21 10:24:08 -07:00
Alex Vandiver 31169526ec scripts: Say "Zulip" rather than "Application". 2021-04-21 10:24:08 -07:00
Alex Vandiver 0de8357820 scripts: Fix path to additional Zulip supervisor files.
The path which contains all of the Zulip supervisor files changed in
3ab9b31d2f to make it easier to purge
now-unwanted supervisor configuration files.  However, the paths that
the zulip upgrade process, and restart-server, look at were not
adjusted.

Fix the supervisor configuration file paths.
2021-04-21 10:24:08 -07:00
Alex Vandiver de41a10d38
upgrade: Install python3-yaml as needed.
3314fefaec started needing `python3-yaml`, but incorrectly claimed
that it was always an indirect dependency; it is a dependency of
`ubuntu-minimal` on 20.04, but not required on 18.04 or Debian.  We
cannot install it in puppet because then is definitionally too late;
it is needed at load time by `zulip-puppet-apply`.

Install `python3-yaml`, but guarded by a simple check so as to not
further slow most installs.

Fixes #18179.
2021-04-21 09:52:56 -07:00
Alex Vandiver 4c8502f7fd upgrade: Show fewer stacktraces.
The stacktraces here are seldom useful -- for the calls to
upgrade-stage-2, we know precisely what was run.  For the `run`
wrapper, the output contains the command that failed, which is
sufficient to identify where in the upgrade process it was.  Showing
more stacktrace below the actual error merely confuses users and
scrolls the real error off of the screen.
2021-04-21 09:51:40 -07:00
Siddharth Asthana d2706fa246 install: Create a .gitconfig file for the zulip user.
For installs which use the `upgrade-zulip-from-git` process, the
deployment directory is a git checkout. This means that an
administrator can, as an emergency tool, run `git revert` and similar
commands -- assuming there is a `~/.gitconfig` set up for the zulip
user.

Add commands to `scripts/lib/install` to create a `~/.gitconfig` file
at installation time.  The `user.name` and `user.email` fields are set
to the hostname and passed-in `--email` value, respectively.

Fixes #18039.
2021-04-20 22:47:20 -07:00
Gaurav Pandey feb720b463 install: Add beta support for debian bullseye for production.
This won't work on a real bullseye system until Bullseye actually
officially releases.

Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Gaurav Pandey 78524d4f87 provision: Add support for debian bullseye.
Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Anders Kaseorg b6b117274c install-node: Upgrade Node.js to 14.16.1 and nvm to 0.38.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Ganesh Pawar c1628e7605 provision: Upgrade support for Fedora to version 33.
Note that the `overwrite_symlink` changes fix a bug introduced in
5c20ee998c, that we need root
permissions to do those operations.
2021-03-22 19:34:18 -07:00
Ganesh Pawar 666ab59b03 pgroonga: Bump pgroonga version to 2.2.8 when building from source. 2021-03-22 19:33:48 -07:00
Ganesh Pawar 7cdb26108c minor: Avoid verbose tar output.
It isn't much helpful and clutters the logs.
2021-03-22 19:33:48 -07:00
Anders Kaseorg 6364e1b5f3 requirements: Upgrade talon fork to 1.4.8.
https://github.com/mailgun/talon/pull/200

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-18 17:10:18 -07:00