Commit Graph

1186 Commits

Author SHA1 Message Date
Anders Kaseorg 684dad8145 tools: Use root-based absolute import for tools.lib, etc.
Mypy can’t follow absolute imports based on directories other than the
root.  This was hiding some type errors due to ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:21:52 -07:00
Anders Kaseorg 7d71a1a31a setup: Add missing __init__.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:20:39 -07:00
Alex Vandiver 88c19bf65a puppet: Catch when a comma is left out of puppet_classes.
With two space-separated classes in `puppet_classes`, the second one
is silently ignored.  With three of more, puppet generates the
following very opaque error message:

```
Error: Could not parse for environment production: This
Name has no effect. A value was produced and then forgotten (one or
more preceding expressions may have the wrong form)
```

Catch when this has happened, and give an error message to the user.

Fixes #18992.
2021-06-28 20:58:56 -04:00
Anders Kaseorg 0ba9114c22 install-yarn: Rewrite Yarn installer.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-27 16:58:17 -07:00
Gaurav Pandey af08bcdb3f management: Delete send_stats command.
This command is part of a statsd infrastructure that we stopped
supporting years ago. Its only purpose for some time has been to
provide sample code for how the restart script might trigger a
notification to a graphing system, which doesn't justify maintaining
it.

Fixes part of #18898.
2021-06-25 09:13:48 -07:00
Anders Kaseorg 91bfebca7d install: Replace wget with curl.
curl uses Happy Eyeballs to avoid long timeouts on systems with broken
IPv6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-25 09:05:07 -07:00
Anders Kaseorg 3b60b25446 ci: Remove bullseye hack.
base-files 11.1 marked bullseye as Debian 11 in /etc/os-release.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Anders Kaseorg bf361e9951 ci: Remove uses of VERSION_CODENAME.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-24 14:35:51 -07:00
Tim Abbott 83738f7e6d install: Use a period at end of root error message. 2021-06-23 08:42:46 -07:00
Gaurav Pandey faae845366 upgrade: Modify upgrade scripts to handle failure.
The current `upgrade-zulip` and `upgrade-zulip-from-git`
bash scripts exit with a zero status even if the
upgrade commands exit with a non-zero status.
Hence add `set -e` command which exits the script with
the same status as the non-zero command.

For pipe commands however, the net status of a command
is the status of the last command, hence if the other parts
fail, the net status is only determined by the last command.
This is the case with our main /lib/upgrade-zulip* command
in the scripts whose status is determined by the `tee` command
instead. Hence add a small condition to get the status of the
actual upgrade command and exit the script if it fails with
a non-zero command.

We also check whether the script is being run as root, matching the
install script logic.
2021-06-23 08:42:20 -07:00
Tim Abbott 28d49edee3 script: Add --no-headings option to purge-old-deployments.
This parameter is somewhat useful, and adding this also fixes a
regression where purge-old-deployments would crash since the changes
around c5580607a7 because of
inconsistent supported args lists.
2021-06-17 15:49:23 -07:00
Mateusz Mandera 06c0a29e47 email-mirror-postfix: Choose scheme based on http_only config.
Fixes #16659.
If the server is behind a reverse proxy with http_only=True, the
requests made by email-mirror-postfix need to use http, as https
doesn't work.
2021-06-17 09:06:09 -07:00
Alex Vandiver d51272cc3d puppet: Remove zulip_deliver_scheduled_* from zulip-workers:*.
Staging and other hosts that are `zulip::app_frontend_base` but not
`zulip::app_frontend_once` do not have a
/etc/supervisor/conf.d/zulip/zulip-once.conf and as such do not have
`zulip_deliver_scheduled_emails` or `zulip_deliver_scheduled_messages`
and thus supervisor will fail to reload.

Making the contents of `zulip-workers` contingent on if the server is
_also_ a `-once` server is complicated, and would involve using Concat
fragments, which severely limit readability.

Instead, expel those two from `zulip-workers`; this is somewhat
reasonable, since they are use an entirely different codepath from
zulip_events_*, using the database rather than RabbitMQ for their
queuing.
2021-06-14 17:12:59 -07:00
Riken Shah c5580607a7 purge-old-deployments: Use the `clean_unused_caches.main` function.
We currently run the `clean_unused_caches.py` as a
script to clean the unused caches.

This commit replaces that with
`clean_unused_caches.main` function as it would be
faster.
2021-06-12 07:28:16 -07:00
Riken Shah 45af71e33b clean_unused_caches: Allow the main function to accept `Namespace` args.
This commit will allow us to pass the arguments in the
'clean...' functions when calling the `main` function (in
`provision`). It also changes args parsing
function location to `if __name__ == "__main__"` block as
we wouldn't need it to parse args when we call the
function.
2021-06-12 07:28:16 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Anders Kaseorg d8cb418586 zulip_tools: Flush ‘set -x’-style messages in run.
Otherwise they often get buffered until after the command actually
runs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg 342834ee9c python: Simplify stdio flushing using print(…, flush=True).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg bc169d63a7 install-node: Upgrade Node.js from 14.16.1 to 14.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:02:12 -07:00
Anders Kaseorg 61e1e38a00 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-07 17:57:51 -07:00
Alex Vandiver 1cdf14d195 puppet: Add a teleport server.
See https://goteleport.com/docs/architecture/overview/ for the general
architecture of a Teleport cluster.  This commit adds a Teleport auth[1]
and proxy[2] server.  The auth server serves as a CA for granting
time-bounded access to users and authenticating nodes on the cluster;
the proxy provides access and a management UI.

[1] https://goteleport.com/docs/architecture/authentication/
[2] https://goteleport.com/docs/architecture/proxy/
2021-06-02 18:38:38 -07:00
Alex Vandiver e080a05b05 node_cache: Serialize to structured data before hashing.
Appending data back-to-back without serializing it loses the
information about where the breaks between them lie, which can lead to
different inputs having the same hash.
2021-05-27 22:47:56 -07:00
Alex Vandiver 87a109e3e0 puppet: Pull in pinned puppet modules.
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
2021-05-27 21:14:48 -07:00
Anders Kaseorg cb8d9a1f8a create-db: Default dbuser and dbname to zulip.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-26 17:19:11 -07:00
Alex Vandiver f3eea72c2a setup: Merge multiple setup-apt-repo scripts into one.
This moves the `.asc` files into subdirectories, and writes out the
according `.list` files into them.  It moves from templates to
written-out `.list` files for clarity and ease of
implementation (Debian and Ubuntu need different templates for
`zulip`), and as a way of making explicit which releases are supported
for each list.  For the special-case of the PGroonga signing key, we
source an additional file within the directory.

This simplifies the process for adding another class of `.list` file.
2021-05-26 14:42:29 -07:00
Adam Birds 4539899cae installer: Add support for custom database user and dbname.
Add support for custom database names and database users, which can be
set with the `--postgresql-database-name` and
`--postgresql-database-user` install script options. If these
parameters aren't provided, then the defaults remain "zulip".

Fixes #17662.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-05-25 13:56:05 -07:00
Alex Vandiver 7ff3c9f966 upgrade-zulip: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Alex Vandiver 1d59330cbc postgresql-init-db: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:04 -07:00
Alex Vandiver 54c222d3f8 settings: Support arbitrary database user and dbname.
This adds basic support for `postgresql.database_user` and
`postgresql.database_name` settings in `zulip.conf`; the defaults if
unspecified are left as `zulip`.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Adam Birds 21cc186105 installer: Add run_psql_as_postgres function zulip_tools.py.
Add a helper `run_psql_as_postgres` function in
`scripts/lib/zulip_tools.py`.  This is preparatory refactoring for the
work to add custom database and user names.
2021-05-24 16:58:11 -07:00
Alex Vandiver 81644f110e install: $ZULIP_ADMINISTRATOR may be unset for non-frontend hosts. 2021-05-23 13:29:23 -07:00
Anders Kaseorg 09f6ba1971 install: Run git config commands from a known readable cwd.
Fixes this error when running the installer from a directory that
isn’t world-readable:

+ su zulip -c 'git config --global user.email anders@zulip.com'
fatal: cannot come back to cwd: Permission denied

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Anders Kaseorg bc45525369 postgresql-init-db: Fix installation from world-unreadable directory.
This reverts part of commit 476524c0c1
(#18215), to fix this error when running the installer from a
directory that isn’t world-readable:

+ '[' -e /var/run/supervisor.sock ']'
+++ dirname /root/zulip-server-4.1/scripts/setup/postgresql-init-db
++ dirname /root/zulip-server-4.1/scripts/setup
+ su zulip -c /root/zulip-server-4.1/scripts/stop-server
bash: /root/zulip-server-4.1/scripts/stop-server: Permission denied

Zulip installation failed (exit code 126)!

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 22:00:29 -07:00
Anders Kaseorg 6766a3f780 purge-old-deployments: Check /srv/zulip.git existence before pruning it.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 20:36:27 -07:00
Tim Abbott de47feab43 scripts: Fix check for services running when upgrading.
When upgrading from a pre-4.0 release, scripts/stop-server logic would
check whether supervisord configuration files were present to
determine what it needed to restart, but only considered paths to
those files that are introduced in Zulip 4.0.

Fixed #18493.
2021-05-13 18:57:19 -07:00
Anders Kaseorg 3f83b843c2 upgrade-zulip-from-git: Create deployment directories with git worktree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 13:42:05 -07:00
Tim Abbott 03420831b0 upgrade-zulip-from-git: Fetch tags from upstream repository.
This ensures that the `git describe` queries that we run for caching
Zulip's Git version are guaranteed to include recent releases.

This change ensures that we have accurate output even if we're pointed
at a fork of Zulip that never updates its tags.

Additionally, it will make it possible to record the
`git merge-base upstream/master` in future commits.

Note that because we run this code before unpacking the new version,
the pre-upgrade version of this code runs.

As a result, we cannot assume that the upstream repository exists.
2021-05-13 11:17:25 -07:00
Alex Vandiver 3ccb77da74 install: Tell NVM to not change $PATH earlier.
This removes a possible window where an installer error could leave
`nvm` in a state where it had prepended the full path to the
newly-installed `npm` to `$PATH`; we would like to avoid `nvm`
fiddling with path whenever possible (ref ebe930ab2c).
2021-05-11 11:25:34 -10:00
Anders Kaseorg 9ba48c4ed3 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Robert Imschweiler 534d78232c scripts: Add {start,stop,restart}-server support for postgresql role.
During the upgrade process of a postgresql-only Zulip installation,
(`puppet_classes = zulip::profile::postgresql` in
`/etc/zulip/zulip.conf`) either `scripts/start-server` or
`scripts/stop-server` fail because they try to handle supervisor
services that are not available (e.g. Tornado) since only
`/etc/supervisor/conf.d/zulip/zulip_db.conf` is present and not
`/etc/supervisor/conf.d/zulip/zulip.conf`.

While this wasn't previously supported, it's a pretty reasonable thing
to do, and can be readily supported by just adding a few conditionals.
2021-05-07 09:41:05 -07:00
Anders Kaseorg 9d57fa9759 puppet: Use pgrep -x to avoid accidental matches.
Matching the full process name (-x without -f) or full command
line (-xf) is less prone to mistakes like matching a random substring
of some other command line or pgrep matching itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 08:54:41 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Alex Vandiver eda9ce2364 locale: Use `C.UTF-8` rather than `en_US.UTF-8`.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.

Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.

Fixes #15819.
2021-05-04 08:51:46 -07:00
Mateusz Mandera dd7f3a1dce upgrade: Use restart-server unless --skip-puppet is used.
In some cases, puppet can end up restarting supervisord services - which
will use code from the old deployment, because when puppet runs,
/home/zulip/deployments/current still points there. Thus restart-server
needs to be used in favor of start-server, unless we know that puppet
has been skipped.
2021-05-03 08:12:54 -07:00
Alex Vandiver ebe930ab2c upgrade: Set an explicit value for PATH.
Previous versions of zulip used `nvm alias default ...` to have `nvm`
prepend the full path to the latest `node` install to the `PATH` in
root's shell.  Unfortunately, this means that `update-prod-static`,
when called from `upgrade-zulip-stage-2` after an upgrade of node in
`install-node`, would still have the full path to the _old_ `node` at
the start of its PATH, because the PATH of `upgrade-zulip-stage-2`
would still be unchanged.

Bootstrap out of this by setting a known-reasonable PATH during
upgrade, and remove the problematic `nvm alias default` behaviour.

Fixes #18258.
2021-05-01 07:16:45 -07:00
Alex Vandiver 49144247dd install: Set explicit value for PATH.
In Debian, becoming root as `su` does not alter the `$PATH`; this can
lead to the root user not having `/usr/sbin` in its path, and thus
the `useradd zulip` step of the installer fails.

Fixes #17441.
2021-05-01 07:16:45 -07:00
Alex Vandiver daabc52a78 restart-server: Reorder supervisorctl calls for less downtime.
Instead of taking the "onion" approach, where all services are
stopped, and then started back up again, default to a rolling restart
across all processes.  This draws out how long the overall "restart"
takes, but minimizes the time that any of the services are down.  This
minimizes user-visible impact and queue buildup.

In cases where speed is more important than minimal impact (for
example, there is already a current outage), a --less-graceful flag is
provided, which brings the services down more suddenly, and back up in
a still-correct order.
2021-04-30 16:47:15 -07:00
Alex Vandiver 4c88da8ed9 scripts: Tool to find the diff to an original settings.py prod template.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour.  This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
2021-04-27 21:50:33 -07:00
Alex Vandiver ae2c377d13 postgresql: Switch to defaulting to PostgreSQL 13. 2021-04-27 16:55:04 -07:00
Robert Imschweiler ba25580b19 clean-unused-caches: Handle non-existent yarn cache. 2021-04-27 10:02:49 -07:00
Riken Shah 1288dcbaaf clean-unused-caches: Add script to remove redundant yarn cache.
This commit removes redundant yarn cache by removing the old
version directories, i.e. All the directory under `~/.cache/yarn`
except `~/.cache/yarn/v6` (current version directory).

Fixes #15964.
2021-04-26 16:28:08 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Alex Vandiver 6db454b252 upgrade: Call start-server rather than restart-server if we stopped it.
This saves a little time, and thus causes a shorter outage window,
since we will not try to stop the services; we know they are already
down.
2021-04-21 10:28:30 -07:00
Alex Vandiver 16650ba239 upgrade: Call ./scripts/stop-server rather than duplicate the logic. 2021-04-21 10:28:30 -07:00
Alex Vandiver ec12a6128a scripts: Add a start-server as well.
In general, `./scripts/restart-server` will already work in any
circumstance where the server is already stopped and needs to be
started.  However, it will output a couple minor warnings, and it is
not readily obvious that it *will* work correctly.

Add an alias for `restart-server` named `start-server`, for
parallelism with `stop-server`, which omits the steps of
`restart-server` which would stop the server first.
2021-04-21 10:24:08 -07:00
Alex Vandiver 476524c0c1 scripts: Add a script to stop the server.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.

Add an explicit tool which only stops the relevant services.  It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.

Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.

Fixes #14959.
2021-04-21 10:24:08 -07:00
Alex Vandiver 31169526ec scripts: Say "Zulip" rather than "Application". 2021-04-21 10:24:08 -07:00
Alex Vandiver 0de8357820 scripts: Fix path to additional Zulip supervisor files.
The path which contains all of the Zulip supervisor files changed in
3ab9b31d2f to make it easier to purge
now-unwanted supervisor configuration files.  However, the paths that
the zulip upgrade process, and restart-server, look at were not
adjusted.

Fix the supervisor configuration file paths.
2021-04-21 10:24:08 -07:00
Alex Vandiver de41a10d38
upgrade: Install python3-yaml as needed.
3314fefaec started needing `python3-yaml`, but incorrectly claimed
that it was always an indirect dependency; it is a dependency of
`ubuntu-minimal` on 20.04, but not required on 18.04 or Debian.  We
cannot install it in puppet because then is definitionally too late;
it is needed at load time by `zulip-puppet-apply`.

Install `python3-yaml`, but guarded by a simple check so as to not
further slow most installs.

Fixes #18179.
2021-04-21 09:52:56 -07:00
Alex Vandiver 4c8502f7fd upgrade: Show fewer stacktraces.
The stacktraces here are seldom useful -- for the calls to
upgrade-stage-2, we know precisely what was run.  For the `run`
wrapper, the output contains the command that failed, which is
sufficient to identify where in the upgrade process it was.  Showing
more stacktrace below the actual error merely confuses users and
scrolls the real error off of the screen.
2021-04-21 09:51:40 -07:00
Siddharth Asthana d2706fa246 install: Create a .gitconfig file for the zulip user.
For installs which use the `upgrade-zulip-from-git` process, the
deployment directory is a git checkout. This means that an
administrator can, as an emergency tool, run `git revert` and similar
commands -- assuming there is a `~/.gitconfig` set up for the zulip
user.

Add commands to `scripts/lib/install` to create a `~/.gitconfig` file
at installation time.  The `user.name` and `user.email` fields are set
to the hostname and passed-in `--email` value, respectively.

Fixes #18039.
2021-04-20 22:47:20 -07:00
Gaurav Pandey feb720b463 install: Add beta support for debian bullseye for production.
This won't work on a real bullseye system until Bullseye actually
officially releases.

Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Gaurav Pandey 78524d4f87 provision: Add support for debian bullseye.
Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Anders Kaseorg b6b117274c install-node: Upgrade Node.js to 14.16.1 and nvm to 0.38.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Ganesh Pawar c1628e7605 provision: Upgrade support for Fedora to version 33.
Note that the `overwrite_symlink` changes fix a bug introduced in
5c20ee998c, that we need root
permissions to do those operations.
2021-03-22 19:34:18 -07:00
Ganesh Pawar 666ab59b03 pgroonga: Bump pgroonga version to 2.2.8 when building from source. 2021-03-22 19:33:48 -07:00
Ganesh Pawar 7cdb26108c minor: Avoid verbose tar output.
It isn't much helpful and clutters the logs.
2021-03-22 19:33:48 -07:00
Anders Kaseorg 6364e1b5f3 requirements: Upgrade talon fork to 1.4.8.
https://github.com/mailgun/talon/pull/200

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-18 17:10:18 -07:00
Alex Vandiver 3314fefaec puppet: Do not require a venv for zulip-puppet-apply.
0663b23d54 changed zulip-puppet-apply to
use the venv, because it began using `yaml` to parse the output of
puppet to determine if changes would happen.

However, not every install ends with a venv; notably, non-frontend
servers do not have one.  Attempting to run zulip-puppet-apply on them
hence now fails.

Remove this dependency on the venv, by installing a system
python3-yaml package -- though in reality, this package is already an
indirect dependency of the system.  Especially since pyyaml is quite
stable, we're not using it in any interesting way, and it does not
actually add to the dependencies, it is preferable to parsing the YAML
by hand in this instance.
2021-03-14 17:50:57 -07:00
Alex Vandiver 52f155873f puppet: Ensure that all `scripts/lib/install` packages are installed.
These have all been required packages for some time, but this helps
keep the install-time list more clearly a subset of the upgrade-time
list.
2021-03-14 17:50:57 -07:00
Anders Kaseorg d393ac5034 update-prod-static: Remove unused --prev-deploy option.
It’s unused since commit 079ddae4c8
(#12676).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-04 18:04:57 -08:00
Anders Kaseorg 25bb98dcf5 install-node: Upgrade Node.js from 14.15.1 to 14.16.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg ccad00b7e9 provision: Suppress exception chaining for CalledProcessError retries.
When exception is raised inside an exception handler, Python 3
helpfully prints both tracebacks separated by “During handling of the
above exception, another exception occurred:”.  But when we’re using
an exception handler to retry the same operation, multiple tracebacks
are just noise.  Suppress the earlier one using PEP 409 syntax.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 16:25:03 -08:00
Alex Vandiver 32149c6a1c puppet: Add ksplice uptrack for kernel hotpatches. 2021-02-25 18:05:47 -08:00
Alex Vandiver 0663b23d54 puppet: Only prompt to apply if there are changes to apply.
Since `yaml` is not a module in the standard library, this requires
makings `zulip-puppet-apply` use the venv.
2021-02-23 18:16:02 -08:00
Alex Vandiver d15e6990e5 puppet: Only execute setup-apt-repo if necessary.
This means that in steady-state, `zulip-puppet-apply` is expected to
produce no changes or commands to execute.  The verification step of
`setup-apt-repo` is quite fast, so this cleans up the output for very
little cost.
2021-02-23 18:16:02 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 1a4f70f1bc lint: Convert sudo exclusion to double quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 08:34:43 -08:00
Ganesh Pawar 7eeca9da46 provision: Add provision support for Ubuntu 20.10(Groovy).
PostgreSQL 13 is used when os_version is 20.10.
2021-02-05 09:30:34 -08:00
Anders Kaseorg 948f2ee2ad manage: Quote commands correctly in log_management_command.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-26 13:26:57 -08:00
Alex Vandiver c2526844e9 worker: Remove SignupWorker and friends.
ZULIP_FRIENDS_LIST_ID and MAILCHIMP_API_KEY are not currently used in
production.

This removes the unused 'signups' queue and worker.
2021-01-17 11:16:35 -08:00
Sutou Kouhei 0d3f9fc855 install: Use PGroonga packages built for PostgreSQL packages by PGDG
Because we always use PostgreSQL packages by PGDG since Zulip 3.0.

Fixes #16058.
2020-12-18 15:38:21 -08:00
Anders Kaseorg 77fdac3579 install-node: Upgrade Node.js to 14.15.1 and nvm to 0.37.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Vishnu KS eb008fc864 emails: Use macros for email tags in invitation email. 2020-10-30 11:50:30 -07:00
Anders Kaseorg aaa7b766d8 python: Use universal_newlines to get str from subprocess.
We can replace ‘universal_newlines’ with ‘text’ when we bump our
minimum Python version to 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 86e8d81c7f python: Skip unnecessary decode before JSON parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Tim Abbott c537912a77 puppet: Migrate postgres_backups puppet manifest name. 2020-10-29 11:29:44 -07:00
Alex Vandiver 2332113c97 upgrade: Adjust puppet class names even with --skip-puppet.
The class names need to be renamed even if we are not about to run
puppet ourselves; otherwise, deployments which rely on running puppet
themselves will still have the wrong class names.
2020-10-28 17:49:14 -07:00
Alex Vandiver 6b9d7000b5 puppet: Set proxy environment variables.
These are respected by `urllib`, and thus also `requests`.  We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
2020-10-28 12:17:35 -07:00
Alex Vandiver 97745688ca docs: Link to the new doc home of the email gateway. 2020-10-28 12:13:04 -07:00
Alex Vandiver f1cf730c5b restore-backup: Rename variables to postgresql. 2020-10-28 11:57:03 -07:00
Alex Vandiver 5ee3379ce0 upgrade: Rename variables to postgresql. 2020-10-28 11:57:03 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Anders Kaseorg 23a289ecd5 install-node: Upgrade Node.js to 12.19.0 and Yarn to 1.22.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Anders Kaseorg de5282d2cf install-node: Install npm and npx symlinks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Alex Vandiver 5f3765b872 upgrade: Adjust puppet classes to new names. 2020-10-27 13:29:19 -07:00
Alex Vandiver 16d9dd84b8 upgrade: Switch to using crudini to update zulip.conf contents.
Using `config_file.write()` only writes out what python stored of the
file; as such, it strips all comments and whitespace.

Use `crudini --set`, which only modifies the line whose contents are
changed.
2020-10-27 13:29:19 -07:00
Alex Vandiver 5365af544a puppet: Rename zulip::profile::rabbit to ::rabbitmq. 2020-10-27 13:29:19 -07:00
Alex Vandiver 188af57296 puppet: Rename postgres_appdb to postgresql.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name.  This is minor
cleanup, but enabled by the other renames in the previous commit.
2020-10-27 13:29:19 -07:00
Alex Vandiver 0f25acc7b3 puppet: Rename "voyager"/"dockervoyager" to "standalone"/"docker".
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
2020-10-27 13:29:19 -07:00
Alex Vandiver c2185a81d6 puppet: Move top-level zulip deployments into "profile" directory.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes.  Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.

Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.

[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
2020-10-27 13:29:19 -07:00
Alex Vandiver 7cf737988d queue: Be more explicit about test/real queue division. 2020-10-26 12:32:47 -07:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg 16aa48d9b2 configure-rabbitmq: Wait for RabbitMQ to start up.
Fixes an occasional failure in ‘vagrant up --provision’.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 17:01:00 -07:00
Anders Kaseorg f16aa8f264 configure-rabbitmq: Put the command and flags in one array.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 17:01:00 -07:00
Alex Vandiver 1fa4ef0271 upgrade-postgres: Catch failed pg_upgradecluster exit code.
Because the command is part of a pipe sequence, the exitcode defaults
to the last in the sequence, which is not the most important one here.

Set pipefail, which sets the exit status to the exit code of the last
program in the sequence to exit non-zero, or 0 if all succeeded.  This
prevents the upgrade from barreling onward and setting
`postgres.version` improperly if the database upgrade step failed.
2020-10-15 15:21:30 -07:00
Anders Kaseorg dfaea9df65 shfmt: Reformat shell scripts with shfmt.
https://github.com/mvdan/sh

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-15 15:16:00 -07:00
Anders Kaseorg dd48dbd912 docs: Add spaces to “check out”, “log in”, “set up”, “sign up” as verbs.
“Checkout”, “login”, “setup”, and “signup” are nouns, not verbs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Tim Abbott 5de6f3523c upgrade-postgres: Pass the requested postgres explicitly. 2020-10-01 14:29:24 -07:00
Alex Vandiver 4d65ea256a rabbitmq: Consolidate check_rabbitmq_queue to call rabbitmqctl once.
`rabbitmqctl` tends to be slow; this shaves half a second off the time
to run `check-rabbitmq-consumers` in some cases.
2020-09-29 17:44:44 -07:00
Alex Vandiver c0e240277b tornado: Remove fingerprinting, write out .tmp files always.
Fingerprinting the config is somewhat brittle -- it requires either
custom bootstrapping for old (fingerprint-less) configs, and may have
false-positives.

Since generating the config is lightweight, do so into the .tmp files,
and compare the output to the originals to determine if there are
changes to apply.

In order to both surface errors, as well as notify the user in case a
restart is necessary, we must run it twice.  The `onlyif`
functionality cannot show configuration errors to the user, only
determine if the command runs or not.  We thus run the command once,
judging errors as "interesting" enough to run the actual command,
whose failure will be verbose in Puppet and halt any steps that depend
on it.

Removing the `onlyif` would result in `stage_updated_sharding` showing
up in the output of every Puppet run, which obscures the important
messages it displays when an update to sharding is necessary.
Removing the `command` (e.g. making it an `echo`) would result in
removing the ability to report configuration errors.  We thus have no
choice but to run it twice; this is thankfully low-overhead.
2020-09-25 10:52:40 -07:00
Alex Vandiver 4b3121db0b certbot: Explicitly apt-get update before installing certbot.
There is no guarantee that the apt data is up-to-date, unless we
explicitly update.

Fixes: zulip/docker-zulip#275
2020-09-21 15:26:28 -07:00
Mateusz Mandera e2dcdc2758 queue: Increase allowed expected_time_to_clear_backlog for embed_links.
It's okay for this queue to be a bit slow, and the default limits are
kind of too low for it.
2020-09-21 15:24:04 -07:00
Mateusz Mandera cd9b194d88 queue: Eliminate useless "burst" concept in monitoring.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
2020-09-21 15:24:04 -07:00
Mateusz Mandera 2365a53496 queue: Fix a race condition in monitoring after queue stops being idle.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
2020-09-21 15:22:56 -07:00
Alex Vandiver 2a12fedcf1 tornado: Remove explicit tornado_processes setting; compute it.
We can compute the intended number of processes from the sharding
configuration.  In doing so, also validate that all of the ports are
contiguous.

This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding.  Having behaviour which
changes merely based on if an empty section exists is surprising.

This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.

After this commit, configuring sharding can be done by adding to
`zulip.conf`:

```
[tornado_sharding]
9800 =              # default
9801 = other_realm
```

Followed by running `./scripts/refresh-sharding-and-restart`.
2020-09-18 15:13:40 -07:00
Anders Kaseorg b7874ac82e install-node: Upgrade Node.js to 12.18.4 and Yarn to 1.22.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Alex Vandiver efdaa58c24 supervisor: Use more specific process_name than "port-9800".
Making this include "zulip-tornado" makes it clearer in supervisor
logs.  Without this, one only sees:
```
2020-09-14 03:43:13,788 INFO waiting for port-9807 to stop
2020-09-14 03:43:14,466 INFO stopped: port-9807 (exit status 1)
2020-09-14 03:43:14,469 INFO spawned: 'port-9807' with pid 24289
2020-09-14 03:43:15,470 INFO success: port-9807 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
```
2020-09-14 22:17:51 -07:00
Alex Vandiver 13fb7875e2 nagios: Remove an unnecessary path.append. 2020-09-14 18:20:12 -07:00
Alex Vandiver dd68cc98fd upgrade: Stop in the same order as restart-server.
restart-server explicitly stops the workers first, then the core
services.  Keep that ordering consistently.
2020-09-14 16:27:15 -07:00
Alex Vandiver dc58dec231 restart-server: Start services in opposite order from stop.
`supervisorctl` starts and stops its arguments sequentially, in the
order they are passed[1].  Start them in the opposite order from the
order in which they were stopped -- this puts the dependencies first,
and the most core services (`zulip-django`) last.

While the only "dependency" here is currently thumbor, this sets us up
in case others are added later.

[1] https://github.com/Supervisor/supervisor/blob/master/supervisor/supervisorctl.py#L782
2020-09-14 16:27:15 -07:00
Alex Vandiver 8adf530400 puppet: Generate sharding in puppet, then refresh-sharding-and-restart.
This supports running puppet to pick up new sharding changes, which
will warn of the need to finalize them via
`refresh-sharding-and-restart`, or simply running that directly.
2020-09-14 16:27:15 -07:00
Alex Vandiver bf029d99f1 sharding: Also mark sharding.json 644 for consistency.
There is no reason to limit this to 640; mark it 644 for consistency
with the other file.
2020-09-14 16:27:15 -07:00
Alex Vandiver b5bcff04e5 sharding: Consistent mode for nginx sharding file.
This disagreed between `tornado_sharding.pp` in puppet and
`scripts/refresh-sharding-and-restart`.
2020-09-14 16:27:15 -07:00
Mateusz Mandera aae84197e8 check-rabbitmq-queue: Use list_queues output for current backlog size.
The value in the stats file can get outdated if the queue hasn't done
enough iterations to update the stats file for a while. The queue size
output by rabbitmqctl list_queues is more up to date, and empirically
tends to agree with the value in the stats file (when the stats file is
fresh).
2020-09-11 15:51:07 -07:00
Anders Kaseorg b7b7475672 python: Use standard secrets module to generate random tokens.
There are three functional side effects:

• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.

• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.

• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.

(The base32 alphabet is A-Z 2-7.  We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-09 15:52:57 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bb4fc3c4c7 python: Prefer --flag=option over --flag option.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg 9edcafb7a0 setup_venv: Add missing comma in COMMON_YUM_VENV_DEPENDENCIES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:25:54 -07:00
Anders Kaseorg a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg fbfd4b399d python: Elide action="store" for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 1f2ac1962f python: Elide default=None for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 3c5b39da9c python: Elide nargs for argparse flag arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 1ded51aa9d python: Replace list literal concatenation with * unpacking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Sutou Kouhei ebf4048dd4
create-db.sql: Ensure using en_US.UTF-8 encoding.
PostgreSQL packages for Ubuntu run "initdb" without specifying locale
on installation. It means that the default template
database (template1) is created by the system default locale. If the
system default locale is non UTF-8 compatible encoding such as
en_US.ISO-8859-15, "zulip" database is also created non UTF-8
compatible encoding such as LATIN9.

You can reproduce this case by running the following script:

    apt update

    apt install -y locales
    locale-gen en_US.ISO-8859-15
    update-locale LANG=en_US.ISO-8859-15 LANGUAGE=en_US:

    apt install -y wget
    wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
    tar xf zulip-server-latest.tar.gz
    zulip-server-*/scripts/setup/install \
      --hostname=zulip-test.example.com \
      --email=zulip-test-admin@example.com \
      --self-signed-cert

scripts/setup/install is failed with the following error:

    + ./manage.py migrate --noinput
    Operations to perform:
      Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
    Running migrations:
      Applying contenttypes.0001_initial... OK
      Applying auth.0001_initial... OK
      Applying zerver.0001_initial...Traceback (most recent call last):
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
        return self.cursor.execute(sql)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
        return wrapper_execute(self, super().execute, query, vars)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
        return action(sql, params)
    psycopg2.errors.UntranslatableCharacter: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
    CONTEXT:  line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "./manage.py", line 50, in <module>
        execute_from_command_line(sys.argv)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
        utility.execute()
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
        output = self.handle(*args, **options)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
        res = handle_func(*args, **kwargs)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
        post_migrate_state = executor.migrate(
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
        state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
        state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
        operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 105, in database_forwards
        self._run_sql(schema_editor, self.sql)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 130, in _run_sql
        schema_editor.execute(statement, params=None)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
        cursor.execute(sql, params)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
        return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
        return executor(sql, params, many, context)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
        return self.cursor.execute(sql)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
        return wrapper_execute(self, super().execute, query, vars)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
        return action(sql, params)
    django.db.utils.DataError: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
    CONTEXT:  line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"
2020-08-24 12:24:38 -07:00
Anders Kaseorg 0f608176ad install-node: Upgrade Node.js from 12.18.2 to 12.18.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 18:37:25 -07:00
Aman 7b9fe77bf1 provision: Fix missing <sasl/sasl.h> headers during provision. 2020-08-12 16:19:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 3582183fba setup_venv: Install libyaml-dev.
This will let PyYAML link against LibYAML when PyYAML is next
installed.  Due to virtualenv-clone, that won’t happen until the next
Python package removal anyway, so we don’t bother bumping
PROVISION_VERSION.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 20:58:07 -07:00