Commit Graph

217 Commits

Author SHA1 Message Date
Anders Kaseorg fd2d642f95 postgres-init-db: Fix invocation from a relative path.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:40:37 -08:00
Anders Kaseorg d918626135 restore-backup: Run generate_secrets.py.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:20:43 -08:00
Mateusz Mandera 27b15a9722 install: Don't create internal realm in the installation process. 2020-02-12 12:00:10 -08:00
Tim Abbott dd969b5339 install: Remove references to "Zulip Voyager".
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".

At the time, we just needed a name quickly, but it was never a good
name, just a placeholder.  This removes that placeholder name from
much of the codebase.  A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
2020-01-30 12:40:41 -08:00
Anders Kaseorg 8c733a3f68 create-db.sql: Start by dropping the zulip database if needed.
At some point the PostgreSQL Docker image started creating the zulip
database for us, which caused our CREATE DATABASE to fail.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg 298d45b46a create-db.sql: Handle exception if zulip user already exists.
Fixes #13530.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg a82032a182 generate_secrets: Enable Redis authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg cdda983e90 settings: Support optional memcached authentication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Anders Kaseorg 699626f3cf flush-memcached: Use pylibmc.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:18 -08:00
Anders Kaseorg a78f8647d8 install: Run generate_secrets.py before zulip-puppet-apply.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-05 22:48:08 -08:00
Anders Kaseorg ab211c7acf lint: Tell ShellCheck to look for sourced files at relative paths.
This uses the new -P option of ShellCheck 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-18 03:48:02 -08:00
Anders Kaseorg 347fd80864 generate_secrets: Remove unused initial_password_salt in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 23:06:53 -08:00
Anders Kaseorg 7ebba2901a generate_secrets: Remove unused local_database_password in production.
Fixes #13464.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 13:05:31 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 0af22dad18 flush-memcached: Respect MEMCACHED_LOCATION; handle errors.
Fixes #13238.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-01 16:05:55 -07:00
Anders Kaseorg 2ff87bd888 setup: Update groonga APT repository signing key.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg 2ec946ad4d postgres-init-db: Require an Enter press in confirmation prompt.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-12 16:04:55 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg 1d05a71b5d install, provision: Treat all nonzero exit codes as failure, not just 1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:22:11 -07:00
Anders Kaseorg f97bb6b65c postgres-create-db: Split into non-pgroonga and pgroonga scripts.
Fixes #12655.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-17 16:18:22 -07:00
Tim Abbott 5748bae71d restore-backup: Run configure-rabbitmq to update RABBITMQ_PASSWORD.
Previously, if you restored onto a different production system from
the one where you took the backup, backup restoration would fail
because the generated rabbitmq passwords for the two systems would be
different, and we didn't update the restored system to use the
password from the original system.

Fixes #12114.
2019-06-06 16:38:27 -07:00
Tim Abbott 3c4030a421 restore-backup: Run zulip-puppet-apply before pg_restore.
This should ensure that we apply any special configuration for the
database system (e.g. installing `pgroonga`) before we try to restore
the database contents from the archive.

For pgroonga in particular, this is important so that we can preserve
the configuration of the extension in the `pg_restore` process.

Fixes #12345.
2019-06-06 16:34:28 -07:00
Tim Abbott 13a20a1ab5 restore-backup: Fix error on systems using S3 upload backend.
With the S3 file upload backend, we don't store uploads locally, so
the `uploads` directory in the backup will be empty, and more
importantly, LOCAL_UPLOADS_DIR will be None, which the previous code
crashed on.
2019-06-06 15:08:18 -07:00
Anders Kaseorg 9937734e50 terminate-psql-sessions: Only terminate if we have permissions.
We have been semi-accidentally relying on the fact that terminate-psql-sessions 
fails silently when there are PIDs we don't have permission to terminate.

This actually happens somewhat often, generally when we're doing a series of
operations in quick succession by different users, because postgres processes
live a little longer than the `psql` shell that started them.

As part of adding ON_STOP_ERROR to all of our postgres commands, it makes
sense to enforce we don't fail here, but that means we need to actually filter
the target PIDs to only ones we can actually kill.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 14:41:42 -07:00
Tim Abbott 74143a0801 scripts: Disable terminate-psql-sessions ON_ERROR_STOP for now.
This appears to fail nondeterministically in CI; there's likely a
better fix but this should let us get CI happy again.
2019-04-22 15:49:48 -07:00
Anders Kaseorg 5290519a62 scripts: Always use ON_ERROR_STOP=1 when running psql.
Also use psql -e (--echo-queries) in scripts that use ‘set -x’, so
errors can be traced to a specific query from the output.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:19 -07:00
Anders Kaseorg ed100d694f restore-backup: Open backup tarball as root.
Fixes permission errors when running restore-backup on a tarball
inaccessible to the zulip user.

Fixes #12125.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg 206741117c restore-backup: Reset cwd when switching to postgres user.
Fixes permission errors when running restore-backup from a cwd
inaccessible to the postgres user.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg 460abe82df terminate-psql-sessions: Allow running as postgres user.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Anders Kaseorg 53879c4673 scripts: Rename DEFAULT_USER to POSTGRES_USER.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Anders Kaseorg d03b8308a5 backup: Use tar --transform to arrange the tarball instead of symlinks.
This allows tar to print the real paths in error messages if something
goes wrong.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-04-12 16:48:34 -07:00
Rohitt Vashishtha 327b70cb19 terminate-psql-sessions: Allow running without sudo. 2019-04-04 16:52:20 -07:00
Tim Abbott 042893a448 configure-rabbitmq: Allow running as root.
This makes it possible to run this in production without access to
sudo.
2019-04-04 16:52:20 -07:00
Anders Kaseorg f78970c679 restore-backup: Set the recreated database owner to zulip.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-04 11:50:13 -07:00
Anders Kaseorg 1e2bd553fb setup-certbot: Remove --force-renewal. (#11652)
There’s no reason to do this unless you’re, like, trying to trip the
Let’s Encrypt rate limits (or perhaps trying to manually test this code).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 15:50:38 -08:00
Anders Kaseorg e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00
Anders Kaseorg e984107966 scripts: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:58 -08:00
Anders Kaseorg 981f09d950 terminate-psql-sessions: Fix shell and SQL quoting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:46:50 -08:00
rht a55f5b59f6 generate-self-signed-cert: Generalize to CentOS, Fedora, RHEL. 2019-01-05 15:49:02 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
rht 711b0fda8b scripts: Initialize setup-apt-repo-debathena. 2018-12-11 13:01:26 -08:00
Tim Abbott adf27aae4c python: Remove now-unnecessary str_utils library.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.

Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.

Fixes #10807.
2018-11-27 11:57:54 -08:00