Commit Graph

20 Commits

Author SHA1 Message Date
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bb4fc3c4c7 python: Prefer --flag=option over --flag option.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg 1ded51aa9d python: Replace list literal concatenation with * unpacking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Vishnu KS 18ecf9bcfa backup: Make restore-backup work in docker.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Anders Kaseorg fa2496c229 terminate-psql-sessions: Rely on the caller to set PGHOST, PGUSER.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg d918626135 restore-backup: Run generate_secrets.py.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:20:43 -08:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Tim Abbott 5748bae71d restore-backup: Run configure-rabbitmq to update RABBITMQ_PASSWORD.
Previously, if you restored onto a different production system from
the one where you took the backup, backup restoration would fail
because the generated rabbitmq passwords for the two systems would be
different, and we didn't update the restored system to use the
password from the original system.

Fixes #12114.
2019-06-06 16:38:27 -07:00
Tim Abbott 3c4030a421 restore-backup: Run zulip-puppet-apply before pg_restore.
This should ensure that we apply any special configuration for the
database system (e.g. installing `pgroonga`) before we try to restore
the database contents from the archive.

For pgroonga in particular, this is important so that we can preserve
the configuration of the extension in the `pg_restore` process.

Fixes #12345.
2019-06-06 16:34:28 -07:00
Tim Abbott 13a20a1ab5 restore-backup: Fix error on systems using S3 upload backend.
With the S3 file upload backend, we don't store uploads locally, so
the `uploads` directory in the backup will be empty, and more
importantly, LOCAL_UPLOADS_DIR will be None, which the previous code
crashed on.
2019-06-06 15:08:18 -07:00
Anders Kaseorg ed100d694f restore-backup: Open backup tarball as root.
Fixes permission errors when running restore-backup on a tarball
inaccessible to the zulip user.

Fixes #12125.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg 206741117c restore-backup: Reset cwd when switching to postgres user.
Fixes permission errors when running restore-backup from a cwd
inaccessible to the postgres user.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg d03b8308a5 backup: Use tar --transform to arrange the tarball instead of symlinks.
This allows tar to print the real paths in error messages if something
goes wrong.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-04-12 16:48:34 -07:00
Anders Kaseorg f78970c679 restore-backup: Set the recreated database owner to zulip.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-04 11:50:13 -07:00
Anders Kaseorg e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00