Commit Graph

55 Commits

Author SHA1 Message Date
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Tim Abbott 800e6b1ca6 generate_secrets: Add more comments/documentation. 2020-04-30 10:44:27 -07:00
Steve Howell 34191f2a56 generate_secrets: Avoid unnecessary settings import.
We try to avoid importing Django settings unless
we really need them, since we want this program
to run very quickly during `provision` (when
secrets have already been generated earlier).
2020-04-30 13:52:54 +00:00
Steve Howell 522ee7fd6b generate_secrets: Lazily import crypto module. 2020-04-30 13:49:44 +00:00
Steve Howell 66392afe15 generate_secrets: Extract random_token(). 2020-04-30 13:23:13 +00:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg a82032a182 generate_secrets: Enable Redis authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg a78f8647d8 install: Run generate_secrets.py before zulip-puppet-apply.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-05 22:48:08 -08:00
Anders Kaseorg 347fd80864 generate_secrets: Remove unused initial_password_salt in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 23:06:53 -08:00
Anders Kaseorg 7ebba2901a generate_secrets: Remove unused local_database_password in production.
Fixes #13464.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 13:05:31 -08:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg e984107966 scripts: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:58 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott adf27aae4c python: Remove now-unnecessary str_utils library.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.

Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.

Fixes #10807.
2018-11-27 11:57:54 -08:00
Aditya Bansal e14974ff2c scripts: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott b3e4c702d1 generate_secrets: Fix handling of an empty secrets file.
This is now a condition that happens during installation, because we
now create an empty file for this in puppet.
2018-05-06 22:12:32 -07:00
rht 2b19f17fc9 scripts: Remove u prefix from strings. 2018-02-05 12:11:33 -08:00
Aditya Bansal 79576797b0 thumbor: Auto generate thumbor_key and add it to zulip settings. 2017-11-16 22:34:14 -08:00
Tim Abbott be619fe881 lint: Wrap many very long lines in the Python codebase.
This decreases the maximum line length in our Python codebase to 130.
2017-10-26 17:31:58 -07:00
rht 71188d7b0a scripts: Remove import print_function. 2017-09-29 15:43:30 -07:00
rht b937e1fede Remove the rest of `import six` outside zerver.lib.
We no longer need the six module, now that we're Python 3 only.
2017-09-27 17:09:15 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Tim Abbott b8e7369dee mypy: Remove type: ignores not needed in Python 3. 2017-08-25 11:04:20 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott 53e4d8562b lint: Add a lint check for bare `type: ignore`s. 2017-07-27 16:31:55 -07:00
EdOverflow 86cfed3e81 Update generate_secrets.py 2017-07-11 12:43:46 -07:00
Tim Abbott b70986469c generate_secrets: Fix handling of missing trailing newlines.
When we added support for automatically adding new secrets in
generate_secrets.py, we failed to account for the possibility that a
human editor might have let the secrets file without a trailing
newline.

We address this by adding a leading newline before our new secret.

Fixes #5209.
2017-06-03 23:17:04 -07:00
Tim Abbott 11adbf5783 generate_secrets: Fix placement of mypy type: ignore. 2017-05-17 00:05:57 -07:00
Tim Abbott b01ba5f389 generate_secrets: Fix mypy errors.
I'm pretty sure these errors reflect a problem with Typeshed, but
don't have time to investigate.
2017-05-16 23:28:44 -07:00
Tim Abbott 2c6a91e24a scripts: Make generate_secrets.py idempotent.
Now, generate_secrets.py will never overwrite existing secrets.  In
addition to being a safer model in generate, this fixes 2 significant
issues:

(1) It makes it much easier to preserve secrets like Oauth tokens in a
development environment (previously, provision would destroy them).
(2) It makes it possible to automatically add new secrets as part of
the upgrade process.  In particular, this is useful for the
zulip_org_id settings.

Fixes #4797.
2017-05-16 22:15:25 -07:00
Tim Abbott 03b5200d8b generate_secrets: Reformat list of autogenerated secrets. 2017-05-16 22:15:25 -07:00
Tim Abbott 55a9101573 settings: Add support for ZULIP_ORG secrets.
These can be used to authenticate the current Zulip server to
zulip.org.
2017-04-18 23:00:10 -07:00
Rishi Gupta 2bbfdeeb7b Fix more errors caught by mypy 0.501.
Another set of relatively easy to review changes.
2017-03-03 14:15:38 -08:00
Tim Abbott de99f48ce7 lint: Clean up E401 PEP-8 rule. 2017-01-23 21:36:39 -08:00
anirudhjain75 beaa62cafa mypy: Convert several directories to use typing.Text.
Specifically, these directories are converted: [analytics/, scripts/,
tools/, zerver/management/, zilencer/, zproject/]
2016-12-07 20:51:05 -08:00
Tommy Ip 46b7d54b3e pep8: Fix E701 violations. 2016-11-30 20:45:09 +00:00
Diptanshu8 d7253b144c generate-secrets: Refactor to make development/production explicit.
generate-secrets.py now requires --development for development environment
setup or --production for production environment setup (and one of these
options is mandatory).

This solves the problem that it was somewhat easy to accidentally run
generate-secrets.py without the `-d` option while doing manual development
environment setup.

Fixes: #1911.
2016-10-06 17:12:49 -07:00
Tim Abbott 3839a25c74 Annotate generate_secrets.py. 2016-08-04 15:53:23 -07:00
Tim Abbott 8b285ec0ff puppet: Read camo key from zulip-secrets.conf. 2016-07-31 00:23:24 -07:00
Tim Abbott 35339f5117 Rename local_settings_template to prod_settings_template. 2016-07-19 20:59:59 -07:00
Eklavya Sharma a9835c0ab2 Activate virtualenv in production Python code.
The manage.py change effectively switches the Zulip production server
to use the virtualenv, since all of our supervisord commands for the
various Python services go through manage.py.

Additionally, this migrates the production scripts and Nagios plugins
to use the virtualenv as well.
2016-06-27 19:55:35 -07:00
Tim Abbott f7ce5fc179 generate_secrets: Silence mypy error with configparser.
See https://github.com/python/typeshed/issues/307.
2016-06-20 09:03:09 -07:00