Commit Graph

1070 Commits

Author SHA1 Message Date
Alex Vandiver 0d7dbd1b07 puppet: Apply basic PostgreSQL configuration before pg_upgradecluster.
Running `pg-upgradecluster` runs the `CREATE TEXT SEARCH DICTIONARY`
and `CREATE TEXT SEARCH CONFIGURATION` from
`zerver/migrations/0001_initial.py` on the new PostgreSQL cluster;
this requires that the stopwords file and dictionary exist _prior_
to `pg_upgradecluster` being run.

This causes a minor dependency conflict -- we do not wish to duplicate
the functionality from `zulip::postgres_appdb_base` which configures
those files, but installing all of `zulip::postgres_appdb_tuned` will
attempt to restart PostgreSQL -- which has not configured the cluster
for the new version yet.

In order to split out configuration of the prerequisites for the
application database, and the steps required to run it, we need to be
able to apply only part of the puppet configuration.  Use the
newly-added `--config` argument to provide a more limited `zulip.conf`
which only applies `zulip::postgres_appdb_base` to the new version of
Postgres, creating the required tsearch data files.

This also preserves the property that a failure at any point prior to
the `pg_upgradecluster` is easily recoverable, by re-running
`zulip-puppet-apply`.
2020-07-06 18:30:16 -07:00
Alex Vandiver 17002f2a0e puppet: Allow passing an alternate config path to zulip-puppet-apply.
When temporary configuration changes are desired, this lets one set up
an alternate `zulip.conf` to apply while leaving the true one in place.
2020-07-06 18:30:16 -07:00
Alex Vandiver efe2b6e5cd puppet: Switch `zulip-puppet-apply` to argparse.
This allows additional arguments other than `-f` or `--force`.
2020-07-06 18:30:16 -07:00
Aman Agrawal a486872a8e requirements: Upgrade Thumbor to 7.0.0a5 on Python 3.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 16:09:53 -07:00
Vishnu KS 97403a09d0 install: Create zulip user only if required.
Otherwise, the useradd command will fail during the DigitalOcean
1-Click App installation because the install script is called
twice during the whole process. Plus the Zulip install script
is designed to be idempotent and this bug compromises that.
2020-07-02 14:55:04 -07:00
Anders Kaseorg e3835554a7 postgres-init-db: Read terminate-psql-sessions script as root.
Fixes #15646.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 14:54:36 -07:00
Tim Abbott ab1ee1f061 install: Add a comment on crudini deletion. 2020-07-01 15:13:00 -07:00
Alex Vandiver 6df99677d3 installer: Remove unnecessary nginx restart.
Puppet takes care of this.
2020-07-01 15:07:52 -07:00
Alex Vandiver 2d4fae0ffe installer: Remove out-of-date comment. 2020-07-01 15:07:52 -07:00
Alex Vandiver 2de8400a32 installer: Only set `deploy_type = production` in zulip.conf.
The value is a holdover from when it controlled runtime behavior,
which it no longer does.

Stop taking a DEPLOYMENT_TYPE, which is unused; the python code only
care about if the option exists, not its value.
2020-07-01 15:07:52 -07:00
Alex Vandiver 117d32cd8c installer: Switch to checking dockervoyager as a class, not a deployment.
The DEPLOYMENT_TYPE=dockervoyager is otherwise unused; and always
happens in conjunction with a `zulip::dockervoyager` puppet class.
2020-07-01 15:07:52 -07:00
Alex Vandiver 8236cb52d2 installer: Switch has_* variables for has_class checks.
These are more correct to the sense of "is this a service we
configured for Zulip", and removes potential confusion around the 0/1
values being backwards from how binary is usually interpreted.
2020-07-01 15:07:52 -07:00
Alex Vandiver 2c79909a5d installer: Switch other PUPPET_CLASSES check for has_class. 2020-07-01 15:07:52 -07:00
Alex Vandiver ec2383dcde installer: Move missing_dictionaries configuration to with other config.
It already has been made to explicitly conflict with
`--no-overwrite-settings`, so moving it inside the else block is safe.
2020-07-01 15:07:52 -07:00
Alex Vandiver 9c0fd632bb installer: Use `puppet --write-catalog-summary` to determine classes.
Using checks of `,$PUPPET_CLASSES,` is repetitive and error-prone; it
does not properly deal with `zulip_ops::` classes, for instance, which
include the `zulip::` classes.

As alluded to in ca9d27175b, this can be fixed by inspecting the
classes that would be applied, using `puppet --write-catalog-summary`.
We work around the chicken-and-egg problem alluded to therein by
writing out as complete `zulip.conf` as would be necessary, before
running puppet and removing the sections we then know to not be
needed.

Unfortunately, there are two checks for `$PUPPET_CLASSES` which cannot
be switched to this technique, as they concern errors that we wish to
catch quite early, and thus before we have puppet installed.  Since we
expect failures of those to only concern warnings, and only be
mistakenly omitted for internal `zulip_ops::` classes, this seems a
reasonable risk to admit in exchange for catching common errors early.
2020-07-01 15:07:51 -07:00
Alex Vandiver 64b44a12f5 puppet: Add an exec rule to reload the whole supervisor config.
When supervisor is first installed, it is started automatically, and
creates the socket, owned by root.  Subsequent reconfiguration in
puppet only calls `reread + update`, which is insufficient to apply
the `chown = zulip:zulip` line in `supervisord.conf`, leaving the
socket owned by `root` and the last part of the installation unable to
restart `supervisor` services as the `zulip` user.  The `chown` line
in `scripts/lib/install` exists to paper over this.

Add a separate exec target for changes to `supervisord.conf` itself,
which restarts the full service.  This leaves the default `restart`
action on the service for the lightweight `reread + update` action,
which is more common.

We use `systemctl` only on redhat-esque builds, because CI runs
Ubuntu, but init is not systemd in that context.  `systemctl reload`
is sufficient to re-apply the socket ownership, but a full `restart`
and not `reload` is necessary under `/etc/init.d/supervisor`.
2020-07-01 10:40:54 -07:00
Anders Kaseorg 7f46886696 settings: Split hostname from port more carefully.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Anders Kaseorg fa89d1b266 generate-self-signed-cert: Correct subjectAltName for an IP address.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Alex Vandiver 918fcb9f6f upgrade: Make upgrade-postgres work without systemctl.
The only postgres cluster which need be stopped is the one we are
upgrading.
2020-06-29 17:18:47 -07:00
Alex Vandiver cd290c2c66 installer: Be tighter about the search for postgres server packages. 2020-06-29 13:37:16 -07:00
Alex Vandiver eb6802057a upgrade: Don't prompt in the second apt-get upgrade. 2020-06-26 16:16:12 -07:00
Alex Vandiver b7a135f037 upgrade: Add a tool to upgrade PostgreSQL.
This is based on the existing steps in the documentation, with
additional changes now that the PostgreSQL version is stored in
`/etc/zulip/zulip.conf`.
2020-06-26 16:07:39 -07:00
Alex Vandiver 31f1f10501 installer: Halt if wrong version of PostgreSQL is already installed.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository.  On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.

During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found.  This is safer than attempting to automatically uninstall or
reconfigure existing databases.
2020-06-24 12:57:38 -07:00
Alex Vandiver 814198d649 installer: Abstract out version of postgres installed.
This allows for installing from-scratch with a different pinned
version of PostgreSQL, and provides a single place to change when the
default should increase.
2020-06-24 12:57:38 -07:00
Alex Vandiver ca9d27175b installer: Write PostgreSQL version based on puppet classes.
Using `/etc/init.d/postgresql` as the detection of if Postgres is on
the server is incorrect, because this line runs _before_ puppet and
any packages are installed.  Thus, it cannot tell the difference
between a new Ubuntu one-host first-time-install without PostgreSQL
yet, and one which is merely a front-end and will never have
PostgreSQL.  This leads to failures in first-time installs:

```
Error: Evaluation Error: Error while evaluating a Function Call,
  Could not find template 'zulip/postgresql//postgresql.conf.template.erb'
```

The only way to detect if PostgreSQL will be present in the _end_
state of the install is to examine the puppet classes that are
applied.

To do this, we must inspect `PUPPET_CLASSES`.  Unfortunately, this can
be fragile to subclassing (e.g. `zulip_ops::postgres_appdb`).  We
might desire to use `puppet apply --write-catalog-summary` to deduce
the _applied_ classes, which would unroll the inheritance; however,
this causes a chicken-and-egg problem, because `zulip.conf` must be
already written out (including a value for `postgresql.version`, if
necessary!) before such a puppet run could successfully complete.

Switch to predicating the `postgresql.version` key on the puppet
classes that are known to install postgres.
2020-06-24 12:57:38 -07:00
Alex Vandiver 253246185f installer: Update documentation.
Where appropriate, documentation wording is shared with
docs/production/install.md
2020-06-24 12:57:38 -07:00
Alex Vandiver 85dbb13c56 installer: Abstract out apt/yum divide into a variable.
This check is done in several places, using a somewhat fragile `case`
statement; move it into an explicit variable.
2020-06-24 12:57:38 -07:00
Alex Vandiver 876ee4a8ed installer: Remove code specific to stretch or xenial.
Support for Xenial and Stretch was removed (5154ddafca, 0f4b1076ad,
8944e0ad53, 79acd5ae40, 1219a2e854), but not all codepaths were
updated to remove their conditionals on it.

Remove all code predicated on Xenial or Stretch.  debathena support
was migrated to Bionic, since that appears to be the current state of
existing debathena servers.
2020-06-24 12:57:38 -07:00
Alex Vandiver e4899eae8b installer: Sync the claimed supported distros with the check.
0f4b1076ad removed Ubuntu 16.04 "xenial" and Debian 9 "stretch" from
the printed list of supported operating systems, but left them in the
verification check that controls if that message is printed,
effectively continuing to support them.

Conversely, 439f0d3004 added Ubuntu 20.04 "focal" to the check, but
not to the printed list.

Synchronize to check and print the right supported distributions:
Ubuntu 18.04 "bionic", Ubuntu 20.04 "focal", and Debian 10 "buster".
2020-06-24 12:57:38 -07:00
Alex Vandiver 58cb7cecd8 installer: Remove `--remote-postgres`, redundant with `--no-init-db`.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.

Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
2020-06-24 12:57:38 -07:00
Alex Vandiver 7c6a25a43d installer: Group and unify ordering of installer options.
This also adds the missing `--no-overwrite-settings` option to
`--help`.
2020-06-24 12:57:38 -07:00
Alex Vandiver b165b4144d installer: Prevent flags which conflict with `--no-overwrite-settings`.
Since `--postgres-missing-dictionaries` edits `/etc/zulip/zulip.conf`,
it interferes with the intent of `--no-overwrite-settings`.

Make the two settings conflict, to prevent this unclear state.
2020-06-23 13:40:28 -07:00
Alex Vandiver 7f4a2527c0 installer: Make `--no-overwrite-settings` also preserve `zulip.conf`.
This allows a path through the installer for places that have already
configured `zulip.conf`, by extending the existing flag and behavior.
2020-06-23 13:40:28 -07:00
Alex Vandiver 27100b4507 installer: Fix mis-indentation. 2020-06-23 13:36:26 -07:00
Alex Vandiver 5b7be7ba5d installer: Do not initialize db with --no-init-db.
The `--no-init-db` option previously only controlled if
`initialize-database` was run, which sets up the tables inside the
database.  If PostgreSQL was installed locally, it still attempted to
create the user and empty database.

This fails on hosts which are remote PostgreSQL hosts, and not
application hosts, as:

 - They may already have a local database, and while
 `initialize-datbase` will detect and offer to abort if one is
 found,`--no-init-db` seems like it should be the option to not
 overwrite it

 - `flush-memcached` requires that a local venv be installed, which it
 often is not on non-frontend machines.

Skip the database configuration when run with `--no-init-db`.
2020-06-23 13:36:26 -07:00
Anders Kaseorg a4f2704301 flush-memcached: Replace a type: ignore with an assert.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Tim Abbott 60b800b1ac upgrade-zulip-from-git: Fix setting postgres_version.
The new logic to set postgres_version when upgrading never wrote the
configuration file after making its edit.
2020-06-18 22:01:01 -07:00
Alex Vandiver 49a7a66004 install: Pin new apt-based installs to the latest postgresql.
Since we now support Postgres versions from 10 to 12, we might as well
have new installations start on Postgres 12 to avoid unnecessary
migration/upgrade work.
2020-06-16 17:08:16 -07:00
Alex Vandiver 6979ed9d97 install: Use the apt postgres server packages from postgres.
This allows Debian and Ubuntu administrators to reasonably seamlessly
swap over to more recent version of postgres than ships with their
distribution.
2020-06-16 17:05:46 -07:00
Alex Vandiver 03bffd3938 upgrade-zulip: Pin the postgres version to the OS default.
We would prefer to use the postgres packages from Postgres themselves,
if available.  However, this requires ensures that, for existing
installs, we preserve the same version of postgres as their base
distribution installed.

Move the version-determination logic from being computed at puppet
interpolation time, to being computed at install time and pinned into
zulip.conf.
2020-06-16 17:05:46 -07:00
Alex Vandiver e788ea52d2 upgrade-zulip: Use existing config helper functions. 2020-06-16 17:05:46 -07:00
Aman Agrawal da84b19aea upgrade-zulip: Shutdown servers with <3GB RAM when buiding static.
Fixes #14643.

This is to avoid running out of memory when building static assets
with webpack while server is running in low ram systems.
2020-06-15 22:17:02 -07:00
Aman Agrawal 81195abdbd upgrade-zulip: Extract shutdown call into a function.
This will help us call it as needed.
2020-06-15 22:17:02 -07:00
Vishnu KS 18ecf9bcfa backup: Make restore-backup work in docker.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Anders Kaseorg fa2496c229 terminate-psql-sessions: Rely on the caller to set PGHOST, PGUSER.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Vishnu KS f2ce856b8f scripts: Don't terminate current session in terminate-psql-sessions.
This is a prep commit. Running terminate-psql-sessions command on
docker-zulip results in the script exiting with non-zero exit status
2. This is because the current session also gets terminated while
running terminate-psql-sessions command. To prevent that from happening
we don't terminate the session created by terminate-psql-sessions.
2020-06-15 21:37:14 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 3461db7ef5 python: Convert percent formatting to "".format in certain files.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.

Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg 5050fb19f6 nagios: Don’t crash on missing cron file.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 16:49:32 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Alex Vandiver 4fe0444108 puppet: Install wal-g, not wal-e. 2020-06-11 15:52:43 -07:00
Anders Kaseorg 0e5946ee5a python: Add noqa comments for the specific star imports we allow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 15:36:43 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
arpit551 9e8f1aacb3 certbot: Switch to use certbot from apt.
certbot-auto doesn’t work on Ubuntu 20.04, and won’t be updated; we
migrate to instead using the certbot package shipped with the OS
instead. Also made sure that sure certbot gets installed when running
zulip-puppet-apply, to handle existing systems.
2020-06-08 21:59:29 -07:00
Anders Kaseorg 523907fe1d upgrade-zulip: Add umask override.
We already override the umask in upgrade-zulip-stage-2, but that’s too
late since we’ve already written a bunch of files in stage 1.  I would
have removed the stage 2 override, but the OS upgrade documentation
references running stage 2 directly.

Fixes #15164.  Note that an affected installation will need to upgrade
twice, because the first upgrade uses the old stage 1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 21:57:05 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
rht 07fa25dcd3 setup-yum-repo: Update url of postgresql rpm repo.
The old url is dead.
2020-06-08 11:26:07 -07:00
Anders Kaseorg 0f63753926 install-node: Upgrade Node.js to 12.18.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:06:57 -07:00
Anders Kaseorg 333f7d16c9 logging: Pass more format arguments to logging.
Commit bdc365d0fe (#14852) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
arpit551 439f0d3004 install: Ad production support for Zulip on Ubuntu Focal.
Install script now runs on Focal.  Python 2 is now installed via the
`python2` package in Focal.
2020-05-25 16:58:42 -07:00
arpit551 3971824d04 puppet: suppress puppet warnings with ruby 2.7.
Ubuntu Focal comes with ruby 2.7 and the latest puppet
has some issues with it so to suppress puppet
warnings with ruby 2.7 we added  RUBYOPT = "-W0" in
the environment.
2020-05-25 16:56:11 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-05-20 13:47:20 -07:00
Mateusz Mandera 28a6983b34 check-rabbitmq-queue: Log queue size in "queue stuck" alert. 2020-05-14 11:55:20 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Tim Abbott 800e6b1ca6 generate_secrets: Add more comments/documentation. 2020-04-30 10:44:27 -07:00
Steve Howell adc0ed4206 provision: Avoid shelling out to clean caches.
Yes, it's slightly janky to create an
argparse.Namespace object like this, but it
saves us from shelling out to a script whose
only real value-add is parsing a single
`threshold_days` argument.

This saves about 130ms for a no-op provision.
2020-04-30 17:19:13 +00:00
Steve Howell 34191f2a56 generate_secrets: Avoid unnecessary settings import.
We try to avoid importing Django settings unless
we really need them, since we want this program
to run very quickly during `provision` (when
secrets have already been generated earlier).
2020-04-30 13:52:54 +00:00
Steve Howell 522ee7fd6b generate_secrets: Lazily import crypto module. 2020-04-30 13:49:44 +00:00
Steve Howell 66392afe15 generate_secrets: Extract random_token(). 2020-04-30 13:23:13 +00:00
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
arpit551 fe6be1f61e setup_venv: Remove python2 dependencies for virtual environment.
These dependencies are not needed.
2020-04-22 16:10:59 -07:00
Steve Howell f4942e9927 digest refactor: Clean up names and comments.
We now use `extra_strings` instead of `package_versions`
to allow for more generic digests to be built
(without naming confusion).
2020-04-22 14:41:42 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Aman Agrawal 120144e099 upgrade-zulip: Exit if OS is unsupported.
This is to prevent folks who accidentally try to
upgrade their system to an unsupported platform
from getting into a bad place.
2020-04-22 10:04:36 -07:00
Aman Agrawal 0f4b1076ad scripts: Remove Xenial and Stretch support from installation scripts.
Note that we leave support for them in `setup-apt-repo` and puppet,
since we're still supporting systems using Xenial for non-appserver
puppet rules.
2020-04-22 10:00:38 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Tim Abbott 703fae8980 send_custom_email: Use a special .gitignored directory.
Previously, the send_custom_email code path leaked files in paths that
were not `.gitignored`, under templates/zerver/emails.

This became problematic when we added automated tests for this code
path, as it meant we leaked these files every time `test-backend` ran.

Fix this by ensuring all the files we generate are in this special
subdirectory.
2020-04-21 16:50:11 -07:00
arpit551 e6edf469ee ci: Remove the need of using TRAVIS in env.
Since now we want to use production suites on Circle CI so there
is no need to set TRAVIS in env while running scripts.

CIRCLECI is set default in the enviroment of Circle CI builds
so we can use it directly.

Also Travis CI had rabbitmq-server installed so we had to add workaround
in install script to avoid the error. That workaround is removed.
2020-04-21 14:46:40 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell b280f73c77 provision: Extract path_version_digest(). 2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Steve Howell 067196c508 provision: Simplify `is_force` codepaths.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.

If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.

Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.

For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.

We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
2020-04-17 09:45:59 -07:00
Tim Abbott 382261dc72 upgrade-zulip: Remove tsearch-extras on upgrade.
We stopped using tsearch-extras in Zulip 2.1.0 after Anders figured
out how to achieve its goals with native postgres.  However, we never
did a `DROP EXTENSION` on systems thta had upgraded, which meant that
backups created on systems originally installed with Zulip 2.0.x and
older, and later upgraded to Zulip 2.1.x, could not be restored on
Zulip servers created with a fresh install of Zulip 2.1.x.

We can't do this with a normal database migration, because DROP
EXTENSION has to be done as the postgres user, so we add some custom
migration code in the upgrade-zulip-stage-2 tool.

It's safe to run this whenever tsearch_extras.control is installed because:
* Zulip is AFAIK the only software that ever used tsearch_extras.
* The package was only installed via puppet on production servers configured to
  run a local Zulip database.
* We'll only run this code once per system, because it removes the
  package and thus the control files.

Fixes #13612.
2020-04-15 15:18:44 -07:00
Tim Abbott 6543788225 inline_email_css: Simplify parsing of template names.
Removing the extensions within the Set construction object was a
confusing way to do things.
2020-04-10 15:52:52 -07:00
Tim Abbott b2cf0aaa38 inline_email_css: Configure cssutils on import.
This is necessary for when we later want to call inline_template from
elsewhere.
2020-04-10 15:51:54 -07:00
wowol 78f3125f0a inline_email_css: Extract inline_template function. 2020-04-10 15:49:10 -07:00
wowol f42adba071 inline_email_css: Extract get_all_templates_from_directory. 2020-04-10 15:44:25 -07:00
wowol b3cc93f961 inline_email_css: Move constants to top, remove chdir. 2020-04-10 15:44:24 -07:00
wowol 0b5a87f98f inline_email_css: Extract strip_unnecesary_tags. 2020-04-10 15:34:17 -07:00
wowol 240d8d565c inline_email_css: Extract escape_jinja2_characters. 2020-04-10 15:29:47 -07:00
Tim Abbott 26c65a5f78 inline_email_css: Extract configure_cssutils. 2020-04-10 15:29:47 -07:00
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Mateusz Mandera f5a12e4094 nagios: Fix use of wrong variable in check_rabbitmq_queue.
That was supposed to be queue_name, not queue - the latter leads to
nonsensical results.
2020-04-10 13:48:16 -07:00
Tim Abbott c20b0bd992 check_rabbitmq_queue: Adjust threshholds for some queues. 2020-04-10 13:14:43 -07:00
Anders Kaseorg 038b315d8e python: Further pyupgrade changes following merge conflict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:59:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Mateusz Mandera 96fe2e5a42 nagios: Deduplicate queue list between check-rabbitmq scripts. 2020-04-09 13:41:01 -07:00
Mateusz Mandera 122d0bca83 check-rabbitmq-queue: Add a simple algorithm to analyze queue stats.
This new algorithm is designed to avoid monitoring paging when a queue
simply has bursty behavior.
2020-04-09 13:41:01 -07:00
arpit551 a2dd1fad12 provision: switch Focal to use PGroonga from PPA.
Since Groonga packages for Ubuntu 20.04 Focal are now available
in their ppa so stopped building pgroonga from source.
2020-03-31 15:02:31 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Tim Abbott 0f1bdcc46f restart-server: Restart Tornado processes individually.
After some testing, I've confirmed that this seems to behave
significantly better in terms of the number of failed requests due to
Tornado being the process of restarting compared with the previous
version, as each individual process is only down for a short time,
rather than all of them being down at once.
2020-03-27 06:23:34 -07:00
Anders Kaseorg 586f78cb32 install-node: Upgrade to Node.js 12.16.1, Yarn 1.22.4, nvm 0.35.3.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:34:47 -07:00
Tom Daff 2f213f7c8e
monitoring: Fix check-rabbitmq-consumers.
Missing commas in the definition of all the queues to check meant that it would be looking for queues with concatenated names, rather than the correct ones. Added the commas.
2020-03-25 17:19:16 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
arpit551 25e091a065 setup_path: Fix KeyError in setup_path for virtualenv ≥ 20.
In virtualenv ≥ 20, the site_packages variable was removed from
activate_this.py.  To avoid a KeyError, replace
activate_locals['site_packages'] with os.path.join(venv, 'lib',
python_version), where python_version is the 'pythonX.Y' name of the
directory where site-packages resides in the virtualenv.

Fixes #14025.
2020-03-23 04:06:19 -07:00
Mateusz Mandera ea93810d9a check-rabbitmq-queue: Put user check before rabbitmqctl call. 2020-03-22 18:46:28 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg fd2d642f95 postgres-init-db: Fix invocation from a relative path.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:40:37 -08:00
Anders Kaseorg d918626135 restore-backup: Run generate_secrets.py.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:20:43 -08:00
Mateusz Mandera 27b15a9722 install: Don't create internal realm in the installation process. 2020-02-12 12:00:10 -08:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
Anders Kaseorg 61bf698185 requirements: Fork pip.txt to pip2.txt for Python 2.
The current version of setuptools no longer supports Python 2.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg d2e07ea51b setup_venv: Replace virtualenv_args list with python2 bool.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg e88fac499f dependencies: Upgrade emoji-datasource from 4.0.4 to 5.0.1.
The “Smileys & People” category has been split into “Smilys & Emotion”
and “People & Body”.

Also, fix generate_sha1sum_emoji to read the emoji-datasource-google
version from yarn.lock, since package.json only gives a version range.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-04 21:30:51 -08:00
Mateusz Mandera 4c5a8e6f0c queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
Tim Abbott dd969b5339 install: Remove references to "Zulip Voyager".
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".

At the time, we just needed a name quickly, but it was never a good
name, just a placeholder.  This removes that placeholder name from
much of the codebase.  A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
2020-01-30 12:40:41 -08:00
Tim Abbott 9f414d74fd email_mirror: Rewrite docstrings to focus on current reality.
These docstrings hadn't been properly updated in years, and bad an
awkward mix of a bad version of the user-facing documentation and
details that are no longer true (e.g. references to "Voyager").

(One important detail is that we have real documentation for this
system now).
2020-01-30 12:39:45 -08:00
neiljp (Neil Pilgrim) 6ecf41d0bd clean_venv_cache: Limit search to .txt requirements files.
This both avoids examining all the .in files, but also any additional stray
files, which can cause provision errors.

Fixes #13762.
2020-01-30 12:20:32 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
rht 7c9954afc6 install: Reduce the RAM limit to accommodate some definition of "2GB".
See also the discussion at
https://chat.zulip.org/#narrow/stream/3-backend/topic/2GB.20ram.20gotcha.
2020-01-24 12:35:37 -08:00
rht 6f5cbed5f1 prod install: Ignore reading VERSION_CODENAME if not available. 2020-01-22 23:19:18 -08:00
rht c2dcaf48d8 prod install: Add the CentOS version of SUPERVISOR_CONF_DIR. 2020-01-22 23:19:18 -08:00
Anders Kaseorg 8c733a3f68 create-db.sql: Start by dropping the zulip database if needed.
At some point the PostgreSQL Docker image started creating the zulip
database for us, which caused our CREATE DATABASE to fail.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg 298d45b46a create-db.sql: Handle exception if zulip user already exists.
Fixes #13530.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg a82032a182 generate_secrets: Enable Redis authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg cdda983e90 settings: Support optional memcached authentication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Anders Kaseorg 699626f3cf flush-memcached: Use pylibmc.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:18 -08:00
rht cd3907648d prod install: Use ID_LIKE to help select os family. 2020-01-07 13:25:25 -08:00
rht bc94e8e815 prod install: Use /etc/os-release for Ubuntu/Debian to get os_id, os_version_id. 2020-01-07 13:25:25 -08:00
rht 9898c07e0d prod install: Add the CentOS version of the step to do dist-upgrade. 2020-01-07 13:25:25 -08:00
rht bf76696d67 prod install: Add the CentOS version of the step to install preparatory packages. 2020-01-07 13:25:25 -08:00
rht 6dd5dc32fc prod install: Add the CentOS version of the step to upgrade packages. 2020-01-07 13:25:25 -08:00
rht d88a7bbb42 prod install: Add the CentOS version of the step to update packages. 2020-01-07 13:25:25 -08:00
rht 49d7adb3cb prod install: Parse CentOS os identifications from /etc/os-release. 2020-01-07 13:25:25 -08:00
rht 771f6d213f prod install: Rename os_codename into os_version_id 2020-01-07 13:25:25 -08:00
Anders Kaseorg a78f8647d8 install: Run generate_secrets.py before zulip-puppet-apply.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-05 22:48:08 -08:00
Anders Kaseorg ab211c7acf lint: Tell ShellCheck to look for sourced files at relative paths.
This uses the new -P option of ShellCheck 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-18 03:48:02 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Anders Kaseorg 347fd80864 generate_secrets: Remove unused initial_password_salt in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 23:06:53 -08:00
Anders Kaseorg 7ebba2901a generate_secrets: Remove unused local_database_password in production.
Fixes #13464.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 13:05:31 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg ac49736311 install-node: Upgrade Node 12.11.1 to 12.13.0, Yarn 1.19.0 to 1.19.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
Anders Kaseorg d6377b00c0 node_cache: Don’t retry copying node_modules; let yarn do its thing.
`copytree` throws an error if the target already exists, and we don’t
really want to rerun the copy anyway.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-29 12:30:28 -07:00
Tim Abbott bbc1484253 check-rabbitmq-queue: Adjust threshholds for paging.
Ultimately, this isn't an effective way to monitor this queue; we want
time-based monitoring, not count-based monitoring.  Doing that
properly will likely involve modifying the queue processor to write
something about its status.

But until we add the monitoring we want, it makes sense to leave this
active with low limits.
2019-10-13 22:39:52 -07:00
Anders Kaseorg 775162d687 setup_venv: Use pip install --require-hashes for better security.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:21:18 -07:00
Anders Kaseorg 9182293d50 node_cache: Preserve symlinks when copying an old node_modules tree.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:19:53 -07:00
Anders Kaseorg 8432d97edf setup_venv: Add pkg-config to VENV_DEPENDENCIES.
This is needed on at least Debian 10, otherwise xmlsec fails to
install: `Could not find xmlsec1 config. Are libxmlsec1-dev and
pkg-config installed?`

Also remove libxmlsec1-openssl, which libxmlsec1-dev already depends.

(No changes are needed on RHEL, where libxml2-devel and xmlsec1-devel
already declare a requirement on /usr/bin/pkg-config.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-05 18:24:32 -07:00
Anders Kaseorg 1235dc3bec install-node: Upgrade to Node 12.11.1, Yarn 1.19.0.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-05 18:07:53 -07:00
Anders Kaseorg 0af22dad18 flush-memcached: Respect MEMCACHED_LOCATION; handle errors.
Fixes #13238.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-01 16:05:55 -07:00
ab1nash 71f0fecda7 scripts: Clean up output from 'clean_unused_caches'.
The output log from running clean_unused_caches was too verbose as
part of the `upgrade-zulip` overall output.  While this output is
potentially helpful when running it directly for debugging, it's
certainly redundant for the main production use case.

So a new flag --no-print-headers is introduced.  It suppresses the
header outputs for the subtools.

Fixes #13214.
2019-09-30 10:51:00 -07:00
Mateusz Mandera c42077c12f dependencies: Add dependencies needed for SAML. 2019-09-28 12:15:13 -07:00
Tim Abbott a84bb89bdc scripts: Move mobile i18n code out scripts/.
Like other code that is only used in the development environment, this
doesn't belong in scripts/.
2019-09-24 12:57:42 -07:00
Tim Abbott 27b3c1a312 provision: Move install-shellcheck to proper directory.
Scripts in scripts/ should be exclusively code that used in
production, and this isn't.
2019-09-24 12:54:33 -07:00
Anders Kaseorg 4fdc80a9c7 setup-apt-repo: Install groonga-keyring.
This allows the system to get updates to the Groonga repository
signing key, so `apt update` doesn’t start failing when the key
changes (like it recently did).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg d1e504079d setup-apt-repo: Don’t waste time installing debian-archive-keyring.
debian-archive-keyring is a dependency of the essential package apt,
so it is present in every Debian system.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg 2ff87bd888 setup: Update groonga APT repository signing key.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg 76492b25ae setup_venv: Install pip.txt requirements with --force-reinstall.
virtualenv on Ubuntu 16.04, when creating a new environment, downloads
the current version of setuptools, then replaces its pkg_resources
with an old copy from
/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl.
This causes problems, a simple example of which is reproducible from
the ubuntu:16.04 Docker base image as follows:

    apt-get update
    apt-get -y install python3-virtualenv
    python3 -m virtualenv -p python3 /ve
    /ve/bin/pip install sockjs-tornado
    /ve/bin/pip download sockjs-tornado

→ `AttributeError: '_NamespacePath' object has no attribute 'sort'`

More relevantly, it breaks pip-compile in the same way.  To fix this,
we need to force setuptools to be reinstalled, even if we’re asking
for the same version.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 13:23:58 -07:00
Anders Kaseorg 8d91bebf95 restart-server: Warn if the shell’s PWD goes through an updated symlink.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-21 12:02:15 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Anders Kaseorg 2e1494bdbd setup-apt-repo: Add ca-certificates to pre_setup_deps.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:15:43 -07:00
Anders Kaseorg 2ec946ad4d postgres-init-db: Require an Enter press in confirmation prompt.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-12 16:04:55 -07:00
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg 875002108f setup_venv: Remove CentOS workaround for fixed pycurl bug.
We are installing pycurl 7.43.0.3 which includes the fix.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:38:38 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Anders Kaseorg 9e481e353a .yarnrc: Set ignore-scripts true.
Follow up to #13065, to keep manual yarn invocations consistent with
our automated ones.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 16:15:54 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Anders Kaseorg 43c8c720ef node_cache: Run yarn with --ignore-scripts.
As predicted in https://www.kb.cert.org/vuls/id/319816/, a malicious
worm is beginning to spread across the npm ecosystem through package
postinstall scripts.  Only instead of direct self-replicating code,
the replication vector is the temptation to monetize postinstall
scripts by polluting the console logs with paid advertisements.  The
effect will be the same unless we all put a stop to this while we
still can.

Apply the recommended VU#319816 workaround, which is to disable
lifecycle scripts when installing npm packages.  The only fallout is:

* node-sass can’t run because it uses compiled native code; we replace
  it with Dart Sass.
* phantomjs-prebuilt doesn’t download the binary at install time; we
  tell it to download it in run-casper.
* ttf2woff2 transparently falls back from native code to an Emscripten
  build.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:36:03 -07:00
Anders Kaseorg fa913b7f5d node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Hemanth V. Alluri dac068df31 production: Finish adding production support for Zulip on Debian Buster.
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.

We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.

Fixes #9828.
2019-08-17 12:22:32 -07:00
Anders Kaseorg 86f8bb83ab setup_venv: Don’t pip uninstall typing; it’s no longer installed.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 14:13:40 -07:00
Anders Kaseorg d761bbc2ed node_cache: Remove unused copy_modules parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-14 14:39:04 -07:00
Anders Kaseorg 906af61d20 pythonrc: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg b320b8c37f email-mirror-postfix: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 62a1c3443d mypy: Remove type ignores for python 3 stubs now present in typeshed.
Namely:
* posix stubs
* readfp stub
2019-08-06 23:24:56 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg 1d05a71b5d install, provision: Treat all nonzero exit codes as failure, not just 1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:22:11 -07:00
Anders Kaseorg 9233b3bc75 setup_path_on_import: Always make sure zulip-py3-venv is activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-07-18 14:21:04 -07:00