Commit Graph

722 Commits

Author SHA1 Message Date
Alex Vandiver 876ee4a8ed installer: Remove code specific to stretch or xenial.
Support for Xenial and Stretch was removed (5154ddafca, 0f4b1076ad,
8944e0ad53, 79acd5ae40, 1219a2e854), but not all codepaths were
updated to remove their conditionals on it.

Remove all code predicated on Xenial or Stretch.  debathena support
was migrated to Bionic, since that appears to be the current state of
existing debathena servers.
2020-06-24 12:57:38 -07:00
Alex Vandiver e4899eae8b installer: Sync the claimed supported distros with the check.
0f4b1076ad removed Ubuntu 16.04 "xenial" and Debian 9 "stretch" from
the printed list of supported operating systems, but left them in the
verification check that controls if that message is printed,
effectively continuing to support them.

Conversely, 439f0d3004 added Ubuntu 20.04 "focal" to the check, but
not to the printed list.

Synchronize to check and print the right supported distributions:
Ubuntu 18.04 "bionic", Ubuntu 20.04 "focal", and Debian 10 "buster".
2020-06-24 12:57:38 -07:00
Alex Vandiver 58cb7cecd8 installer: Remove `--remote-postgres`, redundant with `--no-init-db`.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.

Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
2020-06-24 12:57:38 -07:00
Alex Vandiver 7c6a25a43d installer: Group and unify ordering of installer options.
This also adds the missing `--no-overwrite-settings` option to
`--help`.
2020-06-24 12:57:38 -07:00
Alex Vandiver b165b4144d installer: Prevent flags which conflict with `--no-overwrite-settings`.
Since `--postgres-missing-dictionaries` edits `/etc/zulip/zulip.conf`,
it interferes with the intent of `--no-overwrite-settings`.

Make the two settings conflict, to prevent this unclear state.
2020-06-23 13:40:28 -07:00
Alex Vandiver 7f4a2527c0 installer: Make `--no-overwrite-settings` also preserve `zulip.conf`.
This allows a path through the installer for places that have already
configured `zulip.conf`, by extending the existing flag and behavior.
2020-06-23 13:40:28 -07:00
Alex Vandiver 27100b4507 installer: Fix mis-indentation. 2020-06-23 13:36:26 -07:00
Alex Vandiver 5b7be7ba5d installer: Do not initialize db with --no-init-db.
The `--no-init-db` option previously only controlled if
`initialize-database` was run, which sets up the tables inside the
database.  If PostgreSQL was installed locally, it still attempted to
create the user and empty database.

This fails on hosts which are remote PostgreSQL hosts, and not
application hosts, as:

 - They may already have a local database, and while
 `initialize-datbase` will detect and offer to abort if one is
 found,`--no-init-db` seems like it should be the option to not
 overwrite it

 - `flush-memcached` requires that a local venv be installed, which it
 often is not on non-frontend machines.

Skip the database configuration when run with `--no-init-db`.
2020-06-23 13:36:26 -07:00
Tim Abbott 60b800b1ac upgrade-zulip-from-git: Fix setting postgres_version.
The new logic to set postgres_version when upgrading never wrote the
configuration file after making its edit.
2020-06-18 22:01:01 -07:00
Alex Vandiver 49a7a66004 install: Pin new apt-based installs to the latest postgresql.
Since we now support Postgres versions from 10 to 12, we might as well
have new installations start on Postgres 12 to avoid unnecessary
migration/upgrade work.
2020-06-16 17:08:16 -07:00
Alex Vandiver 6979ed9d97 install: Use the apt postgres server packages from postgres.
This allows Debian and Ubuntu administrators to reasonably seamlessly
swap over to more recent version of postgres than ships with their
distribution.
2020-06-16 17:05:46 -07:00
Alex Vandiver 03bffd3938 upgrade-zulip: Pin the postgres version to the OS default.
We would prefer to use the postgres packages from Postgres themselves,
if available.  However, this requires ensures that, for existing
installs, we preserve the same version of postgres as their base
distribution installed.

Move the version-determination logic from being computed at puppet
interpolation time, to being computed at install time and pinned into
zulip.conf.
2020-06-16 17:05:46 -07:00
Alex Vandiver e788ea52d2 upgrade-zulip: Use existing config helper functions. 2020-06-16 17:05:46 -07:00
Aman Agrawal da84b19aea upgrade-zulip: Shutdown servers with <3GB RAM when buiding static.
Fixes #14643.

This is to avoid running out of memory when building static assets
with webpack while server is running in low ram systems.
2020-06-15 22:17:02 -07:00
Aman Agrawal 81195abdbd upgrade-zulip: Extract shutdown call into a function.
This will help us call it as needed.
2020-06-15 22:17:02 -07:00
Vishnu KS 18ecf9bcfa backup: Make restore-backup work in docker.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 3461db7ef5 python: Convert percent formatting to "".format in certain files.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.

Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 0e5946ee5a python: Add noqa comments for the specific star imports we allow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 15:36:43 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
arpit551 9e8f1aacb3 certbot: Switch to use certbot from apt.
certbot-auto doesn’t work on Ubuntu 20.04, and won’t be updated; we
migrate to instead using the certbot package shipped with the OS
instead. Also made sure that sure certbot gets installed when running
zulip-puppet-apply, to handle existing systems.
2020-06-08 21:59:29 -07:00
Anders Kaseorg 523907fe1d upgrade-zulip: Add umask override.
We already override the umask in upgrade-zulip-stage-2, but that’s too
late since we’ve already written a bunch of files in stage 1.  I would
have removed the stage 2 override, but the OS upgrade documentation
references running stage 2 directly.

Fixes #15164.  Note that an affected installation will need to upgrade
twice, because the first upgrade uses the old stage 1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 21:57:05 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
rht 07fa25dcd3 setup-yum-repo: Update url of postgresql rpm repo.
The old url is dead.
2020-06-08 11:26:07 -07:00
Anders Kaseorg 0f63753926 install-node: Upgrade Node.js to 12.18.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:06:57 -07:00
Anders Kaseorg 333f7d16c9 logging: Pass more format arguments to logging.
Commit bdc365d0fe (#14852) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
arpit551 439f0d3004 install: Ad production support for Zulip on Ubuntu Focal.
Install script now runs on Focal.  Python 2 is now installed via the
`python2` package in Focal.
2020-05-25 16:58:42 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-05-20 13:47:20 -07:00
Mateusz Mandera 28a6983b34 check-rabbitmq-queue: Log queue size in "queue stuck" alert. 2020-05-14 11:55:20 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Steve Howell adc0ed4206 provision: Avoid shelling out to clean caches.
Yes, it's slightly janky to create an
argparse.Namespace object like this, but it
saves us from shelling out to a script whose
only real value-add is parsing a single
`threshold_days` argument.

This saves about 130ms for a no-op provision.
2020-04-30 17:19:13 +00:00
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
arpit551 fe6be1f61e setup_venv: Remove python2 dependencies for virtual environment.
These dependencies are not needed.
2020-04-22 16:10:59 -07:00
Steve Howell f4942e9927 digest refactor: Clean up names and comments.
We now use `extra_strings` instead of `package_versions`
to allow for more generic digests to be built
(without naming confusion).
2020-04-22 14:41:42 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Aman Agrawal 120144e099 upgrade-zulip: Exit if OS is unsupported.
This is to prevent folks who accidentally try to
upgrade their system to an unsupported platform
from getting into a bad place.
2020-04-22 10:04:36 -07:00
Aman Agrawal 0f4b1076ad scripts: Remove Xenial and Stretch support from installation scripts.
Note that we leave support for them in `setup-apt-repo` and puppet,
since we're still supporting systems using Xenial for non-appserver
puppet rules.
2020-04-22 10:00:38 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
arpit551 e6edf469ee ci: Remove the need of using TRAVIS in env.
Since now we want to use production suites on Circle CI so there
is no need to set TRAVIS in env while running scripts.

CIRCLECI is set default in the enviroment of Circle CI builds
so we can use it directly.

Also Travis CI had rabbitmq-server installed so we had to add workaround
in install script to avoid the error. That workaround is removed.
2020-04-21 14:46:40 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell b280f73c77 provision: Extract path_version_digest(). 2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Steve Howell 067196c508 provision: Simplify `is_force` codepaths.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.

If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.

Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.

For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.

We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
2020-04-17 09:45:59 -07:00
Tim Abbott 382261dc72 upgrade-zulip: Remove tsearch-extras on upgrade.
We stopped using tsearch-extras in Zulip 2.1.0 after Anders figured
out how to achieve its goals with native postgres.  However, we never
did a `DROP EXTENSION` on systems thta had upgraded, which meant that
backups created on systems originally installed with Zulip 2.0.x and
older, and later upgraded to Zulip 2.1.x, could not be restored on
Zulip servers created with a fresh install of Zulip 2.1.x.

We can't do this with a normal database migration, because DROP
EXTENSION has to be done as the postgres user, so we add some custom
migration code in the upgrade-zulip-stage-2 tool.

It's safe to run this whenever tsearch_extras.control is installed because:
* Zulip is AFAIK the only software that ever used tsearch_extras.
* The package was only installed via puppet on production servers configured to
  run a local Zulip database.
* We'll only run this code once per system, because it removes the
  package and thus the control files.

Fixes #13612.
2020-04-15 15:18:44 -07:00
Mateusz Mandera f5a12e4094 nagios: Fix use of wrong variable in check_rabbitmq_queue.
That was supposed to be queue_name, not queue - the latter leads to
nonsensical results.
2020-04-10 13:48:16 -07:00
Tim Abbott c20b0bd992 check_rabbitmq_queue: Adjust threshholds for some queues. 2020-04-10 13:14:43 -07:00
Anders Kaseorg 038b315d8e python: Further pyupgrade changes following merge conflict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:59:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Mateusz Mandera 122d0bca83 check-rabbitmq-queue: Add a simple algorithm to analyze queue stats.
This new algorithm is designed to avoid monitoring paging when a queue
simply has bursty behavior.
2020-04-09 13:41:01 -07:00
arpit551 a2dd1fad12 provision: switch Focal to use PGroonga from PPA.
Since Groonga packages for Ubuntu 20.04 Focal are now available
in their ppa so stopped building pgroonga from source.
2020-03-31 15:02:31 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Anders Kaseorg 586f78cb32 install-node: Upgrade to Node.js 12.16.1, Yarn 1.22.4, nvm 0.35.3.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:34:47 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
arpit551 25e091a065 setup_path: Fix KeyError in setup_path for virtualenv ≥ 20.
In virtualenv ≥ 20, the site_packages variable was removed from
activate_this.py.  To avoid a KeyError, replace
activate_locals['site_packages'] with os.path.join(venv, 'lib',
python_version), where python_version is the 'pythonX.Y' name of the
directory where site-packages resides in the virtualenv.

Fixes #14025.
2020-03-23 04:06:19 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
Anders Kaseorg 61bf698185 requirements: Fork pip.txt to pip2.txt for Python 2.
The current version of setuptools no longer supports Python 2.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg d2e07ea51b setup_venv: Replace virtualenv_args list with python2 bool.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg e88fac499f dependencies: Upgrade emoji-datasource from 4.0.4 to 5.0.1.
The “Smileys & People” category has been split into “Smilys & Emotion”
and “People & Body”.

Also, fix generate_sha1sum_emoji to read the emoji-datasource-google
version from yarn.lock, since package.json only gives a version range.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-04 21:30:51 -08:00
Tim Abbott 9f414d74fd email_mirror: Rewrite docstrings to focus on current reality.
These docstrings hadn't been properly updated in years, and bad an
awkward mix of a bad version of the user-facing documentation and
details that are no longer true (e.g. references to "Voyager").

(One important detail is that we have real documentation for this
system now).
2020-01-30 12:39:45 -08:00
neiljp (Neil Pilgrim) 6ecf41d0bd clean_venv_cache: Limit search to .txt requirements files.
This both avoids examining all the .in files, but also any additional stray
files, which can cause provision errors.

Fixes #13762.
2020-01-30 12:20:32 -08:00
rht 7c9954afc6 install: Reduce the RAM limit to accommodate some definition of "2GB".
See also the discussion at
https://chat.zulip.org/#narrow/stream/3-backend/topic/2GB.20ram.20gotcha.
2020-01-24 12:35:37 -08:00
rht 6f5cbed5f1 prod install: Ignore reading VERSION_CODENAME if not available. 2020-01-22 23:19:18 -08:00
rht c2dcaf48d8 prod install: Add the CentOS version of SUPERVISOR_CONF_DIR. 2020-01-22 23:19:18 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
rht cd3907648d prod install: Use ID_LIKE to help select os family. 2020-01-07 13:25:25 -08:00
rht bc94e8e815 prod install: Use /etc/os-release for Ubuntu/Debian to get os_id, os_version_id. 2020-01-07 13:25:25 -08:00
rht 9898c07e0d prod install: Add the CentOS version of the step to do dist-upgrade. 2020-01-07 13:25:25 -08:00
rht bf76696d67 prod install: Add the CentOS version of the step to install preparatory packages. 2020-01-07 13:25:25 -08:00
rht 6dd5dc32fc prod install: Add the CentOS version of the step to upgrade packages. 2020-01-07 13:25:25 -08:00
rht d88a7bbb42 prod install: Add the CentOS version of the step to update packages. 2020-01-07 13:25:25 -08:00
rht 49d7adb3cb prod install: Parse CentOS os identifications from /etc/os-release. 2020-01-07 13:25:25 -08:00
rht 771f6d213f prod install: Rename os_codename into os_version_id 2020-01-07 13:25:25 -08:00
Anders Kaseorg a78f8647d8 install: Run generate_secrets.py before zulip-puppet-apply.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-05 22:48:08 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg ac49736311 install-node: Upgrade Node 12.11.1 to 12.13.0, Yarn 1.19.0 to 1.19.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
Anders Kaseorg d6377b00c0 node_cache: Don’t retry copying node_modules; let yarn do its thing.
`copytree` throws an error if the target already exists, and we don’t
really want to rerun the copy anyway.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-29 12:30:28 -07:00
Anders Kaseorg 775162d687 setup_venv: Use pip install --require-hashes for better security.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:21:18 -07:00
Anders Kaseorg 9182293d50 node_cache: Preserve symlinks when copying an old node_modules tree.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:19:53 -07:00
Anders Kaseorg 8432d97edf setup_venv: Add pkg-config to VENV_DEPENDENCIES.
This is needed on at least Debian 10, otherwise xmlsec fails to
install: `Could not find xmlsec1 config. Are libxmlsec1-dev and
pkg-config installed?`

Also remove libxmlsec1-openssl, which libxmlsec1-dev already depends.

(No changes are needed on RHEL, where libxml2-devel and xmlsec1-devel
already declare a requirement on /usr/bin/pkg-config.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-05 18:24:32 -07:00
Anders Kaseorg 1235dc3bec install-node: Upgrade to Node 12.11.1, Yarn 1.19.0.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-05 18:07:53 -07:00
ab1nash 71f0fecda7 scripts: Clean up output from 'clean_unused_caches'.
The output log from running clean_unused_caches was too verbose as
part of the `upgrade-zulip` overall output.  While this output is
potentially helpful when running it directly for debugging, it's
certainly redundant for the main production use case.

So a new flag --no-print-headers is introduced.  It suppresses the
header outputs for the subtools.

Fixes #13214.
2019-09-30 10:51:00 -07:00
Mateusz Mandera c42077c12f dependencies: Add dependencies needed for SAML. 2019-09-28 12:15:13 -07:00
Tim Abbott a84bb89bdc scripts: Move mobile i18n code out scripts/.
Like other code that is only used in the development environment, this
doesn't belong in scripts/.
2019-09-24 12:57:42 -07:00
Tim Abbott 27b3c1a312 provision: Move install-shellcheck to proper directory.
Scripts in scripts/ should be exclusively code that used in
production, and this isn't.
2019-09-24 12:54:33 -07:00
Anders Kaseorg 4fdc80a9c7 setup-apt-repo: Install groonga-keyring.
This allows the system to get updates to the Groonga repository
signing key, so `apt update` doesn’t start failing when the key
changes (like it recently did).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg d1e504079d setup-apt-repo: Don’t waste time installing debian-archive-keyring.
debian-archive-keyring is a dependency of the essential package apt,
so it is present in every Debian system.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg 76492b25ae setup_venv: Install pip.txt requirements with --force-reinstall.
virtualenv on Ubuntu 16.04, when creating a new environment, downloads
the current version of setuptools, then replaces its pkg_resources
with an old copy from
/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl.
This causes problems, a simple example of which is reproducible from
the ubuntu:16.04 Docker base image as follows:

    apt-get update
    apt-get -y install python3-virtualenv
    python3 -m virtualenv -p python3 /ve
    /ve/bin/pip install sockjs-tornado
    /ve/bin/pip download sockjs-tornado

→ `AttributeError: '_NamespacePath' object has no attribute 'sort'`

More relevantly, it breaks pip-compile in the same way.  To fix this,
we need to force setuptools to be reinstalled, even if we’re asking
for the same version.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 13:23:58 -07:00
Anders Kaseorg 2e1494bdbd setup-apt-repo: Add ca-certificates to pre_setup_deps.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:15:43 -07:00
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg 875002108f setup_venv: Remove CentOS workaround for fixed pycurl bug.
We are installing pycurl 7.43.0.3 which includes the fix.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:38:38 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Anders Kaseorg 9e481e353a .yarnrc: Set ignore-scripts true.
Follow up to #13065, to keep manual yarn invocations consistent with
our automated ones.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 16:15:54 -07:00
Anders Kaseorg 43c8c720ef node_cache: Run yarn with --ignore-scripts.
As predicted in https://www.kb.cert.org/vuls/id/319816/, a malicious
worm is beginning to spread across the npm ecosystem through package
postinstall scripts.  Only instead of direct self-replicating code,
the replication vector is the temptation to monetize postinstall
scripts by polluting the console logs with paid advertisements.  The
effect will be the same unless we all put a stop to this while we
still can.

Apply the recommended VU#319816 workaround, which is to disable
lifecycle scripts when installing npm packages.  The only fallout is:

* node-sass can’t run because it uses compiled native code; we replace
  it with Dart Sass.
* phantomjs-prebuilt doesn’t download the binary at install time; we
  tell it to download it in run-casper.
* ttf2woff2 transparently falls back from native code to an Emscripten
  build.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:36:03 -07:00
Anders Kaseorg fa913b7f5d node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Hemanth V. Alluri dac068df31 production: Finish adding production support for Zulip on Debian Buster.
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.

We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.

Fixes #9828.
2019-08-17 12:22:32 -07:00
Anders Kaseorg 86f8bb83ab setup_venv: Don’t pip uninstall typing; it’s no longer installed.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 14:13:40 -07:00
Anders Kaseorg d761bbc2ed node_cache: Remove unused copy_modules parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-14 14:39:04 -07:00
Anders Kaseorg 906af61d20 pythonrc: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg b320b8c37f email-mirror-postfix: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 62a1c3443d mypy: Remove type ignores for python 3 stubs now present in typeshed.
Namely:
* posix stubs
* readfp stub
2019-08-06 23:24:56 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg 9233b3bc75 setup_path_on_import: Always make sure zulip-py3-venv is activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-07-18 14:21:04 -07:00
Anders Kaseorg a45be467ad get_deploy_root: Avoid useless extra realpath call.
The comment that tabbott edited into my commit while wimpifying this
function is wrong on multiple levels.

Firstly, the way in which users might be “running our scripts” was
never relevant.  `__file__` is not the script that the user ran, it’s
zulip_tools.py itself.  What matters is not how the user ran the
script, but rather how zulip_tools was imported.  If zulip_tools was
imported as scripts.lib.zulip_tools, then `__file__` must end with
`scripts/lib/zulip_tools.py`, so running dirname three times on it is
fine.  In fact, in Python ≥ 3.4 (we don’t support anything older),
`__file__` in an imported module is always an absolute path, so it
must end with `scripts/lib/zulip_tools.py` in any case.

(At present, there’s one script that imports lib.zulip_tools, and the
installer runs scripts/lib/zulip_tools.py as a script, but those uses
don’t hit this function.)

Secondly, even if we do care about `__file__` being a funny relative
path, there’s still no reason to have two calls to `realpath`.
`realpath(dirname(dirname(dirname(realpath(…)))))` is equivalent to
`dirname(dirname(dirname(realpath(…)))), as the inner `realpath` has
already canonicalized symlinks at every level.

This version also deals with `__file__` being a funny relative
path (assuming none of scripts, lib, and zulip_tools.py are themselves
symlinks), while making fewer `lstat` calls than either of the above
constructions.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-07-18 14:20:53 -07:00
Sumanth V Rao 2c9f5e3980 tools: Add tool to update API field of local zuliprc file.
This tool can be used to update the API field of local
zuliprc files for dummy users of development server
(iago, prospero, etc) with the correct API key from database.

This tool can be run after provisioning (or similar tools) which change
the API keys in the database.
2019-07-17 16:00:21 -07:00
Amal Rajan 09049f1678 provision: Add provision support for Ubuntu 19.04
This commit adds support for Ubuntu 19.04 in the development
environment.
2019-07-13 16:44:09 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Anders Kaseorg caecd1c2ad install: Disable installation and provisioning on Ubuntu 14.04 Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 15:58:53 -07:00
Puneeth Chaganti 9f916f78da upgrade-zulip-from-git: Fetch tags to make version info more useful.
To be able to show more detailed git information in ZULIP_VERSION, we need
to fetch tags when running `upgrade-zulip-from-git`.
2019-06-23 23:10:47 -07:00
Tim Abbott a716b09a3f upgrade-zulip-from-git: Fix handling of errors in git clone.
Apparently, the `chown -R` would never run if the original clone
attempt had networking errors, leading to inability to use
upgrade-zulip-from-git without manual intervention.
2019-06-18 10:00:50 -07:00
Tim Abbott 9760c8cfc9 tools: Fix postgres-init-*-db cache handling.
Previously, it didn't properly update the stamp files that determine
our caching behavior, so if one ran test-backend afterwards, nothing
would happen.

A secondary issue that this commit does not fix is that provision will
end up rerunning the whole thing.
2019-06-17 16:24:13 -07:00
Wyatt Hoodes 8e626d3db3 zulip_tools: Add a function to get or create the var uuid path. 2019-06-17 13:51:40 -07:00
Tim Abbott d2fbd62b63 install: Fix argument parsing for no-overwrite-settings option.
This had the same bug as our other recent change to the install
script.
2019-06-16 15:24:20 -07:00
Wyatt Hoodes 0b05d91e62 test_runner: Write database ids to file for reference in clean up.
The ids that will be used for each particular run of the test suite are
written to a unique file. Each file will then be used as a time
reference of when the suite was ran.

This change sets up the ability for a complete clean up of potentially
leaked database templates.

Tweaked by tabbott to remove these files after successful database
cleanup.
2019-06-14 15:23:20 -07:00
Vishnu Ks 6e52779ed5 install: Include no-dist-upgrade in args list.
This was missed out in 2e51ac8c49
2019-06-14 14:24:08 -07:00
Tim Abbott 8a53686f41 install: Add --no-overwrite-settings option.
This commit needs more work to make this option reasonable.
2019-06-13 14:39:25 -07:00
ppreethi 86840adda5 version: Show number of commits and commit sha in ZULIP_VERSION.
We use `git describe --tags` to get information about the number of commit since
the last major version, and the sha of the current HEAD. This is added to the
ZULIP_VERSION when a deploy is done from `git`.

Modified heavily by punchagan to:
* to use git describe instead of `git log` and `wc`
* use a separate script to run the git describe command
* write the file with version info to var/ and remove it from the repo

Fixes #4685.
2019-06-07 13:39:12 -07:00
Thomas Ip c6cdcd082e dependencies: Upgrade node to 12.3.1.
API changes:
* The behaviour of Date.toLocaleTimeString() reverts to pre 8.0.0,
  this only affects automated tests. Lots of other API changes but
  we didn't use any of those.
* The internal sorting algorithm changed which causes one of our own
  compare function to miss coverage.
2019-06-05 17:15:52 -07:00
Vishnu Ks 14ed0e283d install: Add option to skip dist-upgrade. 2019-06-05 15:50:02 -07:00
Thomas Ip 9a84a60029 dependencies: Upgrade yarn to 1.16.0. 2019-06-03 12:24:00 -07:00
Anders Kaseorg 90a9dfa7f5 setup-apt-repo: Try to download pre-setup deps before skipping update.
Simulate isn’t enough in some cases.  The error message when this
fails looks sufficiently non-alarming.

LXC:

    default: + apt-get -dy install lsb-release apt-transport-https gnupg
    default: Reading package lists...
    default: Building dependency tree...
    default:
    default: Reading state information...
    default: lsb-release is already the newest version.
    default: gnupg is already the newest version.
    default: The following NEW packages will be installed:
    default:   apt-transport-https
    default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    default: Need to get 25.1 kB of archives.
    default: After this operation, 238 kB of additional disk space will be used.
    default: Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main apt-transport-https amd64 1.0.1ubuntu2.3
    default:   404  Not Found [IP: 91.189.88.161 80]
    default: Err http://security.ubuntu.com/ubuntu/ trusty-security/main apt-transport-https amd64 1.0.1ubuntu2.3
    default:   404  Not Found [IP: 91.189.88.161 80]
    default: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/apt/apt-transport-https_1.0.1ubuntu2.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
    default:
    default: E: Some files failed to download
    default: + apt-get update
    […]
    default: Fetched 4,504 kB in 7s (611 kB/s)
    default: Reading package lists...
    default: + apt-get -y install lsb-release apt-transport-https gnupg
    default: Reading package lists...

Docker:

    default: + apt-get -dy install lsb-release apt-transport-https gnupg
    default: Reading package lists...
    default: Building dependency tree...
    default:
    default: Reading state information...
    default: Package gnupg is not available, but is referred to by another package.
    default: This may mean that the package is missing, has been obsoleted, or
    default: is only available from another source
    default: E: Package 'gnupg' has no installation candidate
    default: + apt-get update
    […]
    default: Fetched 16.2 MB in 5s (3,326 kB/s)
    default: Reading package lists...
    default: + apt-get -y install lsb-release apt-transport-https gnupg
    default: Reading package lists...

(All in green.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 15:25:00 -07:00
Anders Kaseorg 407564086d provision: Move apt-get update from retry handler to setup-apt-repo.
This avoids unnecessarily alarming error messages if the apt cache is
missing.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:38 -07:00
Anders Kaseorg ae524b677d ci: Move backend and production tests to Ubuntu 16.04 (xenial).
This is preparation for dropping support for Trusty in CI.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:15 -07:00
Anders Kaseorg d267884cc1 node_cache: Run yarn install with --frozen-lockfile.
This way a yarn.lock inconsistent with package.json can’t slip through
unnoticed.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-20 19:23:37 -07:00
Puneeth Chaganti 0884764421 url preview: Add python-magic dependency for content-type sniffing. 2019-05-06 12:37:32 -07:00
Mayank Madan 99414e2d96 provision: Add support for Debian 10 Buster. 2019-05-05 17:57:19 -07:00
Mayank Madan a86eccb2b6 pgroonga: Upgrade from 2.1.6 to 2.1.8.
This is necessary to support postgres 11 in Debian Buster.
2019-05-05 17:52:32 -07:00
Tim Abbott 18146ed947 script: Add static asset pipeline option to support OS upgrades.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
2019-04-17 12:58:42 -07:00
Anders Kaseorg 9fc0aa7ffa scripts: Fix exec invocation for in-process virtualenv activation.
activate_this.py has always documented that it should be exec()ed with
locals = globals, and in virtualenv 16.0.0 it raises a NameError
otherwise.

As a simplified demonstration of the weird things that can go wrong
when locals ≠ globals:

    >>> exec('a = 1; print([a])', {}, {})
    [1]
    >>> exec('a = 1; print([a for b in [1]])', {}, {})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
      File "<string>", line 1, in <listcomp>
    NameError: name 'a' is not defined
    >>> exec('a = 1; print([a for b in [1]])', {})
    [1]

Top-level assignments go into locals, but from inside a new scope like
a list comprehension, they’re read out of globals, which doesn’t work.

Fixes #12030.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-16 17:42:44 -07:00
Wyatt Hoodes d67dc2eb94 install-node: Upgrade yarn to 1.15.2.
In addition to upgrading dependencies being generally useful, this may
fix situations where yarn fails but returns a success status code in the 
presence of an HTTP proxy.
2019-04-14 22:36:14 -07:00
Rohitt Vashishtha 22a3b4230c Revert "install: Ensure that sudo is installed."
Now that we have the run_as_root helper function, we don't need to
install sudo to run Zulip in production

This reverts commit a7d7d181ea.

Fixes #10036.
2019-04-04 16:52:20 -07:00
Mayank Madan 264c2e9803 provision: Add provision support for Ubuntu 18.10.
Note that this is only support for Ubuntu 18.10 in the development
environment.
2019-03-09 16:01:46 -08:00
Tim Abbott 0ab8cd9ca3 provision: Refactor handling of dependencies for building pgroonga.
This lets us leave build-pgroonga platform-agnostic.
2019-03-08 15:12:56 -08:00
Tim Abbott 5c20ee998c build-tsearch-extras: Add support for Debian/Ubuntu. 2019-03-08 15:12:56 -08:00
Tim Abbott 8469c7774f scripts: Remove create-zulip-admin.
This was used only by ancient versions of the Docker project.
2019-03-04 15:33:51 -08:00
Tim Abbott 0e7a5d82b4 setup_venv: Remove virtualenv-clone legacy upgrade code.
Few folks will be upgrading from versions of Zulip old enough to not
have virtualenv-clone, and those who are won't be able to use it due
to older dependencies having been removed.
2019-03-04 15:32:15 -08:00
Tim Abbott 4157d15ff1 setup_venv: Fix check for whether virtualenv-clone is unavailable.
We don't need a bare except here.
2019-03-04 15:32:15 -08:00
Tim Abbott 478e215acd setup_venv: Use run_as_root helper.
Also, don't use split() to break up strings to pass into run().
2019-03-04 15:32:15 -08:00
Anders Kaseorg d9adc9d7bc get_dev_uuid_var_path: Fix theoretical shell quoting problem.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-03-04 15:20:07 -08:00
Tim Abbott b3444354aa su_to_zulip: Fix detection of zulip user ID.
Apparently, while upgrade-zulip-from-git always ensures that zulip
deployment directories are owned by the Zulip user, unpack-zulip (aka
the tarball code path) has them owned by root.

The user ID detection logic in su_to_zulip's helper get_zulip_uid was
intended to support both development environments (where the user ID
might vary) and production environments.  For development
environments, the existing code is fine, but given this unpack-zulip
permissions issue, we need to have code to fallback to 'zulip' if the
detection logic detects the "zulip" user has having UID 0.
2019-03-04 14:27:39 -08:00
Rohitt Vashishtha 3d427d02cc scripts/zulip_tools: Use run_as_root instead of subprocess.check_call. 2019-03-01 11:21:16 -08:00
Rohitt Vashishtha ac48925977 scripts: Use run_as_root instead of run([sudo, ...]). 2019-03-01 11:21:16 -08:00
Tim Abbott 57e1307a3a provision: Fix virtualenv-clone handling of success-stamp.
Apparently, virtualenv-clone ends up copying the success-stamp file
that we use to track whether a virtualenv was successfully
provisioned, which results in problems if we get a network error in
the pip install stage afterwards.

The comment explains our fix, but basically we just delete
success-stamp after the clone.

Fixes #11301.
2019-02-16 11:24:10 -08:00
Anders Kaseorg e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00
Anders Kaseorg ebad0b7cbf zulip_tools: Get the zulip uid from the owner of DEPLOY_ROOT.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-11 17:00:37 -08:00
Anders Kaseorg e4832cc9da setup_path_on_import: Don’t re-add BASE_DIR to sys.path.
BASE_DIR must already be in sys.path if we got as far as importing
this.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:17:31 -08:00
Anders Kaseorg 70bfcd3402 zulip_tools: Extract get_deploy_root function.
Modified by tabbott from the original to preserve the implementation;
see https://github.com/zulip/zulip/pull/11295#discussion_r254925032
for why this is correct.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:09:29 -08:00
Anders Kaseorg e984107966 scripts: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:58 -08:00
Anders Kaseorg 4fef10f4e6 install: Improve error handling.
On usage errors (except --help), write usage message to stderr and
exit with nonzero status.

Forbid setting the hostname and email to the example values.  Those
are specifically checked for and would fail later.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 11:18:42 -08:00
Anders Kaseorg 893e1475bb setup_virtualenv: Do not activate the virtualenv on creation.
Instead, manually activate it in the one place where this
functionality was used (tools/lib/provision.py).  This way we avoid
trying to activate the Python 2 thumbor virtualenv from Python 3.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-31 17:55:04 -08:00
Tim Abbott df436a55b4 scripts: Recommend apt update after enabling universe.
One needs to manually do an apt update after add-apt-repository, or it
won't actually work.
2019-01-26 12:29:05 -08:00
Anders Kaseorg 5ba84d75ce create-zulip-admin: Fix shell quoting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:46:50 -08:00
Harshit Bansal 50ef91bb08 scripts: Add argparse option to `restart-zerver` for `--fill-cache`.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.

Fixes: #10900.
2019-01-14 15:20:01 -08:00
Anders Kaseorg b9f7b08af2 setup_path_on_import: Check that the virtualenv is compatible.
The site_packages variable points to (e.g.)
zulip-py3-venv/lib/python3.4/site-packages.  If that doesn’t exist,
we’re probably running the wrong Python version.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:09 -08:00
Anders Kaseorg 7a58f574bf setup_path_on_import: Skip activate_this if already activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:07 -08:00
Anders Kaseorg 2ad1a27b28 create-production-venv: Remove Python 2 support.
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:05 -08:00
rht 91348a6aee create-thumbor-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht 68c20c610e create-production-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht 15763f8545 provision: Include DISTRIB_FAMILY in parse_lsb_release output. 2019-01-07 18:52:09 -08:00
rht d8a081b94c setup-yum-repo: Add prod flag for doing IUS install on CentOS. 2019-01-05 15:49:02 -08:00
rht 105732ab1f parse_lsb_release: Fix vendor name matching for CentOS. 2019-01-04 14:09:48 -08:00
rht d3139266c8 setup_venv: Uninstall typing on Python >3.4.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
2019-01-01 19:34:48 -08:00
rht c190959b58 provision: Move CentOS & Fedora venv deps construction to setup_venv.py.
They will be reused in production setup.
2019-01-01 19:34:46 -08:00
rht 17ffbdabe2 provision: Use virtualenv instead of python-virtualenv on Fedora.
The package name changed at some point.
2019-01-01 19:34:24 -08:00
rht 75134ef614 provision: Use generic python3 on Fedora dependencies.
It might be cleaner to add new variables in setup_venv.py, but we can
do that later.
2019-01-01 19:34:11 -08:00
rht acbb174100 provision: Add RHEL 7 support. 2018-12-18 17:13:56 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
rht 295cbe7b9c provision: Add build-pgroonga and use it for Fedora. 2018-12-17 16:23:57 -08:00
rht b732fe819e provision: Add Fedora support. 2018-12-17 16:23:44 -08:00
rht 888388cf89 setup-yum-repo: Add -y flag to `yum update`. 2018-12-17 11:49:43 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
Tim Abbott 47c4a2d21a upgrade-zulip-from-git: Remove buggy typing import.
This broke running this on Python 3.4, where typing isn't in stdlib.
2018-12-17 10:33:52 -08:00
rht 30834b7dd8 build-tsearch-extras: Abstract out postgres version. 2018-12-16 13:26:29 -08:00
rht 3afb299aec setup-yum-repo: Abstract out postgres version. 2018-12-16 13:21:55 -08:00
rht 903bce9bdd provision: Install pycurl manually on CentOS.
This works around a pycurl bug on EL7 where the fix hasn't been
released yet.
2018-12-16 10:09:07 -08:00
rht 25a8dbef11 provision: Build tsearch_extras from source for CentOS support. 2018-12-16 10:04:27 -08:00
rht 448303b3f0 provision: Initialize setup-yum-repo. 2018-12-16 10:04:27 -08:00
shubham-padia 3231306736 upgrade-zulip: Use deploy_options specified in zulip.conf.
Fixes #10534.
upgrade-zulip can now also accept the same deploy options as
upgrade-zulip-from-git and pass it as arguments to
upgrade-zulip-stage-2.
2018-12-16 07:52:47 -08:00
shubham-padia 29dce7c9b9 upgrade-zulip-from-git: Refactor deploy_options logic to zulip_tools.py.
This a preparatory commit moving the deploy_options logic to
zulip_tools.py so it can be imported and used in upgrade-zulip.
2018-12-16 07:52:47 -08:00