Commit Graph

439 Commits

Author SHA1 Message Date
Wyatt Hoodes 0b05d91e62 test_runner: Write database ids to file for reference in clean up.
The ids that will be used for each particular run of the test suite are
written to a unique file. Each file will then be used as a time
reference of when the suite was ran.

This change sets up the ability for a complete clean up of potentially
leaked database templates.

Tweaked by tabbott to remove these files after successful database
cleanup.
2019-06-14 15:23:20 -07:00
Vishnu Ks 6e52779ed5 install: Include no-dist-upgrade in args list.
This was missed out in 2e51ac8c49
2019-06-14 14:24:08 -07:00
Tim Abbott 8a53686f41 install: Add --no-overwrite-settings option.
This commit needs more work to make this option reasonable.
2019-06-13 14:39:25 -07:00
ppreethi 86840adda5 version: Show number of commits and commit sha in ZULIP_VERSION.
We use `git describe --tags` to get information about the number of commit since
the last major version, and the sha of the current HEAD. This is added to the
ZULIP_VERSION when a deploy is done from `git`.

Modified heavily by punchagan to:
* to use git describe instead of `git log` and `wc`
* use a separate script to run the git describe command
* write the file with version info to var/ and remove it from the repo

Fixes #4685.
2019-06-07 13:39:12 -07:00
Thomas Ip c6cdcd082e dependencies: Upgrade node to 12.3.1.
API changes:
* The behaviour of Date.toLocaleTimeString() reverts to pre 8.0.0,
  this only affects automated tests. Lots of other API changes but
  we didn't use any of those.
* The internal sorting algorithm changed which causes one of our own
  compare function to miss coverage.
2019-06-05 17:15:52 -07:00
Vishnu Ks 14ed0e283d install: Add option to skip dist-upgrade. 2019-06-05 15:50:02 -07:00
Thomas Ip 9a84a60029 dependencies: Upgrade yarn to 1.16.0. 2019-06-03 12:24:00 -07:00
Anders Kaseorg 90a9dfa7f5 setup-apt-repo: Try to download pre-setup deps before skipping update.
Simulate isn’t enough in some cases.  The error message when this
fails looks sufficiently non-alarming.

LXC:

    default: + apt-get -dy install lsb-release apt-transport-https gnupg
    default: Reading package lists...
    default: Building dependency tree...
    default:
    default: Reading state information...
    default: lsb-release is already the newest version.
    default: gnupg is already the newest version.
    default: The following NEW packages will be installed:
    default:   apt-transport-https
    default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    default: Need to get 25.1 kB of archives.
    default: After this operation, 238 kB of additional disk space will be used.
    default: Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main apt-transport-https amd64 1.0.1ubuntu2.3
    default:   404  Not Found [IP: 91.189.88.161 80]
    default: Err http://security.ubuntu.com/ubuntu/ trusty-security/main apt-transport-https amd64 1.0.1ubuntu2.3
    default:   404  Not Found [IP: 91.189.88.161 80]
    default: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/apt/apt-transport-https_1.0.1ubuntu2.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
    default:
    default: E: Some files failed to download
    default: + apt-get update
    […]
    default: Fetched 4,504 kB in 7s (611 kB/s)
    default: Reading package lists...
    default: + apt-get -y install lsb-release apt-transport-https gnupg
    default: Reading package lists...

Docker:

    default: + apt-get -dy install lsb-release apt-transport-https gnupg
    default: Reading package lists...
    default: Building dependency tree...
    default:
    default: Reading state information...
    default: Package gnupg is not available, but is referred to by another package.
    default: This may mean that the package is missing, has been obsoleted, or
    default: is only available from another source
    default: E: Package 'gnupg' has no installation candidate
    default: + apt-get update
    […]
    default: Fetched 16.2 MB in 5s (3,326 kB/s)
    default: Reading package lists...
    default: + apt-get -y install lsb-release apt-transport-https gnupg
    default: Reading package lists...

(All in green.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 15:25:00 -07:00
Anders Kaseorg 407564086d provision: Move apt-get update from retry handler to setup-apt-repo.
This avoids unnecessarily alarming error messages if the apt cache is
missing.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:38 -07:00
Anders Kaseorg ae524b677d ci: Move backend and production tests to Ubuntu 16.04 (xenial).
This is preparation for dropping support for Trusty in CI.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:15 -07:00
Anders Kaseorg d267884cc1 node_cache: Run yarn install with --frozen-lockfile.
This way a yarn.lock inconsistent with package.json can’t slip through
unnoticed.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-20 19:23:37 -07:00
Puneeth Chaganti 0884764421 url preview: Add python-magic dependency for content-type sniffing. 2019-05-06 12:37:32 -07:00
Mayank Madan 99414e2d96 provision: Add support for Debian 10 Buster. 2019-05-05 17:57:19 -07:00
Mayank Madan a86eccb2b6 pgroonga: Upgrade from 2.1.6 to 2.1.8.
This is necessary to support postgres 11 in Debian Buster.
2019-05-05 17:52:32 -07:00
Tim Abbott 18146ed947 script: Add static asset pipeline option to support OS upgrades.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
2019-04-17 12:58:42 -07:00
Anders Kaseorg 9fc0aa7ffa scripts: Fix exec invocation for in-process virtualenv activation.
activate_this.py has always documented that it should be exec()ed with
locals = globals, and in virtualenv 16.0.0 it raises a NameError
otherwise.

As a simplified demonstration of the weird things that can go wrong
when locals ≠ globals:

    >>> exec('a = 1; print([a])', {}, {})
    [1]
    >>> exec('a = 1; print([a for b in [1]])', {}, {})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
      File "<string>", line 1, in <listcomp>
    NameError: name 'a' is not defined
    >>> exec('a = 1; print([a for b in [1]])', {})
    [1]

Top-level assignments go into locals, but from inside a new scope like
a list comprehension, they’re read out of globals, which doesn’t work.

Fixes #12030.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-16 17:42:44 -07:00
Wyatt Hoodes d67dc2eb94 install-node: Upgrade yarn to 1.15.2.
In addition to upgrading dependencies being generally useful, this may
fix situations where yarn fails but returns a success status code in the 
presence of an HTTP proxy.
2019-04-14 22:36:14 -07:00
Rohitt Vashishtha 22a3b4230c Revert "install: Ensure that sudo is installed."
Now that we have the run_as_root helper function, we don't need to
install sudo to run Zulip in production

This reverts commit a7d7d181ea.

Fixes #10036.
2019-04-04 16:52:20 -07:00
Mayank Madan 264c2e9803 provision: Add provision support for Ubuntu 18.10.
Note that this is only support for Ubuntu 18.10 in the development
environment.
2019-03-09 16:01:46 -08:00
Tim Abbott 0ab8cd9ca3 provision: Refactor handling of dependencies for building pgroonga.
This lets us leave build-pgroonga platform-agnostic.
2019-03-08 15:12:56 -08:00
Tim Abbott 5c20ee998c build-tsearch-extras: Add support for Debian/Ubuntu. 2019-03-08 15:12:56 -08:00
Tim Abbott 8469c7774f scripts: Remove create-zulip-admin.
This was used only by ancient versions of the Docker project.
2019-03-04 15:33:51 -08:00
Tim Abbott 0e7a5d82b4 setup_venv: Remove virtualenv-clone legacy upgrade code.
Few folks will be upgrading from versions of Zulip old enough to not
have virtualenv-clone, and those who are won't be able to use it due
to older dependencies having been removed.
2019-03-04 15:32:15 -08:00
Tim Abbott 4157d15ff1 setup_venv: Fix check for whether virtualenv-clone is unavailable.
We don't need a bare except here.
2019-03-04 15:32:15 -08:00
Tim Abbott 478e215acd setup_venv: Use run_as_root helper.
Also, don't use split() to break up strings to pass into run().
2019-03-04 15:32:15 -08:00
Anders Kaseorg d9adc9d7bc get_dev_uuid_var_path: Fix theoretical shell quoting problem.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-03-04 15:20:07 -08:00
Tim Abbott b3444354aa su_to_zulip: Fix detection of zulip user ID.
Apparently, while upgrade-zulip-from-git always ensures that zulip
deployment directories are owned by the Zulip user, unpack-zulip (aka
the tarball code path) has them owned by root.

The user ID detection logic in su_to_zulip's helper get_zulip_uid was
intended to support both development environments (where the user ID
might vary) and production environments.  For development
environments, the existing code is fine, but given this unpack-zulip
permissions issue, we need to have code to fallback to 'zulip' if the
detection logic detects the "zulip" user has having UID 0.
2019-03-04 14:27:39 -08:00
Rohitt Vashishtha 3d427d02cc scripts/zulip_tools: Use run_as_root instead of subprocess.check_call. 2019-03-01 11:21:16 -08:00
Rohitt Vashishtha ac48925977 scripts: Use run_as_root instead of run([sudo, ...]). 2019-03-01 11:21:16 -08:00
Tim Abbott 57e1307a3a provision: Fix virtualenv-clone handling of success-stamp.
Apparently, virtualenv-clone ends up copying the success-stamp file
that we use to track whether a virtualenv was successfully
provisioned, which results in problems if we get a network error in
the pip install stage afterwards.

The comment explains our fix, but basically we just delete
success-stamp after the clone.

Fixes #11301.
2019-02-16 11:24:10 -08:00
Anders Kaseorg e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00
Anders Kaseorg ebad0b7cbf zulip_tools: Get the zulip uid from the owner of DEPLOY_ROOT.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-11 17:00:37 -08:00
Anders Kaseorg e4832cc9da setup_path_on_import: Don’t re-add BASE_DIR to sys.path.
BASE_DIR must already be in sys.path if we got as far as importing
this.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:17:31 -08:00
Anders Kaseorg 70bfcd3402 zulip_tools: Extract get_deploy_root function.
Modified by tabbott from the original to preserve the implementation;
see https://github.com/zulip/zulip/pull/11295#discussion_r254925032
for why this is correct.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:09:29 -08:00
Anders Kaseorg e984107966 scripts: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:58 -08:00
Anders Kaseorg 4fef10f4e6 install: Improve error handling.
On usage errors (except --help), write usage message to stderr and
exit with nonzero status.

Forbid setting the hostname and email to the example values.  Those
are specifically checked for and would fail later.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 11:18:42 -08:00
Anders Kaseorg 893e1475bb setup_virtualenv: Do not activate the virtualenv on creation.
Instead, manually activate it in the one place where this
functionality was used (tools/lib/provision.py).  This way we avoid
trying to activate the Python 2 thumbor virtualenv from Python 3.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-31 17:55:04 -08:00
Tim Abbott df436a55b4 scripts: Recommend apt update after enabling universe.
One needs to manually do an apt update after add-apt-repository, or it
won't actually work.
2019-01-26 12:29:05 -08:00
Anders Kaseorg 5ba84d75ce create-zulip-admin: Fix shell quoting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:46:50 -08:00
Harshit Bansal 50ef91bb08 scripts: Add argparse option to `restart-zerver` for `--fill-cache`.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.

Fixes: #10900.
2019-01-14 15:20:01 -08:00
Anders Kaseorg b9f7b08af2 setup_path_on_import: Check that the virtualenv is compatible.
The site_packages variable points to (e.g.)
zulip-py3-venv/lib/python3.4/site-packages.  If that doesn’t exist,
we’re probably running the wrong Python version.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:09 -08:00
Anders Kaseorg 7a58f574bf setup_path_on_import: Skip activate_this if already activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:07 -08:00
Anders Kaseorg 2ad1a27b28 create-production-venv: Remove Python 2 support.
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:05 -08:00
rht 91348a6aee create-thumbor-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht 68c20c610e create-production-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht 15763f8545 provision: Include DISTRIB_FAMILY in parse_lsb_release output. 2019-01-07 18:52:09 -08:00
rht d8a081b94c setup-yum-repo: Add prod flag for doing IUS install on CentOS. 2019-01-05 15:49:02 -08:00
rht 105732ab1f parse_lsb_release: Fix vendor name matching for CentOS. 2019-01-04 14:09:48 -08:00
rht d3139266c8 setup_venv: Uninstall typing on Python >3.4.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
2019-01-01 19:34:48 -08:00
rht c190959b58 provision: Move CentOS & Fedora venv deps construction to setup_venv.py.
They will be reused in production setup.
2019-01-01 19:34:46 -08:00