Commit Graph

551 Commits

Author SHA1 Message Date
Anders Kaseorg 5ba84d75ce create-zulip-admin: Fix shell quoting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:46:50 -08:00
Harshit Bansal 50ef91bb08 scripts: Add argparse option to `restart-zerver` for `--fill-cache`.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.

Fixes: #10900.
2019-01-14 15:20:01 -08:00
Anders Kaseorg b9f7b08af2 setup_path_on_import: Check that the virtualenv is compatible.
The site_packages variable points to (e.g.)
zulip-py3-venv/lib/python3.4/site-packages.  If that doesn’t exist,
we’re probably running the wrong Python version.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:09 -08:00
Anders Kaseorg 7a58f574bf setup_path_on_import: Skip activate_this if already activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:07 -08:00
Anders Kaseorg 2ad1a27b28 create-production-venv: Remove Python 2 support.
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:05 -08:00
rht 91348a6aee create-thumbor-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht 68c20c610e create-production-venv: Generalize to CentOS, Fedora, RHEL. 2019-01-07 18:52:27 -08:00
rht 15763f8545 provision: Include DISTRIB_FAMILY in parse_lsb_release output. 2019-01-07 18:52:09 -08:00
rht d8a081b94c setup-yum-repo: Add prod flag for doing IUS install on CentOS. 2019-01-05 15:49:02 -08:00
rht 105732ab1f parse_lsb_release: Fix vendor name matching for CentOS. 2019-01-04 14:09:48 -08:00
rht d3139266c8 setup_venv: Uninstall typing on Python >3.4.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
2019-01-01 19:34:48 -08:00
rht c190959b58 provision: Move CentOS & Fedora venv deps construction to setup_venv.py.
They will be reused in production setup.
2019-01-01 19:34:46 -08:00
rht 17ffbdabe2 provision: Use virtualenv instead of python-virtualenv on Fedora.
The package name changed at some point.
2019-01-01 19:34:24 -08:00
rht 75134ef614 provision: Use generic python3 on Fedora dependencies.
It might be cleaner to add new variables in setup_venv.py, but we can
do that later.
2019-01-01 19:34:11 -08:00
rht acbb174100 provision: Add RHEL 7 support. 2018-12-18 17:13:56 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
rht 295cbe7b9c provision: Add build-pgroonga and use it for Fedora. 2018-12-17 16:23:57 -08:00
rht b732fe819e provision: Add Fedora support. 2018-12-17 16:23:44 -08:00
rht 888388cf89 setup-yum-repo: Add -y flag to `yum update`. 2018-12-17 11:49:43 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
Tim Abbott 47c4a2d21a upgrade-zulip-from-git: Remove buggy typing import.
This broke running this on Python 3.4, where typing isn't in stdlib.
2018-12-17 10:33:52 -08:00
rht 30834b7dd8 build-tsearch-extras: Abstract out postgres version. 2018-12-16 13:26:29 -08:00
rht 3afb299aec setup-yum-repo: Abstract out postgres version. 2018-12-16 13:21:55 -08:00
rht 903bce9bdd provision: Install pycurl manually on CentOS.
This works around a pycurl bug on EL7 where the fix hasn't been
released yet.
2018-12-16 10:09:07 -08:00
rht 25a8dbef11 provision: Build tsearch_extras from source for CentOS support. 2018-12-16 10:04:27 -08:00
rht 448303b3f0 provision: Initialize setup-yum-repo. 2018-12-16 10:04:27 -08:00
shubham-padia 3231306736 upgrade-zulip: Use deploy_options specified in zulip.conf.
Fixes #10534.
upgrade-zulip can now also accept the same deploy options as
upgrade-zulip-from-git and pass it as arguments to
upgrade-zulip-stage-2.
2018-12-16 07:52:47 -08:00
shubham-padia 29dce7c9b9 upgrade-zulip-from-git: Refactor deploy_options logic to zulip_tools.py.
This a preparatory commit moving the deploy_options logic to
zulip_tools.py so it can be imported and used in upgrade-zulip.
2018-12-16 07:52:47 -08:00
rht d54fb5f40d
provision: Add venv dependencies for Centos 7. 2018-12-14 04:59:46 +00:00
Sumanth V Rao 76c6cf8c3a upgrade-zulip-stage-2: Added argument to skip purging old deployments.
This makes it possible to add --skip-purge-old-deployments in the
deploy_options section of /etc/zulip/zulip.conf, and control whether
old deployments are purged automatically on a system.

We still need to do https://github.com/zulip/zulip/issues/10534 and
probably also to add these arguments to be directly passed into
upgrade-zulip, but that can wait for future work.

Fixes #10946.
2018-12-13 10:10:43 -08:00
Anders Kaseorg 149132348f lint: Update ShellCheck to 0.6.0.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-12 08:24:01 -08:00
rht e0ec288928 parse_lsb_release: Add CentOS support. 2018-12-11 13:08:26 -08:00
rht 711b0fda8b scripts: Initialize setup-apt-repo-debathena. 2018-12-11 13:01:26 -08:00
Tim Abbott 0d48f47eba scripts: Make setup-apt-repo a no-op if nothing has changed.
This performance optimization is important for being able to run this
from puppet in our production environment.
2018-12-10 17:36:28 -08:00
Tim Abbott 9423f8e31c node cache: Don't shell out to yarn to get the version.
This can easily be done by just reading the package.json file and
extracting the version value, just like we now do in install-node.
2018-12-07 14:44:52 -08:00
Tim Abbott 630968b632 provision: Use a more efficient approach for getting yarn version.
Since yarn has a package.json conveniently available, we can parse
that with jq, saving the expensive operation of starting up yarn.

This saves ~300ms in a no-op provision.
2018-12-07 14:36:54 -08:00
rht a1ca8c262a setup-apt-repo: Add `set -e` and ensure the sources file exists. 2018-12-07 14:00:59 -08:00
Tim Abbott e13de3e629 scripts: Fix incorrect garbage-collection of emoji/node caches.
Apparently, we were incorrectly expressing the paths in the
caches_in_use data structures for these two cache-cleaning algorithms,
resulting in the default threshhold_days algorithm controlling which
caches could be garbage-collected.  While the emoji one was just a
performance optimization for upgrade-zulip-from-git, it was possible
for the main `node_modules` cache in use in production to be GCed,
resulting in LaTeX rendering being broken.
2018-12-03 11:59:08 -08:00
Tim Abbott 1303f8df20 docs: Fix missing quotes in `su zulip -c` documentation.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
2018-11-30 12:12:27 -08:00
Tim Abbott 31556e45d8 setup-apt-repo: Install gnupg as part of installation.
Apparently, on Debian stretch, the gnupg package isn't installed by
default, which means that our `apt-key add` commands were failing with
these errors on an ultra-minimal Debian installation:

+ apt-key add ./scripts/setup/packagecloud.asc
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
+ apt-key add ./scripts/setup/pgroonga-debian.asc
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

Fixes #10480.
2018-11-30 10:45:04 -08:00
Anders Kaseorg ed0292629b zulip_tools.run: Remove shell=True support.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg d581ad82ca scripts/upgrade-zulip-from-git: Avoid shelling out for mkdir, ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg a61eebd9e3 scripts/lib/unpack-zulip: Avoid shelling out for rm, mv, mkdir, ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg 9bad4003c6 scripts/lib/upgrade-zulip-stage-2: Avoid shelling out for ln.
The original code was actually broken, in that it checked the wrong
path, but it didn't matter because it used `ln -nsf`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:48:23 -08:00
Anders Kaseorg d60b697cc5 scripts/lib/create-production-venv: Avoid shelling out for ln.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:25:01 -08:00
Anders Kaseorg 33a4d12101 scripts: Add zulip_tools.overwrite_symlink function to replace ln -nsf.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:24:59 -08:00
Anders Kaseorg 1597511de0 install: Check whether universe repository is enabled on Ubuntu.
Fixes #10417.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:18:43 -08:00
Tim Abbott 5a56925495 restart-server: Fix restarting server with multiple tornado processes.
Previously, we unconditionally tried to restart the Tornado process
name corresponding to the historically always-true case of a single
Tornado process.  This resulted in Tornado not being automatically
restarted on a production deployment on servers with more than one
Tornado process configured.
2018-11-27 17:20:05 -08:00
Tim Abbott 3e3eb2aa7f scripts: Clarify names of running-as-root assertions.
This should make it more obvious that these functions will exit the
script if the check fails.
2018-11-19 10:58:34 -08:00
Rohitt Vashishtha eae9251cb7 scripts: Make upgrade-zulip-* use root checking from zulip_tools.
This is mostly just a nice code deduplication/cleanup.
2018-11-19 10:58:34 -08:00
Rohitt Vashishtha 767acfa2ac scripts: Add util functions for checking root to zulip_tools. 2018-11-19 10:58:16 -08:00
Rohitt Vashishtha 95ba947f13 setup-cerbot: Allow issuing certificates for multiple domains.
This commit allows specifying Subject Alternative Names to issue certs
for multiple domains using certbot. The first name passed to certbot-auto
becomes the common name for the certificate; common name and the other
names are then added to the SAN field. All of these arguments are now
positional. Also read the following for the certbot syntax reference:

https://community.letsencrypt.org/t/how-to-specify-subject-name-on-san/

Fixes #10674.
2018-11-13 12:47:31 -08:00
Tim Abbott 453c35d0f7 install: Provide a suggestive error message when missing Universe.
By far the dominant cause of errors when installing apt packages is
not having the Universe repository enabled in Ubuntu bionic (this
seems to have started happening a lot recently; I wonder if Ubuntu
changed the defaults for new server installs or something?).

In any case, providing that suggestion in the error output should help
reduce these a lot.
2018-11-12 10:56:39 -08:00
Tim Abbott e7bb833a37 install: Improve some error output for common errors.
This uses `set +x` to hide the `echo` output, and then sets the font
color to red.
2018-11-12 10:55:06 -08:00
Tim Abbott 60ed41081b scripts: Write to upgrade.log in upgrade-zulip-from-git.
This should simplify debugging when doing Git-based upgrades; this log
has long been useful for the main upgrade-zulip tool.
2018-11-06 17:56:01 -08:00
Anders Kaseorg c3bd293eaf Add shellcheck as a linter.
Fixes #10162.

Tweaked by tabbott to cover documentation as well.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-18 15:32:26 -07:00
Anders Kaseorg d0fb34e8af install: Fix shellcheck warnings.
In scripts/lib/install line 71:
ZULIP_PATH="$(readlink -f $(dirname $0)/../..)"
                          ^-- SC2046: Quote this to prevent word splitting.
                                    ^-- SC2086: Double quote to prevent globbing and word splitting.

In scripts/lib/install line 105:
mem_kb=$(cat /proc/meminfo | head -n1 | awk '{print $2}')
             ^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.

In scripts/lib/install line 141:
apt-get -y dist-upgrade $APT_OPTIONS
                        ^-- SC2086: Double quote to prevent globbing and word splitting.

In scripts/lib/install line 145:
    $ADDITIONAL_PACKAGES
    ^-- SC2086: Double quote to prevent globbing and word splitting.

In scripts/lib/install line 254:
    if [ -n "ZULIP_ADMINISTRATOR" ]; then
             ^-- SC2157: Argument to -n is always true due to literal strings.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-18 15:26:21 -07:00
Tim Abbott a7d7d181ea install: Ensure that sudo is installed.
We use it to drop privileges from root to other users in the installer
process (which ideally, we would remove, but it will take some
annoying refactoring).

This should generally be safe to do, since the default sudo
permissions only allow root to use it anyway.

See https://github.com/zulip/zulip/issues/10036 for the follow-up
issue of removing the need to do this.
2018-10-16 15:34:04 -07:00
Tim Abbott 608173657d upgrade-zulip: Clean up legacy google emoji icons.
Because we renamed the "google" iconset to be the modern Google set,
not what is now called the "googleblob" icon set, we need to make sure
that our usually correct policy of not overwriting image files under
`prod-static/` doesn't apply to files potentially being copied in for
the emoji images.

We fix this by just deleting the `images-google-64` directory on
upgrade if it contains the googleblob version of the "hotdog" emoji.

Fixes #10038.
2018-10-12 16:38:49 -07:00
Tim Abbott 9054ce278f locale: Fix incorrect use of locale-gen to generate locales.
Previously, we were having issues installing on Debian Stretch with
non-English locales, because `locale-gen` actually doesn't take a
locale as an argument (and thus `locale-gen en_US.UTF-8` did nothing).
We should instead be calling localedef directly.

Thanks to Tom Daff for debugging this.

Fixes #10629.
2018-10-11 14:42:24 -07:00
Abhilash Verma 0e2322a322 logging: Show timestamp in UTC in non-django production scripts.
Done in pair programming with @aero31aero.

Fixes #9678.
2018-08-20 12:52:40 -07:00
xificurC 9e053c74cf scripts: Allow configuring a custom CA bundle for build process.
For building Zulip in an environment where a custom CA certificate is
required to access the public Internet, one needs to be able to
specify that CA certificate for all network access done by the Zulip
installer/build process.  This change allows configuring that via the
environment.
2018-08-13 15:45:28 -07:00
Tim Abbott b8b97c98fb upgrade-zulip-stage-2: Remove explicit process_fts_updates start.
Thanks to changes in restart-server, this is now already happening there.

(The restart-server changes were required to ensure that if the
upgrade failes and one just does
/home/zulip/deployments/next/restart-server to recover, the right
thing happens; so this is the correct resolution to the conflict).
2018-08-03 15:21:01 -07:00
xificurC 76d6e27d04 install-node: Fail (and show error messages) if wget fails to run. 2018-08-03 10:25:59 -07:00
Tim Abbott ede4f52c20 scripts: Remove docker-functions.sh.
We've determined that it makes sense to keep this code separate in the
docker-zulip project.
2018-08-03 09:17:10 -07:00
Anders Kaseorg 21c60bf6d4 setup-apt-repo: Fix shellcheck warnings.
In scripts/lib/setup-apt-repo line 6:
zulip_source_hash=`sha1sum $SOURCES_FILE`
                  ^-- SC2006: Use $(..) instead of legacy `..`.

In scripts/lib/setup-apt-repo line 10:
SCRIPTS_PATH="$(dirname $(dirname $0))"
                        ^-- SC2046: Quote this to prevent word splitting.
                                  ^-- SC2086: Double quote to prevent globbing and word splitting.

In scripts/lib/setup-apt-repo line 36:
if [ "$zulip_source_hash" = "`sha1sum $SOURCES_FILE`" ] && ! [ -e "$STAMP_FILE" ]; then
                             ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 942bb49c29 install-node: Fix shellcheck warnings.
In scripts/lib/install-node line 34:
    source "$NVM_DIR/nvm.sh"
    ^-- SC1090: Can't follow non-constant source. Use a directive to specify location.

In scripts/lib/install-node line 36:
    export NODE_BIN="$(nvm which default)"
           ^-- SC2155: Declare and assign separately to avoid masking return values.

In scripts/lib/install-node line 39:
    n=$(which node)
        ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 7162ac43a6 create-zulip-admin: Fix shellcheck warnings.
In scripts/lib/create-zulip-admin line 3:
if ([ "$ZULIP_USER_CREATION_ENABLED" == "True" ] || [ "$ZULIP_USER_CREATION_ENABLED" == "true" ]) && \
   ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

In scripts/lib/create-zulip-admin line 4:
   ([ -z "$ZULIP_USER_DOMAIN" ]   || \
   ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 5b4d30abab certbot-maybe-renew: Fix shellcheck warnings.
In scripts/lib/certbot-maybe-renew line 8:
    case "$(echo "$value" | tr A-Z a-z)" in
                               ^-- SC2019: Use '[:upper:]' to support accents and foreign alphabets.
                                   ^-- SC2018: Use '[:lower:]' to support accents and foreign alphabets.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Tim Abbott 35cb7528f9 models: Add new UserMessage flag active_mobile_push_notification.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.

This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.

We include a partial index on UserMessage for this flag.
2018-08-01 16:51:56 -07:00
Tim Abbott b564976ce4 upgrade-zulip: Fix unicode issues with detecting early migrations.
Apparently, our Python 3 conversion for the early-migrations logic
here was incorrect, and as a result we never set
need_create_large_indexes to True (because we were checking whether a
`bytes` was inside a list of `str`s).

The simplest fix would be to just add a `.decode()` in one place, but
this refactor to just decode at the beginning is a lot more readable.
2018-07-31 13:46:06 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Shubham Padia bf6dc4472b models: Add is_private flag to UserMessage and add index for it.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.

This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.

For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.

History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org.  A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
2018-07-30 15:43:55 -07:00
Tim Abbott 30a3c48ff3 thumbor: Fix missing virtualenv-clone package in dependencies.
This package is important in order to avoid scary-looking errors
whenever we upgrade the dependencies in thumbor.txt (where
virtualenv-clone isn't installed in the venv, and then gets installed
by the code we just added a TODO comment to.
2018-07-30 11:55:16 -07:00
Anders Kaseorg a5407e1c7d scripts: Replace node-wrapper with a symlink.
Commit 00e057bf44 (#4727) simplified
node-wrapper to a one-line wrapper script for performance.  Copying
the binary was proposed and rejected because node finds some of its
modules relative to its own path.  But a symlink doesn’t have that
issue, as you can verify with

    node -e 'console.log(require.resolve.paths("foo"))'

(To find its own path, node uses `process.execPath`, which resolves
symlinks, and there’s no plausible reason for that behavior to change.
https://github.com/nodejs/node/blob/v8.11.1/lib/module.js#L708-L717
https://github.com/nodejs/node/blob/v10.7.0/lib/internal/modules/cjs/loader.js#L761-L770)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 11:48:10 -07:00
Anders Kaseorg 09c64f260b scripts/lib/zulip_tools.py: Avoid shelling out for touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg fe76b97e28 scripts/lib/setup_venv.py: Avoid shelling out for cp, touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Tim Abbott ee0f4ca330 locale: Set LANG/LANGUAGE to match LC_ALL.
Apparently, perl at least expects LANG, LANGUAGE, and LC_ALL to be
consistent, and thus apt spits out a bunch of warnings if these are
different.  So if we're forcing LC_ALL in these installer/upgrade
script blocks, we should force the rest too.

I believe this fixes the remaining locale part of #9946.
2018-07-23 23:01:10 -07:00
jeaye 4999474cce install: Add a couple Docker-specific options to the certbot scripts.
--agree-tos is useful for the Docker environment, where we won't have
an interactive shell present for agreeing to the ToS.

--deploy-hook is also useful for the Docker environment; it makes it
possible to customize what deploy hook (if any) we pass into the
underlying cerbot command.
2018-07-23 17:45:59 -07:00
Tim Abbott f228eabd90 install: Make sure the en_US.UTF-8 locale is available.
We need to make sure the en_US.UTF-8 locale has been generated before
setting the locale via the environment.

Fixes part of #9946.
2018-07-18 15:05:27 -07:00
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Tim Abbott 3cc93fd721 setup_venv: Fix missing libssl-dev dependency.
We were already correctly including libssl-dev in Zulip's dependencies
in development environment provisioning, but (at least now) it's
needed to build certain Python packages like pycurl when building a
Zulip virtualenv in production.  I haven't investigated why we didn't
need this on Ubuntu, but one possible reason would be that some other
library in our dependencies list happens to depend on it on Ubuntu.

We fix this by moving the dependency over to the shared
VENV_DEPENDENCIES list.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Tim Abbott 8f9b5633b8 zulip_tools: Fix accessing LSB data on Debian stretch.
Apparently, at least some Debian stretch systems don't have an
/etc/lsb-release, so the optimization that we did in
5d39a0f0fc broke our installer on
Debian.

We fix this, by falling back to calling the lsb_release command on
systems that don't have a faster way to do it.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Joshua Schmidlkofer b1a57d144f thumbor: Add production installer/puppet support.
This commits adds the necessary puppet configuration and
installer/upgrade code for installing and managing the thumbor service
in production.  This configuration is gated by the 'thumbor.pp'
manifest being enabled (which is not yet the default), and so this
commit should have no effect in a default Zulip production environment
(or in the long term, in any Zulip production server that isn't using
thumbor).

Credit for this effort is shared by @TigorC (who initiated the work on
this project), @joshland (who did a great deal of work on this and got
it working during PyCon 2017) and @adnrs96, who completed the work.
2018-07-12 20:37:34 +05:30
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Vishnu Ks 109fa85614 provision: Rename file_hash_updated to file_or_package_hash_updated.
Check for changes in package version as well along
with the files.
2018-06-22 23:40:31 +05:30
Tim Abbott 9d9d59d4b2 clean-unused-caches: Optimize performance.
This saves about 400ms when running clean-unused-caches, basically by
calling its sub-rountines by import (rather than
`subprocess.check_call()`).  The performance optimization seems well worth it.

Fixes #9766.
2018-06-18 07:31:33 -07:00
Vishnu Ks 7b8e79ae48 provision: Refactor hashing of compilemessages into a function.
This allows it to be reused for other tools.

Edited by tabbott to remove the use of "compilemessages" in variable
names.
2018-06-18 06:55:36 -07:00
Greg Price 25c46e3abb tx: Clarify role of the script `process-mobile-i18n`.
This file looks like it's producing some kind of compilation of the
mobile strings, that the mobile app will somehow end up using --
especially as it refers to its output as a "resource file".  In
reality, it compiles statistics to be included in the language-picker
UI in the web app.  Give appropriate names to the identifiers so it's
less confusing.
2018-06-17 17:37:29 -07:00
Raymond Akornor 8f7af5fde5 install-node: Upgrade to yarn 1.7.0. 2018-06-05 08:52:55 -07:00
Tim Abbott 0fabff6dda setup_venv: Clean up VENV_DEPENDENCIES using parsed lsb_release.
This is mostly a cleanup, but it should also save 50ms in the runtime
of create-production-venv.
2018-05-29 10:57:36 -07:00
Raymond Akornor 5d39a0f0fc scripts: Replace calls to lsb release with our own parsing.
This improves the performance of these operations, by saving a ~50ms
Python process startup.  While not a major performance improvement, it
seems worth it, given how often these commands get run.

Fixes #9571.
2018-05-29 10:57:36 -07:00
Tim Abbott 61ee01359e install: Update list of supported distros for installation.
This only changes the install script, not the docs, so it's pretty
low-profile; we'll update the docs after a bit more testing.
2018-05-24 10:44:29 -07:00
Tim Abbott 2655ece96f create-production-venv: Fix missing virtualenv dependency.
On newer distros like Xenial, Stretch, etc., we were incorrectly not
installing the Python 3 version of the virtualenv package.  This was
accidentally working because most base images with Python already have
this package too, but this was failing to install the right
dependencies in our Docker builds, requiring unnecessary manual code.

We fixed this some time ago for provision.py, but not for production.
2018-05-18 16:53:35 -07:00
Tim Abbott 8a66b0b9a9 docker: Set has_appserver=0 correctly.
The docker installer configuration incorrectly had has_appserver set
to 0; this meant that (A) the docker-zulip code needed to copy the
block of code in the installer for the `has_appserver` case into the
Dockerfile (unnecessarily), and (B) one couldn't use `install` from a
Git ref (because the static asset compiler didn't end up in the right
place).

It appears that docker-zulip tried to set this flag in their `install`
command line, but the construction inside `install` meant that didn't
work.
2018-05-15 10:13:44 -07:00
Aditya Bansal cb9d8f6d48 scripts: Change use of typing.Text to str. 2018-05-14 05:16:22 +05:30
Aditya Bansal e14974ff2c scripts: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott 41841221ee scripts: Remove obsolete zesty configuration.
Zesty already reached end-of-life, so we'll never support it.

And in one place, we add support for bionic.
2018-05-05 11:41:57 -07:00
Tim Abbott 1b3b298fa8 install: Allow installing with Debian 9.
For now we just change the script, not the documentation.
2018-05-05 10:49:09 -07:00
Tim Abbott 76fa29085a setup-apt-repo: Clean up setup code for apt repo.
This fixes adding the Ubuntu repositories for Debian, as well as makes
sure that we install the debian-archive-keyring package on Debian,
which is only priority important (and thus might be missing).
2018-05-05 10:03:39 -07:00
Tim Abbott 4ee762a52c apt: Add packagecloud repository for Debian. 2018-05-05 10:03:03 -07:00
Tim Abbott 06cfc591fe setup-apt-repo: Require apt-transport-https be installed.
Doing our apt operations over HTTPS has better security properties.
2018-05-05 10:02:50 -07:00
Greg Price e8be968250 install: Expand error message for missing SSL cert slightly.
It wasn't obvious reading this message that you can perfectly well
bring your own SSL/TLS certificate; unless you read quite a bit
between the lines where we say "could not find", or followed the link
to the detailed docs, the message sounded like you had to either use
--certbot or --self-signed-cert.

So, explicitly mention the BYO option.  Because the "complete chain"
requirement is a bit tricky, don't try to give instructions for it
in this message; just refer the reader to the docs.

Also, drop the logic to identify which of the files is missing; it
certainly makes the code more complex, and I think even the error
message is actually clearer when it just gives the complete list of
required files -- it's much more likely that the reader doesn't know
what's required than that they do and have missed one, and even then
it's easy for them to look for themselves.
2018-04-19 11:08:22 -07:00
Tim Abbott 105eed049e install-node: Fix leaking of $HOME.
This fixes a bug where provision was failing since our most recent
upgrade to yarn/nvm/node.

It turns out my original fix was the correct fix, but to the wrong
third-party tool: nvm, not yarn, was the offender.
2018-04-12 14:32:36 -07:00
Tim Abbott 041fd802b7 Revert "yarn: Revert back to v0.27.5."
This reverts commit d4b88e86cc.
2018-04-12 11:37:00 -07:00
Tim Abbott f6ae57fa70 install-node: Correctly fix yarn installation.
Apparently, new versions of yarn use the HOME environment variable to
figure out where to access their configuration, and sudo apparently
doesn't clear that variable, so install-node was being run with HOME
set to something under /home/vagrant (e.g.).

Fix this by just setting that environment variable correctly.

This replaces 250a036ff8, which
misdiagnosed the issue.
2018-04-12 11:37:00 -07:00
Tim Abbott 250a036ff8 install-node: Fix yarn installation.
It appears that some change in yarn's versioning system means that
installing yarn itself ends up chowning its config directory
incorrectly to be owned by root, preventing `yarn install` from
working later.
2018-04-12 10:42:27 -07:00
Priyank d4b88e86cc yarn: Revert back to v0.27.5.
Revert yarn version back due to some issue with new version that causes
permission issues in ~/.config/yarn directory.

Related discussion: https://chat.zulip.org/#narrow/stream/21-provision-help/topic/EACCES.3A.20permission.20denied.2C.20scandir.20'.2Fhome.2Fvagrant.2F.2Econfig.2Fya
2018-04-12 10:18:59 -07:00
Tim Abbott 9b8dd4f125 install-yarn: Fix buggy status check for the signature.
Apparently, they added a new signing key instance, and so checking
whether the old key exists doesn't work anymore.
2018-04-09 15:09:37 -07:00
Priyank ee078c372f install-node: Upgrade node, yarn, and nvm.
node -> v8.9.4
yarn -> 1.5.1
nvm -> 0.33.8

Also updates a test in timerender.js which depends on time
provided by node which is now changed in newer release.

Some changes have been made in circeci script, we just create ~/.config
directory and chown it to circleci user so installing new version of yarn
does not cause any ci failure on circleci during provision.
2018-04-09 13:56:48 -07:00
Greg Price e792fc6c07 spelling: Correctly write "cannot".
None of these errors were user-facing; mainly in comments, plus
one bit of internal docs and a developer tool.
2018-04-02 15:36:31 -07:00
Tim Abbott 0d35bbc464 install: Install the wget package.
We depend on it for installing node, and it's a standard package, not
a required one, so we do need to explicitly declare the dependency.
2018-03-29 16:03:44 -07:00
Tim Abbott ff92cdaaaf clean-npm-cache: Use islink to check node_modules.
Very old Zulip deployments might have node_modules be an actual
directory, which we should ignore for the purposes of
garbage-collection.
2018-03-29 10:57:41 -07:00
Tim Abbott 838e5880b6 clean-emoji-cache: Stop recalculating paths for other deployments.
This is the analog of 7b2c9223e7 for the
emoji cache; the only difference is that the existing code was working
correctly.  It's still worth changing for improved robustness.
2018-03-28 15:48:29 -07:00
Tim Abbott 7b2c9223e7 clean-npm-cache: Fix buggy garbage-collection logic.
We saw issues with /srv/zulip_npm_cache being cleaned incorrectly by
this tool in production (more correctly, we noticed broken symlinks to
those directories, even from the current deployment).  Print-debugging
showed that indeed older deployments were being ignored, because the
logic for `get_caches_in_use` was totally broken (this was sorta
masked because we also keep the last week's deployments).

The specific bug here turned out to be that we weren't passing the
`production` argument to generate_sha1sum_node_modules, but the
broader problem is that this logic isn't robust to changes in the
hashing algorithm.

Fix this by replacing the broken logic for trying to compute the
correct hash for that deployment with just checking the symlink inside
the deployment to let it self-report.

We can't easily do this same change for clean-venv-cache, because we
use multiple virtualenvs there.  But a similar change could be useful
for the emoji cache as well.

Fixes #8116.
2018-03-28 15:42:02 -07:00
Tim Abbott 3b29d00c69 setup-apt-repo: Fix use of shasum alias for sha1sum.
This is apparently installed by the perl package; I hadn't even known
it existed.  We of course want to use the sha1sum command from
coreutils.

Fixes #8836.
2018-03-27 09:47:14 -07:00
Harshit Bansal 40958e0824 emoji: Switch to 64px 256 color indexed sprite sheets.
This commit switches our emoji infrastructure to use 256 color indexed
64px spritesheets. Earlier we were using non-indexed 32px spritesheets
which were blurry on high dpi displays. These indexed spritesheets not
only provide a crispier display but are also smaller in size.

This commit also removes the `emoji-datasource` package as a dependency
as all the data is now sourced from individual datasource packages.

Fixes: #7862.
2018-03-14 10:28:45 -07:00
Greg Price 1215245be6 install: Check for a supported OS release. 2018-03-06 19:43:02 -08:00
Greg Price 678fcef5bd install: Fix up success message in `--no-init-db` case.
The installation isn't really complete here, and wasn't even when this
was the only success case; the instructions we're giving are for the
next step in the installation.

These instructions don't say what to do in an actual use case for this
option, but decent instructions there will require having a concrete
use case in front of us and designing the flow for it.  At this stage,
just say where we are in the normal flow, and an admin who's chosen to
go off that flow can figure out how they want to vary it from there.
2018-03-06 19:43:02 -08:00
Greg Price f22712e8e8 install: Go straight through by default.
This flips the experimental `--express` option to be the default.

We retain the old behavior, where the script exits before
`initialize-database`, as an option `--no-init-db`; it might be useful
in e.g. a migration scenario (from a Zulip install elsewhere, or
another chat system) where the admin wants to set up the database
separately.

The install instructions are adjusted to match, getting shorter by two
steps and a bunch of words.  I think this opens up opportunities to
refactor the text to simplify things further, too, but leaving that
for another commit.

Also tweak the "production" test suite to match.
2018-03-06 19:43:02 -08:00
Greg Price b1ad4e88d4 install: Switch some more uses of `sudo` to `su`.
Kind of unfortunate because the `sudo` interface for running a command
is objectively better -- a list of arguments, rather than a string to
be re-parsed by the shell.  But some bare-bones machine images lack
`sudo`, so this makes things a bit more portable.
2018-03-06 19:43:02 -08:00
Greg Price eb7eef02e8 install: Suppress initialize-database instructions when redundant.
To do it, add a blob of getopt boilerplate at the top of the script;
and also fix a couple of nits of shell style while here.
2018-03-06 19:43:02 -08:00
Tim Abbott 5ccd3aa71b i18n: Run process-mobile-i18n when syncing translations. 2018-03-02 08:59:00 -08:00
Aditya Bansal b1417860a9 create-thumbor-venv: Specify python 2.7 use for production thumbor venv. 2018-02-21 01:00:40 +05:30
Tim Abbott a1408a9d3a mypy: Remove unnecessary use of typing.Text. 2018-02-20 10:57:59 -08:00
Aditya Bansal b0c0648a4f clean-venv-cache: Handle cleaning up zulip-thumbor-venv properly. 2018-02-20 10:57:32 -08:00
Aditya Bansal 35969edd66 deps: Replace libz-dev with zlib1g-dev since the former was renamed. 2018-02-12 14:40:26 -08:00
Aditya Bansal a137ed9e27 thumbor: Fix issue with dependencies.
We do the following here:
* Remove libjasper-dev from THUMBOR_VENV_DEPENDENCIES.
Reason: This dependancy wasn't really needed by us for using
thumbor. It was a dependancy for using open-cv as Imaging Engine
in thumbor but we use PIL (Pillow now) as Imaging Engine.
* Add zlib1g-dev, libfreetype6-dev to THUMBOR_VENV_DEPENDENCIES.
Reason: These are dependancies of Pillow which are required for it
Pillow to function. Since we use Pillow in thumbor as Imaging Engine
we need these. Stuff before this didn't break because we also use
Pillow in development Environment and have these dependancies
installed from VENV_DEPENDENCIES as well.
2018-02-12 14:40:26 -08:00
Greg Price 6e633f8e2f install: Use readlink -f rather than realpath.
It does exactly the same thing, though the name is less transparent; and
it simplifies the script by avoiding an extra, early `apt-get install`.
2018-02-08 17:22:02 -08:00
Greg Price 8d86a6e331 install: Slightly clean up another couple of bits of style. 2018-02-08 17:14:28 -08:00
Greg Price 218b653930 install: Clean up organization of option parsing.
The parsing of options from environment variables belongs right next
to the normal kind; and `show_help` was unnecessary indirection.
2018-02-08 17:12:36 -08:00
Greg Price f68c485a86 install: Clean up some shell style: use here-docs, and compact getopt. 2018-02-08 17:05:13 -08:00
Greg Price 8f4b7f6e1e install: On --express, check up front for --hostname and --email. 2018-02-08 16:34:49 -08:00
Greg Price 31ec9a4627 install: Implement --self-signed-cert earlier.
Otherwise when puppet tries to install and start nginx, that can
give an error.
2018-02-08 16:34:49 -08:00
Greg Price 70d203c1c8 install: Add experimental option to go straight through more install steps.
We'll make this the normal behavior soon, once we're satisfied with
our arrangements for sending the admin straight to realm creation and
using the app without configuring email.  The instructions in the docs
will also have to change accordingly, of course.
2018-02-05 12:59:12 -08:00
Greg Price 0ba3fc3c2f install: Slightly tighten up CLI parsing.
This causes us to give an error if you pass the installer any
positional arguments, e.g. with `--`.  There's no reason you'd want
to do this, but I accidentally did it by passing an extra `--` to
the `test-install/install` wrapper and spent a few minutes on
confused debugging.
2018-01-24 14:34:30 -08:00
Greg Price f26b34405e install --self-signed-cert: Generate our own, rather than use system's.
This gives us just one way of adopting a self-signed cert, rather than
one script which would generate a new one and an option to another
which would symlink to the system's snakeoil cert.  Now those two
codepaths converge, and do the same thing.

The small advantage of generating our own over the alternative is that
it lets us set the name in the cert to EXTERNAL_HOST, rather than the
system's hostname as embedded in the system snakeoil certs.  Not a big
deal, but might make things go slightly smoother if some browsers are
lenient (in a way that they probably shouldn't be.)
2018-01-23 18:08:52 -08:00
Greg Price d76c2d77f4 install: Tighten an SSL-cert help message.
The option's name now explains for itself some of what we'd
had in prose.
2018-01-23 18:08:52 -08:00
Greg Price 841a5f3152 install: Say --self-signed-cert instead of --snakeoil-cert.
Less evocative, but requires less explanation to document because
it's a well-known term on the Internet.
2018-01-23 18:08:52 -08:00
Greg Price c27f36a82a install: Rename generate-self-signed-cert.
This script generates one certificate.
2018-01-23 18:08:52 -08:00
Greg Price 2a59b2d2ac install: Work around a bug in the (our) Debian package for camo.
Before this fix, the installer has an extremely annoying bug where
when run inside a container with `lxc-attach`, when the installer
finishes, the `lxc-attach` just hangs and doesn't respond even to
C-c or C-z.  The only way to get the terminal back is to root around
from some other terminal to find the PID and kill it; then run
something like `stty sane` to fix the messed-up terminal settings
left behind.

After bisecting pieces of the install script to locate which step
was causing the issue, it comes down to the `service camo restart`.
The comment here indicates that we knew about an annoying bug here
years ago, and just swept it under the rug by skipping this step
when in Travis. >_<

The issue can be reproduced by running simply `service camo restart`
under `lxc-attach` instead of the installer; or `service camo start`,
following a `service camo stop`.  If `lxc-attach` is used to get an
interactive shell, these commands appear to work fine; but then when
that shell exits, the same hang appears.  So, when we start camo
we're evidently leaving some kind of mess that entangles the daemon
with our shell.

Looking at the camo initscript where it starts the daemon, there's
not much code, and one flag jumps out as suspicious:

  start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
    --exec $DAEMON --no-close -c nobody --test > /dev/null 2>&1 \
    || return 1
  start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
    --no-close -c nobody --exec $DAEMON -- \
    $DAEMON_ARGS >> /var/log/camo/camo.log 2>&1 \
    || return 2

What does `--no-close` do?

 -C, --no-close
     Do not close any file descriptor when forcing the daemon
     into  the  background  (since version 1.16.5).  Used for
     debugging purposes to see  the  process  output,  or  to
     redirect  file  descriptors  to  log the process output.

And in fact, looking in /proc/PID/fd while a hang is happening finds
that fd 0 on the camo daemon process, aka stdin, is connected to our
terminal.

So, stop that by denying the initscript our stdin in the first place.
This fixes the problem.

The Debian maintainer turns out to be "Zulip Debian Packaging Team",
at debian@zulip.com; so this package and its bugs are basically ours.
2018-01-22 18:55:46 -08:00
Greg Price cef8549ec6 install: Add --snakeoil-cert option.
This provides a major simplification for non-production installs,
including our own testing (it's already in both the test-install
harness script and the "production" test suite) as well as potential
admins evaluating Zulip.

Ultimately this should probably be the default behavior, with perhaps
something shown to admins on the web as a reminder and link to help on
installing a better certificate.  For now, pending working through
that, just get the behavior in and leave it opt-in.
2018-01-22 18:55:46 -08:00
Greg Price 525b136f10 install: Install curl.
The third-party `install-yarn.sh` script uses `curl`, and we invoke it
in `install-node`.  So we need to install it as a dependency.

We've mostly gotten away with this because it's common for `curl` to
already be installed; but it isn't always.
2018-01-22 18:55:46 -08:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Tim Abbott 15cf87f424 clean-npm-cache: Fix path to yarn in Travis CI.
Apparently, this was checking the wrong path in Travis CI, and thus
never actually running (meaning we'd accumulate every `node_modules`
directory ever in the Travis caches, which in turn resulted in very
slow builds).
2017-11-23 12:01:20 -08:00
Greg Price 64c608a51a install: Clarify how we set locale during install, and why.
This updates commit 11ab545f3 "install: Set the locale ..."
to be somewhat cleaner, and to explain more in the commit message.

In some environments, either pip itself fails or some packages fail to
install, and setting the locale to en_US.UTF-8 resolves the issue.

We heard reports of this kind of behavior with at least two different
sets of symptoms, with 1.7.0 or its release candidates:
  https://chat.zulip.org/#narrow/stream/general/subject/Trusty.201.2E7.20Upgrade/near/302214
  https://chat.zulip.org/#narrow/stream/production.20help/subject/1.2E6.20to.201.2E7/near/306250

In all reported cases, commit 11ab545f3 or equivalent fixed the issue.

Setting LC_CTYPE is redundant when also setting LC_ALL, because LC_ALL
overrides all `LC_*` environment variables; so skip that.  Also move
the line in `install` to a more appropriate spot, and adjust the
comments.
2017-11-22 18:11:20 -08:00
rht 585b71bc00 email-mirror-postfix: Replace optparse with argparse. 2017-11-21 21:34:38 -08:00
Harshit Bansal e75f0c1ee4 requirements: Rename requirements files.
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
2017-11-21 02:38:26 +05:30
Aditya Bansal dd037df2c4 thumbor: Add dependencies and virtualenv setup script.
In this commit we add new dependencies needed for running thumbor.
Also we add the script for creating the virtual environment ready
for thumbor.
Note: Thumbor will use python2 and thus have different virtualenv
dedicated to it.
Credits to @TigorC and @joshland as well for there work on this.
2017-11-16 22:38:29 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
Greg Price 8f387ba4d4 setup-certbot: Add option to choose verification method.
This allows the installer to continue using this script for the
`standalone` method, while the no-argument form now uses the same
`webroot` method as the renewal cron job, suitable for running
by hand to adopt Certbot after initial install.
2017-11-15 21:50:41 -08:00
Greg Price 9adaf3417a install: Enforce a constraint on the options, and expand usage message. 2017-11-15 21:50:41 -08:00
Greg Price 84f956f5f1 certbot: Use --deploy-hook to get the cert actually served.
Certbot replaces the cert files under /etc/letsencrypt/live/,
which our nginx config refers to symlinks to; but it doesn't
tell nginx there's been an update, so nginx keeps serving the
old cert.

This is fine as long as nginx is restarted, or just told to
reload its config, at some point before the cert actually
expires about 30 days later.  Which is probably the common
case, but of course we should make it just work.  So, if we
actually renew a cert, tell nginx to reload its config now.
2017-11-15 21:50:41 -08:00
Greg Price ae901309fc certbot: Control auto-renew with a zulip.conf setting.
This causes the cron job to run only when a Zulip-managed certbot
install is actually set up.

Inside `install`, zulip.conf doesn't yet exist when we run
setup-certbot, so we write the setting later.  But we also give
setup-certbot the ability to write the setting itself, so that we
can recommend it in instructions for adopting certbot in an
existing Zulip installation.
2017-11-15 21:50:41 -08:00
rht 53e37aa511 scripts: Text-wrap long lines exceeding 110. 2017-11-10 16:22:26 -08:00
rht bb46bea44d Remove usage of six.moves.text_type. 2017-11-09 10:00:00 -08:00
derAnfaenger 19bc55aa45 Fix various typos.
The typos and their corrections were found with the
aid of https://github.com/lucasdemarchi/codespell.
2017-11-09 16:26:38 +01:00
rht 01eb22abd9 refactor: Remove all the remaining six.moves import. 2017-11-07 10:51:44 -08:00
rht 9c7d5812ce refactor: Remove six.moves.urllib.parse import. 2017-11-07 10:51:44 -08:00
rht ccf2792c1c refactor: Remove six.moves.configparser import. 2017-11-07 10:51:44 -08:00
neiljp (Neil Pilgrim) 221be326a9 mypy: Set type parameters for IO to Any in scripts/lib/node_cache.py. 2017-11-04 19:47:44 -07:00
Harshit Bansal 48662a79a5 clean-venv-cache: Fix an exception with unusual deployment dirs. 2017-10-31 00:01:09 -07:00
Harshit Bansal 8c9ea94878 scripts: Fix an issue in `purge-old-deployments` script.
We were not including the real path of the symlinks due to which we
were incorrectly deleting deployments pointed by last/current/next.
2017-10-30 23:09:51 -07:00
Harshit Bansal 1871d6fe1f minor: Remove unnecessary path juggling in `get_recent_deployments()`. 2017-10-29 14:38:20 -07:00
Tim Abbott 11ab545f3b install: Set the locale so our dependencies can install.
Many pip packages don't install properly without a US locale.
2017-10-29 11:49:08 -07:00
rht 8b6b4e043f install: Add option to get certs via certbot.
While this doesn't quite complete our plans for certbot support (it's
not documented, etc.), this is a great stride forward.
2017-10-27 17:19:34 -07:00
Shekh Ataul d239f77966 refactor: Replace mkdir_p functions with Python 3 builtin.
This didn't exist in Python 2, but it does in Python 3, so we get to
reap the rewards of dropping Python 2 support.

Fixes #7082.
2017-10-25 11:06:11 -07:00
Tim Abbott 6b1eb647e4 nginx: Fix bugs in new nginx configuration checks. 2017-10-24 14:29:36 -07:00
Tim Abbott 730c77c7df docs: Document scripts/setup/generate-self-signed-certs.
And more generally clean up our non-LetsEncrypt SSL docs.

This should make it a bit easier to setup a Zulip server.
2017-10-24 13:48:14 -07:00
Tim Abbott 1b653409f4 install: Provide a nicer error message for bad nginx configuration.
This also covers missing SSL configuration errors nicely.
2017-10-24 13:39:39 -07:00
rht c8090aafa4 lint: Remove several unused imports. 2017-10-18 10:56:11 -07:00
rht a603a4f9f5 Remove `from __future__ import absolute_import`.
Except in:
- docs/writing-bots-guide.md, because bots are supposed to be Python 2
  compatible
- puppet/zulip_ops/files/zulip-ec2-configure-interfaces, because this
  script is still on python2.7
- tools/lint
- tools/linter_lib
- tools/lister.py

For the latter two, because they might be yanked away to a separate repo
for general use with other FLOSS projects.
2017-10-17 22:59:42 -07:00
Tim Abbott 2ae2a94444 provision: Stop using shared var/ for caching apt state.
This didn't work at all when one did a `vagrant destroy` and then
`vagrant up`, because the cache state would be preserved even though
the machine is gone.

Fixes #5981.
2017-10-17 21:15:58 -07:00
rht 9ab54e5bd7 scripts/lib/install: Add flag to specify key settings.
This should make it easier to script the installation process, and
also conveniently are the options one would want for the --certbot
option.

Significantly modified by tabbott to have a sane right interface,
include --help, and avoid printing all the `set -x` garbage before the
usage notices.
2017-10-03 16:56:45 -07:00
Alexander Trost 6a7d34316c Add scripts to be used in docker-entrypoint.sh.
Based on #450, with commits
restructured by Rein Zustand.
Tweaks by Rein Zustand:
- Replace configure-cert with generate-self-signed-certs
- `mv scripts/lib/create-zulip-admin.sh scripts/lib/create-zulip-admin`
2017-09-30 09:55:48 -07:00
rht 71188d7b0a scripts: Remove import print_function. 2017-09-29 15:43:30 -07:00
Alexander Trost dea507f4b8 Add script to create admin from CLI. 2017-09-27 20:15:41 -07:00
Tim Abbott 358cb40ed1 cache: Add backwards compatibility for emoji cache.
This allows the emoji cache cleaning code to run against old emoji
caches.
2017-09-25 17:06:02 -07:00
Tim Abbott db982672b1 caches: Don't depend on yarn.lock existing.
This allows our cache cleaning code to run on servers that have copies
of Zulip 1.6 and older around.
2017-09-25 17:05:45 -07:00
Tim Abbott 0f19e501a6 caches: Suppress unnecessary output when cleaning caches.
This should make the cache cleaning process a lot less spammy.
2017-09-25 16:34:03 -07:00
Tim Abbott 703b87c116 caches: Move clean-unused-caches to scripts/lib/. 2017-09-25 16:34:03 -07:00
Tim Abbott 86a07baf40 zulip_tools: Skip the lock directory.
This is the one special directory that usually lives in deployments/
and is not a deployment.  Make sure we don't treat it as a deployment.
2017-09-25 15:15:32 -07:00
Harshit Bansal a6caf30ca7 scripts: Fix an issue in `get_recent_deployments()` due to relative paths.
We were checking for whether an item in the deployments directory
represents a directory but were using its relative path which was
causing a false value to be returned for all items irrespective of
their being a directory or not if the script was invoked from some
where other than the deployments directory.
2017-09-25 11:51:24 -07:00
Harshit Bansal 6ff7da04de emoji: Remove `NotoColorEmoji.ttf`.
We no longer use glyphs from `NotoColorEmoji.ttf` so removing this.
2017-09-24 04:51:33 -07:00
Harshit Bansal c8c1c8ef43 emoji: Remove `AndroidEmoji.ttf`. 2017-09-24 04:51:33 -07:00
Harshit Bansal 57161a92a1 scripts: Rearrange the arguments of `purge_unused_caches()`.
This commit re-arranges the arguments of `purge_unused_caches()`
function in order to remain consistent with other similar functions
in the library like `may_be_perform_caching()`.
2017-09-24 04:37:31 -07:00
Harshit Bansal df7ea375c1 scripts: Make default mode of cache-cleaning scripts much less verbose.
Print a detailed report only if `--verbose` flag is specified.

Fixes: #6632.
2017-09-24 04:37:31 -07:00
Harshit Bansal 3e8469a717 zulip_tools: Remove the now unused `GENERIC_CACHE_SCRIPT_PARSER`.
This has been replaced by `parse_cache_script_args()`.
2017-09-24 04:37:31 -07:00
Harshit Bansal 7f752f0942 scripts: Use `parse_cache_script_args()` in various cache cleaning scripts.
Instead of using `GENERIC_CACHE_SCRIPT_PARSER` and defining `parse_args()`
function in each script separately, use `parse_cache_script_args()`.
2017-09-24 04:37:31 -07:00
Harshit Bansal fe80330708 zulip_tools: Add `parse_cache_script_args()`.
This function will replace the repetitive definition of `parse_args()`
in various cache cleaning scripts. Also adds a `--verbose` argument
to the parser.
2017-09-24 04:37:31 -07:00
Harshit Bansal 4e6b68d02f zulip_tools: Change `purge_unused_caches()` API.
Instead of accepting individual arguments, accept `argparse.Namespace`
object as an argument.
2017-09-24 04:37:31 -07:00
Tim Abbott 0a91a5510c install: Fix check for whether update-prod-static is needed.
The previous version seems to be created without update-prod-static.
2017-09-22 19:52:40 -07:00
Tim Abbott 304bd86173 install: Support installing a Zulip server from a Git checkout.
Historically, one has needed to build a release tarball in order to
use/test the Zulip installer, but you could upgrade a Zulip server
from Git.  However, the only reason for that requirement was that we
didn't run `tools/update-prod-static` as part of the install script if
it's required.  A good test for that case is whether we're in a Git
repository, but a better one is to check whether the prod-static
content exists in the tarball paths.

Fixes #3704.
2017-09-22 15:47:42 -07:00
Juliana Bacelar 928dd06cc8 linter: Add lint rule banning 'import os.path' 2017-09-22 10:32:21 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Harshit Bansal 20f062f726 zulip_tools.py: Extract `may_be_perform_purging()` function.
Based on the `dry_run` flag, this function either purges the list
of directories passed to them or prints a listing of the directories
it would have purged/kept_back, had the `dry_run` flag been false.
2017-09-16 08:28:57 -07:00
Tim Abbott 600b164130 setup-apt-repo: Fix failure to rerun properly on provision failure.
Apparently, the refactoring to make this script only run when changes
are present was buggy, in that if `apt-get update` failed, running
provision against wouldn't rerun `apt-get update`, resulting in a
broken state that requires expertise to fix.  This closes that gap, by
using a stamp file to ensure we always successfully update apt before
proceeding.

It doesn't fix existing installations.
2017-09-12 07:08:54 -07:00
Tim Abbott 1a1df29053 get_recent_deployments: Skip uwsgi socket and friends.
This fixes an exception when running clean-venv-caches in production.
2017-08-27 18:18:53 -07:00
Tim Abbott e38ac00f3c caches: Move cache cleaning helper tools under scripts/lib.
This helps keep the root of scripts/ uncluttered.
2017-08-27 17:59:49 -07:00