Commit Graph

340 Commits

Author SHA1 Message Date
Abhilash Verma 0e2322a322 logging: Show timestamp in UTC in non-django production scripts.
Done in pair programming with @aero31aero.

Fixes #9678.
2018-08-20 12:52:40 -07:00
xificurC 9e053c74cf scripts: Allow configuring a custom CA bundle for build process.
For building Zulip in an environment where a custom CA certificate is
required to access the public Internet, one needs to be able to
specify that CA certificate for all network access done by the Zulip
installer/build process.  This change allows configuring that via the
environment.
2018-08-13 15:45:28 -07:00
Tim Abbott b8b97c98fb upgrade-zulip-stage-2: Remove explicit process_fts_updates start.
Thanks to changes in restart-server, this is now already happening there.

(The restart-server changes were required to ensure that if the
upgrade failes and one just does
/home/zulip/deployments/next/restart-server to recover, the right
thing happens; so this is the correct resolution to the conflict).
2018-08-03 15:21:01 -07:00
xificurC 76d6e27d04 install-node: Fail (and show error messages) if wget fails to run. 2018-08-03 10:25:59 -07:00
Tim Abbott ede4f52c20 scripts: Remove docker-functions.sh.
We've determined that it makes sense to keep this code separate in the
docker-zulip project.
2018-08-03 09:17:10 -07:00
Anders Kaseorg 21c60bf6d4 setup-apt-repo: Fix shellcheck warnings.
In scripts/lib/setup-apt-repo line 6:
zulip_source_hash=`sha1sum $SOURCES_FILE`
                  ^-- SC2006: Use $(..) instead of legacy `..`.

In scripts/lib/setup-apt-repo line 10:
SCRIPTS_PATH="$(dirname $(dirname $0))"
                        ^-- SC2046: Quote this to prevent word splitting.
                                  ^-- SC2086: Double quote to prevent globbing and word splitting.

In scripts/lib/setup-apt-repo line 36:
if [ "$zulip_source_hash" = "`sha1sum $SOURCES_FILE`" ] && ! [ -e "$STAMP_FILE" ]; then
                             ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 942bb49c29 install-node: Fix shellcheck warnings.
In scripts/lib/install-node line 34:
    source "$NVM_DIR/nvm.sh"
    ^-- SC1090: Can't follow non-constant source. Use a directive to specify location.

In scripts/lib/install-node line 36:
    export NODE_BIN="$(nvm which default)"
           ^-- SC2155: Declare and assign separately to avoid masking return values.

In scripts/lib/install-node line 39:
    n=$(which node)
        ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 7162ac43a6 create-zulip-admin: Fix shellcheck warnings.
In scripts/lib/create-zulip-admin line 3:
if ([ "$ZULIP_USER_CREATION_ENABLED" == "True" ] || [ "$ZULIP_USER_CREATION_ENABLED" == "true" ]) && \
   ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

In scripts/lib/create-zulip-admin line 4:
   ([ -z "$ZULIP_USER_DOMAIN" ]   || \
   ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 5b4d30abab certbot-maybe-renew: Fix shellcheck warnings.
In scripts/lib/certbot-maybe-renew line 8:
    case "$(echo "$value" | tr A-Z a-z)" in
                               ^-- SC2019: Use '[:upper:]' to support accents and foreign alphabets.
                                   ^-- SC2018: Use '[:lower:]' to support accents and foreign alphabets.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Tim Abbott 35cb7528f9 models: Add new UserMessage flag active_mobile_push_notification.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.

This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.

We include a partial index on UserMessage for this flag.
2018-08-01 16:51:56 -07:00
Tim Abbott b564976ce4 upgrade-zulip: Fix unicode issues with detecting early migrations.
Apparently, our Python 3 conversion for the early-migrations logic
here was incorrect, and as a result we never set
need_create_large_indexes to True (because we were checking whether a
`bytes` was inside a list of `str`s).

The simplest fix would be to just add a `.decode()` in one place, but
this refactor to just decode at the beginning is a lot more readable.
2018-07-31 13:46:06 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Shubham Padia bf6dc4472b models: Add is_private flag to UserMessage and add index for it.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.

This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.

For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.

History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org.  A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
2018-07-30 15:43:55 -07:00
Tim Abbott 30a3c48ff3 thumbor: Fix missing virtualenv-clone package in dependencies.
This package is important in order to avoid scary-looking errors
whenever we upgrade the dependencies in thumbor.txt (where
virtualenv-clone isn't installed in the venv, and then gets installed
by the code we just added a TODO comment to.
2018-07-30 11:55:16 -07:00
Anders Kaseorg a5407e1c7d scripts: Replace node-wrapper with a symlink.
Commit 00e057bf44 (#4727) simplified
node-wrapper to a one-line wrapper script for performance.  Copying
the binary was proposed and rejected because node finds some of its
modules relative to its own path.  But a symlink doesn’t have that
issue, as you can verify with

    node -e 'console.log(require.resolve.paths("foo"))'

(To find its own path, node uses `process.execPath`, which resolves
symlinks, and there’s no plausible reason for that behavior to change.
https://github.com/nodejs/node/blob/v8.11.1/lib/module.js#L708-L717
https://github.com/nodejs/node/blob/v10.7.0/lib/internal/modules/cjs/loader.js#L761-L770)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 11:48:10 -07:00
Anders Kaseorg 09c64f260b scripts/lib/zulip_tools.py: Avoid shelling out for touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg fe76b97e28 scripts/lib/setup_venv.py: Avoid shelling out for cp, touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Tim Abbott ee0f4ca330 locale: Set LANG/LANGUAGE to match LC_ALL.
Apparently, perl at least expects LANG, LANGUAGE, and LC_ALL to be
consistent, and thus apt spits out a bunch of warnings if these are
different.  So if we're forcing LC_ALL in these installer/upgrade
script blocks, we should force the rest too.

I believe this fixes the remaining locale part of #9946.
2018-07-23 23:01:10 -07:00
jeaye 4999474cce install: Add a couple Docker-specific options to the certbot scripts.
--agree-tos is useful for the Docker environment, where we won't have
an interactive shell present for agreeing to the ToS.

--deploy-hook is also useful for the Docker environment; it makes it
possible to customize what deploy hook (if any) we pass into the
underlying cerbot command.
2018-07-23 17:45:59 -07:00
Tim Abbott f228eabd90 install: Make sure the en_US.UTF-8 locale is available.
We need to make sure the en_US.UTF-8 locale has been generated before
setting the locale via the environment.

Fixes part of #9946.
2018-07-18 15:05:27 -07:00
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Tim Abbott 3cc93fd721 setup_venv: Fix missing libssl-dev dependency.
We were already correctly including libssl-dev in Zulip's dependencies
in development environment provisioning, but (at least now) it's
needed to build certain Python packages like pycurl when building a
Zulip virtualenv in production.  I haven't investigated why we didn't
need this on Ubuntu, but one possible reason would be that some other
library in our dependencies list happens to depend on it on Ubuntu.

We fix this by moving the dependency over to the shared
VENV_DEPENDENCIES list.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Tim Abbott 8f9b5633b8 zulip_tools: Fix accessing LSB data on Debian stretch.
Apparently, at least some Debian stretch systems don't have an
/etc/lsb-release, so the optimization that we did in
5d39a0f0fc broke our installer on
Debian.

We fix this, by falling back to calling the lsb_release command on
systems that don't have a faster way to do it.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Joshua Schmidlkofer b1a57d144f thumbor: Add production installer/puppet support.
This commits adds the necessary puppet configuration and
installer/upgrade code for installing and managing the thumbor service
in production.  This configuration is gated by the 'thumbor.pp'
manifest being enabled (which is not yet the default), and so this
commit should have no effect in a default Zulip production environment
(or in the long term, in any Zulip production server that isn't using
thumbor).

Credit for this effort is shared by @TigorC (who initiated the work on
this project), @joshland (who did a great deal of work on this and got
it working during PyCon 2017) and @adnrs96, who completed the work.
2018-07-12 20:37:34 +05:30
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Vishnu Ks 109fa85614 provision: Rename file_hash_updated to file_or_package_hash_updated.
Check for changes in package version as well along
with the files.
2018-06-22 23:40:31 +05:30
Tim Abbott 9d9d59d4b2 clean-unused-caches: Optimize performance.
This saves about 400ms when running clean-unused-caches, basically by
calling its sub-rountines by import (rather than
`subprocess.check_call()`).  The performance optimization seems well worth it.

Fixes #9766.
2018-06-18 07:31:33 -07:00
Vishnu Ks 7b8e79ae48 provision: Refactor hashing of compilemessages into a function.
This allows it to be reused for other tools.

Edited by tabbott to remove the use of "compilemessages" in variable
names.
2018-06-18 06:55:36 -07:00
Greg Price 25c46e3abb tx: Clarify role of the script `process-mobile-i18n`.
This file looks like it's producing some kind of compilation of the
mobile strings, that the mobile app will somehow end up using --
especially as it refers to its output as a "resource file".  In
reality, it compiles statistics to be included in the language-picker
UI in the web app.  Give appropriate names to the identifiers so it's
less confusing.
2018-06-17 17:37:29 -07:00
Raymond Akornor 8f7af5fde5 install-node: Upgrade to yarn 1.7.0. 2018-06-05 08:52:55 -07:00
Tim Abbott 0fabff6dda setup_venv: Clean up VENV_DEPENDENCIES using parsed lsb_release.
This is mostly a cleanup, but it should also save 50ms in the runtime
of create-production-venv.
2018-05-29 10:57:36 -07:00
Raymond Akornor 5d39a0f0fc scripts: Replace calls to lsb release with our own parsing.
This improves the performance of these operations, by saving a ~50ms
Python process startup.  While not a major performance improvement, it
seems worth it, given how often these commands get run.

Fixes #9571.
2018-05-29 10:57:36 -07:00
Tim Abbott 61ee01359e install: Update list of supported distros for installation.
This only changes the install script, not the docs, so it's pretty
low-profile; we'll update the docs after a bit more testing.
2018-05-24 10:44:29 -07:00
Tim Abbott 2655ece96f create-production-venv: Fix missing virtualenv dependency.
On newer distros like Xenial, Stretch, etc., we were incorrectly not
installing the Python 3 version of the virtualenv package.  This was
accidentally working because most base images with Python already have
this package too, but this was failing to install the right
dependencies in our Docker builds, requiring unnecessary manual code.

We fixed this some time ago for provision.py, but not for production.
2018-05-18 16:53:35 -07:00
Tim Abbott 8a66b0b9a9 docker: Set has_appserver=0 correctly.
The docker installer configuration incorrectly had has_appserver set
to 0; this meant that (A) the docker-zulip code needed to copy the
block of code in the installer for the `has_appserver` case into the
Dockerfile (unnecessarily), and (B) one couldn't use `install` from a
Git ref (because the static asset compiler didn't end up in the right
place).

It appears that docker-zulip tried to set this flag in their `install`
command line, but the construction inside `install` meant that didn't
work.
2018-05-15 10:13:44 -07:00
Aditya Bansal cb9d8f6d48 scripts: Change use of typing.Text to str. 2018-05-14 05:16:22 +05:30
Aditya Bansal e14974ff2c scripts: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott 41841221ee scripts: Remove obsolete zesty configuration.
Zesty already reached end-of-life, so we'll never support it.

And in one place, we add support for bionic.
2018-05-05 11:41:57 -07:00
Tim Abbott 1b3b298fa8 install: Allow installing with Debian 9.
For now we just change the script, not the documentation.
2018-05-05 10:49:09 -07:00
Tim Abbott 76fa29085a setup-apt-repo: Clean up setup code for apt repo.
This fixes adding the Ubuntu repositories for Debian, as well as makes
sure that we install the debian-archive-keyring package on Debian,
which is only priority important (and thus might be missing).
2018-05-05 10:03:39 -07:00
Tim Abbott 4ee762a52c apt: Add packagecloud repository for Debian. 2018-05-05 10:03:03 -07:00
Tim Abbott 06cfc591fe setup-apt-repo: Require apt-transport-https be installed.
Doing our apt operations over HTTPS has better security properties.
2018-05-05 10:02:50 -07:00
Greg Price e8be968250 install: Expand error message for missing SSL cert slightly.
It wasn't obvious reading this message that you can perfectly well
bring your own SSL/TLS certificate; unless you read quite a bit
between the lines where we say "could not find", or followed the link
to the detailed docs, the message sounded like you had to either use
--certbot or --self-signed-cert.

So, explicitly mention the BYO option.  Because the "complete chain"
requirement is a bit tricky, don't try to give instructions for it
in this message; just refer the reader to the docs.

Also, drop the logic to identify which of the files is missing; it
certainly makes the code more complex, and I think even the error
message is actually clearer when it just gives the complete list of
required files -- it's much more likely that the reader doesn't know
what's required than that they do and have missed one, and even then
it's easy for them to look for themselves.
2018-04-19 11:08:22 -07:00
Tim Abbott 105eed049e install-node: Fix leaking of $HOME.
This fixes a bug where provision was failing since our most recent
upgrade to yarn/nvm/node.

It turns out my original fix was the correct fix, but to the wrong
third-party tool: nvm, not yarn, was the offender.
2018-04-12 14:32:36 -07:00
Tim Abbott 041fd802b7 Revert "yarn: Revert back to v0.27.5."
This reverts commit d4b88e86cc.
2018-04-12 11:37:00 -07:00
Tim Abbott f6ae57fa70 install-node: Correctly fix yarn installation.
Apparently, new versions of yarn use the HOME environment variable to
figure out where to access their configuration, and sudo apparently
doesn't clear that variable, so install-node was being run with HOME
set to something under /home/vagrant (e.g.).

Fix this by just setting that environment variable correctly.

This replaces 250a036ff8, which
misdiagnosed the issue.
2018-04-12 11:37:00 -07:00
Tim Abbott 250a036ff8 install-node: Fix yarn installation.
It appears that some change in yarn's versioning system means that
installing yarn itself ends up chowning its config directory
incorrectly to be owned by root, preventing `yarn install` from
working later.
2018-04-12 10:42:27 -07:00
Priyank d4b88e86cc yarn: Revert back to v0.27.5.
Revert yarn version back due to some issue with new version that causes
permission issues in ~/.config/yarn directory.

Related discussion: https://chat.zulip.org/#narrow/stream/21-provision-help/topic/EACCES.3A.20permission.20denied.2C.20scandir.20'.2Fhome.2Fvagrant.2F.2Econfig.2Fya
2018-04-12 10:18:59 -07:00
Tim Abbott 9b8dd4f125 install-yarn: Fix buggy status check for the signature.
Apparently, they added a new signing key instance, and so checking
whether the old key exists doesn't work anymore.
2018-04-09 15:09:37 -07:00
Priyank ee078c372f install-node: Upgrade node, yarn, and nvm.
node -> v8.9.4
yarn -> 1.5.1
nvm -> 0.33.8

Also updates a test in timerender.js which depends on time
provided by node which is now changed in newer release.

Some changes have been made in circeci script, we just create ~/.config
directory and chown it to circleci user so installing new version of yarn
does not cause any ci failure on circleci during provision.
2018-04-09 13:56:48 -07:00