Commit Graph

527 Commits

Author SHA1 Message Date
Jason Michalski 24fbd7205d docker: Add postgres docker build with full text search.
This is multi-stage build which first builds tsearch-extras with the
current version of postgres and then configs postgres for zulip. The
zulip config installs the hunspell dictionaries, stop words file,
tsearch-extras, and creates the initial database.

**Testing Plan:**
1) `docker-compose up` the existing config.
2) Build the new image
3) Edit docker-compose.yml to use the new image id
4) `docker-compose up` and verify full text search is still working.
2018-05-16 13:14:24 -07:00
Tim Abbott 8a66b0b9a9 docker: Set has_appserver=0 correctly.
The docker installer configuration incorrectly had has_appserver set
to 0; this meant that (A) the docker-zulip code needed to copy the
block of code in the installer for the `has_appserver` case into the
Dockerfile (unnecessarily), and (B) one couldn't use `install` from a
Git ref (because the static asset compiler didn't end up in the right
place).

It appears that docker-zulip tried to set this flag in their `install`
command line, but the construction inside `install` meant that didn't
work.
2018-05-15 10:13:44 -07:00
Aditya Bansal cb9d8f6d48 scripts: Change use of typing.Text to str. 2018-05-14 05:16:22 +05:30
Aditya Bansal e14974ff2c scripts: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott b3e4c702d1 generate_secrets: Fix handling of an empty secrets file.
This is now a condition that happens during installation, because we
now create an empty file for this in puppet.
2018-05-06 22:12:32 -07:00
Tim Abbott 41841221ee scripts: Remove obsolete zesty configuration.
Zesty already reached end-of-life, so we'll never support it.

And in one place, we add support for bionic.
2018-05-05 11:41:57 -07:00
Tim Abbott 1b3b298fa8 install: Allow installing with Debian 9.
For now we just change the script, not the documentation.
2018-05-05 10:49:09 -07:00
Tim Abbott 76fa29085a setup-apt-repo: Clean up setup code for apt repo.
This fixes adding the Ubuntu repositories for Debian, as well as makes
sure that we install the debian-archive-keyring package on Debian,
which is only priority important (and thus might be missing).
2018-05-05 10:03:39 -07:00
Tim Abbott 4ee762a52c apt: Add packagecloud repository for Debian. 2018-05-05 10:03:03 -07:00
Tim Abbott 06cfc591fe setup-apt-repo: Require apt-transport-https be installed.
Doing our apt operations over HTTPS has better security properties.
2018-05-05 10:02:50 -07:00
Greg Price e8be968250 install: Expand error message for missing SSL cert slightly.
It wasn't obvious reading this message that you can perfectly well
bring your own SSL/TLS certificate; unless you read quite a bit
between the lines where we say "could not find", or followed the link
to the detailed docs, the message sounded like you had to either use
--certbot or --self-signed-cert.

So, explicitly mention the BYO option.  Because the "complete chain"
requirement is a bit tricky, don't try to give instructions for it
in this message; just refer the reader to the docs.

Also, drop the logic to identify which of the files is missing; it
certainly makes the code more complex, and I think even the error
message is actually clearer when it just gives the complete list of
required files -- it's much more likely that the reader doesn't know
what's required than that they do and have missed one, and even then
it's easy for them to look for themselves.
2018-04-19 11:08:22 -07:00
Tim Abbott 105eed049e install-node: Fix leaking of $HOME.
This fixes a bug where provision was failing since our most recent
upgrade to yarn/nvm/node.

It turns out my original fix was the correct fix, but to the wrong
third-party tool: nvm, not yarn, was the offender.
2018-04-12 14:32:36 -07:00
Tim Abbott 041fd802b7 Revert "yarn: Revert back to v0.27.5."
This reverts commit d4b88e86cc.
2018-04-12 11:37:00 -07:00
Tim Abbott f6ae57fa70 install-node: Correctly fix yarn installation.
Apparently, new versions of yarn use the HOME environment variable to
figure out where to access their configuration, and sudo apparently
doesn't clear that variable, so install-node was being run with HOME
set to something under /home/vagrant (e.g.).

Fix this by just setting that environment variable correctly.

This replaces 250a036ff8, which
misdiagnosed the issue.
2018-04-12 11:37:00 -07:00
Tim Abbott 250a036ff8 install-node: Fix yarn installation.
It appears that some change in yarn's versioning system means that
installing yarn itself ends up chowning its config directory
incorrectly to be owned by root, preventing `yarn install` from
working later.
2018-04-12 10:42:27 -07:00
Priyank d4b88e86cc yarn: Revert back to v0.27.5.
Revert yarn version back due to some issue with new version that causes
permission issues in ~/.config/yarn directory.

Related discussion: https://chat.zulip.org/#narrow/stream/21-provision-help/topic/EACCES.3A.20permission.20denied.2C.20scandir.20'.2Fhome.2Fvagrant.2F.2Econfig.2Fya
2018-04-12 10:18:59 -07:00
Rhea Parekh ccefaf7b26 scripts: Remove the depreciated script 'postgres-reset-sequences'. 2018-04-10 13:07:14 -07:00
Tim Abbott 9b8dd4f125 install-yarn: Fix buggy status check for the signature.
Apparently, they added a new signing key instance, and so checking
whether the old key exists doesn't work anymore.
2018-04-09 15:09:37 -07:00
Priyank ee078c372f install-node: Upgrade node, yarn, and nvm.
node -> v8.9.4
yarn -> 1.5.1
nvm -> 0.33.8

Also updates a test in timerender.js which depends on time
provided by node which is now changed in newer release.

Some changes have been made in circeci script, we just create ~/.config
directory and chown it to circleci user so installing new version of yarn
does not cause any ci failure on circleci during provision.
2018-04-09 13:56:48 -07:00
Greg Price e792fc6c07 spelling: Correctly write "cannot".
None of these errors were user-facing; mainly in comments, plus
one bit of internal docs and a developer tool.
2018-04-02 15:36:31 -07:00
Tim Abbott 0d35bbc464 install: Install the wget package.
We depend on it for installing node, and it's a standard package, not
a required one, so we do need to explicitly declare the dependency.
2018-03-29 16:03:44 -07:00
Tim Abbott ff92cdaaaf clean-npm-cache: Use islink to check node_modules.
Very old Zulip deployments might have node_modules be an actual
directory, which we should ignore for the purposes of
garbage-collection.
2018-03-29 10:57:41 -07:00
Tim Abbott 5fb3bff67e certbot: Don't prompt when installing apt packages.
The comment included in this commit explains the somewhat messy
situation that requires running certbot twice as part of this
installer.

Fixes #8486.
2018-03-28 17:09:38 -07:00
Tim Abbott 838e5880b6 clean-emoji-cache: Stop recalculating paths for other deployments.
This is the analog of 7b2c9223e7 for the
emoji cache; the only difference is that the existing code was working
correctly.  It's still worth changing for improved robustness.
2018-03-28 15:48:29 -07:00
Tim Abbott 7b2c9223e7 clean-npm-cache: Fix buggy garbage-collection logic.
We saw issues with /srv/zulip_npm_cache being cleaned incorrectly by
this tool in production (more correctly, we noticed broken symlinks to
those directories, even from the current deployment).  Print-debugging
showed that indeed older deployments were being ignored, because the
logic for `get_caches_in_use` was totally broken (this was sorta
masked because we also keep the last week's deployments).

The specific bug here turned out to be that we weren't passing the
`production` argument to generate_sha1sum_node_modules, but the
broader problem is that this logic isn't robust to changes in the
hashing algorithm.

Fix this by replacing the broken logic for trying to compute the
correct hash for that deployment with just checking the symlink inside
the deployment to let it self-report.

We can't easily do this same change for clean-venv-cache, because we
use multiple virtualenvs there.  But a similar change could be useful
for the emoji cache as well.

Fixes #8116.
2018-03-28 15:42:02 -07:00
Tim Abbott 3b29d00c69 setup-apt-repo: Fix use of shasum alias for sha1sum.
This is apparently installed by the perl package; I hadn't even known
it existed.  We of course want to use the sha1sum command from
coreutils.

Fixes #8836.
2018-03-27 09:47:14 -07:00
Harshit Bansal 40958e0824 emoji: Switch to 64px 256 color indexed sprite sheets.
This commit switches our emoji infrastructure to use 256 color indexed
64px spritesheets. Earlier we were using non-indexed 32px spritesheets
which were blurry on high dpi displays. These indexed spritesheets not
only provide a crispier display but are also smaller in size.

This commit also removes the `emoji-datasource` package as a dependency
as all the data is now sourced from individual datasource packages.

Fixes: #7862.
2018-03-14 10:28:45 -07:00
Greg Price 1215245be6 install: Check for a supported OS release. 2018-03-06 19:43:02 -08:00
Greg Price 678fcef5bd install: Fix up success message in `--no-init-db` case.
The installation isn't really complete here, and wasn't even when this
was the only success case; the instructions we're giving are for the
next step in the installation.

These instructions don't say what to do in an actual use case for this
option, but decent instructions there will require having a concrete
use case in front of us and designing the flow for it.  At this stage,
just say where we are in the normal flow, and an admin who's chosen to
go off that flow can figure out how they want to vary it from there.
2018-03-06 19:43:02 -08:00
Greg Price f22712e8e8 install: Go straight through by default.
This flips the experimental `--express` option to be the default.

We retain the old behavior, where the script exits before
`initialize-database`, as an option `--no-init-db`; it might be useful
in e.g. a migration scenario (from a Zulip install elsewhere, or
another chat system) where the admin wants to set up the database
separately.

The install instructions are adjusted to match, getting shorter by two
steps and a bunch of words.  I think this opens up opportunities to
refactor the text to simplify things further, too, but leaving that
for another commit.

Also tweak the "production" test suite to match.
2018-03-06 19:43:02 -08:00
Greg Price b1ad4e88d4 install: Switch some more uses of `sudo` to `su`.
Kind of unfortunate because the `sudo` interface for running a command
is objectively better -- a list of arguments, rather than a string to
be re-parsed by the shell.  But some bare-bones machine images lack
`sudo`, so this makes things a bit more portable.
2018-03-06 19:43:02 -08:00
Greg Price eb7eef02e8 install: Suppress initialize-database instructions when redundant.
To do it, add a blob of getopt boilerplate at the top of the script;
and also fix a couple of nits of shell style while here.
2018-03-06 19:43:02 -08:00
Tim Abbott 5ccd3aa71b i18n: Run process-mobile-i18n when syncing translations. 2018-03-02 08:59:00 -08:00
Greg Price 4475950ddf queue: Restore prematurely-cut upgrade path.
Revert c8f034e9a "queue: Remove missedmessage_email_senders code."
As the comment in the code says, it ensures a smooth upgrade path
from 1.7.x; we can delete it in master after 1.8.0 is released.
The removal commit was merged early due to a communication failure.
2018-02-28 11:15:53 -08:00
Umair Khan c8f034e9a0 queue: Remove missedmessage_email_senders code.
After 68513952fb, all emails are sent through email_senders queue.
This commit removes code related to the legacy queue.
2018-02-21 16:43:56 -08:00
Aditya Bansal b1417860a9 create-thumbor-venv: Specify python 2.7 use for production thumbor venv. 2018-02-21 01:00:40 +05:30
Tim Abbott a1408a9d3a mypy: Remove unnecessary use of typing.Text. 2018-02-20 10:57:59 -08:00
Aditya Bansal b0c0648a4f clean-venv-cache: Handle cleaning up zulip-thumbor-venv properly. 2018-02-20 10:57:32 -08:00
Aditya Bansal 35969edd66 deps: Replace libz-dev with zlib1g-dev since the former was renamed. 2018-02-12 14:40:26 -08:00
Aditya Bansal a137ed9e27 thumbor: Fix issue with dependencies.
We do the following here:
* Remove libjasper-dev from THUMBOR_VENV_DEPENDENCIES.
Reason: This dependancy wasn't really needed by us for using
thumbor. It was a dependancy for using open-cv as Imaging Engine
in thumbor but we use PIL (Pillow now) as Imaging Engine.
* Add zlib1g-dev, libfreetype6-dev to THUMBOR_VENV_DEPENDENCIES.
Reason: These are dependancies of Pillow which are required for it
Pillow to function. Since we use Pillow in thumbor as Imaging Engine
we need these. Stuff before this didn't break because we also use
Pillow in development Environment and have these dependancies
installed from VENV_DEPENDENCIES as well.
2018-02-12 14:40:26 -08:00
Greg Price 6e633f8e2f install: Use readlink -f rather than realpath.
It does exactly the same thing, though the name is less transparent; and
it simplifies the script by avoiding an extra, early `apt-get install`.
2018-02-08 17:22:02 -08:00
Greg Price 8d86a6e331 install: Slightly clean up another couple of bits of style. 2018-02-08 17:14:28 -08:00
Greg Price 218b653930 install: Clean up organization of option parsing.
The parsing of options from environment variables belongs right next
to the normal kind; and `show_help` was unnecessary indirection.
2018-02-08 17:12:36 -08:00
Greg Price 08135e3666 install: Don't try to chown a self-signed cert to zulip.
The zulip user has no need to see this file; it's used by nginx.
And when we set up the cert early in install, there's no zulip user
yet anyway, so this fails.
2018-02-08 17:08:16 -08:00
Greg Price f68c485a86 install: Clean up some shell style: use here-docs, and compact getopt. 2018-02-08 17:05:13 -08:00
Greg Price 8f4b7f6e1e install: On --express, check up front for --hostname and --email. 2018-02-08 16:34:49 -08:00
Greg Price 31ec9a4627 install: Implement --self-signed-cert earlier.
Otherwise when puppet tries to install and start nginx, that can
give an error.
2018-02-08 16:34:49 -08:00
Greg Price 70d203c1c8 install: Add experimental option to go straight through more install steps.
We'll make this the normal behavior soon, once we're satisfied with
our arrangements for sending the admin straight to realm creation and
using the app without configuring email.  The instructions in the docs
will also have to change accordingly, of course.
2018-02-05 12:59:12 -08:00
rht 2b19f17fc9 scripts: Remove u prefix from strings. 2018-02-05 12:11:33 -08:00
Umair Khan 8b33c46165 i18n: Add script to process mobile locales.
This script iterates over all the mobile.json resources and creates a
single file at static/locale/mobile_info.json which contains total and
not-translated strings information against each language. After doing
this, it deletes all the mobile i18n resources downloaded by
tools/sync-translations because we neither want to check them in our
repository nor we want to make our repository dirty.
2018-01-26 16:55:40 -08:00