Commit Graph

1022 Commits

Author SHA1 Message Date
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg 5050fb19f6 nagios: Don’t crash on missing cron file.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 16:49:32 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Alex Vandiver 4fe0444108 puppet: Install wal-g, not wal-e. 2020-06-11 15:52:43 -07:00
Anders Kaseorg 0e5946ee5a python: Add noqa comments for the specific star imports we allow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 15:36:43 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
arpit551 9e8f1aacb3 certbot: Switch to use certbot from apt.
certbot-auto doesn’t work on Ubuntu 20.04, and won’t be updated; we
migrate to instead using the certbot package shipped with the OS
instead. Also made sure that sure certbot gets installed when running
zulip-puppet-apply, to handle existing systems.
2020-06-08 21:59:29 -07:00
Anders Kaseorg 523907fe1d upgrade-zulip: Add umask override.
We already override the umask in upgrade-zulip-stage-2, but that’s too
late since we’ve already written a bunch of files in stage 1.  I would
have removed the stage 2 override, but the OS upgrade documentation
references running stage 2 directly.

Fixes #15164.  Note that an affected installation will need to upgrade
twice, because the first upgrade uses the old stage 1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 21:57:05 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
rht 07fa25dcd3 setup-yum-repo: Update url of postgresql rpm repo.
The old url is dead.
2020-06-08 11:26:07 -07:00
Anders Kaseorg 0f63753926 install-node: Upgrade Node.js to 12.18.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:06:57 -07:00
Anders Kaseorg 333f7d16c9 logging: Pass more format arguments to logging.
Commit bdc365d0fe (#14852) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
arpit551 439f0d3004 install: Ad production support for Zulip on Ubuntu Focal.
Install script now runs on Focal.  Python 2 is now installed via the
`python2` package in Focal.
2020-05-25 16:58:42 -07:00
arpit551 3971824d04 puppet: suppress puppet warnings with ruby 2.7.
Ubuntu Focal comes with ruby 2.7 and the latest puppet
has some issues with it so to suppress puppet
warnings with ruby 2.7 we added  RUBYOPT = "-W0" in
the environment.
2020-05-25 16:56:11 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-05-20 13:47:20 -07:00
Mateusz Mandera 28a6983b34 check-rabbitmq-queue: Log queue size in "queue stuck" alert. 2020-05-14 11:55:20 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Tim Abbott 800e6b1ca6 generate_secrets: Add more comments/documentation. 2020-04-30 10:44:27 -07:00
Steve Howell adc0ed4206 provision: Avoid shelling out to clean caches.
Yes, it's slightly janky to create an
argparse.Namespace object like this, but it
saves us from shelling out to a script whose
only real value-add is parsing a single
`threshold_days` argument.

This saves about 130ms for a no-op provision.
2020-04-30 17:19:13 +00:00
Steve Howell 34191f2a56 generate_secrets: Avoid unnecessary settings import.
We try to avoid importing Django settings unless
we really need them, since we want this program
to run very quickly during `provision` (when
secrets have already been generated earlier).
2020-04-30 13:52:54 +00:00
Steve Howell 522ee7fd6b generate_secrets: Lazily import crypto module. 2020-04-30 13:49:44 +00:00
Steve Howell 66392afe15 generate_secrets: Extract random_token(). 2020-04-30 13:23:13 +00:00
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
arpit551 fe6be1f61e setup_venv: Remove python2 dependencies for virtual environment.
These dependencies are not needed.
2020-04-22 16:10:59 -07:00
Steve Howell f4942e9927 digest refactor: Clean up names and comments.
We now use `extra_strings` instead of `package_versions`
to allow for more generic digests to be built
(without naming confusion).
2020-04-22 14:41:42 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Aman Agrawal 120144e099 upgrade-zulip: Exit if OS is unsupported.
This is to prevent folks who accidentally try to
upgrade their system to an unsupported platform
from getting into a bad place.
2020-04-22 10:04:36 -07:00
Aman Agrawal 0f4b1076ad scripts: Remove Xenial and Stretch support from installation scripts.
Note that we leave support for them in `setup-apt-repo` and puppet,
since we're still supporting systems using Xenial for non-appserver
puppet rules.
2020-04-22 10:00:38 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Tim Abbott 703fae8980 send_custom_email: Use a special .gitignored directory.
Previously, the send_custom_email code path leaked files in paths that
were not `.gitignored`, under templates/zerver/emails.

This became problematic when we added automated tests for this code
path, as it meant we leaked these files every time `test-backend` ran.

Fix this by ensuring all the files we generate are in this special
subdirectory.
2020-04-21 16:50:11 -07:00
arpit551 e6edf469ee ci: Remove the need of using TRAVIS in env.
Since now we want to use production suites on Circle CI so there
is no need to set TRAVIS in env while running scripts.

CIRCLECI is set default in the enviroment of Circle CI builds
so we can use it directly.

Also Travis CI had rabbitmq-server installed so we had to add workaround
in install script to avoid the error. That workaround is removed.
2020-04-21 14:46:40 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell b280f73c77 provision: Extract path_version_digest(). 2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Steve Howell 067196c508 provision: Simplify `is_force` codepaths.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.

If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.

Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.

For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.

We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
2020-04-17 09:45:59 -07:00
Tim Abbott 382261dc72 upgrade-zulip: Remove tsearch-extras on upgrade.
We stopped using tsearch-extras in Zulip 2.1.0 after Anders figured
out how to achieve its goals with native postgres.  However, we never
did a `DROP EXTENSION` on systems thta had upgraded, which meant that
backups created on systems originally installed with Zulip 2.0.x and
older, and later upgraded to Zulip 2.1.x, could not be restored on
Zulip servers created with a fresh install of Zulip 2.1.x.

We can't do this with a normal database migration, because DROP
EXTENSION has to be done as the postgres user, so we add some custom
migration code in the upgrade-zulip-stage-2 tool.

It's safe to run this whenever tsearch_extras.control is installed because:
* Zulip is AFAIK the only software that ever used tsearch_extras.
* The package was only installed via puppet on production servers configured to
  run a local Zulip database.
* We'll only run this code once per system, because it removes the
  package and thus the control files.

Fixes #13612.
2020-04-15 15:18:44 -07:00
Tim Abbott 6543788225 inline_email_css: Simplify parsing of template names.
Removing the extensions within the Set construction object was a
confusing way to do things.
2020-04-10 15:52:52 -07:00
Tim Abbott b2cf0aaa38 inline_email_css: Configure cssutils on import.
This is necessary for when we later want to call inline_template from
elsewhere.
2020-04-10 15:51:54 -07:00
wowol 78f3125f0a inline_email_css: Extract inline_template function. 2020-04-10 15:49:10 -07:00
wowol f42adba071 inline_email_css: Extract get_all_templates_from_directory. 2020-04-10 15:44:25 -07:00
wowol b3cc93f961 inline_email_css: Move constants to top, remove chdir. 2020-04-10 15:44:24 -07:00
wowol 0b5a87f98f inline_email_css: Extract strip_unnecesary_tags. 2020-04-10 15:34:17 -07:00
wowol 240d8d565c inline_email_css: Extract escape_jinja2_characters. 2020-04-10 15:29:47 -07:00
Tim Abbott 26c65a5f78 inline_email_css: Extract configure_cssutils. 2020-04-10 15:29:47 -07:00
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Mateusz Mandera f5a12e4094 nagios: Fix use of wrong variable in check_rabbitmq_queue.
That was supposed to be queue_name, not queue - the latter leads to
nonsensical results.
2020-04-10 13:48:16 -07:00
Tim Abbott c20b0bd992 check_rabbitmq_queue: Adjust threshholds for some queues. 2020-04-10 13:14:43 -07:00
Anders Kaseorg 038b315d8e python: Further pyupgrade changes following merge conflict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:59:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Mateusz Mandera 96fe2e5a42 nagios: Deduplicate queue list between check-rabbitmq scripts. 2020-04-09 13:41:01 -07:00
Mateusz Mandera 122d0bca83 check-rabbitmq-queue: Add a simple algorithm to analyze queue stats.
This new algorithm is designed to avoid monitoring paging when a queue
simply has bursty behavior.
2020-04-09 13:41:01 -07:00
arpit551 a2dd1fad12 provision: switch Focal to use PGroonga from PPA.
Since Groonga packages for Ubuntu 20.04 Focal are now available
in their ppa so stopped building pgroonga from source.
2020-03-31 15:02:31 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Tim Abbott 0f1bdcc46f restart-server: Restart Tornado processes individually.
After some testing, I've confirmed that this seems to behave
significantly better in terms of the number of failed requests due to
Tornado being the process of restarting compared with the previous
version, as each individual process is only down for a short time,
rather than all of them being down at once.
2020-03-27 06:23:34 -07:00
Anders Kaseorg 586f78cb32 install-node: Upgrade to Node.js 12.16.1, Yarn 1.22.4, nvm 0.35.3.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:34:47 -07:00
Tom Daff 2f213f7c8e
monitoring: Fix check-rabbitmq-consumers.
Missing commas in the definition of all the queues to check meant that it would be looking for queues with concatenated names, rather than the correct ones. Added the commas.
2020-03-25 17:19:16 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
arpit551 25e091a065 setup_path: Fix KeyError in setup_path for virtualenv ≥ 20.
In virtualenv ≥ 20, the site_packages variable was removed from
activate_this.py.  To avoid a KeyError, replace
activate_locals['site_packages'] with os.path.join(venv, 'lib',
python_version), where python_version is the 'pythonX.Y' name of the
directory where site-packages resides in the virtualenv.

Fixes #14025.
2020-03-23 04:06:19 -07:00
Mateusz Mandera ea93810d9a check-rabbitmq-queue: Put user check before rabbitmqctl call. 2020-03-22 18:46:28 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg fd2d642f95 postgres-init-db: Fix invocation from a relative path.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:40:37 -08:00
Anders Kaseorg d918626135 restore-backup: Run generate_secrets.py.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:20:43 -08:00
Mateusz Mandera 27b15a9722 install: Don't create internal realm in the installation process. 2020-02-12 12:00:10 -08:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
Anders Kaseorg 61bf698185 requirements: Fork pip.txt to pip2.txt for Python 2.
The current version of setuptools no longer supports Python 2.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg d2e07ea51b setup_venv: Replace virtualenv_args list with python2 bool.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg e88fac499f dependencies: Upgrade emoji-datasource from 4.0.4 to 5.0.1.
The “Smileys & People” category has been split into “Smilys & Emotion”
and “People & Body”.

Also, fix generate_sha1sum_emoji to read the emoji-datasource-google
version from yarn.lock, since package.json only gives a version range.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-04 21:30:51 -08:00
Mateusz Mandera 4c5a8e6f0c queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
Tim Abbott dd969b5339 install: Remove references to "Zulip Voyager".
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".

At the time, we just needed a name quickly, but it was never a good
name, just a placeholder.  This removes that placeholder name from
much of the codebase.  A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
2020-01-30 12:40:41 -08:00
Tim Abbott 9f414d74fd email_mirror: Rewrite docstrings to focus on current reality.
These docstrings hadn't been properly updated in years, and bad an
awkward mix of a bad version of the user-facing documentation and
details that are no longer true (e.g. references to "Voyager").

(One important detail is that we have real documentation for this
system now).
2020-01-30 12:39:45 -08:00
neiljp (Neil Pilgrim) 6ecf41d0bd clean_venv_cache: Limit search to .txt requirements files.
This both avoids examining all the .in files, but also any additional stray
files, which can cause provision errors.

Fixes #13762.
2020-01-30 12:20:32 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
rht 7c9954afc6 install: Reduce the RAM limit to accommodate some definition of "2GB".
See also the discussion at
https://chat.zulip.org/#narrow/stream/3-backend/topic/2GB.20ram.20gotcha.
2020-01-24 12:35:37 -08:00
rht 6f5cbed5f1 prod install: Ignore reading VERSION_CODENAME if not available. 2020-01-22 23:19:18 -08:00
rht c2dcaf48d8 prod install: Add the CentOS version of SUPERVISOR_CONF_DIR. 2020-01-22 23:19:18 -08:00
Anders Kaseorg 8c733a3f68 create-db.sql: Start by dropping the zulip database if needed.
At some point the PostgreSQL Docker image started creating the zulip
database for us, which caused our CREATE DATABASE to fail.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg 298d45b46a create-db.sql: Handle exception if zulip user already exists.
Fixes #13530.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg a82032a182 generate_secrets: Enable Redis authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg cdda983e90 settings: Support optional memcached authentication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Anders Kaseorg 699626f3cf flush-memcached: Use pylibmc.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:18 -08:00
rht cd3907648d prod install: Use ID_LIKE to help select os family. 2020-01-07 13:25:25 -08:00
rht bc94e8e815 prod install: Use /etc/os-release for Ubuntu/Debian to get os_id, os_version_id. 2020-01-07 13:25:25 -08:00
rht 9898c07e0d prod install: Add the CentOS version of the step to do dist-upgrade. 2020-01-07 13:25:25 -08:00
rht bf76696d67 prod install: Add the CentOS version of the step to install preparatory packages. 2020-01-07 13:25:25 -08:00
rht 6dd5dc32fc prod install: Add the CentOS version of the step to upgrade packages. 2020-01-07 13:25:25 -08:00
rht d88a7bbb42 prod install: Add the CentOS version of the step to update packages. 2020-01-07 13:25:25 -08:00
rht 49d7adb3cb prod install: Parse CentOS os identifications from /etc/os-release. 2020-01-07 13:25:25 -08:00
rht 771f6d213f prod install: Rename os_codename into os_version_id 2020-01-07 13:25:25 -08:00
Anders Kaseorg a78f8647d8 install: Run generate_secrets.py before zulip-puppet-apply.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-05 22:48:08 -08:00
Anders Kaseorg ab211c7acf lint: Tell ShellCheck to look for sourced files at relative paths.
This uses the new -P option of ShellCheck 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-18 03:48:02 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Anders Kaseorg 347fd80864 generate_secrets: Remove unused initial_password_salt in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 23:06:53 -08:00
Anders Kaseorg 7ebba2901a generate_secrets: Remove unused local_database_password in production.
Fixes #13464.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-09 13:05:31 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg ac49736311 install-node: Upgrade Node 12.11.1 to 12.13.0, Yarn 1.19.0 to 1.19.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
Anders Kaseorg d6377b00c0 node_cache: Don’t retry copying node_modules; let yarn do its thing.
`copytree` throws an error if the target already exists, and we don’t
really want to rerun the copy anyway.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-29 12:30:28 -07:00
Tim Abbott bbc1484253 check-rabbitmq-queue: Adjust threshholds for paging.
Ultimately, this isn't an effective way to monitor this queue; we want
time-based monitoring, not count-based monitoring.  Doing that
properly will likely involve modifying the queue processor to write
something about its status.

But until we add the monitoring we want, it makes sense to leave this
active with low limits.
2019-10-13 22:39:52 -07:00
Anders Kaseorg 775162d687 setup_venv: Use pip install --require-hashes for better security.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:21:18 -07:00
Anders Kaseorg 9182293d50 node_cache: Preserve symlinks when copying an old node_modules tree.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:19:53 -07:00
Anders Kaseorg 8432d97edf setup_venv: Add pkg-config to VENV_DEPENDENCIES.
This is needed on at least Debian 10, otherwise xmlsec fails to
install: `Could not find xmlsec1 config. Are libxmlsec1-dev and
pkg-config installed?`

Also remove libxmlsec1-openssl, which libxmlsec1-dev already depends.

(No changes are needed on RHEL, where libxml2-devel and xmlsec1-devel
already declare a requirement on /usr/bin/pkg-config.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-05 18:24:32 -07:00
Anders Kaseorg 1235dc3bec install-node: Upgrade to Node 12.11.1, Yarn 1.19.0.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-05 18:07:53 -07:00
Anders Kaseorg 0af22dad18 flush-memcached: Respect MEMCACHED_LOCATION; handle errors.
Fixes #13238.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-01 16:05:55 -07:00
ab1nash 71f0fecda7 scripts: Clean up output from 'clean_unused_caches'.
The output log from running clean_unused_caches was too verbose as
part of the `upgrade-zulip` overall output.  While this output is
potentially helpful when running it directly for debugging, it's
certainly redundant for the main production use case.

So a new flag --no-print-headers is introduced.  It suppresses the
header outputs for the subtools.

Fixes #13214.
2019-09-30 10:51:00 -07:00
Mateusz Mandera c42077c12f dependencies: Add dependencies needed for SAML. 2019-09-28 12:15:13 -07:00
Tim Abbott a84bb89bdc scripts: Move mobile i18n code out scripts/.
Like other code that is only used in the development environment, this
doesn't belong in scripts/.
2019-09-24 12:57:42 -07:00
Tim Abbott 27b3c1a312 provision: Move install-shellcheck to proper directory.
Scripts in scripts/ should be exclusively code that used in
production, and this isn't.
2019-09-24 12:54:33 -07:00
Anders Kaseorg 4fdc80a9c7 setup-apt-repo: Install groonga-keyring.
This allows the system to get updates to the Groonga repository
signing key, so `apt update` doesn’t start failing when the key
changes (like it recently did).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg d1e504079d setup-apt-repo: Don’t waste time installing debian-archive-keyring.
debian-archive-keyring is a dependency of the essential package apt,
so it is present in every Debian system.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg 2ff87bd888 setup: Update groonga APT repository signing key.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 16:01:39 -07:00
Anders Kaseorg 76492b25ae setup_venv: Install pip.txt requirements with --force-reinstall.
virtualenv on Ubuntu 16.04, when creating a new environment, downloads
the current version of setuptools, then replaces its pkg_resources
with an old copy from
/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl.
This causes problems, a simple example of which is reproducible from
the ubuntu:16.04 Docker base image as follows:

    apt-get update
    apt-get -y install python3-virtualenv
    python3 -m virtualenv -p python3 /ve
    /ve/bin/pip install sockjs-tornado
    /ve/bin/pip download sockjs-tornado

→ `AttributeError: '_NamespacePath' object has no attribute 'sort'`

More relevantly, it breaks pip-compile in the same way.  To fix this,
we need to force setuptools to be reinstalled, even if we’re asking
for the same version.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 13:23:58 -07:00
Anders Kaseorg 8d91bebf95 restart-server: Warn if the shell’s PWD goes through an updated symlink.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-21 12:02:15 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Anders Kaseorg 2e1494bdbd setup-apt-repo: Add ca-certificates to pre_setup_deps.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:15:43 -07:00
Anders Kaseorg 2ec946ad4d postgres-init-db: Require an Enter press in confirmation prompt.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-12 16:04:55 -07:00
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg 875002108f setup_venv: Remove CentOS workaround for fixed pycurl bug.
We are installing pycurl 7.43.0.3 which includes the fix.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:38:38 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Anders Kaseorg 9e481e353a .yarnrc: Set ignore-scripts true.
Follow up to #13065, to keep manual yarn invocations consistent with
our automated ones.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 16:15:54 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Anders Kaseorg 43c8c720ef node_cache: Run yarn with --ignore-scripts.
As predicted in https://www.kb.cert.org/vuls/id/319816/, a malicious
worm is beginning to spread across the npm ecosystem through package
postinstall scripts.  Only instead of direct self-replicating code,
the replication vector is the temptation to monetize postinstall
scripts by polluting the console logs with paid advertisements.  The
effect will be the same unless we all put a stop to this while we
still can.

Apply the recommended VU#319816 workaround, which is to disable
lifecycle scripts when installing npm packages.  The only fallout is:

* node-sass can’t run because it uses compiled native code; we replace
  it with Dart Sass.
* phantomjs-prebuilt doesn’t download the binary at install time; we
  tell it to download it in run-casper.
* ttf2woff2 transparently falls back from native code to an Emscripten
  build.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:36:03 -07:00
Anders Kaseorg fa913b7f5d node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Hemanth V. Alluri dac068df31 production: Finish adding production support for Zulip on Debian Buster.
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.

We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.

Fixes #9828.
2019-08-17 12:22:32 -07:00
Anders Kaseorg 86f8bb83ab setup_venv: Don’t pip uninstall typing; it’s no longer installed.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 14:13:40 -07:00
Anders Kaseorg d761bbc2ed node_cache: Remove unused copy_modules parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-14 14:39:04 -07:00
Anders Kaseorg 906af61d20 pythonrc: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg b320b8c37f email-mirror-postfix: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 62a1c3443d mypy: Remove type ignores for python 3 stubs now present in typeshed.
Namely:
* posix stubs
* readfp stub
2019-08-06 23:24:56 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg 1d05a71b5d install, provision: Treat all nonzero exit codes as failure, not just 1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:22:11 -07:00
Anders Kaseorg 9233b3bc75 setup_path_on_import: Always make sure zulip-py3-venv is activated.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-07-18 14:21:04 -07:00
Anders Kaseorg a45be467ad get_deploy_root: Avoid useless extra realpath call.
The comment that tabbott edited into my commit while wimpifying this
function is wrong on multiple levels.

Firstly, the way in which users might be “running our scripts” was
never relevant.  `__file__` is not the script that the user ran, it’s
zulip_tools.py itself.  What matters is not how the user ran the
script, but rather how zulip_tools was imported.  If zulip_tools was
imported as scripts.lib.zulip_tools, then `__file__` must end with
`scripts/lib/zulip_tools.py`, so running dirname three times on it is
fine.  In fact, in Python ≥ 3.4 (we don’t support anything older),
`__file__` in an imported module is always an absolute path, so it
must end with `scripts/lib/zulip_tools.py` in any case.

(At present, there’s one script that imports lib.zulip_tools, and the
installer runs scripts/lib/zulip_tools.py as a script, but those uses
don’t hit this function.)

Secondly, even if we do care about `__file__` being a funny relative
path, there’s still no reason to have two calls to `realpath`.
`realpath(dirname(dirname(dirname(realpath(…)))))` is equivalent to
`dirname(dirname(dirname(realpath(…)))), as the inner `realpath` has
already canonicalized symlinks at every level.

This version also deals with `__file__` being a funny relative
path (assuming none of scripts, lib, and zulip_tools.py are themselves
symlinks), while making fewer `lstat` calls than either of the above
constructions.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-07-18 14:20:53 -07:00
Anders Kaseorg f97bb6b65c postgres-create-db: Split into non-pgroonga and pgroonga scripts.
Fixes #12655.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-17 16:18:22 -07:00
Sumanth V Rao 2c9f5e3980 tools: Add tool to update API field of local zuliprc file.
This tool can be used to update the API field of local
zuliprc files for dummy users of development server
(iago, prospero, etc) with the correct API key from database.

This tool can be run after provisioning (or similar tools) which change
the API keys in the database.
2019-07-17 16:00:21 -07:00
Amal Rajan 09049f1678 provision: Add provision support for Ubuntu 19.04
This commit adds support for Ubuntu 19.04 in the development
environment.
2019-07-13 16:44:09 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Anders Kaseorg caecd1c2ad install: Disable installation and provisioning on Ubuntu 14.04 Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 15:58:53 -07:00
Puneeth Chaganti 9f916f78da upgrade-zulip-from-git: Fetch tags to make version info more useful.
To be able to show more detailed git information in ZULIP_VERSION, we need
to fetch tags when running `upgrade-zulip-from-git`.
2019-06-23 23:10:47 -07:00
Tim Abbott a716b09a3f upgrade-zulip-from-git: Fix handling of errors in git clone.
Apparently, the `chown -R` would never run if the original clone
attempt had networking errors, leading to inability to use
upgrade-zulip-from-git without manual intervention.
2019-06-18 10:00:50 -07:00
Tim Abbott 9760c8cfc9 tools: Fix postgres-init-*-db cache handling.
Previously, it didn't properly update the stamp files that determine
our caching behavior, so if one ran test-backend afterwards, nothing
would happen.

A secondary issue that this commit does not fix is that provision will
end up rerunning the whole thing.
2019-06-17 16:24:13 -07:00
Wyatt Hoodes 8e626d3db3 zulip_tools: Add a function to get or create the var uuid path. 2019-06-17 13:51:40 -07:00
Tim Abbott d2fbd62b63 install: Fix argument parsing for no-overwrite-settings option.
This had the same bug as our other recent change to the install
script.
2019-06-16 15:24:20 -07:00
Wyatt Hoodes 0b05d91e62 test_runner: Write database ids to file for reference in clean up.
The ids that will be used for each particular run of the test suite are
written to a unique file. Each file will then be used as a time
reference of when the suite was ran.

This change sets up the ability for a complete clean up of potentially
leaked database templates.

Tweaked by tabbott to remove these files after successful database
cleanup.
2019-06-14 15:23:20 -07:00
Vishnu Ks 6e52779ed5 install: Include no-dist-upgrade in args list.
This was missed out in 2e51ac8c49
2019-06-14 14:24:08 -07:00
Tim Abbott 8a53686f41 install: Add --no-overwrite-settings option.
This commit needs more work to make this option reasonable.
2019-06-13 14:39:25 -07:00
ppreethi 86840adda5 version: Show number of commits and commit sha in ZULIP_VERSION.
We use `git describe --tags` to get information about the number of commit since
the last major version, and the sha of the current HEAD. This is added to the
ZULIP_VERSION when a deploy is done from `git`.

Modified heavily by punchagan to:
* to use git describe instead of `git log` and `wc`
* use a separate script to run the git describe command
* write the file with version info to var/ and remove it from the repo

Fixes #4685.
2019-06-07 13:39:12 -07:00
Tim Abbott 5748bae71d restore-backup: Run configure-rabbitmq to update RABBITMQ_PASSWORD.
Previously, if you restored onto a different production system from
the one where you took the backup, backup restoration would fail
because the generated rabbitmq passwords for the two systems would be
different, and we didn't update the restored system to use the
password from the original system.

Fixes #12114.
2019-06-06 16:38:27 -07:00
Tim Abbott 3c4030a421 restore-backup: Run zulip-puppet-apply before pg_restore.
This should ensure that we apply any special configuration for the
database system (e.g. installing `pgroonga`) before we try to restore
the database contents from the archive.

For pgroonga in particular, this is important so that we can preserve
the configuration of the extension in the `pg_restore` process.

Fixes #12345.
2019-06-06 16:34:28 -07:00
Tim Abbott 13a20a1ab5 restore-backup: Fix error on systems using S3 upload backend.
With the S3 file upload backend, we don't store uploads locally, so
the `uploads` directory in the backup will be empty, and more
importantly, LOCAL_UPLOADS_DIR will be None, which the previous code
crashed on.
2019-06-06 15:08:18 -07:00
Thomas Ip c6cdcd082e dependencies: Upgrade node to 12.3.1.
API changes:
* The behaviour of Date.toLocaleTimeString() reverts to pre 8.0.0,
  this only affects automated tests. Lots of other API changes but
  we didn't use any of those.
* The internal sorting algorithm changed which causes one of our own
  compare function to miss coverage.
2019-06-05 17:15:52 -07:00
Vishnu Ks 14ed0e283d install: Add option to skip dist-upgrade. 2019-06-05 15:50:02 -07:00
Thomas Ip 9a84a60029 dependencies: Upgrade yarn to 1.16.0. 2019-06-03 12:24:00 -07:00
Anders Kaseorg 90a9dfa7f5 setup-apt-repo: Try to download pre-setup deps before skipping update.
Simulate isn’t enough in some cases.  The error message when this
fails looks sufficiently non-alarming.

LXC:

    default: + apt-get -dy install lsb-release apt-transport-https gnupg
    default: Reading package lists...
    default: Building dependency tree...
    default:
    default: Reading state information...
    default: lsb-release is already the newest version.
    default: gnupg is already the newest version.
    default: The following NEW packages will be installed:
    default:   apt-transport-https
    default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    default: Need to get 25.1 kB of archives.
    default: After this operation, 238 kB of additional disk space will be used.
    default: Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main apt-transport-https amd64 1.0.1ubuntu2.3
    default:   404  Not Found [IP: 91.189.88.161 80]
    default: Err http://security.ubuntu.com/ubuntu/ trusty-security/main apt-transport-https amd64 1.0.1ubuntu2.3
    default:   404  Not Found [IP: 91.189.88.161 80]
    default: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/apt/apt-transport-https_1.0.1ubuntu2.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
    default:
    default: E: Some files failed to download
    default: + apt-get update
    […]
    default: Fetched 4,504 kB in 7s (611 kB/s)
    default: Reading package lists...
    default: + apt-get -y install lsb-release apt-transport-https gnupg
    default: Reading package lists...

Docker:

    default: + apt-get -dy install lsb-release apt-transport-https gnupg
    default: Reading package lists...
    default: Building dependency tree...
    default:
    default: Reading state information...
    default: Package gnupg is not available, but is referred to by another package.
    default: This may mean that the package is missing, has been obsoleted, or
    default: is only available from another source
    default: E: Package 'gnupg' has no installation candidate
    default: + apt-get update
    […]
    default: Fetched 16.2 MB in 5s (3,326 kB/s)
    default: Reading package lists...
    default: + apt-get -y install lsb-release apt-transport-https gnupg
    default: Reading package lists...

(All in green.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 15:25:00 -07:00
Anders Kaseorg 407564086d provision: Move apt-get update from retry handler to setup-apt-repo.
This avoids unnecessarily alarming error messages if the apt cache is
missing.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:38 -07:00
Anders Kaseorg ae524b677d ci: Move backend and production tests to Ubuntu 16.04 (xenial).
This is preparation for dropping support for Trusty in CI.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:15 -07:00
Anders Kaseorg d267884cc1 node_cache: Run yarn install with --frozen-lockfile.
This way a yarn.lock inconsistent with package.json can’t slip through
unnoticed.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-20 19:23:37 -07:00
Puneeth Chaganti 0884764421 url preview: Add python-magic dependency for content-type sniffing. 2019-05-06 12:37:32 -07:00
Mayank Madan 99414e2d96 provision: Add support for Debian 10 Buster. 2019-05-05 17:57:19 -07:00
Mayank Madan a86eccb2b6 pgroonga: Upgrade from 2.1.6 to 2.1.8.
This is necessary to support postgres 11 in Debian Buster.
2019-05-05 17:52:32 -07:00
Anders Kaseorg 9937734e50 terminate-psql-sessions: Only terminate if we have permissions.
We have been semi-accidentally relying on the fact that terminate-psql-sessions 
fails silently when there are PIDs we don't have permission to terminate.

This actually happens somewhat often, generally when we're doing a series of
operations in quick succession by different users, because postgres processes
live a little longer than the `psql` shell that started them.

As part of adding ON_STOP_ERROR to all of our postgres commands, it makes
sense to enforce we don't fail here, but that means we need to actually filter
the target PIDs to only ones we can actually kill.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 14:41:42 -07:00
Tim Abbott 74143a0801 scripts: Disable terminate-psql-sessions ON_ERROR_STOP for now.
This appears to fail nondeterministically in CI; there's likely a
better fix but this should let us get CI happy again.
2019-04-22 15:49:48 -07:00
Anders Kaseorg 5290519a62 scripts: Always use ON_ERROR_STOP=1 when running psql.
Also use psql -e (--echo-queries) in scripts that use ‘set -x’, so
errors can be traced to a specific query from the output.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:19 -07:00
Tim Abbott 18146ed947 script: Add static asset pipeline option to support OS upgrades.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
2019-04-17 12:58:42 -07:00
Anders Kaseorg ed100d694f restore-backup: Open backup tarball as root.
Fixes permission errors when running restore-backup on a tarball
inaccessible to the zulip user.

Fixes #12125.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg 206741117c restore-backup: Reset cwd when switching to postgres user.
Fixes permission errors when running restore-backup from a cwd
inaccessible to the postgres user.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-17 12:26:03 -07:00
Anders Kaseorg 9fc0aa7ffa scripts: Fix exec invocation for in-process virtualenv activation.
activate_this.py has always documented that it should be exec()ed with
locals = globals, and in virtualenv 16.0.0 it raises a NameError
otherwise.

As a simplified demonstration of the weird things that can go wrong
when locals ≠ globals:

    >>> exec('a = 1; print([a])', {}, {})
    [1]
    >>> exec('a = 1; print([a for b in [1]])', {}, {})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
      File "<string>", line 1, in <listcomp>
    NameError: name 'a' is not defined
    >>> exec('a = 1; print([a for b in [1]])', {})
    [1]

Top-level assignments go into locals, but from inside a new scope like
a list comprehension, they’re read out of globals, which doesn’t work.

Fixes #12030.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-16 17:42:44 -07:00
Wyatt Hoodes d67dc2eb94 install-node: Upgrade yarn to 1.15.2.
In addition to upgrading dependencies being generally useful, this may
fix situations where yarn fails but returns a success status code in the 
presence of an HTTP proxy.
2019-04-14 22:36:14 -07:00
Tim Abbott ad81f700a1 scripts: Remove nagios overrides for missedmessage_emails.
Since 5cec566cb9, the
missedmessage_emails queue no longer is expected to grow a backlog
over time.
2019-04-13 20:43:07 -07:00
Puneeth Chaganti 9876f1b14e check_rabbitmq_queue: Fix the time period when we ignore long queues.
The commit 87d1809657 changed the time when
digests are sent by 3 hours to account for moving from the US East Coast to the
West Coast, but didn't change the time period exception in the
`check-rabbitmq-queue` script.

Closes #5415
2019-04-13 20:43:07 -07:00
Anders Kaseorg 460abe82df terminate-psql-sessions: Allow running as postgres user.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Anders Kaseorg 53879c4673 scripts: Rename DEFAULT_USER to POSTGRES_USER.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Anders Kaseorg d03b8308a5 backup: Use tar --transform to arrange the tarball instead of symlinks.
This allows tar to print the real paths in error messages if something
goes wrong.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-04-12 16:48:34 -07:00
Rohitt Vashishtha 22a3b4230c Revert "install: Ensure that sudo is installed."
Now that we have the run_as_root helper function, we don't need to
install sudo to run Zulip in production

This reverts commit a7d7d181ea.

Fixes #10036.
2019-04-04 16:52:20 -07:00
Rohitt Vashishtha 327b70cb19 terminate-psql-sessions: Allow running without sudo. 2019-04-04 16:52:20 -07:00
Tim Abbott 042893a448 configure-rabbitmq: Allow running as root.
This makes it possible to run this in production without access to
sudo.
2019-04-04 16:52:20 -07:00
Anders Kaseorg f78970c679 restore-backup: Set the recreated database owner to zulip.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-04 11:50:13 -07:00
Mayank Madan 264c2e9803 provision: Add provision support for Ubuntu 18.10.
Note that this is only support for Ubuntu 18.10 in the development
environment.
2019-03-09 16:01:46 -08:00
Tim Abbott 0ab8cd9ca3 provision: Refactor handling of dependencies for building pgroonga.
This lets us leave build-pgroonga platform-agnostic.
2019-03-08 15:12:56 -08:00
Tim Abbott 5c20ee998c build-tsearch-extras: Add support for Debian/Ubuntu. 2019-03-08 15:12:56 -08:00