Commit Graph

931 Commits

Author SHA1 Message Date
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Alex Vandiver 8b1d49dbc7 puppet: Rename "wiki" realm to "monitoring".
This is vestigial.

It requires manually altering the `htdigest` file (not stored in this
repo) to change the digest realm from `wiki` to `monitoring`, and will
re-prompt users for their passwords if the browsers currently store
them.
2020-05-30 12:26:21 -07:00
Alex Vandiver b33aa8da7f postgresql: Update setup-disks to use `service postgresql`.
Using `service postgresql` makes it no longer linked to the specific
version/cluster that is on the host.
2020-05-30 12:14:24 -07:00
Alex Vandiver 4e370cda75 postgresql: Update setup-disks to drop /mnt disabling.
Hosts do not start out with a `/mnt`; there is no need to disable it.
2020-05-30 12:14:24 -07:00
Alex Vandiver a7d85b7e69 postgresql: Update setup-disks to not move /tmp.
Drop the change to move `/tmp` onto the local disk.  Doing this move
confuses `resolved` until there is a restart, and has no clear
benefits.  The change came in during bf82fadc95, but does not describe
the reasoning; it is particularly puzzling, since postgresql stores
its temporary files under `$PGDATA/base/pgsql_tmp`.
2020-05-30 12:14:24 -07:00
Alex Vandiver 481613a344 postgresql: Update setup-disks to not use RAID.
Do not RAID the disks together.  This was previously done when they
were spinning media, for reliability; running them on an SSD obviates
this sufficiently.  This means that updating the initramfs is also not
necessary.
2020-05-30 12:14:24 -07:00
Alex Vandiver b537563bc1 postgresql: Set the current primary host. 2020-05-30 12:14:24 -07:00
Alex Vandiver ad2918ea51 puppet: Remove `postgres_other` nagios hostgroup.
This no longer has any rules specific to it.  We leave the `postgres`
munin group (which now only contains `postgres_appdb`) as
future-proofing, and so that `postgres_appdb` matches to the puppet
manifest of the same name.
2020-05-28 17:24:35 -07:00
Alex Vandiver 2c73fbdcb6 puppet: Remove munin monitoring for no-longer-used "postgres_other".
The `wiki` and `trac` products are no longer used.
2020-05-28 17:24:35 -07:00
Tim Abbott 0b93e09e72 puppet: Add nginx configuration for blog.zulip.org move. 2020-05-26 14:47:05 -07:00
Anders Kaseorg f5b33f9398 python: Further pyupgrade changes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 333f7d16c9 logging: Pass more format arguments to logging.
Commit bdc365d0fe (#14852) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
Anders Kaseorg 824d97987b process_fts_updates: Use cursor.execute correctly.
Commit b501d04f6a (#14841) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
arpit551 439f0d3004 install: Ad production support for Zulip on Ubuntu Focal.
Install script now runs on Focal.  Python 2 is now installed via the
`python2` package in Focal.
2020-05-25 16:58:42 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-05-20 13:47:20 -07:00
Tim Abbott cdd3b7efbc tornado: Configure upstreams for TORNADO_PROCESSES. 2020-05-20 13:43:48 -07:00
Tim Abbott c3d3324295 puppet: Add link to the sources for Zephyr patches. 2020-05-19 20:54:11 -07:00
Tim Abbott a35e71ebbc puppet: Update package name for boto-on-python3.
The python3-boto3 package is the maintained fork that supports Python
3; it was renamed in Ubuntu Bionic from the original Ubuntu Xenial name.
2020-05-19 20:25:11 -07:00
Tim Abbott 1c28770810 puppet: Fix apt_repo_debathena setup_file path.
There was a typo introduced here when scripts_path was added.
2020-05-19 20:21:30 -07:00
Tim Abbott c43b3d95e2 puppet: Switch env-wal-e to use wal-g rather than wal-e.
wal-g is the modern reimplementation of wal-e that supports current
postgres.  It requires a bit of extra configuration to specify the AWS
region.
2020-05-15 16:45:36 -07:00
Anders Kaseorg fcca4a38b6 puppet: Work around memcached SASL configuration path bug.
memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.

https://bugs.launchpad.net/ubuntu/+source/memcached/+bug/1878721

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-14 23:25:24 -07:00
Tim Abbott b3c5f2c13e puppet: Remove check_postgres_replication_lag hostname hardcoding.
Since this runs on the Nagios server, which already has the relevant
hostnames defined in zulip.conf, we can just read it from there.
2020-05-11 23:42:36 -07:00
Tim Abbott 225bbf3633 puppet: Update check_postgres_replication_lag for postgres 10.
These functions were renamed in postgres 10.
2020-05-11 15:59:23 -07:00
Tim Abbott d8ea649869 puppet: Cast tornado_processes to Integer.
This is the latest mechanism in puppet for turning a string into an
integer.

We update an adjacent comment while we're at it.
2020-05-11 00:54:48 -07:00
Tim Abbott 6319c181eb puppet: Use actual name for the bind9-host package.
Using the `host` virtual package confused Puppet into reporting it was
doing work every time one did a puppet run, resulting in unnecessarily
spammy output.
2020-05-11 00:51:53 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Tim Abbott 21a04e2dbc puppet: Use nice to deprioritize various processes.
Our priority hierarchy is:
(1) Tornado and base services like memcached, redis, etc.
(2) Django and message sender queue workers.
(3) Everything else.

Ideally, we'd have something a bit more fine-grained (e.g. some queue
workers are potentially in the sending path, while others aren't), but
this should have a big impact on ensuring Tornado gets the resources
it needs during load spikes.

I think this has a good chance of causing some load spikes that would
previously have resulted in a user-facing delivery delays no longer
having any significant user-facing impact.
2020-05-10 23:28:25 -07:00
shubhamgupta2956 9cd8644c7c uploads: Add support for ".jpe" file extension.
Currently when the user uploads files with ".jpe" file extension, the
markdown is converted to link but the image is not embedded.

This commit adds the support for ".jpe" file extension.

Fixes #14863
2020-05-10 22:55:52 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Anders Kaseorg 708c6f4f11 puppet: Finally vanquish the cursed integer conversion conditional.
We no longer support Puppet 3.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Tim Abbott 50d8d61d3c puppet: Remove unnecssary/broken ;.
This breaks the Xenial build, which we're removing soon, but it's
unnecessary in any case.
2020-05-07 16:23:37 -07:00
Tim Abbott 03991d098a puppet: Add optional postgres version override.
This makes it convenient to run an alternative postgres version.
2020-05-07 09:33:24 -07:00
Mateusz Mandera 4643e48f60 retention: Add a daily cron job.
This will run archive_messages management command at 6am every day, 1
hour after soft_deactivate_users (which runs at 5am).
2020-05-05 10:11:38 -07:00
Tim Abbott 4034f6f99e nagios: Fix check_postgres_replication_lag.
This expects to be run outside a virtualenv and thus without
typing_extensions available.
2020-05-03 00:14:54 -07:00
Tim Abbott 4f3976b917 process_fts_updates: Clean up logging output.
This saves a couple lines of spammy output in the run-dev.py startup
experience, and will be better output in production as well.
2020-05-01 11:51:20 -07:00
Anders Kaseorg c0ffa71fa9 nginx: Replace unanchored regexes in location directives.
We could anchor the regexes, but there’s no need for the power (and
responsibility) of regexes here.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 16:58:19 -07:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg 09ea778db1 nginx: Listen for ACME challenges on port 80 too.
This should make Certbot renewals more reliable.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-23 16:22:04 -07:00
Aman Agrawal 2dc6d09c2a python3-upgrade: Move python2 scripts to run on python3. 2020-04-22 16:13:15 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
pemontto fd34bc5161
puppet: Allow /etc/zulip to be a symlink.
This PR updates the puppet manifest to allow /etc/zulip to be a symlink. The current behaviour overwrites /etc/zulip if it is link to another directory, which is problematic with docker-zulip and 
in particular the `LINK_SETTINGS_TO_DATA` setting.
2020-04-17 12:45:05 -07:00
Tim Abbott 777a3b6c18 puppet: Fix nagios check to not require typing_extensions. 2020-04-16 17:56:05 -07:00
Tim Abbott e1ce53ac46 puppet: Update nagios checks for disk to exclude kernel filesystems.
The fact that we have to explicitly list these is almost certainly a
bug in check_disk, but at least this works.
2020-04-16 17:49:29 -07:00
Tim Abbott cfbb617f5c puppet: Update nagios configuration for checking local disk. 2020-04-16 17:48:36 -07:00
Tim Abbott 9821dfa9fc puppet: The letsencrypt package is debian is now certbot.
It was an alias starting with Ubuntu Xenial, and will eventually be
removed.
2020-04-16 17:30:01 -07:00
Tim Abbott 8e5a866122 puppet: Update tuning for load average monitoring. 2020-04-16 16:47:05 -07:00
Tim Abbott b1ff823798 puppet: Remove old zulipbot configuration.
We haven't used zulipbot hosted here for years.
2020-04-16 16:18:48 -07:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 00:48:02 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00