0fb6779899
docker: Make user and dbname configurable in process_fts_updates.
2017-03-23 14:28:21 -07:00
17c8527856
nagios: Clean up check_send_receive_time arguments.
2017-03-15 12:52:27 -07:00
0ff1f3d663
nagios: Error on heartbeat events in check_send_receive_time.
...
It was probably going to fail anyway if those show up, but this
produces a clearer failure mode.
2017-03-15 12:51:19 -07:00
6a5e98b77e
puppet: Increase MaxStartups SSH configuration.
2017-03-08 22:28:16 -08:00
1e441b8d7c
uwsgi: Add master mode to the main uwsgi process.
...
- Enable `master` parameter for `uswgi` configuration.
It allows cleaning leaked processes if the parent
process is closed unexpectedly or with SIGKILL command.
Child processes follow to the master and kill themselves
after the main process.
Fixes #3855
2017-03-07 21:35:51 -08:00
6a801db1c2
missed-emails-sending: Move email sending to separate queue worker.
...
- Add new 'missedmessage_email_senders' queue for sending missed messages emails.
- Add the new worker to process 'missedmessage_email_senders' queue.
- Split aggregation missed messages and sending missed messages email
to separate queue workers.
- Adapt tests for sending missed emails to the new logic.
Fixes #2607
2017-03-07 20:08:40 -08:00
67219cf660
puppet: Use restart-server for weekly server restarts.
...
Using `supervisorctl restart all` carried longer downtime (since it
just restarts everything at the same moment) and was less under our
control; I'm not sure it had any advantages.
2017-03-05 11:36:10 -08:00
ab2ce9d5e9
lint: Fix whitespace error with recent mypy annotations.
2017-03-04 15:41:11 -08:00
75e81253f2
mypy: Work around several new mypy bugs in 0.501.
2017-03-04 15:33:39 -08:00
a3a03bd6a5
mypy: Added Dict, List and Set imports.
...
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
3d07ac0c49
Change timezone-naive datetimes to use timezone.now() where safe to do so.
...
Change timezone-naive datetimes to use timezone.now() in cases where there
is no change in behavior.
2017-03-01 22:54:28 -08:00
0218422e96
Use time.time() instead of datetime.now() to measure elapsed time.
...
Both because it is more idiomatic and because we will soon start enforcing
that all datetimes in Zulip are timezone aware.
2017-03-01 22:54:28 -08:00
aed3632cbb
puppet: Convert remaining queue workers to _ style.
2017-02-19 16:18:37 -08:00
4b54307d94
puppet: Generate most of main supervisor config with template.
2017-02-19 16:18:37 -08:00
ae09d55e46
puppet: Move zulip.conf to be a template file.
2017-02-19 16:18:37 -08:00
a2d935a2ee
puppet: Fix PostgreSQL user to create PGroonga extension
...
"root" user isn't a PostgreSQL administrator. "postgres" is a PostgreSQL
administrator.
2017-02-08 12:57:56 -08:00
fa8045a484
puppet: Add websockets Nagios test to configuration.
...
Since browser clients send messages via websockets and not the API,
this is an important element in making sure mission-critical Zulip
functionality is working.
2017-02-08 11:13:19 -08:00
8db13d0bb9
check_send_receive_time: Use a different state file for websockets.
...
Otherwise, the two Nagios checks will fight over the same state file
if both are in use.
2017-02-08 11:13:19 -08:00
ba5f454be5
puppet: Extract zulip::analytics.
...
I'm not altogether happy with this (a better solution would be
database-level locking), but I think it solves the immediate problem
of folks with 2 servers being very likely to run analytics on both of
them.
2017-02-07 12:29:15 -08:00
70388b17d2
puppet: Add missing dependency on ssl-cert.
2017-02-06 15:51:38 -08:00
76f3d02590
analytics: Add cron job to run analytics jobs.
...
This adds a cron job to update the Zulip analytics counts, complete
with locking etc.
Substantially tweaked by tabbott.
2017-02-01 17:02:46 -08:00
2fb51ff876
puppet: Use SIGINT to restart uwsgi.
...
This results in a brief service interruption (not a graceful restart),
but fixes a bug where on a `supervisorctl restart zulip-django`, we'd
end up leaking a bunch of uwsgi processes.
The mechanism was that sending SIGHUP to uwsgi was a command for it to
gracefully restart, so it'd start doing that (whereas supervisor
expected it to be dying)... and then supervisor would start up the new
uwsgi process group, resulting in 2 uwsgi process groups running.
This, in turn, led to a memory leak that could eventually result in
OOM kills.
2017-01-28 22:26:12 -08:00
36d54cf5ff
Replace references to zulip.com/dist with zulip.org/dist.
...
Now that zulip.org has all the files to distribute, there's no reason
to still point to the soon-to-be-decommissioned zulip.com/dist.
2017-01-28 17:56:25 -08:00
0ce29d7ad6
Remove some some duplicate words in copy.
2017-01-23 23:15:04 -08:00
4e171ce787
lint: Clean up E126 PEP-8 rule.
2017-01-23 22:06:13 -08:00
d6e38e2a5c
lint: Clean up E123 PEP-8 rule.
2017-01-23 21:34:26 -08:00
bde2da7dfd
lint: clean up PEP-8 W391 rule.
2017-01-23 20:39:02 -08:00
bbd853e208
puppet: Add redirect to https to zulip.org.
2017-01-22 21:52:50 -08:00
44776c43a1
puppet: Add configuration for zulip.org website.
...
This puppet configuration, plus cloning the zulip.github.io repo and
letsencrypt key setup, is all we need to run a zulip.org server.
2017-01-22 21:48:48 -08:00
9de75f5167
Fixed typos with separate
2017-01-12 04:52:05 -08:00
ff8639f9db
Fixed typos with threshold.
2017-01-12 04:50:20 -08:00
5008f45112
Fixed typo in munin.conf.erb
2017-01-12 04:49:19 -08:00
3e32102016
nagios: Fix various critical issues not tagged as pageable.
2017-01-06 21:49:20 -08:00
edebf7619b
puppet: Add PAM common_session disabling systemd-login.
...
This fixes a weird problem with systemd where logging into a server
via ssh frequently has a 15s+ lag.
2017-01-06 21:49:15 -08:00
93c2c19775
nagios: Increase process count limits.
2017-01-06 21:49:15 -08:00
2c6cb37385
munin: Add default munin configuration template.
2017-01-06 21:44:57 -08:00
9ab8e7ba34
nagios: Disable swap checks for servers with no swap.
2017-01-06 21:39:07 -08:00
3e01ed1f73
nagios: Increase NTP max_check_attempts.
...
NTP often suffers from brief interruptions of service that lead to
spurious Nagios alerts; it makes sense to suppress these.
2017-01-06 21:32:43 -08:00
e4420b08d2
zulip_ops: Disable unattended upgrades of security packages.
...
Since Zulip does not handle e.g. postgres server restarts gracefully,
it's best for a system administrator to manually trigger security
updates.
2017-01-06 21:30:56 -08:00
6f9c73d0e5
zmirror: Update Debathena release in configuration.
...
The zulip_ops configuration is now for xenial, not obsolete wheezy.
2017-01-06 21:30:41 -08:00
bd9176d1d9
nagios: Remove some default files.
...
Nagios ships with a bunch of default configuration files that one
needs to delete in order to configure it.
2017-01-06 21:25:12 -08:00
7083899e77
zulip_ops: Add postgres config for enabling Nagios.
...
The old zulip_ops Nagios configuration depended on Nagios having the
ability to login as the zulip user (with essentially full write
access); this configuration is helpful for limiting nagios to special
"nagios" user with more limited credentials.
2017-01-06 21:24:24 -08:00
204edb0f85
zulip_ops: Cleanup pg_hba.conf configuration.
2017-01-06 21:23:51 -08:00
30c57eb2ae
zulip_ops: Add basic .emacs for production.
2017-01-06 21:20:21 -08:00
eb87d04168
puppet: Remove xxxxx password hardcoding in recovery.conf.
2017-01-06 21:20:21 -08:00
6404a1a5ff
zulip_ops: Add nagios-plugins-contrib.
...
This has a number of useful nagios plugins.
2017-01-06 21:19:59 -08:00
f7b77008ef
zulip_ops: Add aptitude dependency.
...
This is useful for `aptitude why`.
2017-01-06 21:19:50 -08:00
2510a51a8a
zulip_ops: Add letsencrypt dependency.
2017-01-06 21:19:31 -08:00
65774e1c4f
zulip_ops: use check_postgres package from apt.
2017-01-06 21:18:55 -08:00
165b4d3126
nagios: Fix check_send_receive_time threshholds.
...
Previously, the CRITICAL state would never fire (because x > 6 =>
x > 3). Additionally, 6s is not so unusually high as to deserve being
immediately pageable.
2017-01-06 21:16:37 -08:00
0d8c18a6dd
nagios-plugins: Add websocket checking to nagios message sending test.
...
- Add websocket client to create connection with SockJS websocket server.
It contains callback method to launch after connection setup.
- Add '--websocket' parameter to 'check_send_receive_time' script to
check websocket connection.
- Add testing websocket connection to production installation checking.
- Add cronjob to launch websocket connection nagios test.
This makes it possible for Zulip Nagios monitoring to check for
problems impacting the websockets sending code path, which is what all
web users use.
2016-12-30 15:36:37 -08:00
336a041ac0
Django 1.10: Use uWSGI.
...
Fixes : #1121
With some tweaks by tabbott to make the number of processes configurable.
2016-12-13 21:40:43 -08:00
c93f1d4eda
Add oembed/Open Graph/Meta tags data retrieval from inline links.
...
This change adds support for displaying inline open graph previews for
links posted into Zulip.
It is designed to interact correctly with message editing.
This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.
By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.
Eventually, we may want to make this manageable via a (set of?)
per-realm settings. E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
144d82305d
mypy: Annotate puppet/zulip_ops.
2016-12-03 11:00:25 -08:00
adebc75740
pep8: Fix E502 violations
2016-12-03 10:56:36 -08:00
8c0c12c1d9
pep8: Fix E303 violations.
2016-12-02 15:34:11 -08:00
c5316b4002
lint: Fix E127 pep8 violations.
...
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
6b0df43463
pep8: Fix E125.
2016-11-30 20:03:29 -08:00
46b7d54b3e
pep8: Fix E701 violations.
2016-11-30 20:45:09 +00:00
7a2282986a
pep8: Fix E225 pep8 violations.
2016-11-28 15:21:15 -08:00
092fe4fecb
puppet: Write rabbitmq-env.conf before installing rabbitmq-server
...
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 18:54:12 -08:00
207cf6302b
Always start python via shebang lines.
...
This is preparation for supporting using Python 3 in production.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
1ea8abe493
Replace python -u with PYTHONUNBUFFERED=1
...
(Why is -u needed at all? I’m not sure, but test-run-dev spins forever
“Polling run-dev...” without it.)
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
d1dc2cf30e
Mark scripts executable
...
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
18e49bec3a
puppet: Add another missing dependency on postgresql-common
...
The postgres group must exist before we give files to it.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 11:23:32 -08:00
f78eb9d82d
puppet: Add missing dependency on postgresql-common.
...
postgresql-$(version) depends on postgresql-common depends on ssl-cert,
which creates the ssl-cert group.
2016-11-21 07:38:45 -08:00
2e65dc1206
puppet: make check_send_receive_time target host configurable.
2016-11-02 23:40:53 -07:00
9578908601
check_send_receive_time: Stop hardcoding EXTERNAL_API_URI.
2016-11-02 23:35:08 -07:00
eceaf36001
setup_disks: Fix postgres RAID setup to work correctly on Xenial.
...
Nobody's going to run this on Wheezy again.
2016-10-28 11:04:08 -07:00
9b7a3f040c
Remove now-unused /json/get_events endpoint.
2016-10-27 21:34:58 -07:00
4fbe201187
puppet: Automate autossh process monitoring maintenance.
...
Previously, the Zulip Nagios configuration effectively hardcoded the
count for how many system should have autossh connections.
2016-10-26 00:49:03 -07:00
6bdb10b71b
puppet: Update emacs dependency to emacs-nox metapackage.
...
This way, one doesn't need to keep updating the dependency every time
a new major emacs release comes out.
2016-10-26 00:42:22 -07:00
11b5d203f7
sshd_config: Increase MaxStartups.
...
This fixes connection problems when using the full Zulip recommended
Nagios configuration against a given server.
2016-10-26 00:41:03 -07:00
73f54dd0cb
sshd_config: Add updates from Xenial upstream.
...
It seems worth updating this to match the Linux distro this
configuration targets.
2016-10-26 00:40:44 -07:00
0a5a2c4eda
nagios: Automate authorized users list maintenance.
2016-10-26 00:37:29 -07:00
fa4998db59
puppet: Add zulip_zephyr_mirror plugins.
2016-10-26 00:35:57 -07:00
ac4f28050c
zmirror: Remove unnecessary krb5-clients dependency.
...
I'm pretty sure krb5-clients isn't needed to run the Zephyr mirroring
service.
2016-10-26 00:35:11 -07:00
d490e83645
puppet: Upgrade nagios cgi.cfg with modern defaults.
2016-10-26 00:31:41 -07:00
1159ad4857
puppet: Upgrade nagios.cfg with modern defaults.
2016-10-26 00:31:41 -07:00
73178e5e5a
puppet: Run check_send_receive_time via a cron job.
...
This allows the actual nagios work involved with
check_send_receive_time nagios checks to be done by an unprivileged
"nagios" user rather than the "zulip" user.
2016-10-26 00:26:52 -07:00
96cf330649
puppet: ssh as the nagios user instead of zulip user.
...
This is a follow-up to 4f58fef54b
2016-10-26 00:23:47 -07:00
a350d43683
puppet: Add recovery.conf configuration to postgres_slave.pp.
...
This file is needed to run a valid postgres slave; it's not clear why
this wasn't installed in the original zulip.com configuration.
2016-10-26 00:22:57 -07:00
c3727c9886
nagios: Remove old zulip.com trac/git/replica servers.
...
These are unlikely to be relevant to anyone.
2016-10-26 00:21:53 -07:00
383f39b543
nagios: Enable allow_empty_hostgroup_assignment.
...
This fixes the configuration being broken when we remove some of the
old zulip.com hosts that are unlikely to be of interest to anyone.
2016-10-26 00:19:21 -07:00
4f58fef54b
zulip_ops: Use nagios user for all Nagios checks.
...
There's no reason these Nagios checks needs to run as the
semi-priviliged Zulip user.
2016-10-26 00:17:26 -07:00
32d244dbe5
puppet: Add Nagios checks for other consumers.
2016-10-26 00:11:08 -07:00
f1fa4397f3
puppet: Fix package deps for zulip-ec2-configure-interfaces.
2016-10-26 00:11:08 -07:00
3448ab4c7a
zulip-ec2-configure: Fix network IDs for Ubuntu Xenial.
2016-10-26 00:11:08 -07:00
91da4bd59b
puppet: Add check_cron_file generic helper.
2016-10-26 00:11:08 -07:00
080dd8c987
nagios: Ignore kthreads in check_procs tests.
...
Modern Linux can have a lot of kernel threads not doing anything.
Since this isn't interesting from a monitoring perpsective, we ignore
these.
2016-10-26 00:10:40 -07:00
4c9a283542
puppet: Remove configuration for old builder host.
...
I don't think this configuration was ever even used; it's just
clutter.
2016-10-26 00:01:52 -07:00
f9ad75f58e
puppet: Remove configuration for old zulip.com bots host.
...
This configuration didn't do anything anyway and just clutters the
repo.
2016-10-26 00:01:29 -07:00
9d4f3f1e1b
puppet: Replace zulip_ops postgres configs with postgres_appdb_tuned.
...
There's no longer a reason to have copies of forked postgres
configuration files in our repository, since some time ago we merged
the features of these configuration files into the main
postgres_appdb_tuned.pp.
2016-10-25 23:58:53 -07:00
105ea972f6
puppet: Remove now-unncessary kernel.shm sysctl values.
...
With modern Linux and postgres, these settings are not required.
2016-10-25 23:58:33 -07:00
2227e77cce
puppet: Remove Dropbox usernames from Nagios config.
2016-10-25 23:55:42 -07:00
8584c05d80
zulip_ops: Remove unnecessary loadbalancer stanzas.
2016-10-25 23:52:37 -07:00
624ee3989f
puppet: Remove old Dropbox certificates.
2016-10-25 23:52:30 -07:00
f0bb78ba2d
puppet: Fix iptables-persistent->netfilter-persistent rename.
2016-10-25 23:45:21 -07:00
c4ca7ee6e1
puppet: Move Apache sites files to correct paths.
...
Apache now actually requires its configuration files have names ending
with .conf.
2016-10-25 23:44:28 -07:00
2b8324b778
emoji: Fix caching permissions issues.
...
Previously, you needed to be root to update the emoji cache, which
caused problems with how Zulip is upgraded in production.
2016-10-25 17:52:19 -07:00
8e82257444
Fix node_cache code to not require root.
...
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
56c0b80067
nginx: Fix JavaScript not being compressed properly.
...
Apparently, we weren't actually compressing our JavaScript being sent
over the wire due to incorrect nginx configuration.
2016-10-23 20:06:02 -07:00
a5a03c2e0b
zulip_ops: Include zulip::apt_repository.
...
This replaces the old wheezy configuration.
2016-10-16 20:13:35 -07:00
8c68c6f09b
zulip_ops: Remove wheezy apt repo.
...
Nobody uses wheezy anymore, and the configuration wasn't even
conditional on the OS version.
2016-10-16 20:13:35 -07:00
5210b0a6a4
zulip_ops: Cleanup old redis configuration.
...
One can now just use the improved configuration we've merged into the
main Zulip repo.
2016-10-16 20:13:35 -07:00
869f0724ce
zulip_ops: Remove humbughq.com nginx configuration.
...
The humbughq.com name hasn't been the product's name since 2013, and
it's nice to finish clearing it out of the repository.
2016-10-16 20:13:29 -07:00
29448fb47b
zulip_ops: Remove old Zulip, Inc. trac configuration.
...
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
771e03cfa7
zulip_ops: Remove old Zulip, Inc. mediawiki configuration.
...
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
36e336edc3
puppet: Rename zulip_internal to zulip_ops.
...
The old "zulip_internal" name was from back when Zulip, Inc. had two
distributions of Zulip, the enterprise distribution in puppet/zulip/
and the "internal" SAAS distribution in puppet/zulip_internal. I
think the name is a bit confusing in the new fully open-source Zulip
work, so we're replacing it with "zulip_ops". I don't think the new
name is perfect, but it's better.
In the following commits, we'll delete a bunch of pieces of Zulip,
Inc.'s infrastructure that don't exist anymore and thus are no longer
useful (e.g. the old Trac configuration), with the goal of cleaning
the repository of as much unnecessary content as possible.
2016-10-16 19:23:27 -07:00
2e50dce9dd
check_queue_worker_errors: Don't import zproject.settings directly.
2016-10-15 22:53:28 -07:00
913d913c90
puppet: Annotate Nagios plugins.
2016-10-15 22:44:15 -07:00
458e455b75
Annotate process_fts_updates.
2016-10-15 22:44:15 -07:00
a4b2a6b6d4
puppet: Fix missing dependency for supervisor enabling.
...
You can't run `systemctl enable` on a service before the package
containing the service is installed.
2016-10-06 21:44:10 -07:00
058587da77
Remove extra new lines at the ends of Zulip authoried files.
...
Fixes #1627 .
[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
12ddb1a36f
puppet: Fix buggy logrotate configuration.
2016-09-08 16:51:17 -07:00
683f49aa99
Support full text search for all languages using pgroonga.
...
This adds support for using PGroonga to back the Zulip full-text
search feature. Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.
Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.
PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.
Fixes #615 .
[docs and tests tweaked by tabbott]
2016-08-26 21:04:03 -07:00
546c23fd1f
puppet: Activate pgroonga extension after installation.
...
The pgroonga extension needs to be activated using postgres root
access, so we make that happen from puppet.
2016-08-25 18:37:45 -07:00
9818a760b5
Install pgroonga in development and (optionally) in prod.
...
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
682d78bc30
puppet: Add default logrotate configuration.
...
This should prevent Zulip from eating all the disk now that we are
using the race-free WatchedFileHandler rather than
TimedRotatingFileHandler.
2016-08-24 14:30:54 -07:00
1c40df9363
static_asset_compiler: Use correct package name on trusty.
2016-08-22 23:47:03 -07:00
97b622dffa
puppet: Fix postgres_appdb_tuned manifest on trusty.
2016-08-22 23:41:07 -07:00
750e43518f
puppet: Ensure supervisord starts on boot with systemd.
...
This is important to ensure that Zulip comes up properly after a
reboot on Ubuntu Xenial.
2016-08-22 23:25:54 -07:00
2be597dd09
puppet: Move support for accepting a loadbalancer to zulip manifests.
...
Previously, we didn't have an easy process for how one should do this.
Now, it's an (undocumented) zulip.conf setting!
2016-08-22 17:19:03 -07:00
88a123d5e0
Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
...
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.
Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.
Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.
Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
852af83d3c
check_postgres_replication_lag: Fix psql command line.
...
This allows the plugin to be run as users other than the "zulip" user,
where the "zulip" database would not be the default.
2016-08-12 13:19:08 -07:00
6496fe2a53
travis: Remove rabbitmq nodename dependency on hostname.
...
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.
We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.
To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:
[rabbitmq]
nodename = zulip@localhost
However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer. So for now, this just
fixes our Travis CI problems.
Fixes : #1579 .
2016-08-12 09:38:23 -07:00
2581ac166c
puppet: Automatically scale default memcached memory allocation.
...
Previously, we used a fixed memcached memory allocation of 512MB,
regardless of the size of the server. While that is a good allocation
for a server with 4GB of RAM, for servers with less, we should
decrease the allocation, and for a large server with much more RAM, we
should increase it. We still support the user overriding the
configuration setting, but this produces more sensible defaults.
2016-08-11 20:27:45 -07:00
2e0e8193d4
puppet: Make memcached memory allocation a variable.
2016-08-11 20:27:45 -07:00
3edf880c9b
Remove moreutils from zulip_internal::base.
...
We recently added this to zulip::base, so it would be a duplicate.
2016-08-11 18:12:59 -07:00
9afb1c7a71
puppet: Stop wholesale replacing /etc/redis/redis.conf.
...
Zulip had only patches the redis configuration in one small way, which
resulted in unnecessary portability issues for using Redis on
different versions of Linux. We replace this with just a adding an
include mechanism to the redis config.
While we're at it, we configure this to take advantage of the
new REDIS_PASSWORD secret to automatically configure redis passwords.
2016-08-11 17:26:04 -07:00
cb21584ffe
check_send_receive_time: Support being run in a cron job.
2016-08-11 14:48:21 -07:00
4dcbaf1e6b
check_send_receive_time: Cleanup unnecessary print statement.
2016-08-11 14:48:21 -07:00
1d6ebd2b3d
check_fts_update_log: Fix psycopg2 arguments.
...
* Fixes passing a string argument rather than an actual Python
argument.
* Switches to hardcoding the database to connect to rather than the
user, so this check can be run as an arbitrary user.
2016-08-03 14:58:46 -07:00
1c65508624
puppet: Add postgres user to Zulip group.
2016-08-02 14:59:25 -07:00
04fc3ff1e1
pg_backup_and_purge: Don't try to use a virtualenv.
...
This makes this actually work if the postgres server is different from
the Zulip app frontend.
2016-08-02 14:59:25 -07:00
4bdd9d3769
check_postgres_backup: Don't try to use a virtualenv.
...
The dependencies for this tool are installed using apt.
2016-08-02 14:59:25 -07:00
a5115d54ee
env-wal-e: Rename s3_backup_bucket to s3_backups_bucket.
...
This makes it consistent with the other variables in this file.
2016-08-02 14:59:15 -07:00
ff80daef16
puppet: Move zulipsecret into its own file.
...
Apparently puppet doesn't support declaring multiple functions in the
same file.
2016-08-02 14:55:51 -07:00
75b5d021fa
Remove unused puppet-common third-party module code.
2016-07-31 19:24:42 -07:00
6158acb41b
nagios: Fix path to check_debian_packages.
2016-07-31 14:25:07 -07:00
6954dd84ab
postgres_appdb_tuned: Add SSL certificate path configs.
2016-07-31 14:25:07 -07:00
d66f6b8176
Fix postgresql configuration template for Ubuntu Xenial.
...
I incorrectly assumed that this was working because Travis CI passed
the build, whereas in fact Travis CI only tests the Trusty templates.
2016-07-31 14:24:34 -07:00
8b285ec0ff
puppet: Read camo key from zulip-secrets.conf.
2016-07-31 00:23:24 -07:00
2b40309029
puppet: Add new zulipsecret function.
2016-07-31 00:23:24 -07:00
298e040bac
puppet: Strip newlines in zulipconf function.
...
This avoids creating unnecessary extra newlines in config generated
using this.
2016-07-31 00:23:24 -07:00
16a4ce1bd2
puppet: Fix Nagios check_disk flagging special filesystems.
2016-07-30 23:57:31 -07:00
c764b46cef
puppet: Fix missing base include in nagios manifest.
2016-07-30 23:48:41 -07:00
bc9bdd53aa
puppet: Use hostgroup for postgres_other monitoring.
2016-07-30 23:46:05 -07:00
6b34bee806
puppet: Remove unused ancient humbug-default apache site.
2016-07-30 23:43:55 -07:00
217faed3b3
Update check_postgres plugin to version 2.22.
2016-07-30 23:19:40 -07:00
1f549dcfab
check_fts_update_log: Fix use on non-appbd machines.
2016-07-30 21:48:17 -07:00
35edd8c9dd
puppet: Move crudini dependency to base.pp.
...
This allows us to use the new crudini-based zulipconf function in any
of our manifests in the future.
2016-07-30 21:23:12 -07:00
922a4acdc5
puppet: Add zulip.conf setting for listen_addresses tuning.
2016-07-30 21:23:12 -07:00
b2e32e6e3f
puppet: Add postgres replication option for postgres_appdb_tuned.pp.
...
This adds a setting that controls whether to include the Zulip default
streaming replication configuration in the postgres configuration file.
2016-07-30 21:23:12 -07:00
3ab0295061
puppet: Add zulip.conf setting for effective_io_concurrency tuning.
2016-07-30 21:23:12 -07:00
f3db368a3c
puppet: Add zulip.conf setting for random_page_cost tuning.
2016-07-30 21:23:12 -07:00
63c757eac3
puppet: add zulipconf function for reading settings from zulip.conf.
...
This makes it convenient for us to have optional user-defined settings
in the main Zulip puppet configuration.
2016-07-30 21:23:12 -07:00
3b3b5c7c16
Rename management command email-mirror to email_mirror.
...
All other zulip management command names have underscores, so
rename email-mirror to email_mirror.
This will also make it possible to import this module, which will
help in writing tests for it.
2016-07-28 14:52:09 -07:00
69528790a5
check_worker_memory: Fix handling of no queue workers running.
2016-07-28 13:26:31 -07:00
408d070170
puppet/: Make subprocess calls unicode-aware.
2016-07-26 12:06:41 -07:00
3d87c376c2
Make zulip-ec2-configure-interfaces pass mypy check.
2016-07-24 12:52:58 +05:30
0a5aa2ccc7
Make check_send_receive_time pass mypy check.
2016-07-24 12:41:06 +05:30
2a209e46dd
puppet: Fix remaining hardcoding of postgres 9.1 in manifests.
2016-07-20 21:11:22 -07:00
d529bc12ef
pg_backup_and_purge: Fix hardcoding of postgres version.
2016-07-19 19:19:42 -07:00
bdb1ce04a2
puppet: Move pg_backup_and_purge to main config.
2016-07-19 19:19:42 -07:00
81136ff092
env-wal-e: Eliminate hardcoding of AWS keys.
...
Pre-Zulip being open sourced, this file just had the AWS keys for
backups hardcoded.
Instead, these are simply read from zulip-secrets.conf.
2016-07-19 19:19:42 -07:00
3023745ed7
puppet: Move env-wal-e to Zulip main distribution.
...
This doesn't yet include the dependencies for env-wal-e.
2016-07-19 19:19:42 -07:00
d0dcc8bf26
puppet: Create static_asset_compiler manifest.
2016-07-19 16:38:09 -07:00
601ab24d2a
zulip_internal::app_frontend: Remove now-unnecessary dependencies.
...
These Python packages are no longer relevant, since we install all of
our Python dependencies via virtualenvs.
2016-07-19 16:37:50 -07:00
77ccc0d87f
Patch httpoxy security issue.
2016-07-18 10:24:03 -07:00
da36947400
Change unbuffering strategy in runtornado.py.
...
runtornado unbuffers its output using
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0).
This is not python 3 compatible since we can't specify
buffering on a text stream in python 3. So use the '-u'
option of python when calling runtornado.py to make output
unbuffered.
2016-07-17 10:31:15 -07:00
ace8ae8301
puppet: Fix supervisor restart rule catching itself in pgrep.
2016-07-12 10:39:33 -07:00
dfc9e75342
nginx: Enable shared SSL session cache.
2016-07-10 08:30:10 -07:00
ee47da8790
nginx: Disable RC4 and prefer server cipher order.
2016-07-10 08:30:05 -07:00
ea089518ee
Add Zulip customizations to Postgres 9.5 config file.
...
This includes reasonable tuning of memory usage parameters based on
the values that pgtune would use, roughly.
2016-07-08 16:16:12 -07:00
e818dff0b0
postgres: Remove comment matching Puppet ERB syntax.
2016-07-08 16:16:12 -07:00
6ba659aeec
Add a stock Ubuntu Xenial Postgres 9.5 configuration file.
2016-07-08 16:16:12 -07:00
0f5e62e994
puppet: Don't use pgtune on Linux versions where it doesn't exist.
2016-07-08 16:16:12 -07:00
903f728587
Scope postgresql.conf templates by postgres version.
2016-07-08 16:16:12 -07:00
ffe79e0d50
Fix EPMD restart being attempted on every puppet apply.
2016-07-08 16:16:11 -07:00
a137bf15ed
Wrap some lines with length greater than 120.
...
With some tweaks by tabbott.
2016-07-06 14:35:16 -07:00
809a45394f
puppet: Start supervisord if it isn't running on restart.
2016-06-28 22:09:36 -07:00
7a98d7bd24
manifests/base.pp: Remove apt python package dependencies.
2016-06-27 19:55:35 -07:00
ebad1415c0
app_frontend_base.pp: Remove apt python package dependencies.
2016-06-27 19:55:35 -07:00
a9835c0ab2
Activate virtualenv in production Python code.
...
The manage.py change effectively switches the Zulip production server
to use the virtualenv, since all of our supervisord commands for the
various Python services go through manage.py.
Additionally, this migrates the production scripts and Nagios plugins
to use the virtualenv as well.
2016-06-27 19:55:35 -07:00
46c2416cc8
puppet: Add postgres version map for xenial.
...
This is an early step towards Ubuntu Xenial support in production.
2016-06-17 16:41:42 -07:00
94e4b39112
Replace python2.7 by python everywhere.
2016-05-29 05:03:08 -07:00
149938d468
Change shebangs from python2.7 to python.
2016-05-29 05:03:08 -07:00
b6bd5445bc
End puppet/.../process_fts_updates gracefully.
...
Catch KeyboardInterrupt in puppet/zulip/files/postgresql/process_fts_updates,
print a message and exit.
2016-05-24 13:28:05 -07:00
3441f0848c
Annotate pg_backup_and_purge.py.
2016-05-18 17:10:17 +05:30
5359e6b0d4
Convert Zulip to use Jinja2 templates.
...
This results in a substantial performance improvement for all of
Zulip's backend templates.
Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.
Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`
Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
instead of `RequestContext` object. There are two reasons to do this:
1. `RequestContext` is not compatible with Jinja2
2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
includes a custom backend and a template renderer, compressors for js
and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.
Fixes #620 .
2016-05-09 09:55:18 -07:00
00ccf147cd
check_nagios_send_time: Add --insecure option for use in tests.
2016-05-08 17:35:50 -07:00
c0d38f42f1
supervisor: Use 127.0.0.1 rather than localhost for tornado.
...
In theory these should be the same, but in misconfigured environments
(such at Travis CI) where /etc/hosts has multiple entries for
"localhost", 127.0.0.1 is safer than "localhost".
2016-05-08 17:35:50 -07:00
cb81a59e38
Move write-rabbitmq-consumers-state-file to scripts/nagios/.
2016-05-07 19:37:06 -07:00
2761c012e5
Move rabbitmq consumer checks from bots/ to scripts/nagios/.
2016-05-07 19:37:06 -07:00
be6566dc5c
nagios: Move cron_file_helper from bots/ to scripts/lib.
...
This ensures the tool is available in Zulip production deployments.
2016-05-07 19:37:06 -07:00
52c1e8ac7d
Run a local camo server in voyager production environments.
...
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS. We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
2016-05-02 17:21:31 -07:00
5195d1ecb7
Fix missing newlines at ends of JSON files.
2016-04-14 10:48:52 -07:00
caba24b2af
Fix existing trailing whitespace in bash scripts.
2016-04-08 11:52:11 -07:00
a7f83c9e05
Fix check_postgres_replication_lag nagios command.
2016-04-06 15:30:51 -07:00
991341867c
nagios: Remove unnecessary dependency on netcat.
2016-04-06 15:30:40 -07:00
c92221dcd3
Remove old humbughq apache configuration.
2016-04-06 15:30:36 -07:00
4855296771
puppet: Migrate check_postgres plugins to postgres_common.pp.
2016-04-06 15:20:36 -07:00
69a8925076
check_user_zephyr_mirror_liveness: Fix importing settings.
2016-04-05 13:27:04 -07:00
55172e2e0c
Remove old zulip_internal nagios_plugins installation.
2016-04-05 13:27:04 -07:00
934e8641ee
Migrate Zephyr mirror Nagios plugins to subdirectory.
2016-04-05 13:27:04 -07:00
7b753e5882
Migrate check_debian_packages to zulip/.
2016-04-05 13:27:04 -07:00
2da9fc56d6
Migrate check_pg_replication_lag to zulip/.
2016-04-05 13:27:04 -07:00
c2e210ca0d
Migrate check_website_response.sh to new zulip::nagios.pp.
2016-04-05 13:27:04 -07:00
eb72cecd9e
Migrate check_fts_update_log to zulip/.
2016-04-05 13:27:04 -07:00
92d696d007
Migrate check_postgres plugins to zulip/.
2016-04-05 13:27:04 -07:00
e155ecdc49
Migrate check_rabbitmq plugins to zulip/.
2016-04-05 13:27:04 -07:00
3ed7d658f8
Migrate check_send_receive_time to zulip/.
2016-04-05 13:27:04 -07:00
ca45ec3f3f
Migrate check_email_deliverer plugins to zulip/.
2016-04-05 13:27:04 -07:00
4e10424512
Migrate check_worker_memory to zulip/.
2016-04-05 13:27:04 -07:00
59b46278be
Move check_queue_worker_errors into subdirectory.
...
This fixes an issue where this worker wasn't even being installed
properly in a way that sets us up for doing further reorganization of
the Zulip Nagios plugins.
2016-04-05 13:27:04 -07:00
6f20c43097
Move dependency on nagios_plugins into base.pp.
2016-04-05 13:27:04 -07:00
2fe0700f55
Update memcache -> remote cache in inline documentation.
2016-03-31 12:56:42 -07:00
8c0b110e9a
Add python-typing dependency.
2016-03-30 21:50:25 -07:00
c16749d783
Add missing dependency on netcat in both dev and prod.
...
Fixes #474 .
2016-03-29 21:30:48 -07:00
2adf6d822f
puppet: Fix process_queue command lines to use the new argument style.
...
cd2348e9aeFixes #586 .
2016-03-27 23:17:16 -07:00
965f923ac3
Remove postgres2 configuration
2016-03-23 20:41:42 -07:00
ae2560a027
Add postgres3 configuration
2016-03-23 20:41:25 -07:00
6137ae9902
Fix incorrect shell quoting in check_worker_memory.
2016-03-23 20:40:06 -07:00
210c2897e7
Fix check_worker_memory regular expression.
2016-03-23 20:40:02 -07:00
b06739df11
Move email digest triggering to default zulip config.
...
Previously, even though the Zulip digest emails were documented in the
settings, the cron job to run the script that actually sends the daily
digest emails wasn't included in the non-zulip.com part of the Zulip
production distribution. The overall consequence is that digest
emails didn't work for non-zulip.com users. This fixes that issue by
moving that cron job into the zulip manifests.
[commit message details expanded by tabbott]
2016-03-19 10:34:41 -07:00
37f9520666
Make the remaining ambiguous divisions python 3 compatible.
...
Refer to #256
2016-03-12 10:53:51 -08:00
b9e792c4e6
Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six
...
Refer to #256
2016-03-10 22:03:58 -08:00
d3b63f9a2d
Apply Python 3 futurize transform libmodernize.fixes.fix_file
...
Refer to #256
2016-03-10 22:02:34 -08:00
c59185e119
Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import
...
Refer #256
2016-03-10 22:02:17 -08:00
e2d5ec1868
Apply Python 3 futurize transform lib2to3.fixes.fix_ws_comma
2016-03-10 22:02:04 -08:00
4fb549abe8
Apply Python 3 futurize transform lib2to3.fixes.fix_idioms
...
Refer to #256
2016-03-10 22:02:01 -08:00
f3d387e727
Apply Python 3 futurize transform lib2to3.fixes.fix_except
2016-03-10 22:01:50 -08:00
4887a79d21
Make the nginx log directory owned by zulip
...
This is required to make log2zulip not error out when reading the nginx
error.log.
2016-02-02 23:05:44 -08:00
757e89260e
Migrate use of StringIO to Python 2+3 compatible six.moves.cStringIO.
...
And add a check for additional usage of the original StringIO module.
2016-01-26 21:09:43 -08:00
1c73c992dd
Fix missing puppet dependencies on postgres package.
2016-01-26 20:32:33 -08:00
2e16b44b24
puppet: Use $postgres_version in postgres template.
...
This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.
2016-01-26 20:32:33 -08:00
806aa986b7
puppet: Use a variable to configure the postgres version.
2016-01-26 20:32:33 -08:00
a3ac56efe2
puppet: Make apt repository conditional on the Ubuntu version.
...
We still will need to address this in the install script as well.
2016-01-26 20:32:33 -08:00
f6c59feb05
Document the puppet configuration somewhat in zulip::voyager.
2016-01-26 20:32:33 -08:00
345b5254d7
puppet: Move default nginx configuration out of voyager.pp.
2016-01-26 20:32:33 -08:00
dd61e3f97d
puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.
2016-01-26 20:32:33 -08:00
c3153274c1
puppet: Move memcached into its own puppet module.
2016-01-26 20:32:33 -08:00
8a0e07fe1a
puppet: Rename app_frontend.pp to app_frontend_base.pp.
...
This will enable us to move the remaining app-frontend related content
out of voyager.pp.
2016-01-26 20:32:33 -08:00
91286d00aa
puppet: Move Zulip apt repository to its own manifest.
2016-01-26 20:32:33 -08:00
69dd17dfb6
puppet: Move prod-static creation from voyager.pp to app_frontend.pp.
...
Every app frontend will need this directory and this should help
enable more modular puppet rules.
2016-01-26 20:32:33 -08:00
702f501638
puppet: Move tuned postgres configuration out of voyager.pp.
...
This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.
2016-01-26 20:32:28 -08:00
d5f04bd20b
Rename zulip::postgres_appdb to zulip::postgres_appdb_base.
...
The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.
2016-01-26 20:30:12 -08:00
3f27573cb2
puppet: Move several debugging tool dependencies out of base.pp.
2016-01-26 20:30:11 -08:00
2b0394d807
Add documentation explaining what process_fts_updates does.
2016-01-11 08:56:03 -08:00
Alexander Trost
Tim Abbott
K.Kanakhin
Raghav Jajodia
Rishi Gupta
Kouhei Sutou
umkay
Eitan Adler
JefftheBest1
Tim Abbott
Umair Khan
Igor Tokarev
Jason Le
bulat22101
Sidhant Bhavnani
Rafid Aslam
Bickio
Tommy Ip
Anders Kaseorg
Sahil Dua
Eklavya Sharma
Alex Gaynor
Taranjeet
Ryan Moore
Zev Benjamin
Josh Mandel