Commit Graph

1011 Commits

Author SHA1 Message Date
Tim Abbott 0d35bbc464 install: Install the wget package.
We depend on it for installing node, and it's a standard package, not
a required one, so we do need to explicitly declare the dependency.
2018-03-29 16:03:44 -07:00
Ghislain Antony Vaillant 00dd86967b puppet: Add Debian 9 package names to definitions.
This doesn't add support for Debian 9, but it will save time for folks
working on Stretch support in the future.
2018-03-28 12:33:45 -07:00
neiljp (Neil Pilgrim) 090b47ed19 mypy: Add explicit Optional for default=None parameters in various files. 2018-03-28 12:31:51 -07:00
neiljp (Neil Pilgrim) f32f3cbf72 mypy: Amend zulip-ec2-configure-interfaces to avoid None. 2018-03-23 11:39:54 -07:00
Tim Abbott d98be2f19f puppet: Only run analytics Nagios checks on machine running cron.
Running this on additional machines would be redundant; additionally,
the FillState checker cron job runs only on cron systems, so this will
crash on other app frontends.
2018-03-06 13:38:27 -08:00
Tim Abbott 8e8faab006 puppet: Move clearsessions cron job to app_frontend_once.
While this is a different system than I'd written up in #8004, I think
this is a better solution to the general problem of cron jobs to run
on just one server.

Fixes #8004.
2018-03-06 13:35:51 -08:00
Tim Abbott 9a74ef5056 puppet: Move some cron jobs to app_frontend_once.pp.
Several cron jobs had incorrectly ended up in the app_frontend.pp
template, and thus would only run on voyager instances.
2018-03-06 13:35:51 -08:00
Tim Abbott 3ae645ed12 puppet: Rename analytics.pp to app_frontend_once.pp. 2018-03-06 13:35:51 -08:00
Tim Abbott ad7f38ab3b puppet: Move analytics cron job to analytics.pp.
This better groups it with the related code.
2018-03-06 13:35:51 -08:00
Tim Abbott 24b6106c9c puppet: Dsiable checking for evictions in memcached nagios.
Zulip's caching model for message history is such that it is normal
and healthy for there to eventually be a nontrivial volume of
evictions.
2018-03-06 13:34:02 -08:00
Greg Price 4475950ddf queue: Restore prematurely-cut upgrade path.
Revert c8f034e9a "queue: Remove missedmessage_email_senders code."
As the comment in the code says, it ensures a smooth upgrade path
from 1.7.x; we can delete it in master after 1.8.0 is released.
The removal commit was merged early due to a communication failure.
2018-02-28 11:15:53 -08:00
Tim Abbott 65767e5226 localhost_sso: Fix missing enabling of mod_wsgi.
This is apparently required on Ubuntu Xenial, at least.
2018-02-22 10:09:29 -08:00
Umair Khan c8f034e9a0 queue: Remove missedmessage_email_senders code.
After 68513952fb, all emails are sent through email_senders queue.
This commit removes code related to the legacy queue.
2018-02-21 16:43:56 -08:00
Aditya Bansal efe8545303 local-uploads: Start running authentication checks on file requests.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.

NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.

Fixes: #320 and #291 partially.
2018-02-16 05:06:37 +05:30
Greg Price 20c734c90a puppet: Fix type error in new Nagios check for analytics state. 2018-02-09 17:46:46 -08:00
Tim Abbott 005b0fb566 puppet: Clean up ssh authorized_keys configuration rules. 2018-02-09 16:37:03 -08:00
Tim Abbott aca25b6f0a puppet: Move ssh configuration to use notify.
This handles more correctly the case where we're using the upstream
sshd_config file.
2018-02-09 16:37:03 -08:00
Tim Abbott 486de8abfc puppet: Edit some rules to support chat.zulip.org.
This should make it possible to use the zulip_ops base rules
successfully on chat.zulip.org.  Many of the changes in this commit
are hacks and probably can be cleaned up later, but given that we plan
to drop trusty support soon, it's likely that most of them will simply
be deleted then.
2018-02-09 16:37:03 -08:00
Rishi Gupta 1d581a9c6e nagios: Add nagios check for analytics state.
This should help us detect issues where the analytics cron jobs aren't
running properly.

The cron/nagios part of the implementation done by tabbott.
2018-02-09 16:36:05 -08:00
Greg Price 7df29e7a7c puppet: Only use those "modern" options when on xenial.
On trusty, we of course have an older version -- 1.4.14 -- and it is
not so modern, so this just gives an error.
2018-02-08 18:11:52 -08:00
Greg Price 23e6a2e579 puppet: Update memcached config to turn on this decade's technology.
We've been running this change on zulipchat.com for a couple of months
now.  Before then, we used to regularly get exceptions like this:

     File "./zerver/views/messages.py", line 749, in get_messages_backend
       setter=stringify_message_dict)
     File "./zerver/lib/cache.py", line 275, in generic_bulk_cached_fetch
       cache_set_many(items_for_remote_cache)
     File "./zerver/lib/cache.py", line 215, in cache_set_many
       get_cache_backend(cache_name).set_many(items, timeout=timeout)
     File "/home/zulip/deployments/2017-09-28-21-04-12/zulip-py3-venv/lib/python3.5/site-packages/django/core/cache/backends/memcached.py", line 150, in set_many
       self._cache.set_multi(safe_data, self.get_backend_timeout(timeout))
   pylibmc.Error: error 48 from memcached_set_multi

This error means memcached was unable to find space for the new value.
You might think that because memcached provides an LRU cache, this
shouldn't happen because it would just evict something... but in fact
  * memcached splits its data into "slabs" by object size, and
  * until recently, once a 1MiB "chunk" is allocated to a given "slab"
    i.e. size class, it wouldn't be reclaimed to allocate to another.

So once the cache has been filled up with objects of some distribution
of sizes, if some objects come in that would go in a different size
class, we have no chunks for that size class / slab, and can't get one.
And that's exactly what was happening on zulipchat.com.

Useful background can be found in
  https://github.com/memcached/memcached/wiki/ServerMaint#slab-imbalance
  https://github.com/memcached/memcached/wiki/ReleaseNotes1411
  https://github.com/memcached/memcached/wiki/ReleaseNotes1425
  https://github.com/memcached/memcached/wiki/ReleaseNotes150
We're already running v1.4.25, which provides an "automover" that should
be well equipped to fix this; v1.5.0 turns it on by default.

With this commit, adopt the "modern start line" recommended in the
release notes for our v1.4.25, including turning on the automover.
2018-02-08 16:34:49 -08:00
Vishnu Ks bf2961418b puppet: Remove comment about period of soft deactivate users.
This often becomes wrong over time as it is currently.
2018-01-24 17:15:08 -08:00
Vishnu Ks a11b742984 messages: Calculate value of first visible message ID using cron job.
[greg: Fixed buggy time conversion in estimate_recent_messages.]
2018-01-24 17:15:08 -08:00
Tim Abbott 9ed2a94b8c nagios: Add configuration designed for full-stack servers.
This doesn't yet pass all Nagios checks correctly, and still has a few
flaws:
* The ideal setup code for the `nagios` user in the database isn't included.
* Some of the other details are a bit off; we need to split some host roles.

But it's better than nothing, and we can iterate from here.
2018-01-24 14:16:03 -08:00
Aditya Bansal dd0e6c8025 reminders: Fix issue with log file permissions in production. 2018-01-24 03:33:40 +05:30
Tim Abbott 2365b13b68 puppet: Move postgres Nagios plugin to main postgres-common.
This plugins package is required in order to use Nagios checks to
verify the Zulip postgres database, and thus belongs in the default
package set.
2018-01-23 10:31:48 -08:00
Aditya Bansal ec1297c1e8 schedulemessages: Add delivery system for scheduled message. 2018-01-10 09:18:02 -05:00
Umair Khan 68513952fb email-worker: Create EmailSendingWorker.
This commit just copies all the code from MissedMessageSendingWorker
class to a new EmailSendingWorker class. All the logic to send an email
through a queue was already there. This commit only makes the logic
generic. It does so by creating a special purpose queue called
'email_senders' to send any type of email. To make
MissedMessageSendingWorker still work we derive it from
EmailSendingWorker. All the tests that were testing
MissedMessageSendingWorker now run against EmailSendingWorker.
2017-12-20 19:36:27 -08:00
Tim Abbott f423dc4930 check_send_receive_time: Fix parsing bug.
This was a regression introduced with the argparse migration.
2017-11-27 14:01:30 -08:00
rht e55898850a Replace optparse with argparse in remaining tools.
Tweaked by tabbott to fix various bugs with the usage output.
2017-11-21 21:34:38 -08:00
Vishnu Ks 766511e519 actions: Mark all messages as read when user unsubscribes from stream.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak.  While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.

Fixes #7095.
2017-11-21 20:09:17 -08:00
Greg Price ae901309fc certbot: Control auto-renew with a zulip.conf setting.
This causes the cron job to run only when a Zulip-managed certbot
install is actually set up.

Inside `install`, zulip.conf doesn't yet exist when we run
setup-certbot, so we write the setting later.  But we also give
setup-certbot the ability to write the setting itself, so that we
can recommend it in instructions for adopting certbot in an
existing Zulip installation.
2017-11-15 21:50:41 -08:00
Greg Price dacf65b301 certbot: Move verification webroot under /var/lib/zulip .
If we were making an old-fashioned webroot where hand-written static
HTML files went, somewhere under `/srv` would be most appropriate.
Here, this webroot is really more of an implementation detail of the
certbot set up by the Zulip installer/packaging, containing transient
state.  So someplace under `/var` is appropriate, and specifically
under `/var/lib/zulip` in order to properly namespace it.

For background on `/var/www` and friends, see the top couple of answers
on
  https://unix.stackexchange.com/questions/47436/why-web-server-var-www
2017-11-15 21:50:41 -08:00
Tim Abbott 2afc3b9e50 certbot: Move path to /usr/local/sbin.
[greg: fixed typo bug]
2017-11-15 21:50:41 -08:00
rht 97ec56276c certbot: Add certbot renew cron job to puppet.
Tweaked by tabbott to use the proper command.
2017-11-15 21:50:41 -08:00
Tim Abbott 94554c65da certbot: Modify nginx configuration to support automated renewal. 2017-11-08 12:32:26 -08:00
Tim Abbott 62bb465896 puppet: Modify lb0 nginx configuration. 2017-11-08 12:32:26 -08:00
rht ccf2792c1c refactor: Remove six.moves.configparser import. 2017-11-07 10:51:44 -08:00
rht 549a26860f refactor: Remove six.moves.range import. 2017-11-07 10:46:42 -08:00
Tim Abbott acb0b6ee43 process_fts_updates: Fix pgroonga search in development.
For some reason, we have the USING_PGROONGA setting on in development
right now.  I'm going to disable that in another commit to match what
we're doing in production, but we'll still want that setting to work
in development.

The problem here was that process_fts_updates only attempted to read
the USING_PGROONGA setting from a /etc/zulip/zulip.conf source, and
thus would just not be updating the index in development.
2017-10-30 11:44:04 -07:00
Tim Abbott 0d1194811f mypy: Remove ignores for a few typeshed bugs fixed upstream. 2017-10-27 17:09:00 -07:00
Tim Abbott 89b97e7480 python3: Fix REMOTE_USER Apache configuration for Python 3.
We were previously still installing the Python 2 version of mod_wsgi,
which of course doesn't work and can't use the Zulip virtualenv.
2017-10-24 11:48:14 -07:00
Tim Abbott 15f3d5f714 nginx: Fix some buggy gzip compression configuration.
We weren't compressing SVG, while at the same time were incorrectly
compressing octet-stream (Which meant downloading .tar.gz files in
Chrome would get double-compressed).
2017-10-20 11:01:28 -07:00
Tim Abbott 540cae19a8 puppet: Remove obsolete sparkle configuration.
Sparkle was the auto-update system used by the legacy desktop app.  We
haven't been capable of using it for auto-update in years, so there's
no reason to keep around the configuration.

The new Electron app uses a different system anyway.
2017-10-19 16:35:55 -07:00
rht b57289aacd py3: Remove all `from __future__ import print_function.
Except for these files:
- tools/linter_lib/*
- tools/lib
- tools/lister.py
2017-10-18 12:07:19 -07:00
rht 2f3ae84e5a py3: Remove all `__future__ import division`. 2017-10-17 23:09:12 -07:00
Tim Abbott 6a5cb0e48c puppet: Make problems with Zephyr mirroring pageable.
Generally this indicates sending messages is completely broken.
2017-10-12 00:16:32 -07:00
rht de30400fc5 pg_backup_and_purge.py: Remove .py extension. 2017-10-08 15:32:43 -07:00
Tim Abbott 47c5aae5b2 log2zulip: Enforce using python 3 in cron job.
We aren't guaranteed to have the Zulip dependencies installed on
Python 2.
2017-10-06 16:37:17 -07:00
Tim Abbott 0f2e4a55c0 soft deactivation: Shorten management command name.
This command is really for soft deactivation; there's just an undo
feature.
2017-10-06 08:48:43 -07:00
Tim Abbott f2055397c1 nagios: Update apache configuration to be generated.
Since this is basically just stock Apache configuration for Nagios
with a hostname put in, we can just fetch the hostname from our
configuration.
2017-10-05 21:51:29 -07:00
Tim Abbott 3af01bed85 puppet: Simplify zulip_ops nginx configuration.
Whatever dist/ functionality this had in 2014 is now served by
zulip.org, and since this serves as a sample, it should be as simple
as possible.

Previously, this was more cluttered than it needed to be.
2017-10-05 21:17:57 -07:00
Tim Abbott e6e7bcf6e1 nagios: Move camo_check_url into configuration. 2017-10-05 21:09:24 -07:00
Tim Abbott 82cee4fde9 check_worker_memory: Increase limits for what leaking means.
The old limits were such that these would sometimes oscillated too
high and page erroneously.  The purpose of this check is to prevent
large memory leaks, and will still achieve that with a higher limit.
2017-10-05 20:54:03 -07:00
Tim Abbott 1c453fdf2a puppet: Add redis_password file for Nagios.
This allows the Nagios user to access redis without having full access
to the redis system.  Ideally, this would eventually use a password
that only has statistics read access, but I'm not sure redis supports
that.
2017-10-05 20:42:07 -07:00
Tim Abbott 13a36d9af3 puppet: Make old redis_tunnel configuration usable.
This old puppet configuration was never really used, and regardless
hardcoded an ancient zulip.net hostname.  We fix this to use the
zulipconf system to get the host domain (though not, at present, the
hostname).
2017-10-05 20:40:22 -07:00
Tim Abbott 96c3014da0 nagios: Automate configuration of outgoing email with msmtp.
Now we no longer need to check in a bunch of hostnames in order to
configure Nagios.
2017-10-05 20:29:47 -07:00
Tim Abbott 5b4c260c3f puppet: Add munin apache auth configuration.
This is completely stock configuration, and seems to be required for
munin to run properly.
2017-10-05 20:17:12 -07:00
Tim Abbott ba7be4102e puppet: Update munin tunnels configuration to use zulipconf.
This eliminates another old hardcoding of zulip.net.
2017-10-05 20:14:43 -07:00
Tim Abbott 162eaf8917 nagios: Modify check for swap to allow no swap.
If a machine is configured with no swap intentationally, that
shouldn't be a Nagios problem.  This alert is intended to flag
machines which are swapping.
2017-10-05 20:07:44 -07:00
Tim Abbott 80a16bf873 nagios: Fix path to source zulip_nagios.cfg.
Arguably, we should make this a symlink, but it's probably a good idea
to have every change in the production Nagios configuration go through
the zulip-puppet-apply diff experience.
2017-10-05 20:06:50 -07:00
Tim Abbott 886a8853ac nagios: Move server-specific config into hostgroups.
These new hostgroups exist so we can eliminate explicit references to
individual hosts in services.cfg.
2017-10-05 20:06:48 -07:00
Tim Abbott b6ce9583a9 nagios: Fetch list of hosts from zulip.conf.
This makes this much more configurable and much less hardcoded.
2017-10-05 20:06:30 -07:00
Tim Abbott 5193936bc3 nagios: Add Memcached and Redis monitoring.
These are standard Nagios plugins that might be sometimes helpful.
2017-10-05 20:06:16 -07:00
Tim Abbott f7d554d533 nagios: Rename zmirror2 to zmirrorp in configuration.
The "p" stands for "personals", aka zephyr private messages, which is
what this host manages.
2017-10-05 20:06:08 -07:00
Tim Abbott 062d280914 puppet: Clean up unnecessary pagerduty_nagios.cfg. 2017-10-05 19:23:33 -07:00
Tim Abbott 7e328ba865 nagios: Move email addresses for contacts into variables. 2017-10-05 19:23:33 -07:00
Tim Abbott 6017d3dec5 puppet: Move contacts.cfg to be a template. 2017-10-05 19:23:33 -07:00
Tim Abbott 09aec3e467 puppet: Move hosts.cfg to be managed by a template. 2017-10-05 19:23:33 -07:00
Tim Abbott 692f4b77d1 puppet: Remove messy Nagios crontab. 2017-10-05 19:23:33 -07:00
Tim Abbott 26982ff55f puppet: Remove pageduty_nagios.pl.
This hasn't been used in like 4 years, and clutters the repo.
2017-10-05 18:46:09 -07:00
Tim Abbott 5a80c029a2 nagios: Update path to sync_public_streams to match new config. 2017-10-05 13:34:27 -07:00
Tim Abbott fdd021fd6a zephyr-mirror: Update supervisor configuration for repository split.
This now points to the path of the integration in the new package.
2017-10-05 13:18:37 -07:00
Tim Abbott 1eff717146 zephyr-mirror: Update cron job to use python-zulip-api.
This is a deferred follow-up project to the repository split.
2017-10-05 13:07:45 -07:00
Alexander Trost b33126a3e5 Add Docker-specific puppet files.
Based on #450, with commits
restructured by Rein Zustand.
Additionally, tweaked by Rein Zustand for typo mechanics and syntax.
2017-09-30 09:55:48 -07:00
hollywoodno ee7d4808fe emails: Add log for outgoing emails.
Fixes #5900.
2017-09-25 15:37:09 -07:00
Tim Abbott 3708f5708a upgrade: Include zulip::static_asset_compiler by default.
Since we've found that it's fairly frequent that we want to recommend
to developers that they upgrade to a version of Zulip from Git, it
makes sense to include that by default.
2017-09-22 11:38:20 -07:00
Tim Abbott 64e950ae48 puppet: Add curl as a dependency.
It's needed by scripts/install-yarn.sh.  This hadn't been discovered
because most systems end up having curl installed even though it isn't
technically a required package.
2017-09-19 19:14:25 -07:00
Tim Abbott 6c4006625d nginx: Update zulip.org nginx configuration.
* Added zulipbot to configuration.
* Added redirect to zulipchat.com for homepage.
2017-09-16 02:09:53 -07:00
Tim Abbott 2d11a67975 soft_deactivation: Change cron job to run daily.
This seems like a more consistent default model than weekly.
2017-08-27 18:40:31 -07:00
Aditya Bansal b232563e12 soft-deactivation: Add cron job for weekly soft deactivating users. 2017-08-27 11:33:06 -07:00
Greg Price d02101a401 APNs: Rip out the existing, broken implementation.
This code empirically doesn't work.  It's not entirely clear why, even
having done quite a bit of debugging; partly because the code is quite
convoluted, and because it shows the symptoms of people making changes
over time without really understanding how it was supposed to work.

Moreover, this code targets an old version of the APNs provider API.
Apple deprecated that in 2015, in favor of a shiny new one which uses
HTTP/2 to meet the same needs for concurrency and scale that the old
one had to do a bunch of ad-hoc protocol design for.

So, rip this code out.  We'll build a pathway to the new API from
scratch; it's not that complicated.
2017-08-26 14:16:05 -07:00
Tim Abbott d449346bb5 puppet: Fix process leaks in small-server configuration.
Whenever you restarted supervisord services, we'd end up leaking one
process from the process_queue group, eventually resulting in running
out of memory.

Fixes #6184.
2017-08-25 12:36:29 -07:00
Tim Abbott f91a2a8ef2 supervisord: Improve memory footprint for small servers.
This cuts the number of uwsgi and message_sender works for smaller
servers with only 2GB of RAM.
2017-08-25 12:34:41 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott 989cfb1963 mypy: Fix process_fts_updates annotations. 2017-08-15 22:04:52 -07:00
Tim Abbott a38d846e8b process_fts_updates: Fix check for USING_PGROONGA to not use settings.
The Zulip server's settings are only available if process-fts-updates
is running is on the same server as a Zulip production deployment.  So
we instead check whether we have pgroonga configured in
/etc/zulip/zulip.conf.
2017-08-15 20:33:36 -07:00
Greg Price e4d1d22e9f py3: Explicitly keep our wal-e PostgreSQL replication on Python 2.
On `trusty` there is no package for `boto` or `gevent` on Python 3, both
of which are dependencies of `wal-e` (at the version we've pinned.)  This
is something used only on database servers and only in a replication
scenario, and it doesn't involve any of our code outside the wal-e repo,
so the Python version it uses is quite independent of the Zulip
application server itself and the rest of our code.  For now, keep it
explicitly on Python 2 while we move forward for most everything else.
2017-08-15 17:30:31 -07:00
Greg Price 2a4d851a7c py3: Explicitly keep one boto-using ops script on Python 2.
This script in `zulip_ops` is handy for managing EC2 instances.  It uses
`boto`, which isn't available in `trusty` for Python 3.  The use of
`boto` here isn't particularly deep, so we could replace it with some
more manual HTTP calls if it comes to that.  For now, just mark it to
stay on Python 2 while we move the app and all the rest of the ops code
(except this and another straggler or two) to Python 3.

Also make a comment on this package in the Puppet manifest clearer
about what it specifically refers to.
2017-08-15 17:30:31 -07:00
Greg Price 61666a9262 zulip_ops: Delete the long-disused `stats1.zulip.net` config and its dependencies.
This consists of the `zulip_ops::stats` Puppet class, which has apparently
not been used since 2014, and a number of files that I believe were
only used for that.  Also a couple of tiny loose ends in other files.
2017-08-15 17:30:31 -07:00
Greg Price 0042fc0c19 py3: Move `python-gevent` dependency to narrow its scope.
This is only actually used in our `wal-e` setup, which is in
zulip_ops::postgres_common.  (In fact the only mentions of `gevent` in
our whole Git history are for `wal-e`.)  So remove where we mention it
on the broader zulip::postgres_common module, and move it where it's
needed.

This follows up on 98cef0ab4 by eliminating the only dependency
outside of the `zulip_ops` Puppet tree on a system Python-library
package which isn't available in `trusty` for Python 3.
2017-08-15 17:30:31 -07:00
Greg Price e469578a55 py3: Fix up (almost) all script invocations to rely on shebangs.
This follows up on 207cf6302 from last year to clean up cases that
have apparently popped up since then.  Invoking the scripts directly
makes a cleaner command line in any case, and moreover is essential
to how we control running a Zulip install as either Python 2 or 3
(soon, how we always ensure it runs as Python 3.)

One exception: we're currently forcing `provision` in dev to run
Python 3, while still running both Python 2 and Python 3 jobs in CI.
We use a non-shebang invocation to do the forcing of Python 3.
2017-08-15 17:30:31 -07:00
Greg Price 98cef0ab48 py3: Augment all mentions of system Python packages to include Python 3.
In some of these contexts, we may still be *using* the Python 2
version, but at least this should eliminate running into
`ImportError`s one by one in scripts that run outside a virtualenv,
as we update their shebangs to refer to Python 3.

Several Python libraries we use don't come in Python 3 versions on
trusty: gevent, boto, twisted, django, django-tagging, whisper.
The latter two don't come in Python 3 versions even on xenial.
So some work required before we can actually switch the code that
relies on those libraries to run as Python 3 -- probably the best
solution will be to backport them all in our apt repo.  (All but
`whisper` are packaged in zesty; `whisper` upstream just grew Python 3
support this year.)
2017-08-09 14:07:05 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Greg Price b8089bdd1c api: Update log2zulip cron job to find the script at its new path. 2017-07-31 21:24:02 -07:00
Greg Price c127630dcf Delete some obsolete usage-stats tools.
These are no longer useful, with our spiffy new analytics framework,
and we haven't in fact been using them for some time, while the
`active-user-stats` cron job does cause regular mail from cron.
Just delete them.
2017-07-31 17:06:15 -07:00
Tim Abbott 2317819e47 nginx: Use the Django 404 page for files under static/.
This elimintes the need for us to maintain duplicate copies of the
Zulip 404 error pages.

Fixes #5382.
2017-07-18 09:55:30 -07:00
neiljp (Neil Pilgrim) 8cabce9f5e mypy: For EC2, Ensure to_configure is passed a not-None argument. 2017-07-17 16:57:42 -07:00
neiljp (Neil Pilgrim) ba51958c40 mypy: For EC2, pre-assign address & gateway to enable assertion. 2017-07-17 16:57:42 -07:00
neiljp (Neil Pilgrim) fd941e8f88 mypy: For EC2, make guess_gateway return None if address is None. 2017-07-17 16:57:42 -07:00
Tim Abbott 215e3389e2 logrotate: Add a few more Django files to log rotation. 2017-07-16 10:49:15 -07:00
Tim Abbott 05723acb3c logrotate: Explicitly declare users/modes for log files. 2017-07-16 10:49:15 -07:00
Tim Abbott cef6285eee logrotate: Fix whitespace in logrotate configuration. 2017-07-16 10:49:11 -07:00
Tim Abbott 728b852b28 logrotate: Dramatically decrease maximum size of logs.
It's rare that there's value in having the log files get this big, and
these changes mean these log files should never consume more than a
few gigabytes.

And in particular, the server.log is far more important than the other
log files, and grows much faster, so we might as well spend most of
the space we are spending on that.

I estimate that the total size of log files from this is going to be
under 1-2GB, since 75MB (compressed size) * 10 (compressed logs) +
500MB (uncompressed size) = 1.25GB from server.log, and the rest is
negligible.

Fixes part of #5724.
2017-07-07 12:58:33 -07:00
Tim Abbott 6355f26b91 supervisor: Dramatically decrease default log file max sizes.
Most of these log files are useless except a few minutes after an
event happens, and the aggregate effect of the originals size limits
meant that Zulip's logs could consume many gigabytes of disk.

The new logging strategy should limit our usage from supervisor logs
to at most 3 Gigabytes:
* 20 * 3 = 60MB per queue worker => <1GB.
* 100 * 10 = 1GB for Django and Tornado logs.

Fixes part of #5724.
2017-07-07 12:30:24 -07:00
cursiv 2b7938b8ed nginx: Add missing HTTP request methods to header.
This will make life easier if trying to write a Zulip client inside a
browser using the API.
2017-07-06 14:38:07 -07:00
rht 940cf9db3b Run queue processors multithreaded in production if system memory <3.5GB.
While running queue processors multithreaded will limit the
performance available to very small systems, it's easy to fix that by
adding more RAM, and previously, Zulip didn't work on such systems at
all, so this is unambiguously an improvement there.

Fixes #32.
Fixes #34.

(Commit message expanded significantly by tabbott.)
2017-06-03 12:19:58 -07:00
Aditya Bansal aa433f4342 pep8: Add compliance with rule E261 check_send_receive_time. 2017-05-31 17:07:15 -07:00
Aditya Bansal 5989c88545 pep8: Make compliant zulip-ec2-configure-interfaces with rule E261. 2017-05-31 17:07:15 -07:00
Aditya Bansal 6b8e85e065 pep8: Make compliant check_zephyr_mirror with rule E261. 2017-05-31 17:07:15 -07:00
Aditya Bansal 49ae51f23a pep8: Make compliant check_user_zephyr_mirror_liveness with rule E261. 2017-05-31 17:07:15 -07:00
Aditya Bansal 6d0927ed0b pep8: Add compliance with rule E261 to check_personal_zephyr_mirrors. 2017-05-31 17:07:15 -07:00
Tim Abbott c5bc1265f3 bots: Move log2zulip into api/integrations. 2017-05-26 15:15:56 -07:00
Tim Abbott 55f69c677a puppet: Remove obsolete zuliprc.nagios file.
This hasn't done anything for years.
2017-05-26 15:14:12 -07:00
Reid Barton ccb4c5c26f bots: Move zephyr-related files to api/integrations/zephyr/. 2017-05-26 15:07:02 -07:00
Elliott Jin 0ec9e54954 bots: Add queue and QueueProcessingWorker for embedded bots. 2017-05-25 15:00:51 -07:00
Konstantin Gukov dd76222a3f Fetch system bots using new get_system_bot function.
This eliminate a bunch of uninteresting calls to
get_user_profile_by_email.
2017-05-23 10:30:40 -07:00
Tim Abbott 1d1b0894a3 zulip_ops: Add logrotate configuration from main zulip. 2017-05-15 21:54:35 -07:00
Tim Abbott e049ea01b1 puppet: Update munin configuration to work with modern munin. 2017-05-15 21:49:53 -07:00
Tim Abbott 6871c227cd puppet: Add missing restarts of Nagios on config updates. 2017-05-15 21:49:53 -07:00
Tim Abbott e94586adb4 puppet: Fix path to events_deliver_enqueued_emails.log. 2017-05-09 22:29:58 -07:00
Aditya Bansal b677689fea pep8: Add compliance with rule E261 to pg_backup_and_purge.py. 2017-05-07 23:21:50 -07:00
vaibhav 8881b5eb9f Outgoing Webhook System: Check for @-mentioned outgoing webhook bots.
Also puts them into a processing queue, though the queue processor
does nothing.

Rewritten by tabbott to avoid unnecessary database queries in
do_send_messages.
2017-05-02 09:22:04 -07:00
K.Kanakhin e3e52e7284 email-mirror: Move postfix email mirror integration to separate script.
This fixes a performance problem where we were previously starting up
a full Django process (~0.7s even on a fast machine) every time a new
email came in, potentially allowing users to accidentally DoS a Zulip
server.  Now, we just post over HTTPS, allowing the existing thread
pool support to do its job.

- Add script wrapper to communicate postfix pipe with django web server
  over HTTP(S). It uses shared_secret authentication mode.
- Add django view to process messages from email mirror server.
- Clean management command `email-mirror`. Left just functional
  for cron email processing.
- Add routes for new tornado view.
- Change pipe script in master process postfix config template
  based on updated script.
- Add tests.

Tweaked by tabbott to adjust the directory and set better defaults.

Fixes #2421.
2017-04-24 21:24:23 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
Alexander Trost 0fb6779899 docker: Make user and dbname configurable in process_fts_updates. 2017-03-23 14:28:21 -07:00
Tim Abbott 17c8527856 nagios: Clean up check_send_receive_time arguments. 2017-03-15 12:52:27 -07:00
Tim Abbott 0ff1f3d663 nagios: Error on heartbeat events in check_send_receive_time.
It was probably going to fail anyway if those show up, but this
produces a clearer failure mode.
2017-03-15 12:51:19 -07:00
Tim Abbott 6a5e98b77e puppet: Increase MaxStartups SSH configuration. 2017-03-08 22:28:16 -08:00
K.Kanakhin 1e441b8d7c uwsgi: Add master mode to the main uwsgi process.
- Enable `master` parameter for `uswgi` configuration.
  It allows cleaning leaked processes if the parent
  process is closed unexpectedly or with SIGKILL command.
  Child processes follow to the master and kill themselves
  after the main process.

Fixes #3855
2017-03-07 21:35:51 -08:00
K.Kanakhin 6a801db1c2 missed-emails-sending: Move email sending to separate queue worker.
- Add new 'missedmessage_email_senders' queue for sending missed messages emails.
- Add the new worker to process 'missedmessage_email_senders' queue.
- Split aggregation missed messages and sending missed messages email
  to separate queue workers.
- Adapt tests for sending missed emails to the new logic.

Fixes #2607
2017-03-07 20:08:40 -08:00
Tim Abbott 67219cf660 puppet: Use restart-server for weekly server restarts.
Using `supervisorctl restart all` carried longer downtime (since it
just restarts everything at the same moment) and was less under our
control; I'm not sure it had any advantages.
2017-03-05 11:36:10 -08:00
Tim Abbott ab2ce9d5e9 lint: Fix whitespace error with recent mypy annotations. 2017-03-04 15:41:11 -08:00
Tim Abbott 75e81253f2 mypy: Work around several new mypy bugs in 0.501. 2017-03-04 15:33:39 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 3d07ac0c49 Change timezone-naive datetimes to use timezone.now() where safe to do so.
Change timezone-naive datetimes to use timezone.now() in cases where there
is no change in behavior.
2017-03-01 22:54:28 -08:00
Rishi Gupta 0218422e96 Use time.time() instead of datetime.now() to measure elapsed time.
Both because it is more idiomatic and because we will soon start enforcing
that all datetimes in Zulip are timezone aware.
2017-03-01 22:54:28 -08:00
Tim Abbott aed3632cbb puppet: Convert remaining queue workers to _ style. 2017-02-19 16:18:37 -08:00
Tim Abbott 4b54307d94 puppet: Generate most of main supervisor config with template. 2017-02-19 16:18:37 -08:00
Tim Abbott ae09d55e46 puppet: Move zulip.conf to be a template file. 2017-02-19 16:18:37 -08:00
Kouhei Sutou a2d935a2ee puppet: Fix PostgreSQL user to create PGroonga extension
"root" user isn't a PostgreSQL administrator. "postgres" is a PostgreSQL
administrator.
2017-02-08 12:57:56 -08:00
Tim Abbott fa8045a484 puppet: Add websockets Nagios test to configuration.
Since browser clients send messages via websockets and not the API,
this is an important element in making sure mission-critical Zulip
functionality is working.
2017-02-08 11:13:19 -08:00
Tim Abbott 8db13d0bb9 check_send_receive_time: Use a different state file for websockets.
Otherwise, the two Nagios checks will fight over the same state file
if both are in use.
2017-02-08 11:13:19 -08:00
Tim Abbott ba5f454be5 puppet: Extract zulip::analytics.
I'm not altogether happy with this (a better solution would be
database-level locking), but I think it solves the immediate problem
of folks with 2 servers being very likely to run analytics on both of
them.
2017-02-07 12:29:15 -08:00
Tim Abbott 70388b17d2 puppet: Add missing dependency on ssl-cert. 2017-02-06 15:51:38 -08:00
umkay 76f3d02590 analytics: Add cron job to run analytics jobs.
This adds a cron job to update the Zulip analytics counts, complete
with locking etc.

Substantially tweaked by tabbott.
2017-02-01 17:02:46 -08:00
Tim Abbott 2fb51ff876 puppet: Use SIGINT to restart uwsgi.
This results in a brief service interruption (not a graceful restart),
but fixes a bug where on a `supervisorctl restart zulip-django`, we'd
end up leaking a bunch of uwsgi processes.

The mechanism was that sending SIGHUP to uwsgi was a command for it to
gracefully restart, so it'd start doing that (whereas supervisor
expected it to be dying)... and then supervisor would start up the new
uwsgi process group, resulting in 2 uwsgi process groups running.

This, in turn, led to a memory leak that could eventually result in
OOM kills.
2017-01-28 22:26:12 -08:00
Tim Abbott 36d54cf5ff Replace references to zulip.com/dist with zulip.org/dist.
Now that zulip.org has all the files to distribute, there's no reason
to still point to the soon-to-be-decommissioned zulip.com/dist.
2017-01-28 17:56:25 -08:00
Eitan Adler 0ce29d7ad6 Remove some some duplicate words in copy. 2017-01-23 23:15:04 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott d6e38e2a5c lint: Clean up E123 PEP-8 rule. 2017-01-23 21:34:26 -08:00
Tim Abbott bde2da7dfd lint: clean up PEP-8 W391 rule. 2017-01-23 20:39:02 -08:00
Tim Abbott bbd853e208 puppet: Add redirect to https to zulip.org. 2017-01-22 21:52:50 -08:00
Tim Abbott 44776c43a1 puppet: Add configuration for zulip.org website.
This puppet configuration, plus cloning the zulip.github.io repo and
letsencrypt key setup, is all we need to run a zulip.org server.
2017-01-22 21:48:48 -08:00
JefftheBest1 9de75f5167 Fixed typos with separate 2017-01-12 04:52:05 -08:00
JefftheBest1 ff8639f9db Fixed typos with threshold. 2017-01-12 04:50:20 -08:00
JefftheBest1 5008f45112 Fixed typo in munin.conf.erb 2017-01-12 04:49:19 -08:00
Tim Abbott 3e32102016 nagios: Fix various critical issues not tagged as pageable. 2017-01-06 21:49:20 -08:00
Tim Abbott edebf7619b puppet: Add PAM common_session disabling systemd-login.
This fixes a weird problem with systemd where logging into a server
via ssh frequently has a 15s+ lag.
2017-01-06 21:49:15 -08:00
Tim Abbott 93c2c19775 nagios: Increase process count limits. 2017-01-06 21:49:15 -08:00
Tim Abbott 2c6cb37385 munin: Add default munin configuration template. 2017-01-06 21:44:57 -08:00
Tim Abbott 9ab8e7ba34 nagios: Disable swap checks for servers with no swap. 2017-01-06 21:39:07 -08:00
Tim Abbott 3e01ed1f73 nagios: Increase NTP max_check_attempts.
NTP often suffers from brief interruptions of service that lead to
spurious Nagios alerts; it makes sense to suppress these.
2017-01-06 21:32:43 -08:00
Tim Abbott e4420b08d2 zulip_ops: Disable unattended upgrades of security packages.
Since Zulip does not handle e.g. postgres server restarts gracefully,
it's best for a system administrator to manually trigger security
updates.
2017-01-06 21:30:56 -08:00
Tim Abbott 6f9c73d0e5 zmirror: Update Debathena release in configuration.
The zulip_ops configuration is now for xenial, not obsolete wheezy.
2017-01-06 21:30:41 -08:00
Tim Abbott bd9176d1d9 nagios: Remove some default files.
Nagios ships with a bunch of default configuration files that one
needs to delete in order to configure it.
2017-01-06 21:25:12 -08:00
Tim Abbott 7083899e77 zulip_ops: Add postgres config for enabling Nagios.
The old zulip_ops Nagios configuration depended on Nagios having the
ability to login as the zulip user (with essentially full write
access); this configuration is helpful for limiting nagios to special
"nagios" user with more limited credentials.
2017-01-06 21:24:24 -08:00
Tim Abbott 204edb0f85 zulip_ops: Cleanup pg_hba.conf configuration. 2017-01-06 21:23:51 -08:00
Tim Abbott 30c57eb2ae zulip_ops: Add basic .emacs for production. 2017-01-06 21:20:21 -08:00
Tim Abbott eb87d04168 puppet: Remove xxxxx password hardcoding in recovery.conf. 2017-01-06 21:20:21 -08:00
Tim Abbott 6404a1a5ff zulip_ops: Add nagios-plugins-contrib.
This has a number of useful nagios plugins.
2017-01-06 21:19:59 -08:00
Tim Abbott f7b77008ef zulip_ops: Add aptitude dependency.
This is useful for `aptitude why`.
2017-01-06 21:19:50 -08:00
Tim Abbott 2510a51a8a zulip_ops: Add letsencrypt dependency. 2017-01-06 21:19:31 -08:00
Tim Abbott 65774e1c4f zulip_ops: use check_postgres package from apt. 2017-01-06 21:18:55 -08:00
Tim Abbott 165b4d3126 nagios: Fix check_send_receive_time threshholds.
Previously, the CRITICAL state would never fire (because x > 6 =>
x > 3).  Additionally, 6s is not so unusually high as to deserve being
immediately pageable.
2017-01-06 21:16:37 -08:00
K.Kanakhin 0d8c18a6dd nagios-plugins: Add websocket checking to nagios message sending test.
- Add websocket client to create connection with SockJS websocket server.
  It contains callback method to launch after connection setup.
- Add '--websocket' parameter to 'check_send_receive_time' script to
  check websocket connection.
- Add testing  websocket connection to production installation checking.
- Add cronjob to launch websocket connection nagios test.

This makes it possible for Zulip Nagios monitoring to check for
problems impacting the websockets sending code path, which is what all
web users use.
2016-12-30 15:36:37 -08:00
Umair Khan 336a041ac0 Django 1.10: Use uWSGI.
Fixes: #1121

With some tweaks by tabbott to make the number of processes configurable.
2016-12-13 21:40:43 -08:00
Igor Tokarev c93f1d4eda Add oembed/Open Graph/Meta tags data retrieval from inline links.
This change adds support for displaying inline open graph previews for
links posted into Zulip.

It is designed to interact correctly with message editing.

This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.

By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.

Eventually, we may want to make this manageable via a (set of?)
per-realm settings.  E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
Jason Le 144d82305d mypy: Annotate puppet/zulip_ops. 2016-12-03 11:00:25 -08:00
bulat22101 adebc75740 pep8: Fix E502 violations 2016-12-03 10:56:36 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Bickio 6b0df43463 pep8: Fix E125. 2016-11-30 20:03:29 -08:00
Tommy Ip 46b7d54b3e pep8: Fix E701 violations. 2016-11-30 20:45:09 +00:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Anders Kaseorg 092fe4fecb puppet: Write rabbitmq-env.conf before installing rabbitmq-server
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 18:54:12 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Anders Kaseorg 1ea8abe493 Replace python -u with PYTHONUNBUFFERED=1
(Why is -u needed at all?  I’m not sure, but test-run-dev spins forever
“Polling run-dev...” without it.)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Anders Kaseorg d1dc2cf30e Mark scripts executable
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Anders Kaseorg 18e49bec3a puppet: Add another missing dependency on postgresql-common
The postgres group must exist before we give files to it.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 11:23:32 -08:00
Tim Abbott f78eb9d82d puppet: Add missing dependency on postgresql-common.
postgresql-$(version) depends on postgresql-common depends on ssl-cert,
which creates the ssl-cert group.
2016-11-21 07:38:45 -08:00
Tim Abbott 2e65dc1206 puppet: make check_send_receive_time target host configurable. 2016-11-02 23:40:53 -07:00
Tim Abbott 9578908601 check_send_receive_time: Stop hardcoding EXTERNAL_API_URI. 2016-11-02 23:35:08 -07:00
Tim Abbott eceaf36001 setup_disks: Fix postgres RAID setup to work correctly on Xenial.
Nobody's going to run this on Wheezy again.
2016-10-28 11:04:08 -07:00
Tim Abbott 9b7a3f040c Remove now-unused /json/get_events endpoint. 2016-10-27 21:34:58 -07:00
Tim Abbott 4fbe201187 puppet: Automate autossh process monitoring maintenance.
Previously, the Zulip Nagios configuration effectively hardcoded the
count for how many system should have autossh connections.
2016-10-26 00:49:03 -07:00
Tim Abbott 6bdb10b71b puppet: Update emacs dependency to emacs-nox metapackage.
This way, one doesn't need to keep updating the dependency every time
a new major emacs release comes out.
2016-10-26 00:42:22 -07:00
Tim Abbott 11b5d203f7 sshd_config: Increase MaxStartups.
This fixes connection problems when using the full Zulip recommended
Nagios configuration against a given server.
2016-10-26 00:41:03 -07:00
Tim Abbott 73f54dd0cb sshd_config: Add updates from Xenial upstream.
It seems worth updating this to match the Linux distro this
configuration targets.
2016-10-26 00:40:44 -07:00
Tim Abbott 0a5a2c4eda nagios: Automate authorized users list maintenance. 2016-10-26 00:37:29 -07:00
Tim Abbott fa4998db59 puppet: Add zulip_zephyr_mirror plugins. 2016-10-26 00:35:57 -07:00
Tim Abbott ac4f28050c zmirror: Remove unnecessary krb5-clients dependency.
I'm pretty sure krb5-clients isn't needed to run the Zephyr mirroring
service.
2016-10-26 00:35:11 -07:00
Tim Abbott d490e83645 puppet: Upgrade nagios cgi.cfg with modern defaults. 2016-10-26 00:31:41 -07:00
Tim Abbott 1159ad4857 puppet: Upgrade nagios.cfg with modern defaults. 2016-10-26 00:31:41 -07:00
Tim Abbott 73178e5e5a puppet: Run check_send_receive_time via a cron job.
This allows the actual nagios work involved with
check_send_receive_time nagios checks to be done by an unprivileged
"nagios" user rather than the "zulip" user.
2016-10-26 00:26:52 -07:00
Tim Abbott 96cf330649 puppet: ssh as the nagios user instead of zulip user.
This is a follow-up to 4f58fef54b,
touching services.cfg instead of commands.cfg.
2016-10-26 00:23:47 -07:00
Tim Abbott a350d43683 puppet: Add recovery.conf configuration to postgres_slave.pp.
This file is needed to run a valid postgres slave; it's not clear why
this wasn't installed in the original zulip.com configuration.
2016-10-26 00:22:57 -07:00
Tim Abbott c3727c9886 nagios: Remove old zulip.com trac/git/replica servers.
These are unlikely to be relevant to anyone.
2016-10-26 00:21:53 -07:00
Tim Abbott 383f39b543 nagios: Enable allow_empty_hostgroup_assignment.
This fixes the configuration being broken when we remove some of the
old zulip.com hosts that are unlikely to be of interest to anyone.
2016-10-26 00:19:21 -07:00
Tim Abbott 4f58fef54b zulip_ops: Use nagios user for all Nagios checks.
There's no reason these Nagios checks needs to run as the
semi-priviliged Zulip user.
2016-10-26 00:17:26 -07:00
Tim Abbott 32d244dbe5 puppet: Add Nagios checks for other consumers. 2016-10-26 00:11:08 -07:00
Tim Abbott f1fa4397f3 puppet: Fix package deps for zulip-ec2-configure-interfaces. 2016-10-26 00:11:08 -07:00
Tim Abbott 3448ab4c7a zulip-ec2-configure: Fix network IDs for Ubuntu Xenial. 2016-10-26 00:11:08 -07:00
Tim Abbott 91da4bd59b puppet: Add check_cron_file generic helper. 2016-10-26 00:11:08 -07:00
Tim Abbott 080dd8c987 nagios: Ignore kthreads in check_procs tests.
Modern Linux can have a lot of kernel threads not doing anything.
Since this isn't interesting from a monitoring perpsective, we ignore
these.
2016-10-26 00:10:40 -07:00
Tim Abbott 4c9a283542 puppet: Remove configuration for old builder host.
I don't think this configuration was ever even used; it's just
clutter.
2016-10-26 00:01:52 -07:00
Tim Abbott f9ad75f58e puppet: Remove configuration for old zulip.com bots host.
This configuration didn't do anything anyway and just clutters the
repo.
2016-10-26 00:01:29 -07:00
Tim Abbott 9d4f3f1e1b puppet: Replace zulip_ops postgres configs with postgres_appdb_tuned.
There's no longer a reason to have copies of forked postgres
configuration files in our repository, since some time ago we merged
the features of these configuration files into the main
postgres_appdb_tuned.pp.
2016-10-25 23:58:53 -07:00
Tim Abbott 105ea972f6 puppet: Remove now-unncessary kernel.shm sysctl values.
With modern Linux and postgres, these settings are not required.
2016-10-25 23:58:33 -07:00
Tim Abbott 2227e77cce puppet: Remove Dropbox usernames from Nagios config. 2016-10-25 23:55:42 -07:00
Tim Abbott 8584c05d80 zulip_ops: Remove unnecessary loadbalancer stanzas. 2016-10-25 23:52:37 -07:00
Tim Abbott 624ee3989f puppet: Remove old Dropbox certificates. 2016-10-25 23:52:30 -07:00
Tim Abbott f0bb78ba2d puppet: Fix iptables-persistent->netfilter-persistent rename. 2016-10-25 23:45:21 -07:00
Tim Abbott c4ca7ee6e1 puppet: Move Apache sites files to correct paths.
Apache now actually requires its configuration files have names ending
with .conf.
2016-10-25 23:44:28 -07:00
Tim Abbott 2b8324b778 emoji: Fix caching permissions issues.
Previously, you needed to be root to update the emoji cache, which
caused problems with how Zulip is upgraded in production.
2016-10-25 17:52:19 -07:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 56c0b80067 nginx: Fix JavaScript not being compressed properly.
Apparently, we weren't actually compressing our JavaScript being sent
over the wire due to incorrect nginx configuration.
2016-10-23 20:06:02 -07:00
Tim Abbott a5a03c2e0b zulip_ops: Include zulip::apt_repository.
This replaces the old wheezy configuration.
2016-10-16 20:13:35 -07:00
Tim Abbott 8c68c6f09b zulip_ops: Remove wheezy apt repo.
Nobody uses wheezy anymore, and the configuration wasn't even
conditional on the OS version.
2016-10-16 20:13:35 -07:00
Tim Abbott 5210b0a6a4 zulip_ops: Cleanup old redis configuration.
One can now just use the improved configuration we've merged into the
main Zulip repo.
2016-10-16 20:13:35 -07:00
Tim Abbott 869f0724ce zulip_ops: Remove humbughq.com nginx configuration.
The humbughq.com name hasn't been the product's name since 2013, and
it's nice to finish clearing it out of the repository.
2016-10-16 20:13:29 -07:00
Tim Abbott 29448fb47b zulip_ops: Remove old Zulip, Inc. trac configuration.
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
Tim Abbott 771e03cfa7 zulip_ops: Remove old Zulip, Inc. mediawiki configuration.
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
Tim Abbott 36e336edc3 puppet: Rename zulip_internal to zulip_ops.
The old "zulip_internal" name was from back when Zulip, Inc. had two
distributions of Zulip, the enterprise distribution in puppet/zulip/
and the "internal" SAAS distribution in puppet/zulip_internal.  I
think the name is a bit confusing in the new fully open-source Zulip
work, so we're replacing it with "zulip_ops".  I don't think the new
name is perfect, but it's better.

In the following commits, we'll delete a bunch of pieces of Zulip,
Inc.'s infrastructure that don't exist anymore and thus are no longer
useful (e.g. the old Trac configuration), with the goal of cleaning
the repository of as much unnecessary content as possible.
2016-10-16 19:23:27 -07:00
Tim Abbott 2e50dce9dd check_queue_worker_errors: Don't import zproject.settings directly. 2016-10-15 22:53:28 -07:00
Tim Abbott 913d913c90 puppet: Annotate Nagios plugins. 2016-10-15 22:44:15 -07:00
Tim Abbott 458e455b75 Annotate process_fts_updates. 2016-10-15 22:44:15 -07:00
Tim Abbott a4b2a6b6d4 puppet: Fix missing dependency for supervisor enabling.
You can't run `systemctl enable` on a service before the package
containing the service is installed.
2016-10-06 21:44:10 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
Tim Abbott 12ddb1a36f puppet: Fix buggy logrotate configuration. 2016-09-08 16:51:17 -07:00
Kouhei Sutou 683f49aa99 Support full text search for all languages using pgroonga.
This adds support for using PGroonga to back the Zulip full-text
search feature.  Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.

Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.

PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.

Fixes #615.

[docs and tests tweaked by tabbott]
2016-08-26 21:04:03 -07:00
Tim Abbott 546c23fd1f puppet: Activate pgroonga extension after installation.
The pgroonga extension needs to be activated using postgres root
access, so we make that happen from puppet.
2016-08-25 18:37:45 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
Tim Abbott 682d78bc30 puppet: Add default logrotate configuration.
This should prevent Zulip from eating all the disk now that we are
using the race-free WatchedFileHandler rather than
TimedRotatingFileHandler.
2016-08-24 14:30:54 -07:00
Tim Abbott 1c40df9363 static_asset_compiler: Use correct package name on trusty. 2016-08-22 23:47:03 -07:00
Tim Abbott 97b622dffa puppet: Fix postgres_appdb_tuned manifest on trusty. 2016-08-22 23:41:07 -07:00
Tim Abbott 750e43518f puppet: Ensure supervisord starts on boot with systemd.
This is important to ensure that Zulip comes up properly after a
reboot on Ubuntu Xenial.
2016-08-22 23:25:54 -07:00
Tim Abbott 2be597dd09 puppet: Move support for accepting a loadbalancer to zulip manifests.
Previously, we didn't have an easy process for how one should do this.

Now, it's an (undocumented) zulip.conf setting!
2016-08-22 17:19:03 -07:00
Tim Abbott 88a123d5e0 Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.

Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.

Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.

Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
Tim Abbott 852af83d3c check_postgres_replication_lag: Fix psql command line.
This allows the plugin to be run as users other than the "zulip" user,
where the "zulip" database would not be the default.
2016-08-12 13:19:08 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Tim Abbott 2581ac166c puppet: Automatically scale default memcached memory allocation.
Previously, we used a fixed memcached memory allocation of 512MB,
regardless of the size of the server.  While that is a good allocation
for a server with 4GB of RAM, for servers with less, we should
decrease the allocation, and for a large server with much more RAM, we
should increase it.  We still support the user overriding the
configuration setting, but this produces more sensible defaults.
2016-08-11 20:27:45 -07:00
Tim Abbott 2e0e8193d4 puppet: Make memcached memory allocation a variable. 2016-08-11 20:27:45 -07:00
Tim Abbott 3edf880c9b Remove moreutils from zulip_internal::base.
We recently added this to zulip::base, so it would be a duplicate.
2016-08-11 18:12:59 -07:00
Tim Abbott 9afb1c7a71 puppet: Stop wholesale replacing /etc/redis/redis.conf.
Zulip had only patches the redis configuration in one small way, which
resulted in unnecessary portability issues for using Redis on
different versions of Linux.  We replace this with just a adding an
include mechanism to the redis config.

While we're at it, we configure this to take advantage of the
new REDIS_PASSWORD secret to automatically configure redis passwords.
2016-08-11 17:26:04 -07:00
Tim Abbott cb21584ffe check_send_receive_time: Support being run in a cron job. 2016-08-11 14:48:21 -07:00
Tim Abbott 4dcbaf1e6b check_send_receive_time: Cleanup unnecessary print statement. 2016-08-11 14:48:21 -07:00
Tim Abbott 1d6ebd2b3d check_fts_update_log: Fix psycopg2 arguments.
* Fixes passing a string argument rather than an actual Python
  argument.
* Switches to hardcoding the database to connect to rather than the
  user, so this check can be run as an arbitrary user.
2016-08-03 14:58:46 -07:00
Tim Abbott 1c65508624 puppet: Add postgres user to Zulip group. 2016-08-02 14:59:25 -07:00
Tim Abbott 04fc3ff1e1 pg_backup_and_purge: Don't try to use a virtualenv.
This makes this actually work if the postgres server is different from
the Zulip app frontend.
2016-08-02 14:59:25 -07:00
Tim Abbott 4bdd9d3769 check_postgres_backup: Don't try to use a virtualenv.
The dependencies for this tool are installed using apt.
2016-08-02 14:59:25 -07:00
Tim Abbott a5115d54ee env-wal-e: Rename s3_backup_bucket to s3_backups_bucket.
This makes it consistent with the other variables in this file.
2016-08-02 14:59:15 -07:00
Tim Abbott ff80daef16 puppet: Move zulipsecret into its own file.
Apparently puppet doesn't support declaring multiple functions in the
same file.
2016-08-02 14:55:51 -07:00
Tim Abbott 75b5d021fa Remove unused puppet-common third-party module code. 2016-07-31 19:24:42 -07:00
Tim Abbott 6158acb41b nagios: Fix path to check_debian_packages. 2016-07-31 14:25:07 -07:00
Tim Abbott 6954dd84ab postgres_appdb_tuned: Add SSL certificate path configs. 2016-07-31 14:25:07 -07:00
Tim Abbott d66f6b8176 Fix postgresql configuration template for Ubuntu Xenial.
I incorrectly assumed that this was working because Travis CI passed
the build, whereas in fact Travis CI only tests the Trusty templates.
2016-07-31 14:24:34 -07:00
Tim Abbott 8b285ec0ff puppet: Read camo key from zulip-secrets.conf. 2016-07-31 00:23:24 -07:00
Tim Abbott 2b40309029 puppet: Add new zulipsecret function. 2016-07-31 00:23:24 -07:00
Tim Abbott 298e040bac puppet: Strip newlines in zulipconf function.
This avoids creating unnecessary extra newlines in config generated
using this.
2016-07-31 00:23:24 -07:00
Tim Abbott 16a4ce1bd2 puppet: Fix Nagios check_disk flagging special filesystems. 2016-07-30 23:57:31 -07:00
Tim Abbott c764b46cef puppet: Fix missing base include in nagios manifest. 2016-07-30 23:48:41 -07:00
Tim Abbott bc9bdd53aa puppet: Use hostgroup for postgres_other monitoring. 2016-07-30 23:46:05 -07:00
Tim Abbott 6b34bee806 puppet: Remove unused ancient humbug-default apache site. 2016-07-30 23:43:55 -07:00
Tim Abbott 217faed3b3 Update check_postgres plugin to version 2.22. 2016-07-30 23:19:40 -07:00
Tim Abbott 1f549dcfab check_fts_update_log: Fix use on non-appbd machines. 2016-07-30 21:48:17 -07:00
Tim Abbott 35edd8c9dd puppet: Move crudini dependency to base.pp.
This allows us to use the new crudini-based zulipconf function in any
of our manifests in the future.
2016-07-30 21:23:12 -07:00
Tim Abbott 922a4acdc5 puppet: Add zulip.conf setting for listen_addresses tuning. 2016-07-30 21:23:12 -07:00
Tim Abbott b2e32e6e3f puppet: Add postgres replication option for postgres_appdb_tuned.pp.
This adds a setting that controls whether to include the Zulip default
streaming replication configuration in the postgres configuration file.
2016-07-30 21:23:12 -07:00
Tim Abbott 3ab0295061 puppet: Add zulip.conf setting for effective_io_concurrency tuning. 2016-07-30 21:23:12 -07:00
Tim Abbott f3db368a3c puppet: Add zulip.conf setting for random_page_cost tuning. 2016-07-30 21:23:12 -07:00
Tim Abbott 63c757eac3 puppet: add zulipconf function for reading settings from zulip.conf.
This makes it convenient for us to have optional user-defined settings
in the main Zulip puppet configuration.
2016-07-30 21:23:12 -07:00
Eklavya Sharma 3b3b5c7c16 Rename management command email-mirror to email_mirror.
All other zulip management command names have underscores, so
rename email-mirror to email_mirror.

This will also make it possible to import this module, which will
help in writing tests for it.
2016-07-28 14:52:09 -07:00
Tim Abbott 69528790a5 check_worker_memory: Fix handling of no queue workers running. 2016-07-28 13:26:31 -07:00
Eklavya Sharma 408d070170 puppet/: Make subprocess calls unicode-aware. 2016-07-26 12:06:41 -07:00
Eklavya Sharma 3d87c376c2 Make zulip-ec2-configure-interfaces pass mypy check. 2016-07-24 12:52:58 +05:30
Eklavya Sharma 0a5aa2ccc7 Make check_send_receive_time pass mypy check. 2016-07-24 12:41:06 +05:30
Tim Abbott 2a209e46dd puppet: Fix remaining hardcoding of postgres 9.1 in manifests. 2016-07-20 21:11:22 -07:00
Tim Abbott d529bc12ef pg_backup_and_purge: Fix hardcoding of postgres version. 2016-07-19 19:19:42 -07:00
Tim Abbott bdb1ce04a2 puppet: Move pg_backup_and_purge to main config. 2016-07-19 19:19:42 -07:00
Tim Abbott 81136ff092 env-wal-e: Eliminate hardcoding of AWS keys.
Pre-Zulip being open sourced, this file just had the AWS keys for
backups hardcoded.

Instead, these are simply read from zulip-secrets.conf.
2016-07-19 19:19:42 -07:00
Tim Abbott 3023745ed7 puppet: Move env-wal-e to Zulip main distribution.
This doesn't yet include the dependencies for env-wal-e.
2016-07-19 19:19:42 -07:00
Tim Abbott d0dcc8bf26 puppet: Create static_asset_compiler manifest. 2016-07-19 16:38:09 -07:00
Tim Abbott 601ab24d2a zulip_internal::app_frontend: Remove now-unnecessary dependencies.
These Python packages are no longer relevant, since we install all of
our Python dependencies via virtualenvs.
2016-07-19 16:37:50 -07:00
Tim Abbott 77ccc0d87f Patch httpoxy security issue. 2016-07-18 10:24:03 -07:00
Eklavya Sharma da36947400 Change unbuffering strategy in runtornado.py.
runtornado unbuffers its output using
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0).
This is not python 3 compatible since we can't specify
buffering on a text stream in python 3.  So use the '-u'
option of python when calling runtornado.py to make output
unbuffered.
2016-07-17 10:31:15 -07:00
Tim Abbott ace8ae8301 puppet: Fix supervisor restart rule catching itself in pgrep. 2016-07-12 10:39:33 -07:00
Tim Abbott dfc9e75342 nginx: Enable shared SSL session cache. 2016-07-10 08:30:10 -07:00
Alex Gaynor ee47da8790 nginx: Disable RC4 and prefer server cipher order. 2016-07-10 08:30:05 -07:00
Tim Abbott ea089518ee Add Zulip customizations to Postgres 9.5 config file.
This includes reasonable tuning of memory usage parameters based on
the values that pgtune would use, roughly.
2016-07-08 16:16:12 -07:00
Tim Abbott e818dff0b0 postgres: Remove comment matching Puppet ERB syntax. 2016-07-08 16:16:12 -07:00
Tim Abbott 6ba659aeec Add a stock Ubuntu Xenial Postgres 9.5 configuration file. 2016-07-08 16:16:12 -07:00
Tim Abbott 0f5e62e994 puppet: Don't use pgtune on Linux versions where it doesn't exist. 2016-07-08 16:16:12 -07:00
Tim Abbott 903f728587 Scope postgresql.conf templates by postgres version. 2016-07-08 16:16:12 -07:00
Tim Abbott ffe79e0d50 Fix EPMD restart being attempted on every puppet apply. 2016-07-08 16:16:11 -07:00
Taranjeet a137bf15ed Wrap some lines with length greater than 120.
With some tweaks by tabbott.
2016-07-06 14:35:16 -07:00
Tim Abbott 809a45394f puppet: Start supervisord if it isn't running on restart. 2016-06-28 22:09:36 -07:00
Eklavya Sharma 7a98d7bd24 manifests/base.pp: Remove apt python package dependencies. 2016-06-27 19:55:35 -07:00
Eklavya Sharma ebad1415c0 app_frontend_base.pp: Remove apt python package dependencies. 2016-06-27 19:55:35 -07:00
Eklavya Sharma a9835c0ab2 Activate virtualenv in production Python code.
The manage.py change effectively switches the Zulip production server
to use the virtualenv, since all of our supervisord commands for the
various Python services go through manage.py.

Additionally, this migrates the production scripts and Nagios plugins
to use the virtualenv as well.
2016-06-27 19:55:35 -07:00
Tim Abbott 46c2416cc8 puppet: Add postgres version map for xenial.
This is an early step towards Ubuntu Xenial support in production.
2016-06-17 16:41:42 -07:00
Eklavya Sharma 94e4b39112 Replace python2.7 by python everywhere. 2016-05-29 05:03:08 -07:00
Eklavya Sharma 149938d468 Change shebangs from python2.7 to python. 2016-05-29 05:03:08 -07:00
Eklavya Sharma b6bd5445bc End puppet/.../process_fts_updates gracefully.
Catch KeyboardInterrupt in puppet/zulip/files/postgresql/process_fts_updates,
print a message and exit.
2016-05-24 13:28:05 -07:00
Eklavya Sharma 3441f0848c Annotate pg_backup_and_purge.py. 2016-05-18 17:10:17 +05:30
Umair Khan 5359e6b0d4 Convert Zulip to use Jinja2 templates.
This results in a substantial performance improvement for all of
Zulip's backend templates.

Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.

Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`

Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
  instead of `RequestContext` object. There are two reasons to do this:
    1. `RequestContext` is not compatible with Jinja2
    2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
  includes a custom backend and a template renderer, compressors for js
  and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.

Fixes #620.
2016-05-09 09:55:18 -07:00
Tim Abbott 00ccf147cd check_nagios_send_time: Add --insecure option for use in tests. 2016-05-08 17:35:50 -07:00
Tim Abbott c0d38f42f1 supervisor: Use 127.0.0.1 rather than localhost for tornado.
In theory these should be the same, but in misconfigured environments
(such at Travis CI) where /etc/hosts has multiple entries for
"localhost", 127.0.0.1 is safer than "localhost".
2016-05-08 17:35:50 -07:00
Tim Abbott cb81a59e38 Move write-rabbitmq-consumers-state-file to scripts/nagios/. 2016-05-07 19:37:06 -07:00
Tim Abbott 2761c012e5 Move rabbitmq consumer checks from bots/ to scripts/nagios/. 2016-05-07 19:37:06 -07:00
Tim Abbott be6566dc5c nagios: Move cron_file_helper from bots/ to scripts/lib.
This ensures the tool is available in Zulip production deployments.
2016-05-07 19:37:06 -07:00
Tim Abbott 52c1e8ac7d Run a local camo server in voyager production environments.
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS.  We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
2016-05-02 17:21:31 -07:00
Tim Abbott 5195d1ecb7 Fix missing newlines at ends of JSON files. 2016-04-14 10:48:52 -07:00
Tim Abbott caba24b2af Fix existing trailing whitespace in bash scripts. 2016-04-08 11:52:11 -07:00
Tim Abbott a7f83c9e05 Fix check_postgres_replication_lag nagios command. 2016-04-06 15:30:51 -07:00
Tim Abbott 991341867c nagios: Remove unnecessary dependency on netcat. 2016-04-06 15:30:40 -07:00
Tim Abbott c92221dcd3 Remove old humbughq apache configuration. 2016-04-06 15:30:36 -07:00
Tim Abbott 4855296771 puppet: Migrate check_postgres plugins to postgres_common.pp. 2016-04-06 15:20:36 -07:00
Tim Abbott 69a8925076 check_user_zephyr_mirror_liveness: Fix importing settings. 2016-04-05 13:27:04 -07:00
Tim Abbott 55172e2e0c Remove old zulip_internal nagios_plugins installation. 2016-04-05 13:27:04 -07:00
Tim Abbott 934e8641ee Migrate Zephyr mirror Nagios plugins to subdirectory. 2016-04-05 13:27:04 -07:00
Tim Abbott 7b753e5882 Migrate check_debian_packages to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 2da9fc56d6 Migrate check_pg_replication_lag to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott c2e210ca0d Migrate check_website_response.sh to new zulip::nagios.pp. 2016-04-05 13:27:04 -07:00
Tim Abbott eb72cecd9e Migrate check_fts_update_log to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 92d696d007 Migrate check_postgres plugins to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott e155ecdc49 Migrate check_rabbitmq plugins to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 3ed7d658f8 Migrate check_send_receive_time to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott ca45ec3f3f Migrate check_email_deliverer plugins to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 4e10424512 Migrate check_worker_memory to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 59b46278be Move check_queue_worker_errors into subdirectory.
This fixes an issue where this worker wasn't even being installed
properly in a way that sets us up for doing further reorganization of
the Zulip Nagios plugins.
2016-04-05 13:27:04 -07:00
Tim Abbott 6f20c43097 Move dependency on nagios_plugins into base.pp. 2016-04-05 13:27:04 -07:00
Ryan Moore 2fe0700f55 Update memcache -> remote cache in inline documentation. 2016-03-31 12:56:42 -07:00
Tim Abbott 8c0b110e9a Add python-typing dependency. 2016-03-30 21:50:25 -07:00
Tim Abbott c16749d783 Add missing dependency on netcat in both dev and prod.
Fixes #474.
2016-03-29 21:30:48 -07:00
Tim Abbott 2adf6d822f puppet: Fix process_queue command lines to use the new argument style.
cd2348e9ae broke installing Zulip in
production since it didn't correctly update the puppet configuration
to call the process_queue script using the new argument format.

This commit isn't ideal in that I'd prefer to not require updating
puppet in sync with the actual running code, but we don't have a great
mechanism for doing that.

Fixes #586.
2016-03-27 23:17:16 -07:00
Zev Benjamin 965f923ac3 Remove postgres2 configuration 2016-03-23 20:41:42 -07:00
Zev Benjamin ae2560a027 Add postgres3 configuration 2016-03-23 20:41:25 -07:00
Tim Abbott 6137ae9902 Fix incorrect shell quoting in check_worker_memory. 2016-03-23 20:40:06 -07:00
Tim Abbott 210c2897e7 Fix check_worker_memory regular expression. 2016-03-23 20:40:02 -07:00
Josh Mandel b06739df11 Move email digest triggering to default zulip config.
Previously, even though the Zulip digest emails were documented in the
settings, the cron job to run the script that actually sends the daily
digest emails wasn't included in the non-zulip.com part of the Zulip
production distribution.  The overall consequence is that digest
emails didn't work for non-zulip.com users.  This fixes that issue by
moving that cron job into the zulip manifests.

[commit message details expanded by tabbott]
2016-03-19 10:34:41 -07:00
Eklavya Sharma 37f9520666 Make the remaining ambiguous divisions python 3 compatible.
Refer to #256
2016-03-12 10:53:51 -08:00
Eklavya Sharma b9e792c4e6 Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six
Refer to #256
2016-03-10 22:03:58 -08:00
Eklavya Sharma d3b63f9a2d Apply Python 3 futurize transform libmodernize.fixes.fix_file
Refer to #256
2016-03-10 22:02:34 -08:00
Eklavya Sharma c59185e119 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import
Refer #256
2016-03-10 22:02:17 -08:00
Eklavya Sharma e2d5ec1868 Apply Python 3 futurize transform lib2to3.fixes.fix_ws_comma 2016-03-10 22:02:04 -08:00
Eklavya Sharma 4fb549abe8 Apply Python 3 futurize transform lib2to3.fixes.fix_idioms
Refer to #256
2016-03-10 22:02:01 -08:00
Eklavya Sharma f3d387e727 Apply Python 3 futurize transform lib2to3.fixes.fix_except 2016-03-10 22:01:50 -08:00
Zev Benjamin 4887a79d21 Make the nginx log directory owned by zulip
This is required to make log2zulip not error out when reading the nginx
error.log.
2016-02-02 23:05:44 -08:00
Tim Abbott 757e89260e Migrate use of StringIO to Python 2+3 compatible six.moves.cStringIO.
And add a check for additional usage of the original StringIO module.
2016-01-26 21:09:43 -08:00
Tim Abbott 1c73c992dd Fix missing puppet dependencies on postgres package. 2016-01-26 20:32:33 -08:00
Tim Abbott 2e16b44b24 puppet: Use $postgres_version in postgres template.
This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.
2016-01-26 20:32:33 -08:00
Tim Abbott 806aa986b7 puppet: Use a variable to configure the postgres version. 2016-01-26 20:32:33 -08:00
Tim Abbott a3ac56efe2 puppet: Make apt repository conditional on the Ubuntu version.
We still will need to address this in the install script as well.
2016-01-26 20:32:33 -08:00
Tim Abbott f6c59feb05 Document the puppet configuration somewhat in zulip::voyager. 2016-01-26 20:32:33 -08:00
Tim Abbott 345b5254d7 puppet: Move default nginx configuration out of voyager.pp. 2016-01-26 20:32:33 -08:00
Tim Abbott dd61e3f97d puppet: Move memcached and rabbitmq include out of app_frontend_base.pp. 2016-01-26 20:32:33 -08:00
Tim Abbott c3153274c1 puppet: Move memcached into its own puppet module. 2016-01-26 20:32:33 -08:00
Tim Abbott 8a0e07fe1a puppet: Rename app_frontend.pp to app_frontend_base.pp.
This will enable us to move the remaining app-frontend related content
out of voyager.pp.
2016-01-26 20:32:33 -08:00
Tim Abbott 91286d00aa puppet: Move Zulip apt repository to its own manifest. 2016-01-26 20:32:33 -08:00
Tim Abbott 69dd17dfb6 puppet: Move prod-static creation from voyager.pp to app_frontend.pp.
Every app frontend will need this directory and this should help
enable more modular puppet rules.
2016-01-26 20:32:33 -08:00
Tim Abbott 702f501638 puppet: Move tuned postgres configuration out of voyager.pp.
This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.
2016-01-26 20:32:28 -08:00
Tim Abbott d5f04bd20b Rename zulip::postgres_appdb to zulip::postgres_appdb_base.
The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.
2016-01-26 20:30:12 -08:00
Tim Abbott 3f27573cb2 puppet: Move several debugging tool dependencies out of base.pp. 2016-01-26 20:30:11 -08:00
Tim Abbott 2b0394d807 Add documentation explaining what process_fts_updates does. 2016-01-11 08:56:03 -08:00
Tim Abbott 0162dc4bc0 process_fts_updates: Cleanup and document new settings import logic. 2016-01-11 08:56:03 -08:00
Javier Ros a6a47aacde Add easy support for using a remote postgres database. 2016-01-11 08:56:03 -08:00
Tim Abbott a79e89b28f Cleanup remaining usage of % comprehensions without explicit tuples. 2015-12-05 15:29:42 -08:00
Tim Abbott 07419104a5 nginx: Enable gzip compression on main content types.
Apparently, previously nginx was only compressing text/html content.
This should result in a substantial savings in network traffic -- some
quick testing I did found it cut the total data transferred for
loading a logged-in zulip.com instance from 3MB to 1.2MB.
2015-11-24 05:21:49 -08:00
Luke Faraone db5c460cfc Run django.setup() in nagios plugins to avoid exceptions on 1.8
If running on Django 1.8, running these plugins would die with the below. A fix
for this is to run `django.setup()` before interacting with Django.

Refs:
    https://docs.djangoproject.com/en/1.8/ref/applications/#troubleshooting

```

Traceback (most recent call last):
  File "/usr/lib/nagios/plugins/check_send_receive_time", line 103, in <module>
    sender = get_user_profile_by_email(settings.NAGIOS_SEND_BOT)
  File "/home/zulip/deployments/current/zerver/lib/cache.py", line 113, in func_with_caching
    val = func(*args, **kwargs)
  File "/home/zulip/deployments/current/zerver/models.py", line 1073, in get_user_profile_by_email
    return UserProfile.objects.select_related().get(email__iexact=email.strip())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 977, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
    sql, params = self.as_sql()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 378, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 48, in pre_sql_setup
    self.setup_query()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 39, in setup_query
    self.select, self.klass_info, self.annotation_col_map = self.get_select()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 206, in get_select
    related_klass_infos = self.get_related_selections(select)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 700, in get_related_selections
    [f.name], opts, root_alias)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1471, in setup_joins
    names, opts, allow_many, fail_on_missing=True)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1372, in names_to_path
    if field.is_relation and not field.related_model:
  File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 60, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 110, in related_model
    apps.check_models_ready()
  File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
```
2015-11-23 18:54:08 +00:00
Tim Abbott f3783fb4a1 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import. 2015-11-01 09:26:16 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Guillaume Simon 956fd7c420 puppet: Ensure rabbitmq-server and epmd services are running.
[tabbott@mit.edu: Added a few comments]
2015-09-30 09:21:45 -07:00
Waseem Daher 6bb9b129f7 Update Zulip support email to zulip-devel@googlegroups.com.
Ideally some of these templates should really point to the
local installation's support email address, but this is a
good start.

Exceptions:
* Where to report security incidents
* MIT Zephyr-related pages
* zulip.com terms and conditions
2015-09-29 17:59:47 -07:00
Tim Abbott a625ca49ec puppet: Move /var/lib/nagios_state creation to zulip::base.pp.
Previously, in Zulip voyager, the cron jobs would spew error emails
every time they ran, due to this directory not existing.

This also tightens the permissions for the folder and avoids needing
to create a nagios user for Zulip voyager; it should be writeable by
both root and the zulip user and world-readable (and thus readable by
the Nagios user on zulip.com systems).
2015-09-26 21:44:23 -07:00
Tim Abbott 9748780192 Remove unnecessary puppet.conf configuration.
Fixes #23.
2015-09-26 21:34:19 -07:00
Tim Abbott bc3f096918 Update redis config to be supported on Trusty.
Previously our redis config was built for precise.

Synced from redis-server 2:2.8.4-2 plus our one change, which is
disabling saving to disk, so just put that at the bottom for maximum
obviousness.

I wish there was a better way to represent the fact that this is all
we're doing, since this will make life more difficult for running on
precise as well.

Fixes #28.
2015-09-26 21:33:55 -07:00
Anders Kaseorg 0d12dfd06f Improve shell quoting hygiene
Most of these problems were found by ShellCheck
(http://www.shellcheck.net).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2015-09-25 23:25:08 -04:00
Adam Smith f0add4638c Update the puppet nginx file to match apache
this updates the nginx configuration file to match the installation documentation requirements and apache configuration file.
2015-09-25 19:34:58 -05:00
Tim Abbott f4be74dafc Postgres configuration updates for Postgres 9.3.
(imported from commit d643e742f3e582ecf1cc4db9635039b938e18a06)
2015-09-25 09:45:26 -07:00
Tim Abbott abb1a13e31 Switch to new Trusty PPA for Voyager install process.
(imported from commit 91fdc16f121938d36c389f5b285f0da942558759)
2015-09-25 09:45:26 -07:00
Tim Abbott 3c3238d8e1 Fix puppet module name for voyager module.
(imported from commit c9db2eac98d77f94b432d96363c75cee01f2870d)
2015-09-25 09:45:26 -07:00
Zev Benjamin 81fe34d011 Stop deleting server logs and increase the size before rotation
(imported from commit c9bba20c5d7525b859833aa5eaa6ce210434970c)
2015-09-25 03:58:31 -07:00
Tim Abbott fcf4731ed9 python-django-south is no longer required with Django 1.8.
(imported from commit 4b07d0e7a8d926c2a412ec2d468b7b1437790e8a)
2015-09-25 00:22:15 -07:00
Tim Abbott 8269b4dc76 Upgrade Zulip voyager to postgres 9.3 in preparation for trusty.
(imported from commit 1dd8bd8fe1361979b3a52f595d2ff0c92bb03e05)
2015-09-25 00:22:15 -07:00
Tim Abbott ae4cb6df35 Make python-postmonkey not a required package.
This is in some ways a regression, but because we don't have
python-postmonkey packaged right now, this is required to make the
Zulip production installation process work on Trusty.

(imported from commit 539d253eb7fedc20bf02cc1f0674e9345beebf48)
2015-09-25 00:22:14 -07:00
Tim Abbott 2fb9560476 puppet: enterprise=>voyager for single-server configuration.
(imported from commit 7cc47ac7c470239cd00f69cfd3137b39a0d06e3a)
2015-09-24 22:40:59 -07:00
Tim Abbott 3e5e40a3ed Remove redis tunnel from staging configuration.
(imported from commit 855cec4607fb6dc3913db8d0ddab2184849b135e)
2015-09-24 19:34:28 -07:00
Tim Abbott 95056c6681 Rename zulip-enteprise.{key,combined-chain.crt} to drop -enterprise.
(imported from commit 6f51577866c36a768b1e63df59e5f8a6f4c64cf9)
2015-09-24 19:08:52 -07:00
Tim Abbott eff0d31dcd Fix existing tab-based whitespace in the codebase.
(imported from commit 3bd8f51d1bb303455597d238af47b1b5541aed04)
2015-09-19 23:23:02 -07:00