Commit Graph

798 Commits

Author SHA1 Message Date
Tim Abbott 91da4bd59b puppet: Add check_cron_file generic helper. 2016-10-26 00:11:08 -07:00
Tim Abbott 080dd8c987 nagios: Ignore kthreads in check_procs tests.
Modern Linux can have a lot of kernel threads not doing anything.
Since this isn't interesting from a monitoring perpsective, we ignore
these.
2016-10-26 00:10:40 -07:00
Tim Abbott 4c9a283542 puppet: Remove configuration for old builder host.
I don't think this configuration was ever even used; it's just
clutter.
2016-10-26 00:01:52 -07:00
Tim Abbott f9ad75f58e puppet: Remove configuration for old zulip.com bots host.
This configuration didn't do anything anyway and just clutters the
repo.
2016-10-26 00:01:29 -07:00
Tim Abbott 9d4f3f1e1b puppet: Replace zulip_ops postgres configs with postgres_appdb_tuned.
There's no longer a reason to have copies of forked postgres
configuration files in our repository, since some time ago we merged
the features of these configuration files into the main
postgres_appdb_tuned.pp.
2016-10-25 23:58:53 -07:00
Tim Abbott 105ea972f6 puppet: Remove now-unncessary kernel.shm sysctl values.
With modern Linux and postgres, these settings are not required.
2016-10-25 23:58:33 -07:00
Tim Abbott 2227e77cce puppet: Remove Dropbox usernames from Nagios config. 2016-10-25 23:55:42 -07:00
Tim Abbott 8584c05d80 zulip_ops: Remove unnecessary loadbalancer stanzas. 2016-10-25 23:52:37 -07:00
Tim Abbott 624ee3989f puppet: Remove old Dropbox certificates. 2016-10-25 23:52:30 -07:00
Tim Abbott f0bb78ba2d puppet: Fix iptables-persistent->netfilter-persistent rename. 2016-10-25 23:45:21 -07:00
Tim Abbott c4ca7ee6e1 puppet: Move Apache sites files to correct paths.
Apache now actually requires its configuration files have names ending
with .conf.
2016-10-25 23:44:28 -07:00
Tim Abbott 2b8324b778 emoji: Fix caching permissions issues.
Previously, you needed to be root to update the emoji cache, which
caused problems with how Zulip is upgraded in production.
2016-10-25 17:52:19 -07:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 56c0b80067 nginx: Fix JavaScript not being compressed properly.
Apparently, we weren't actually compressing our JavaScript being sent
over the wire due to incorrect nginx configuration.
2016-10-23 20:06:02 -07:00
Tim Abbott a5a03c2e0b zulip_ops: Include zulip::apt_repository.
This replaces the old wheezy configuration.
2016-10-16 20:13:35 -07:00
Tim Abbott 8c68c6f09b zulip_ops: Remove wheezy apt repo.
Nobody uses wheezy anymore, and the configuration wasn't even
conditional on the OS version.
2016-10-16 20:13:35 -07:00
Tim Abbott 5210b0a6a4 zulip_ops: Cleanup old redis configuration.
One can now just use the improved configuration we've merged into the
main Zulip repo.
2016-10-16 20:13:35 -07:00
Tim Abbott 869f0724ce zulip_ops: Remove humbughq.com nginx configuration.
The humbughq.com name hasn't been the product's name since 2013, and
it's nice to finish clearing it out of the repository.
2016-10-16 20:13:29 -07:00
Tim Abbott 29448fb47b zulip_ops: Remove old Zulip, Inc. trac configuration.
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
Tim Abbott 771e03cfa7 zulip_ops: Remove old Zulip, Inc. mediawiki configuration.
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
Tim Abbott 36e336edc3 puppet: Rename zulip_internal to zulip_ops.
The old "zulip_internal" name was from back when Zulip, Inc. had two
distributions of Zulip, the enterprise distribution in puppet/zulip/
and the "internal" SAAS distribution in puppet/zulip_internal.  I
think the name is a bit confusing in the new fully open-source Zulip
work, so we're replacing it with "zulip_ops".  I don't think the new
name is perfect, but it's better.

In the following commits, we'll delete a bunch of pieces of Zulip,
Inc.'s infrastructure that don't exist anymore and thus are no longer
useful (e.g. the old Trac configuration), with the goal of cleaning
the repository of as much unnecessary content as possible.
2016-10-16 19:23:27 -07:00
Tim Abbott 2e50dce9dd check_queue_worker_errors: Don't import zproject.settings directly. 2016-10-15 22:53:28 -07:00
Tim Abbott 913d913c90 puppet: Annotate Nagios plugins. 2016-10-15 22:44:15 -07:00
Tim Abbott 458e455b75 Annotate process_fts_updates. 2016-10-15 22:44:15 -07:00
Tim Abbott a4b2a6b6d4 puppet: Fix missing dependency for supervisor enabling.
You can't run `systemctl enable` on a service before the package
containing the service is installed.
2016-10-06 21:44:10 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
Tim Abbott 12ddb1a36f puppet: Fix buggy logrotate configuration. 2016-09-08 16:51:17 -07:00
Kouhei Sutou 683f49aa99 Support full text search for all languages using pgroonga.
This adds support for using PGroonga to back the Zulip full-text
search feature.  Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.

Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.

PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.

Fixes #615.

[docs and tests tweaked by tabbott]
2016-08-26 21:04:03 -07:00
Tim Abbott 546c23fd1f puppet: Activate pgroonga extension after installation.
The pgroonga extension needs to be activated using postgres root
access, so we make that happen from puppet.
2016-08-25 18:37:45 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
Tim Abbott 682d78bc30 puppet: Add default logrotate configuration.
This should prevent Zulip from eating all the disk now that we are
using the race-free WatchedFileHandler rather than
TimedRotatingFileHandler.
2016-08-24 14:30:54 -07:00
Tim Abbott 1c40df9363 static_asset_compiler: Use correct package name on trusty. 2016-08-22 23:47:03 -07:00
Tim Abbott 97b622dffa puppet: Fix postgres_appdb_tuned manifest on trusty. 2016-08-22 23:41:07 -07:00
Tim Abbott 750e43518f puppet: Ensure supervisord starts on boot with systemd.
This is important to ensure that Zulip comes up properly after a
reboot on Ubuntu Xenial.
2016-08-22 23:25:54 -07:00
Tim Abbott 2be597dd09 puppet: Move support for accepting a loadbalancer to zulip manifests.
Previously, we didn't have an easy process for how one should do this.

Now, it's an (undocumented) zulip.conf setting!
2016-08-22 17:19:03 -07:00
Tim Abbott 88a123d5e0 Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.

Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.

Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.

Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
Tim Abbott 852af83d3c check_postgres_replication_lag: Fix psql command line.
This allows the plugin to be run as users other than the "zulip" user,
where the "zulip" database would not be the default.
2016-08-12 13:19:08 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Tim Abbott 2581ac166c puppet: Automatically scale default memcached memory allocation.
Previously, we used a fixed memcached memory allocation of 512MB,
regardless of the size of the server.  While that is a good allocation
for a server with 4GB of RAM, for servers with less, we should
decrease the allocation, and for a large server with much more RAM, we
should increase it.  We still support the user overriding the
configuration setting, but this produces more sensible defaults.
2016-08-11 20:27:45 -07:00
Tim Abbott 2e0e8193d4 puppet: Make memcached memory allocation a variable. 2016-08-11 20:27:45 -07:00
Tim Abbott 3edf880c9b Remove moreutils from zulip_internal::base.
We recently added this to zulip::base, so it would be a duplicate.
2016-08-11 18:12:59 -07:00
Tim Abbott 9afb1c7a71 puppet: Stop wholesale replacing /etc/redis/redis.conf.
Zulip had only patches the redis configuration in one small way, which
resulted in unnecessary portability issues for using Redis on
different versions of Linux.  We replace this with just a adding an
include mechanism to the redis config.

While we're at it, we configure this to take advantage of the
new REDIS_PASSWORD secret to automatically configure redis passwords.
2016-08-11 17:26:04 -07:00
Tim Abbott cb21584ffe check_send_receive_time: Support being run in a cron job. 2016-08-11 14:48:21 -07:00
Tim Abbott 4dcbaf1e6b check_send_receive_time: Cleanup unnecessary print statement. 2016-08-11 14:48:21 -07:00
Tim Abbott 1d6ebd2b3d check_fts_update_log: Fix psycopg2 arguments.
* Fixes passing a string argument rather than an actual Python
  argument.
* Switches to hardcoding the database to connect to rather than the
  user, so this check can be run as an arbitrary user.
2016-08-03 14:58:46 -07:00
Tim Abbott 1c65508624 puppet: Add postgres user to Zulip group. 2016-08-02 14:59:25 -07:00
Tim Abbott 04fc3ff1e1 pg_backup_and_purge: Don't try to use a virtualenv.
This makes this actually work if the postgres server is different from
the Zulip app frontend.
2016-08-02 14:59:25 -07:00
Tim Abbott 4bdd9d3769 check_postgres_backup: Don't try to use a virtualenv.
The dependencies for this tool are installed using apt.
2016-08-02 14:59:25 -07:00
Tim Abbott a5115d54ee env-wal-e: Rename s3_backup_bucket to s3_backups_bucket.
This makes it consistent with the other variables in this file.
2016-08-02 14:59:15 -07:00
Tim Abbott ff80daef16 puppet: Move zulipsecret into its own file.
Apparently puppet doesn't support declaring multiple functions in the
same file.
2016-08-02 14:55:51 -07:00
Tim Abbott 75b5d021fa Remove unused puppet-common third-party module code. 2016-07-31 19:24:42 -07:00
Tim Abbott 6158acb41b nagios: Fix path to check_debian_packages. 2016-07-31 14:25:07 -07:00
Tim Abbott 6954dd84ab postgres_appdb_tuned: Add SSL certificate path configs. 2016-07-31 14:25:07 -07:00
Tim Abbott d66f6b8176 Fix postgresql configuration template for Ubuntu Xenial.
I incorrectly assumed that this was working because Travis CI passed
the build, whereas in fact Travis CI only tests the Trusty templates.
2016-07-31 14:24:34 -07:00
Tim Abbott 8b285ec0ff puppet: Read camo key from zulip-secrets.conf. 2016-07-31 00:23:24 -07:00
Tim Abbott 2b40309029 puppet: Add new zulipsecret function. 2016-07-31 00:23:24 -07:00
Tim Abbott 298e040bac puppet: Strip newlines in zulipconf function.
This avoids creating unnecessary extra newlines in config generated
using this.
2016-07-31 00:23:24 -07:00
Tim Abbott 16a4ce1bd2 puppet: Fix Nagios check_disk flagging special filesystems. 2016-07-30 23:57:31 -07:00
Tim Abbott c764b46cef puppet: Fix missing base include in nagios manifest. 2016-07-30 23:48:41 -07:00
Tim Abbott bc9bdd53aa puppet: Use hostgroup for postgres_other monitoring. 2016-07-30 23:46:05 -07:00
Tim Abbott 6b34bee806 puppet: Remove unused ancient humbug-default apache site. 2016-07-30 23:43:55 -07:00
Tim Abbott 217faed3b3 Update check_postgres plugin to version 2.22. 2016-07-30 23:19:40 -07:00
Tim Abbott 1f549dcfab check_fts_update_log: Fix use on non-appbd machines. 2016-07-30 21:48:17 -07:00
Tim Abbott 35edd8c9dd puppet: Move crudini dependency to base.pp.
This allows us to use the new crudini-based zulipconf function in any
of our manifests in the future.
2016-07-30 21:23:12 -07:00
Tim Abbott 922a4acdc5 puppet: Add zulip.conf setting for listen_addresses tuning. 2016-07-30 21:23:12 -07:00
Tim Abbott b2e32e6e3f puppet: Add postgres replication option for postgres_appdb_tuned.pp.
This adds a setting that controls whether to include the Zulip default
streaming replication configuration in the postgres configuration file.
2016-07-30 21:23:12 -07:00
Tim Abbott 3ab0295061 puppet: Add zulip.conf setting for effective_io_concurrency tuning. 2016-07-30 21:23:12 -07:00
Tim Abbott f3db368a3c puppet: Add zulip.conf setting for random_page_cost tuning. 2016-07-30 21:23:12 -07:00
Tim Abbott 63c757eac3 puppet: add zulipconf function for reading settings from zulip.conf.
This makes it convenient for us to have optional user-defined settings
in the main Zulip puppet configuration.
2016-07-30 21:23:12 -07:00
Eklavya Sharma 3b3b5c7c16 Rename management command email-mirror to email_mirror.
All other zulip management command names have underscores, so
rename email-mirror to email_mirror.

This will also make it possible to import this module, which will
help in writing tests for it.
2016-07-28 14:52:09 -07:00
Tim Abbott 69528790a5 check_worker_memory: Fix handling of no queue workers running. 2016-07-28 13:26:31 -07:00
Eklavya Sharma 408d070170 puppet/: Make subprocess calls unicode-aware. 2016-07-26 12:06:41 -07:00
Eklavya Sharma 3d87c376c2 Make zulip-ec2-configure-interfaces pass mypy check. 2016-07-24 12:52:58 +05:30
Eklavya Sharma 0a5aa2ccc7 Make check_send_receive_time pass mypy check. 2016-07-24 12:41:06 +05:30
Tim Abbott 2a209e46dd puppet: Fix remaining hardcoding of postgres 9.1 in manifests. 2016-07-20 21:11:22 -07:00
Tim Abbott d529bc12ef pg_backup_and_purge: Fix hardcoding of postgres version. 2016-07-19 19:19:42 -07:00
Tim Abbott bdb1ce04a2 puppet: Move pg_backup_and_purge to main config. 2016-07-19 19:19:42 -07:00
Tim Abbott 81136ff092 env-wal-e: Eliminate hardcoding of AWS keys.
Pre-Zulip being open sourced, this file just had the AWS keys for
backups hardcoded.

Instead, these are simply read from zulip-secrets.conf.
2016-07-19 19:19:42 -07:00
Tim Abbott 3023745ed7 puppet: Move env-wal-e to Zulip main distribution.
This doesn't yet include the dependencies for env-wal-e.
2016-07-19 19:19:42 -07:00
Tim Abbott d0dcc8bf26 puppet: Create static_asset_compiler manifest. 2016-07-19 16:38:09 -07:00
Tim Abbott 601ab24d2a zulip_internal::app_frontend: Remove now-unnecessary dependencies.
These Python packages are no longer relevant, since we install all of
our Python dependencies via virtualenvs.
2016-07-19 16:37:50 -07:00
Tim Abbott 77ccc0d87f Patch httpoxy security issue. 2016-07-18 10:24:03 -07:00
Eklavya Sharma da36947400 Change unbuffering strategy in runtornado.py.
runtornado unbuffers its output using
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0).
This is not python 3 compatible since we can't specify
buffering on a text stream in python 3.  So use the '-u'
option of python when calling runtornado.py to make output
unbuffered.
2016-07-17 10:31:15 -07:00
Tim Abbott ace8ae8301 puppet: Fix supervisor restart rule catching itself in pgrep. 2016-07-12 10:39:33 -07:00
Tim Abbott dfc9e75342 nginx: Enable shared SSL session cache. 2016-07-10 08:30:10 -07:00
Alex Gaynor ee47da8790 nginx: Disable RC4 and prefer server cipher order. 2016-07-10 08:30:05 -07:00
Tim Abbott ea089518ee Add Zulip customizations to Postgres 9.5 config file.
This includes reasonable tuning of memory usage parameters based on
the values that pgtune would use, roughly.
2016-07-08 16:16:12 -07:00
Tim Abbott e818dff0b0 postgres: Remove comment matching Puppet ERB syntax. 2016-07-08 16:16:12 -07:00
Tim Abbott 6ba659aeec Add a stock Ubuntu Xenial Postgres 9.5 configuration file. 2016-07-08 16:16:12 -07:00
Tim Abbott 0f5e62e994 puppet: Don't use pgtune on Linux versions where it doesn't exist. 2016-07-08 16:16:12 -07:00
Tim Abbott 903f728587 Scope postgresql.conf templates by postgres version. 2016-07-08 16:16:12 -07:00
Tim Abbott ffe79e0d50 Fix EPMD restart being attempted on every puppet apply. 2016-07-08 16:16:11 -07:00
Taranjeet a137bf15ed Wrap some lines with length greater than 120.
With some tweaks by tabbott.
2016-07-06 14:35:16 -07:00
Tim Abbott 809a45394f puppet: Start supervisord if it isn't running on restart. 2016-06-28 22:09:36 -07:00
Eklavya Sharma 7a98d7bd24 manifests/base.pp: Remove apt python package dependencies. 2016-06-27 19:55:35 -07:00
Eklavya Sharma ebad1415c0 app_frontend_base.pp: Remove apt python package dependencies. 2016-06-27 19:55:35 -07:00
Eklavya Sharma a9835c0ab2 Activate virtualenv in production Python code.
The manage.py change effectively switches the Zulip production server
to use the virtualenv, since all of our supervisord commands for the
various Python services go through manage.py.

Additionally, this migrates the production scripts and Nagios plugins
to use the virtualenv as well.
2016-06-27 19:55:35 -07:00
Tim Abbott 46c2416cc8 puppet: Add postgres version map for xenial.
This is an early step towards Ubuntu Xenial support in production.
2016-06-17 16:41:42 -07:00
Eklavya Sharma 94e4b39112 Replace python2.7 by python everywhere. 2016-05-29 05:03:08 -07:00
Eklavya Sharma 149938d468 Change shebangs from python2.7 to python. 2016-05-29 05:03:08 -07:00
Eklavya Sharma b6bd5445bc End puppet/.../process_fts_updates gracefully.
Catch KeyboardInterrupt in puppet/zulip/files/postgresql/process_fts_updates,
print a message and exit.
2016-05-24 13:28:05 -07:00
Eklavya Sharma 3441f0848c Annotate pg_backup_and_purge.py. 2016-05-18 17:10:17 +05:30
Umair Khan 5359e6b0d4 Convert Zulip to use Jinja2 templates.
This results in a substantial performance improvement for all of
Zulip's backend templates.

Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.

Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`

Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
  instead of `RequestContext` object. There are two reasons to do this:
    1. `RequestContext` is not compatible with Jinja2
    2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
  includes a custom backend and a template renderer, compressors for js
  and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.

Fixes #620.
2016-05-09 09:55:18 -07:00
Tim Abbott 00ccf147cd check_nagios_send_time: Add --insecure option for use in tests. 2016-05-08 17:35:50 -07:00
Tim Abbott c0d38f42f1 supervisor: Use 127.0.0.1 rather than localhost for tornado.
In theory these should be the same, but in misconfigured environments
(such at Travis CI) where /etc/hosts has multiple entries for
"localhost", 127.0.0.1 is safer than "localhost".
2016-05-08 17:35:50 -07:00
Tim Abbott cb81a59e38 Move write-rabbitmq-consumers-state-file to scripts/nagios/. 2016-05-07 19:37:06 -07:00
Tim Abbott 2761c012e5 Move rabbitmq consumer checks from bots/ to scripts/nagios/. 2016-05-07 19:37:06 -07:00
Tim Abbott be6566dc5c nagios: Move cron_file_helper from bots/ to scripts/lib.
This ensures the tool is available in Zulip production deployments.
2016-05-07 19:37:06 -07:00
Tim Abbott 52c1e8ac7d Run a local camo server in voyager production environments.
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS.  We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
2016-05-02 17:21:31 -07:00
Tim Abbott 5195d1ecb7 Fix missing newlines at ends of JSON files. 2016-04-14 10:48:52 -07:00
Tim Abbott caba24b2af Fix existing trailing whitespace in bash scripts. 2016-04-08 11:52:11 -07:00
Tim Abbott a7f83c9e05 Fix check_postgres_replication_lag nagios command. 2016-04-06 15:30:51 -07:00
Tim Abbott 991341867c nagios: Remove unnecessary dependency on netcat. 2016-04-06 15:30:40 -07:00
Tim Abbott c92221dcd3 Remove old humbughq apache configuration. 2016-04-06 15:30:36 -07:00
Tim Abbott 4855296771 puppet: Migrate check_postgres plugins to postgres_common.pp. 2016-04-06 15:20:36 -07:00
Tim Abbott 69a8925076 check_user_zephyr_mirror_liveness: Fix importing settings. 2016-04-05 13:27:04 -07:00
Tim Abbott 55172e2e0c Remove old zulip_internal nagios_plugins installation. 2016-04-05 13:27:04 -07:00
Tim Abbott 934e8641ee Migrate Zephyr mirror Nagios plugins to subdirectory. 2016-04-05 13:27:04 -07:00
Tim Abbott 7b753e5882 Migrate check_debian_packages to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 2da9fc56d6 Migrate check_pg_replication_lag to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott c2e210ca0d Migrate check_website_response.sh to new zulip::nagios.pp. 2016-04-05 13:27:04 -07:00
Tim Abbott eb72cecd9e Migrate check_fts_update_log to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 92d696d007 Migrate check_postgres plugins to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott e155ecdc49 Migrate check_rabbitmq plugins to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 3ed7d658f8 Migrate check_send_receive_time to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott ca45ec3f3f Migrate check_email_deliverer plugins to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 4e10424512 Migrate check_worker_memory to zulip/. 2016-04-05 13:27:04 -07:00
Tim Abbott 59b46278be Move check_queue_worker_errors into subdirectory.
This fixes an issue where this worker wasn't even being installed
properly in a way that sets us up for doing further reorganization of
the Zulip Nagios plugins.
2016-04-05 13:27:04 -07:00
Tim Abbott 6f20c43097 Move dependency on nagios_plugins into base.pp. 2016-04-05 13:27:04 -07:00
Ryan Moore 2fe0700f55 Update memcache -> remote cache in inline documentation. 2016-03-31 12:56:42 -07:00
Tim Abbott 8c0b110e9a Add python-typing dependency. 2016-03-30 21:50:25 -07:00
Tim Abbott c16749d783 Add missing dependency on netcat in both dev and prod.
Fixes #474.
2016-03-29 21:30:48 -07:00
Tim Abbott 2adf6d822f puppet: Fix process_queue command lines to use the new argument style.
cd2348e9ae broke installing Zulip in
production since it didn't correctly update the puppet configuration
to call the process_queue script using the new argument format.

This commit isn't ideal in that I'd prefer to not require updating
puppet in sync with the actual running code, but we don't have a great
mechanism for doing that.

Fixes #586.
2016-03-27 23:17:16 -07:00
Zev Benjamin 965f923ac3 Remove postgres2 configuration 2016-03-23 20:41:42 -07:00
Zev Benjamin ae2560a027 Add postgres3 configuration 2016-03-23 20:41:25 -07:00
Tim Abbott 6137ae9902 Fix incorrect shell quoting in check_worker_memory. 2016-03-23 20:40:06 -07:00
Tim Abbott 210c2897e7 Fix check_worker_memory regular expression. 2016-03-23 20:40:02 -07:00
Josh Mandel b06739df11 Move email digest triggering to default zulip config.
Previously, even though the Zulip digest emails were documented in the
settings, the cron job to run the script that actually sends the daily
digest emails wasn't included in the non-zulip.com part of the Zulip
production distribution.  The overall consequence is that digest
emails didn't work for non-zulip.com users.  This fixes that issue by
moving that cron job into the zulip manifests.

[commit message details expanded by tabbott]
2016-03-19 10:34:41 -07:00
Eklavya Sharma 37f9520666 Make the remaining ambiguous divisions python 3 compatible.
Refer to #256
2016-03-12 10:53:51 -08:00
Eklavya Sharma b9e792c4e6 Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six
Refer to #256
2016-03-10 22:03:58 -08:00
Eklavya Sharma d3b63f9a2d Apply Python 3 futurize transform libmodernize.fixes.fix_file
Refer to #256
2016-03-10 22:02:34 -08:00
Eklavya Sharma c59185e119 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import
Refer #256
2016-03-10 22:02:17 -08:00
Eklavya Sharma e2d5ec1868 Apply Python 3 futurize transform lib2to3.fixes.fix_ws_comma 2016-03-10 22:02:04 -08:00
Eklavya Sharma 4fb549abe8 Apply Python 3 futurize transform lib2to3.fixes.fix_idioms
Refer to #256
2016-03-10 22:02:01 -08:00
Eklavya Sharma f3d387e727 Apply Python 3 futurize transform lib2to3.fixes.fix_except 2016-03-10 22:01:50 -08:00
Zev Benjamin 4887a79d21 Make the nginx log directory owned by zulip
This is required to make log2zulip not error out when reading the nginx
error.log.
2016-02-02 23:05:44 -08:00
Tim Abbott 757e89260e Migrate use of StringIO to Python 2+3 compatible six.moves.cStringIO.
And add a check for additional usage of the original StringIO module.
2016-01-26 21:09:43 -08:00
Tim Abbott 1c73c992dd Fix missing puppet dependencies on postgres package. 2016-01-26 20:32:33 -08:00
Tim Abbott 2e16b44b24 puppet: Use $postgres_version in postgres template.
This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.
2016-01-26 20:32:33 -08:00
Tim Abbott 806aa986b7 puppet: Use a variable to configure the postgres version. 2016-01-26 20:32:33 -08:00
Tim Abbott a3ac56efe2 puppet: Make apt repository conditional on the Ubuntu version.
We still will need to address this in the install script as well.
2016-01-26 20:32:33 -08:00
Tim Abbott f6c59feb05 Document the puppet configuration somewhat in zulip::voyager. 2016-01-26 20:32:33 -08:00
Tim Abbott 345b5254d7 puppet: Move default nginx configuration out of voyager.pp. 2016-01-26 20:32:33 -08:00
Tim Abbott dd61e3f97d puppet: Move memcached and rabbitmq include out of app_frontend_base.pp. 2016-01-26 20:32:33 -08:00
Tim Abbott c3153274c1 puppet: Move memcached into its own puppet module. 2016-01-26 20:32:33 -08:00
Tim Abbott 8a0e07fe1a puppet: Rename app_frontend.pp to app_frontend_base.pp.
This will enable us to move the remaining app-frontend related content
out of voyager.pp.
2016-01-26 20:32:33 -08:00
Tim Abbott 91286d00aa puppet: Move Zulip apt repository to its own manifest. 2016-01-26 20:32:33 -08:00
Tim Abbott 69dd17dfb6 puppet: Move prod-static creation from voyager.pp to app_frontend.pp.
Every app frontend will need this directory and this should help
enable more modular puppet rules.
2016-01-26 20:32:33 -08:00
Tim Abbott 702f501638 puppet: Move tuned postgres configuration out of voyager.pp.
This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.
2016-01-26 20:32:28 -08:00
Tim Abbott d5f04bd20b Rename zulip::postgres_appdb to zulip::postgres_appdb_base.
The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.
2016-01-26 20:30:12 -08:00
Tim Abbott 3f27573cb2 puppet: Move several debugging tool dependencies out of base.pp. 2016-01-26 20:30:11 -08:00
Tim Abbott 2b0394d807 Add documentation explaining what process_fts_updates does. 2016-01-11 08:56:03 -08:00
Tim Abbott 0162dc4bc0 process_fts_updates: Cleanup and document new settings import logic. 2016-01-11 08:56:03 -08:00
Javier Ros a6a47aacde Add easy support for using a remote postgres database. 2016-01-11 08:56:03 -08:00
Tim Abbott a79e89b28f Cleanup remaining usage of % comprehensions without explicit tuples. 2015-12-05 15:29:42 -08:00
Tim Abbott 07419104a5 nginx: Enable gzip compression on main content types.
Apparently, previously nginx was only compressing text/html content.
This should result in a substantial savings in network traffic -- some
quick testing I did found it cut the total data transferred for
loading a logged-in zulip.com instance from 3MB to 1.2MB.
2015-11-24 05:21:49 -08:00
Luke Faraone db5c460cfc Run django.setup() in nagios plugins to avoid exceptions on 1.8
If running on Django 1.8, running these plugins would die with the below. A fix
for this is to run `django.setup()` before interacting with Django.

Refs:
    https://docs.djangoproject.com/en/1.8/ref/applications/#troubleshooting

```

Traceback (most recent call last):
  File "/usr/lib/nagios/plugins/check_send_receive_time", line 103, in <module>
    sender = get_user_profile_by_email(settings.NAGIOS_SEND_BOT)
  File "/home/zulip/deployments/current/zerver/lib/cache.py", line 113, in func_with_caching
    val = func(*args, **kwargs)
  File "/home/zulip/deployments/current/zerver/models.py", line 1073, in get_user_profile_by_email
    return UserProfile.objects.select_related().get(email__iexact=email.strip())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 977, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
    sql, params = self.as_sql()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 378, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 48, in pre_sql_setup
    self.setup_query()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 39, in setup_query
    self.select, self.klass_info, self.annotation_col_map = self.get_select()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 206, in get_select
    related_klass_infos = self.get_related_selections(select)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 700, in get_related_selections
    [f.name], opts, root_alias)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1471, in setup_joins
    names, opts, allow_many, fail_on_missing=True)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1372, in names_to_path
    if field.is_relation and not field.related_model:
  File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 60, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 110, in related_model
    apps.check_models_ready()
  File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
```
2015-11-23 18:54:08 +00:00
Tim Abbott f3783fb4a1 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import. 2015-11-01 09:26:16 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Guillaume Simon 956fd7c420 puppet: Ensure rabbitmq-server and epmd services are running.
[tabbott@mit.edu: Added a few comments]
2015-09-30 09:21:45 -07:00
Waseem Daher 6bb9b129f7 Update Zulip support email to zulip-devel@googlegroups.com.
Ideally some of these templates should really point to the
local installation's support email address, but this is a
good start.

Exceptions:
* Where to report security incidents
* MIT Zephyr-related pages
* zulip.com terms and conditions
2015-09-29 17:59:47 -07:00
Tim Abbott a625ca49ec puppet: Move /var/lib/nagios_state creation to zulip::base.pp.
Previously, in Zulip voyager, the cron jobs would spew error emails
every time they ran, due to this directory not existing.

This also tightens the permissions for the folder and avoids needing
to create a nagios user for Zulip voyager; it should be writeable by
both root and the zulip user and world-readable (and thus readable by
the Nagios user on zulip.com systems).
2015-09-26 21:44:23 -07:00
Tim Abbott 9748780192 Remove unnecessary puppet.conf configuration.
Fixes #23.
2015-09-26 21:34:19 -07:00
Tim Abbott bc3f096918 Update redis config to be supported on Trusty.
Previously our redis config was built for precise.

Synced from redis-server 2:2.8.4-2 plus our one change, which is
disabling saving to disk, so just put that at the bottom for maximum
obviousness.

I wish there was a better way to represent the fact that this is all
we're doing, since this will make life more difficult for running on
precise as well.

Fixes #28.
2015-09-26 21:33:55 -07:00
Anders Kaseorg 0d12dfd06f Improve shell quoting hygiene
Most of these problems were found by ShellCheck
(http://www.shellcheck.net).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2015-09-25 23:25:08 -04:00
Adam Smith f0add4638c Update the puppet nginx file to match apache
this updates the nginx configuration file to match the installation documentation requirements and apache configuration file.
2015-09-25 19:34:58 -05:00
Tim Abbott f4be74dafc Postgres configuration updates for Postgres 9.3.
(imported from commit d643e742f3e582ecf1cc4db9635039b938e18a06)
2015-09-25 09:45:26 -07:00
Tim Abbott abb1a13e31 Switch to new Trusty PPA for Voyager install process.
(imported from commit 91fdc16f121938d36c389f5b285f0da942558759)
2015-09-25 09:45:26 -07:00
Tim Abbott 3c3238d8e1 Fix puppet module name for voyager module.
(imported from commit c9db2eac98d77f94b432d96363c75cee01f2870d)
2015-09-25 09:45:26 -07:00
Zev Benjamin 81fe34d011 Stop deleting server logs and increase the size before rotation
(imported from commit c9bba20c5d7525b859833aa5eaa6ce210434970c)
2015-09-25 03:58:31 -07:00
Tim Abbott fcf4731ed9 python-django-south is no longer required with Django 1.8.
(imported from commit 4b07d0e7a8d926c2a412ec2d468b7b1437790e8a)
2015-09-25 00:22:15 -07:00
Tim Abbott 8269b4dc76 Upgrade Zulip voyager to postgres 9.3 in preparation for trusty.
(imported from commit 1dd8bd8fe1361979b3a52f595d2ff0c92bb03e05)
2015-09-25 00:22:15 -07:00
Tim Abbott ae4cb6df35 Make python-postmonkey not a required package.
This is in some ways a regression, but because we don't have
python-postmonkey packaged right now, this is required to make the
Zulip production installation process work on Trusty.

(imported from commit 539d253eb7fedc20bf02cc1f0674e9345beebf48)
2015-09-25 00:22:14 -07:00
Tim Abbott 2fb9560476 puppet: enterprise=>voyager for single-server configuration.
(imported from commit 7cc47ac7c470239cd00f69cfd3137b39a0d06e3a)
2015-09-24 22:40:59 -07:00
Tim Abbott 3e5e40a3ed Remove redis tunnel from staging configuration.
(imported from commit 855cec4607fb6dc3913db8d0ddab2184849b135e)
2015-09-24 19:34:28 -07:00
Tim Abbott 95056c6681 Rename zulip-enteprise.{key,combined-chain.crt} to drop -enterprise.
(imported from commit 6f51577866c36a768b1e63df59e5f8a6f4c64cf9)
2015-09-24 19:08:52 -07:00
Tim Abbott eff0d31dcd Fix existing tab-based whitespace in the codebase.
(imported from commit 3bd8f51d1bb303455597d238af47b1b5541aed04)
2015-09-19 23:23:02 -07:00
Tim Abbott b7bb55932a Remove unused customer29.zulip.com nginx configuration.
(imported from commit cbba9437d872bd480a276aeaa8103f2da5210f88)
2015-09-19 21:54:51 -07:00
David Roe e3f38acbce Enterprise => Voyager.
(imported from commit 41b9a67301aeaf5fd40bbbb8f34a326ca98431fd)
2015-08-21 10:33:35 -07:00
Reid Barton ab9539cffe Remove OpenID authentication
(imported from commit 70a859041a851ed10dc40cfc068330e472d2ed09)
2015-08-20 23:52:48 -07:00
Jonathan Dahan e95dfd78ed Fix production deployment script.
(imported from commit 5b8398a0075dc297d3f86c65ce2ef43977e57ecb)
2015-08-20 20:50:41 -07:00
Tim Abbott e4fce10f46 Give root higher nofile limits in limits.conf.
(imported from commit f7fd9ce325449931f8866f6696f8484769c39a14)
2015-08-16 13:39:48 -07:00
Zev Benjamin 8ecb555a98 Update log2zulip bot config
(imported from commit c25802d55cd50c1a81050466d9a59a29501aca87)
2015-02-05 22:46:30 -08:00
Luke Faraone d3ebfbd042 Redirect c.d.n users to google auth
(imported from commit 72d6d486593155e346e7e856149fca3712868d89)
2015-02-05 21:40:23 -08:00
Luke Faraone e241e6e833 [manual] Support chat.dropboxer.net hostname.
This needs to be deployed on both prod and lb0 to be functional

DEPLOY INSTRUCTIONS: restart carefully

(imported from commit d97a450754608357418c80e5b3c7b3bbcd1d09fb)
2015-02-05 17:55:29 -08:00
Luke Faraone acccd2d5be [manual] Cert for chat.dropboxer.net from Tom Cook.
DEPLOY INSTRUCTIONS: Copy the cert over and restart carefully.

(imported from commit e7d96b231a70de89d3d8c5641df70354e7b7319d)
2015-02-05 17:55:23 -08:00
Luke Faraone e8f63e2041 [manual] external-content now shares www IP
This is safe because we have the wildcard-all cert.

DEPLOY INSTRUCTIONS: Change the CNAME in R53 for external-content.zulipcdn.net
to the same as www.zulip.com

(imported from commit 075984943ce3a3b17518b913ea650992e45f705e)
2015-02-05 17:49:35 -08:00
Luke Faraone 8a8356221e puppet: remove dbx.zulip.com -- HW project
(imported from commit 6edc3f1af44969427b2a12bee1983367855b4b88)
2015-02-05 17:48:47 -08:00
Luke Faraone 0f27554423 puppet: TLS cipher changes
This has been live on lb0 for a few months.

(imported from commit 32688aa82a891dd34ca1e7fbcdd84f18b25880c6)
2015-02-05 17:19:51 -08:00
Tim Abbott 87d1809657 Send digest emails 3 hours later now that we're on the west coast.
(imported from commit c7d5d0b30ac21314518132b5cf7dd9d7b2780f99)
2015-01-12 22:18:49 -08:00
Luke Faraone b2688bcf43 Add dbx to nginx config
(imported from commit 48f633eb1f4e949cfdb3ededc78bb774163fde19)
2014-10-06 15:27:55 -07:00
Luke Faraone d515f92d20 CORS headers for hack week API access
This has been live on staging for a while.

(imported from commit bd8b7ef500887fefc10656121fde389dca2aa217)
2014-10-06 15:27:55 -07:00
Jason Michalski 0bc2f43e32 Add a new postfix virtual address for missed message emails
The one time use email addresses are prefixed with mm and need be sent
to the local zulip user to be picked up by the email mirror.

(imported from commit e17cfe6855ab7886f25ded52790b8f31df955ef2)
2014-10-06 15:27:54 -07:00
acrefoot f62b9a199c fix typo in rules.v4 for iptables
(imported from commit 198259fec740c9e025ed56fd6e0b5d42f8d4323c)
2014-06-10 23:29:10 -07:00
Luke Faraone 5507575b7e Switch install scripts to use apt.zulip.net for ops systems
(imported from commit 3ff243459237ac673c5e8ae23077e628fa07a6b9)
2014-06-10 13:16:27 -07:00
Zev Benjamin 23c108a05c nagios: Check HTTPS instead of HTTP
(imported from commit ba0bb76d9bea6661e5396308eb431ff95ef51771)
2014-06-05 17:30:15 -07:00
Luke Faraone b383884019 Change expected autossh processes to 10
(imported from commit 41b06ce3f7cded7a29101a6de2d471bdffab5bcc)
2014-05-15 10:49:54 -07:00
Luke Faraone ef8b6e5a42 New certificates from Dropbox.
Thanks Tom Cook for getting these through Digicert!

We no longer need separate wildcard certificates, etc, because we have SAN star
certs.

(imported from commit 40a8961da51b6a0ae90c68b40b2af6d59cb5cf9f)
2014-05-05 18:24:14 -07:00
Zev Benjamin b190a24f40 puppet: Convert zmirror2 to use apt module for debathena sources
(imported from commit 67c71e15b5e5cbdffaa12953eaa7c0c72e7b107a)
2014-04-17 17:12:36 -07:00
Zev Benjamin 08a6969f48 puppet: Fix zmirror apt source resource
The resource is called "apt::source", not "apt::sources_list"

(imported from commit 5ace3d9a62e361b3c7f0b54bf69ac91c1136bb6d)
2014-04-17 17:12:36 -07:00
Zev Benjamin 50b763b12b puppet: Make iptables rules a puppet template
This allows us to specify different rules for the zmirror machines, which need
ports open for Zephyr.

(imported from commit f3c061e9492cbb99783f156debccf03161347e47)
2014-04-17 17:12:36 -07:00
Tim Abbott 52424cd67a Improve Zephyr personal mirror Nagios check.
(imported from commit 713f8f1ff1463537b46d493ac3571c2727d85379)
2014-04-11 16:44:13 -07:00
Zev Benjamin caef3f8bf3 [puppet] Allow Zulip to be loaded in an iframe on all domains
This removes "X-Frame-Options DENY" from our nginx config.  We need to be able
to load Zulip in an iframe for embedding and we decided that it doesn't actually
provide much protection.

(imported from commit 5bc363693db949010f6163cb3000c12229618a83)
2014-04-07 14:42:02 -07:00
Zev Benjamin bd20b295e1 [manual] Add python-pyjwt dependency
This will be used in the next commit.

(imported from commit 559fedd41fb731575ba7201dfbcef45d03461ef2)
2014-04-04 16:51:32 -07:00
Zev Benjamin 8518801601 Untabify app_frontend.pp
(imported from commit 2555cafa3bf3f9ef3551e4beab7ca2ad17ed05d4)
2014-04-04 16:51:31 -07:00
Zev Benjamin 286bd3005d nagios: Disable idle transaction checks
We apparently still have some process that occationally sits idle in a
transaction for a while, which makes this alert super noisy.

(imported from commit 074b04ad746bac0da1b8714763538d1ce22da64e)
2014-03-17 14:17:43 -04:00
Jessica McKellar 1c06e37245 [puppet] Change digest email send time by 1 hour for daylight savings.
(imported from commit 8ac1747f15e10b850f73ce28ea4579bee62ba1a4)
2014-03-14 20:48:49 -04:00
Zev Benjamin f7b64827e4 nagios: Don't check txn_time on trac
Doing so requires superuser privileges because check_postgres.pl only connects
to one database for that action.  We could theoretically work around this, but I
don't think it's worthwhile for non-production DBs.

(imported from commit 3ab06e4dd6f844c81128b81709cdc3cdfbe37c47)
2014-03-14 20:48:46 -04:00
Zev Benjamin d445386adc nagios: Add Postgres check for disabled triggers
(imported from commit 08ff85aecfc44c9226e637383464fae4d2b8997a)
2014-03-14 20:48:44 -04:00
Zev Benjamin 1653541e83 nagios: Re-enable Postgres transaction time checks
We believe these will generally no longer be disruptive now that we have
autocommit enabled.

(imported from commit c8c1301e0d4b188d6708173cd8c8b16279e3d910)
2014-03-14 20:48:44 -04:00
Tim Abbott 12309c61b6 Remove Nagios monitoring for the old email mirror.
(imported from commit fc4d95b12d5ee29438a2d3e7d8d694e8aa21f202)
2014-03-12 21:15:19 -04:00
Jessica McKellar e7ef654b45 [puppet]: Adjust zmirror Nagios checks to be more tolerant of a bad network.
We get a lot of alerts and sometimes pages due to network blips.

(imported from commit 4766585e71533b8551d49fa61bc4653114a65457)
2014-03-11 13:06:16 -04:00
Jessica McKellar 8a8efdc537 Use /usr/bin/env python for process_fts_updates.
`/usr/bin/env python` is almost always preferred over specifying the
specific python to run (and this script doesn't work for me on OSX
with /usr/bin/python specified).

(imported from commit 531e6062ba0ac1f25e3c681bb5cf83a918d0e3e7)
2014-03-03 16:08:32 -05:00
Luke Faraone e80c4f13b9 Don't include an "@" in mailname(5)
(imported from commit 84486373966c90d5fe2dd4af5966ae614628188f)
2014-02-27 20:25:34 -05:00
Leo Franchi 8ec1d9b552 [manual] Release desktop app 0.4.4
This requires a puppet apply on prod, as well as manually
updating the symlinks of Zulip-latest and Humbug-latest on
prod0

(imported from commit c5ef8cd0e2d156144531b35af9a8c5226f5bf750)
2014-02-27 20:25:26 -05:00
Leo Franchi acec697fe7 Report unnarrow times as well as narrow times
(imported from commit b3a889aa11dc112508c5a1d213f68e5223a879fc)
2014-02-13 14:45:22 -05:00
Zev Benjamin 41e3a89398 [manual] puppet: Puppetize Munin
To deploy this, the zulip_internal::base and zulip_internal::munin classes must
be added to nagios.zulip.net.

(imported from commit 50d6a4ed19fcc9c62c7104977d69043bf5b9bbf9)
2014-02-13 13:26:40 -05:00
Leo Franchi 2efaf75b25 Release desktop app v0.4.3
(imported from commit 13f5b79ce483db22cfa136a1318eadc4d04eb746)
2014-02-12 16:18:34 -05:00
Zev Benjamin 32d66d6f73 [manual] Monitor the new redis servers with nagios and munin
We have to start the tunnels up manually and add them to the wiki

(imported from commit aa5f80630a651c3fb33bba321e9d4444b5c498a2)
2014-02-10 13:23:28 -05:00
Zev Benjamin 631783f3cd [manual] Use dedicated Redis server for staging
Before we deploy this commit, we must migrate the data from the staging redis
server to the new, dedicated redis server.  The steps for doing so are the
following:

* Remove the zulip::redis puppet class from staging's zulip.conf
* ssh once from staging to redis-staging.zulip.net so that the host key is known
* Create a tunnel from redis0.zulip.net to staging.zulip.net
  * zulip@redis0:~$ ssh -N -L 127.0.0.1:6380:127.0.0.1:6379 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 staging.zulip.net
* Set the redis instance on redis0.zulip.net to replicate the one on staging.zulip.net
  * redis 127.0.0.1:6379> slaveof 127.0.0.1 6380
* Stop the app on staging
* Stop redis-server on staging
* Promote the redis server on redis0.zulip.net to a master
  * redis 127.0.0.1:6379> slaveof no one
* Do a puppet apply at this commit on staging (this will bring up the tunnel to redis0)
* Deploy this commit to staging (start the app on staging)
* Kill the tunnel from redis0.zulip.net to staging.zulip.net
* Uninstall redis-server on staging

The steps for migrating prod will be the same modulo s/staging/prod0/.

(imported from commit 546d258883ac299d65e896710edd0974b6bd60f8)
2014-02-10 13:23:28 -05:00
Zev Benjamin 1d7976d332 puppet: Add manifest for dedicated Redis server
(imported from commit 894ad5ca005de0fb9a64bfb58da374f72734eb8d)
2014-02-10 13:23:28 -05:00
Zev Benjamin 4d91bb39d3 [manual] puppet: Split out redis server configuration from app_frontend
The zulip::redis puppet class should be added to all our frontends' zulip.conf
after this is deployed.  No puppet apply is required.

(imported from commit ccea89f4779c6c49c0cbe837adcb5be21bfe55ab)
2014-02-10 13:23:28 -05:00
Luke Faraone c7565222f0 Fail fast if fqdn is not defined on Enterprise with Postfix
Otherwise, we won't be able to generate valid configuration files.

(imported from commit 5ec1a43fed5991dc609c470b596926a5febcd4c5)
2014-02-07 01:02:06 -05:00
Luke Faraone 602f7f96e5 Move postfix inclusion from public app_frontend to internal manifest
Otherwise, we will enable the postfix config on all frontends,
regardless of whether Enterprise deployments requested it.

(imported from commit 9592be3706adcee7547f6795f32fe7b8d85e71ee)
2014-02-07 01:01:33 -05:00
Luke Faraone 60cfd3cfb0 Accept SMTP connections on hosts.
(imported from commit 524ae3f4362ffea12ff96498ae554322f7fe8a3c)
2014-02-06 12:14:21 -05:00
Luke Faraone 24f8492236 [manual] Enable local email mirror on all frontends.
This removed the cronjob from all app_frontend servers and enables the
local Postfix mail server on the same.

This is a no-op on staging if the parent commit has already been
applied.

To deploy this commit, run a puppet-apply on prod.

(imported from commit 6d3977fd12088abcd33418279e9fa28f9b2a2006)
2014-02-06 10:26:56 -05:00
Luke Faraone 30a6fd3bd7 [manual] Enable postfix email mirror on staging
This will cause us to recieve messages sent to streams.staging.zulip.com
via the local Postfix daemon running on staging.

This commit does not impact prod. To deploy, a puppet-apply is needed on
staging.

(imported from commit 9eaedc28359f55a65b672a2e078c57362897c0de)
2014-02-04 10:38:17 -05:00
Luke Faraone 882047515c [manual] Move polling email mirror to prod from staging
This will allow us to roll out the Postfix-based mirror on staging in
the future without impacting production mirroring.

This branch should be puppet-deployed first on prod, then staging.

(imported from commit eceaa6c02a06f7074cacc19c6439e5928eef3ae4)
2014-02-04 10:38:17 -05:00
Luke Faraone 374acb7f24 [puppet] Move email mirror cron to public module
This way we can reference it in the documentation.

(imported from commit 37d5cbfcfb745e2b44768674f53d7ba450518cd0)
2014-02-04 10:38:17 -05:00
Luke Faraone de56b947d4 Remove unused postfix aliases file.
(imported from commit f40cb5b532aaf6421b9dd55a197644ecf65021a4)
2014-02-04 10:38:17 -05:00
Luke Faraone 38636d5125 Puppet configuration for postfix
(imported from commit 230325f6233c6d32ecab5f9fa3fc102373b22039)
2014-01-31 15:33:15 -05:00
Luke Faraone 760cd7a474 email-mirror: Run queue worker from supervisord
(imported from commit f496046bbc92b3d3b41aa15c3fbdd1d38556d6d0)
2014-01-31 15:33:15 -05:00
Luke Faraone 3263d09939 Convert zmirror to use puppet apt module for debathena sources
(imported from commit 080d59d2ac750d03b55460752d7fe7d02e72611c)
2014-01-31 13:43:04 -05:00
Luke Faraone aa52475e96 Switch to puppetlabs/apt
(imported from commit b2f581280dc7877051ef79d86eac671bfd455ace)
2014-01-31 13:43:04 -05:00
Tim Abbott 532cd061fb [puppet] Raise maximum items per page for trac.
(imported from commit 2ffa5e04c220a87d51cba42ade89874cc43ba584)
2014-01-29 17:22:19 -05:00
Tim Abbott 5108253e97 nagios: Make Zephyr mirroring alerts not pageable.
(imported from commit ab98af762b1edf93703fc865496aedc59ce7bd2d)
2014-01-24 13:53:48 -05:00
Zev Benjamin 759d33fad1 puppet: Check all disks via nagios, not just /
(imported from commit 0bc9fc150e791ce3ccec99688f3593a8678a87c9)
2014-01-23 13:37:27 -05:00
Tim Abbott 57c7634a4e Increase Zulip worker memory limits.
(imported from commit 6969eb1d2db0ee47c7b115b7f9b55ded2c9265dd)
2014-01-22 17:19:19 -05:00
Zev Benjamin c4e1d9f02a puppet: check_postgres_backup: Connect to the 'postgres' database
This allows the utility to run on trac.zulip.net, which doesn't have a 'zulip'
database.

(imported from commit c8eabb89e5e161191d6f2c92ca2b1428b17a9aa0)
2014-01-22 12:07:57 -05:00
Zev Benjamin 49f2657c8d nagios: Add check_postgres checks for the trac and wiki databases
We don't do the sequence check because that requires read access to the database
itself, which the zulip user doesn't have.

(imported from commit fba7604826353b2974e9757f01dcb426297993b3)
2014-01-22 12:07:56 -05:00
Zev Benjamin 3840cf760f nagios: Move a few services from hostgroup postgres -> hostgroup postgres_appdb
(imported from commit 54a738f19f176d36526d40968c379f6357d56e6b)
2014-01-22 12:07:56 -05:00
Zev Benjamin 1ae040c7fb nagios: Specify the db and user for check_postgres via arguments
(imported from commit c3b1a7fe7c63094ed8956ed1bdf4861d747637bd)
2014-01-22 12:07:56 -05:00
Zev Benjamin a974301b8b nagios: Add trac to the postgres_other hostgroup
(imported from commit 7e531b982b8f8961f2201cdc8b88d90d5d238907)
2014-01-22 12:07:56 -05:00
Zev Benjamin 41e274a8e4 nagios: Split postgres hostgroup into more fine-grained groups
(imported from commit ab5fcc0893fb8635defecdf3045a3ffdd5e26f14)
2014-01-22 12:07:56 -05:00
Leo Franchi e734155a1c Mount and make graphite backup drive when creating stats1
(imported from commit f8af032fa314812610d0ec7eb6227ebb0b3c2f32)
2014-01-22 10:49:49 -05:00
Luke Faraone 92ae790130 [manual] Switch listen address to www.humbughq.com for humbughq.com domains
We cannot use SNI for these legacy domains because old plugins still
connect to them.

This commit (along with the three previous commits) requires a lb0 nginx
deployment to function.

(imported from commit f47f3d7b597666508b3817d965fe8ce19d50c2c0)
2014-01-21 11:15:08 -05:00
Luke Faraone e852580a0e Use correct key for humbughq SAN cert.
This is live right now.

(imported from commit 051a44e2962557f3fc293e3e2f2e169a5d6e658c)
2014-01-21 11:15:07 -05:00
Luke Faraone c9158dd3d9 [manual] Use SNI cert instead of wildcard for humbughq
To deploy, the certs need to manually be copied to lb0's /etc/ssl/certs
directory, the nginx config updated, and the server restarted

(imported from commit c70c7678cd010a1b2b0aba830ab3d862005bd627)
2014-01-17 15:03:29 -05:00
Tim Abbott 7ce692b3c3 Restore serving the app on humbughq.
Partially reverts b1a8de8763

(imported from commit ddd9443d527f1e46f78008178b2410374551b8a6)
2014-01-17 15:03:29 -05:00
Luke Faraone 846be23ce2 Load SNI-enabled www.hhq.c cert
This replaces the old www.humbughq.com cert.

Contains these hostnames:
 * www.humbughq.com
 * api.humbughq.com
 * humbughq.com

Generated per 9d674d6a0.

(imported from commit 0ef3f0ff2a02996246868466b5e634ebf45439a2)
2014-01-17 15:03:16 -05:00
Luke Faraone ce50478a1e Move humbughq.com hosts to www.zulip.com IP
These are redirect hosts, so they don't need their own IP.  Supporting
non-SNI clients isn't a priority for us.

(imported from commit b1a8de8763ab944885518c868e4e30307d84c11d)
2014-01-16 15:56:16 -05:00
Luke Faraone 2c86c5c8ee Redirect humbughq domains to www.zulip.com per Waseem.
(imported from commit d5b8e8f33787d2a590516219ca4043b304b80a21)
2014-01-16 15:54:53 -05:00
Luke Faraone b6a2208d84 nginx configuration for customer29 on lb0
(imported from commit 7b6712e3e68aca71e81a6224af7d3f876af6ab1e)
2014-01-16 15:54:53 -05:00
Luke Faraone 8ebf0a414c Remove expired and unused SSL certificates
(imported from commit 7b058878183edc6cca593df6cd4b8cfeb15bab70)
2014-01-16 15:54:53 -05:00
Zev Benjamin 20e4e31dcf puppet: Update env-wal-e to take the S3 bucket to use from /etc/zulip/zulip.conf
This will let us do normal puppet applies on our postgres hosts again.

Crudini is already installed and /etc/zulip/zulip.conf has already been edited
on the relevant hosts.

(imported from commit 8e2b88d2fe2f7b2367ecb73a50a299200fe381a0)
2014-01-16 15:23:21 -05:00
Zev Benjamin ab1aafeb1c puppet: Add python-sqlalchemy dependency
(imported from commit 1ed6a8a730d368a97fad6cd478ec13e75504b789)
2014-01-14 11:47:12 -05:00
Zev Benjamin ef5ed9f9b9 puppet: Add postgresql-9.1-tsearch-extras dependency
Note that this change can not currently be applied on postgres hosts due to the
postgres puppet config currently being slightly broken.

(imported from commit 5d8ddeabfd9612d469a048256d22949c0bfa6aba)
2014-01-14 11:47:12 -05:00
Luke Faraone 16ae70948f Move python-googleapi dep to public Zulip manifest
(imported from commit 20298f82fbd674b3cf6b67b7741bf800b9733f36)
2014-01-13 16:24:21 -05:00
Luke Faraone 3948e1673d [manual] Accept OAuth2 tokens for API login via Google Apps
This is used by the Android app to authenticate without prompting for a
password.

To do so, we implement a custom authentication backend that validates
the ID token provided by Google and then tries to see if we have a
corresponding UserProfile on file for them.

If the attestation is valid but the user is unregistered, we return that
fact by modifying a dictionary passed in as a parameter. We then return
the appropriate error message via the API.

This commit adds a dependency on the "googleapi" module. On Debian-based
systems with the Zulip APT repository:
    sudo apt-get install python-googleapi

For OS X and other platforms:
    pip install googleapi

(imported from commit dbda4e657e5228f081c39af95f956bd32dd20139)
2014-01-13 13:30:55 -05:00
Leo Franchi 20f3b3af8f Fix zulip->zulip_internal puppet path change for apns checker
(imported from commit 1fd43a4f4907c24fcbbda73bbaf3cf092a6cace1)
2014-01-10 21:38:59 -05:00
Leo Franchi 91c54754fb [puppet] Add the apns-token crontab file to puppet
(imported from commit f12001453c9ca924c801a6000927e3ee2696a392)
2014-01-10 21:38:57 -05:00
Zev Benjamin c045644097 puppet: Run check_ntp_time against an NTP pool instead of time.mit.edu
MIT implemented NTP rate-limiting to defend against on-going reflection attacks,
which was causing our nagios checks to fail intermittently.  When the attacks
die down or when external sites fix their NTP configurations, checking against
time.mit.edu will stop failing.  However, there also isn't much of a reason to
stick with checking against a single server.

(imported from commit 2c2a1a04646b880b010cbb4b6d94016b1eccd1a0)
2014-01-06 17:30:09 -05:00
Jessica McKellar 61d660f9f3 [manual] digest: move cron job from staging to all app frontends.
Manual instructions:

This commit requires a puppet apply after deployment on both staging
and prod.

(imported from commit 2d10e33c6db2f5e9cc1204cdd5f2c91833da2a8e)
2013-12-20 12:50:23 -05:00
Tim Abbott bdcc2e5c52 nagios: Set max_check_attempts to 3 for batched queue processors.
(imported from commit ec0ac86726cd6ff3d0fdfcfcb161d3329fca02ac)
2013-12-19 17:31:41 -05:00
Tim Abbott b2d01e2da0 [manual] restart-server: Minimize downtime for message sender worker.
The manual step here is that we need to do the `puppet apply` before
pushing this commit, or `restart-server` will crash.

Previously we shut down everything in one group, which performed
poorly with supervisor's bad performance on restarting many daemons at
once.  Now we shut down the unimportant stuff, then the important
stuff, bring back the important stuff, and then bring back the
unimportant stuff.

This new model has a little over 5s of downtime for the core
user-facing daemons -- which is still far more than would be ideal,
but a lot less than the 13s or so that we had before.

Here's some logs with the current setup for the tornado/django downtime:
2013-12-19 20:16:51,995 restart-server: Stopping daemons
2013-12-19 20:16:53,461 restart-server: Starting daemons
2013-12-19 20:16:57,146 restart-server: Starting workers

Compare with the behavior on master today:
2013-12-19 20:21:45,281 restart-server: Stopping daemons
2013-12-19 20:21:49,225 restart-server: Starting daemons
2013-12-19 20:21:58,463 restart-server: Done!

(imported from commit b2c1ba77f3dc989551d0939779208465a8410435)
2013-12-19 17:21:23 -05:00
Luke Faraone a5775d94ef Install uploads.types in puppet
We also move uploads.types to zulip-include-frontend since its only
needed on the frontends.

(imported from commit cfdf15c0c537f7ea4c239b0f882aeaa561929777)
2013-12-18 16:25:10 -05:00
Leo Franchi 9c82e869c2 [manual] Release OS X desktop app 0.4.2
This reqires a puppet apply as well as a manual move of the installed
files and symlink switch. Leo will do it when it hits master.

(imported from commit e58e52087ad38f1cb8e0e606b82266a93cf91e53)
2013-12-18 16:14:51 -05:00
Jessica McKellar 5e217a1079 Use correct time zone in digest email cron job.
(imported from commit fd470af4b44ffb9696ff3a97372aaf2524a4806b)
2013-12-18 14:31:03 -05:00
Tim Abbott d62ca820db puppet: Fix permissions on /etc/cron.d/log2zulip.
(imported from commit 33ee5ae97b09b3925849940262ecd0bcbce38a3f)
2013-12-17 16:22:14 -05:00
Tim Abbott ae6c17a87d puppet: Stop using /var/log/nginx/zulip.*.log.
It's confusing to have our log data on different files on different
systems (e.g. loadbalancer vs. app).

(imported from commit be701072ee05e2659f146b226a39f33cb4707180)
2013-12-17 16:22:08 -05:00
Tim Abbott 6ccf19bed6 Run log2zulip on load balancers too.
(imported from commit 74c8be20d2d03aa524f05b7681febe9a9be9cdff)
2013-12-17 13:46:00 -05:00
Tim Abbott 8dcf7d4cc3 [puppet] Add log2zulip tool for sending log files to Zulip.
This tool is a little crude; it runs out of a cron job and will
forward to staging a notice about any new lines in the declared log
files, truncating if there are more than 10 lines.

(imported from commit 6748ddff1def0907b061dc278a3a848bd2e933f1)
2013-12-17 11:02:55 -05:00
Jessica McKellar 8bb1caec8f [manual] digest: add the cron job that will trigger digest emails.
Manual deployment instructions:

On staging, do a puppet apply.

No action needs to be taken for the prod deploy.

(imported from commit 0f6e5ab22aaeacfcc69d57de12f2bb6fac6f0635)
2013-12-17 10:47:16 -05:00
Tim Abbott b6acbe040c Fix missing nginx service notifications on configuration changes.
(imported from commit 0bfce276bab3704e508f6c8a58c9434e9fc224cd)
2013-12-16 13:44:50 -05:00
Tim Abbott c872866289 puppet: Fix nginx upstreams for staging.
(imported from commit eb1e6e3b2d35533af4a24015a91201e2414f8e28)
2013-12-16 11:32:05 -05:00
Tim Abbott f8fe9d1dd4 Fix check_worker_memory process list computation.
(imported from commit 9ac58b894ecfd84da6ac8509c0dc2ceb60eedfce)
2013-12-16 10:09:59 -05:00
Luke Faraone 1370c014a5 Clean up logging and documentation in ec2 interfaces script
(imported from commit e55247931cdeb61563f2348ca09f3d7b9fc85f0c)
2013-12-13 18:07:08 -05:00
Luke Faraone 104c2a06ae Set iptables rules for each IP, not just each interface
(imported from commit c24d2123489dc384bf50e379d245807af3488ebf)
2013-12-13 18:07:08 -05:00
Kevin Mehall f929e51776 puppet: Make Camo Nagios check waste less bandwidth
Use http://www.google.com/favicon.ico instead of a 1.7MB animated gif from
imgur.

(imported from commit 94993af35bf87b0f22e6e743a9ba1cc1c5c9a78f)
2013-12-13 17:27:01 -05:00
Tim Abbott 950e4c800b puppet: Declare upstreams properly in app nginx config.
(imported from commit 859eeed0d5b92c1b5b2b0764aba06aebcde8e2e2)
2013-12-12 16:48:52 -05:00
Tim Abbott ae4d214c49 Fix longpolling treatment for api.zulip.com/v1/events.
(imported from commit 78029972938ad7c9aa862330e38965b4b032c935)
2013-12-12 16:03:45 -05:00
Tim Abbott 73f04b21e9 Add zulip.customer29.invalid host.
(imported from commit ea3e7bb465c920b8ec21b7471cd261868f5059e7)
2013-12-12 16:03:45 -05:00
Tim Abbott c21e85e569 Remove staging.humbughq.com loadbalancer config.
The DNS has been disabled for some time.

(imported from commit e054c0fb0b37077d8303eab4d4ffec6ff53e8990)
2013-12-12 16:03:45 -05:00
Kevin Mehall 662edc2558 [manual] Backend support for Android GCM push notifications
This adds a dependency on gcmclient:
http://gcm-client.readthedocs.org/en/latest/gcmclient.html

pip install gcm-client

or

apt-get install python-gcm-client

(imported from commit 9f1fbf1f793e4a27baed85c6f1aa7a7b03106a10)
2013-12-11 15:37:48 -05:00
Luke Faraone b0a0853bd2 Specify full fingerprint rather than short key ID
(imported from commit fc4e9d51c440000e469f8e3882739215a3bcb022)
2013-12-11 10:54:30 -05:00
Luke Faraone 510b3349a7 Switch to downloading keys via SSL in puppet
(imported from commit 05d2b0626338f09370614e916050cfcee7f14829)
2013-12-11 10:54:30 -05:00
Luke Faraone 1b5c1ac021 Update style of client strings.
(imported from commit 1516461cf53b2715de68e01f16bb8a8cc33c48ad)
2013-12-09 11:47:52 -05:00
Leo Franchi e39cc5324b [puppet] Aggregate narrow timing stats
(imported from commit 4eff25635a3cb7687e995ad1127cff68da51329a)
2013-12-07 10:44:54 -05:00
Leo Franchi f70878e6c5 Fix aggregation rules for endtoend time
(imported from commit 29165b09e2d8904ee502cc04610a951d87ef896f)
2013-12-07 10:44:54 -05:00
Tim Abbott abeb29c226 Fix incorrect proxy_pass location for staging longpolling.
(imported from commit a4ac2c5c3416a8d8f748237411df6235f237e893)
2013-12-07 08:02:55 -05:00
Tim Abbott 09a61e8128 nginx: Enable keepalive for communication between lbs and frontends.
(imported from commit a7c8d9dfefbb6e5d01c8050688d831787b31bbd4)
2013-12-07 07:41:45 -05:00
Tim Abbott a337638f7f nginx config: Enable some popular performance-improving features.
(imported from commit 50193d071d091cc864867c9f7d5c9c9fb74b9d92)
2013-12-07 07:41:45 -05:00
Tim Abbott 6663247e22 Set nginx worker_processes automatically based on CPU count.
(imported from commit e48143c1410439d0574bc78bfd64c22a3063d558)
2013-12-07 07:41:45 -05:00
Tim Abbott 1843262672 puppet: Mark all Nagios plugins as executable.
They were being installed as executable anyway, but this will make
running them manually a bit easier.

(imported from commit a1181d2c90770af5aa44b0f65a47a460efdcf2d7)
2013-12-05 15:25:25 -05:00
Tim Abbott 64807c0628 nginx: Ensure zulip-include files are distributed to the right systems.
There were a few recently introduced bugs, and this also cuts down on
our having to review diffs that don't actually affect the relevant
server when doing updates.

(imported from commit 43f3cff9a414bc1632f45a8222012846353e8501)
2013-12-05 15:25:25 -05:00
Tim Abbott 676e9d90ff nginx: Get rid of trailing / in loadbalancer proxy_pass directives.
The trailing "/" actually means "replace the location with /", which
is either useless or actively harmful, depending on the location.

(imported from commit 58b9c4c9e55e3a162ffce49c954bc2182ec57dde)
2013-12-05 15:25:25 -05:00
Tim Abbott cc00ed6d7e nginx: Clean up now-empty 'loadbalancer' include file.
(imported from commit d13b5d91f6b85ba3e0bef7728985d0eba1cae084)
2013-12-05 15:25:25 -05:00
Tim Abbott afaff0c2cf nginx: Set X-Forwarded-For in common proxy configuration.
Previously we sometimes set it to $proxy_add_x_forwarded_for and other
times to $remote_addr, but according to

http://wiki.nginx.org/HttpProxyModule#.24proxy_add_x_forwarded_for

$proxy_add_x_forwarded_for handles this for us -- it will be
$remote_addr if there was no X-Forwarded-For header anyway.

(imported from commit 67dc52250e3e7751b1bf375d1a71d0272475435c)
2013-12-05 15:25:25 -05:00
Tim Abbott afe167ea58 nginx: Use the longpolling proxy configuration on load balancers.
(imported from commit f590e6b1eec2856b5128e310797f8ba58846417a)
2013-12-05 15:25:25 -05:00
Tim Abbott 21a69f2188 nginx: Move common longpolling proxy configuration into include file.
(imported from commit 4ace82824c32cec8c6da8a1a6b8a527dae105a89)
2013-12-05 15:25:24 -05:00
Tim Abbott 9e24558092 nginx: Move common proxy configuration into an include file.
(imported from commit 2ee5afc74fe146f8ee98f18f846342351c61c7f0)
2013-12-05 15:25:24 -05:00
Tim Abbott 3760609f3f Enable /sockjs handling on api.zulip.com (not used yet).
(imported from commit c2581e3243b2129c980fd3dd318eb3d99f3eb593)
2013-12-05 15:25:24 -05:00
Tim Abbott 953c3578dc external-sso.conf: Fix missing proxy_set_header.
(imported from commit 64bcb06cf65f15908ee74d637ab3868916b1dfd7)
2013-12-05 15:25:24 -05:00
Tim Abbott 79910fa2b3 Disable proxy_next_upstream for sockjs in remaining proxy_pass lines.
(imported from commit f14c7962253b34040ed9ab077a58c8b200df5d9d)
2013-12-05 15:25:24 -05:00
Tim Abbott e5be713103 Clean up EXTERNAL_API_HOST usage and defaults.
We now have 2 variablse:
EXTERNAL_API_PATH: e.g. staging.zulip.com/api
EXTERNAL_API_URI: e.g. https://staging.zulip.com/api

The former is primarily needed for certain integrations.

(imported from commit 3878b99a4d835c5fcc2a2c6001bc7eeeaf4c9363)
2013-12-04 15:10:54 -05:00
Tim Abbott b8a151ca4e Revert "[puppet] Add cron job to restart our workers daily."
This reverts commit 0b0180b0751f6c618d877b9c9ffc2b8287254e4d.

(imported from commit a81c552100345d369ffcaf69f28a86dea0893128)
2013-12-04 10:27:45 -05:00
Tim Abbott f7ac58bab5 Revert "Add user_activity_test worker that does nothing."
Now that we've debugged the memory leak, I don't think we need this
anymore.

This reverts commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6.

Conflicts:

	puppet/zulip/files/cron.d/rabbitmq-numconsumers
	puppet/zulip/files/supervisor/conf.d/zulip.conf

(imported from commit ff87f2aebcbc71013fa7a05aedb24e2dcad82ae6)
2013-12-04 10:27:45 -05:00
Tim Abbott 606d8a4f9b Add Nagios check for queue worker memory usage.
This is detect future memory leaks.

(imported from commit 75fd4c2ad41ea71e87a53fb33e2106c5773909d5)
2013-12-04 10:27:44 -05:00
Tim Abbott 850eae3e8e puppet: Disable proxy_next_upstream feature in nginx config.
(imported from commit 84cad76701f9ee40fa9601ae06b3f804948b96d4)
2013-12-03 15:20:45 -05:00
Tim Abbott 5007d4d87a [puppet] Update set_real_ip_from to use lb0's internal IP address.
This is something we forgot to do in the VPC migration, so our IPs
have all been the lb0 IP in our logs :(.

(imported from commit 9d3fc69cf72a84f7bd7c54e50fb1e776a67d971f)
2013-12-03 14:29:34 -05:00
Luke Faraone dc5c410257 Remove django-jstemplate, unused per trac #1973
(imported from commit 97b2e75766e3576f17b7dab0f1a4a00c34a5c2e5)
2013-12-03 11:36:18 -05:00
Leo Franchi 42e23dc82e [manual] Release desktop app 0.4.1 for OS X
This requires a puppet apply on prod0, and an update of the
Zulip-latest.dmg and Humbug-latest.dmg symlinks in
/src/www/dist/apps/mac and /srv/www/dist/apps/sso/mac

(imported from commit e83170a19ac2de6458a0fd43140068fab4135483)
2013-12-02 15:24:32 -05:00
Zev Benjamin 9fa9f81a0a puppet: Add ptop to postgres-common packages
(imported from commit 085bf7d0ef33f8287bea9152e328908a89144b8f)
2013-12-02 14:54:53 -05:00
Zev Benjamin 20cf06013e Add missing newline to crontab
(imported from commit fe14ff383ff6915f0e60d3e6904ecdedc5ff5bf7)
2013-11-28 02:17:07 -05:00
acrefoot eb7be522ac [manual] add supervisor and numconsumer entries for push notifications queue
requires puppet-apply on both staging and prod

(imported from commit 6fc25041b40145d7c62a8bb959a8d25d0dbcb44e)
2013-11-27 18:00:33 -05:00
Leo Franchi d36510e4c3 [manual] Release Zulip Desktop 0.4.0
This requires a puppet apply, and also a manual update of
the Zulip-latest.* symlinks in /srv/www/dist/apps

(imported from commit 991dd6924ba33d81f486e914bcbadfec5b350660)
2013-11-26 17:41:25 -05:00
Tim Abbott 3971f18de8 loadbalancer: Fix missing location-sockjs config.
(imported from commit 27b168e73014d7b7c71fb00ce5b75271393fc491)
2013-11-26 12:22:17 -05:00
Zev Benjamin 7af4b92b98 puppet: Rename app to prod0 in nagios
(imported from commit c2d1c2c06276a816ef33e057d3f859c755490cb3)
2013-11-25 11:43:16 -05:00
Zev Benjamin 9f2af6fd0d puppet: Fix postgres_primary alias
(imported from commit 1cd199224e45700fac03e68c99f9d4f7d9212b45)
2013-11-25 11:43:16 -05:00
Zev Benjamin 847d4dfbca puppet: Specify hosts for the postgres autovac_freeze check via a hostgroup
(imported from commit d0afc1b78015740fa9638563a5672d3400dd5002)
2013-11-23 12:08:49 -05:00
Zev Benjamin 139518ccbe puppet: Remove postgres0 from nagios and munin configs
(imported from commit 6a4eb208b2a344d65d684cf904ba882a5400056d)
2013-11-23 12:06:27 -05:00
Zev Benjamin dacf97db48 puppet: Use peer authentication for Postgres nagios checks
(imported from commit d8f02d5320d6f8b97fd82cd3f0ca65f6e5c42b03)
2013-11-23 10:01:15 -05:00
Zev Benjamin 3454680e4b puppet: Add VPC subnets to pg_hba.conf
(imported from commit 633bf08bfe2f3695bd6c9ed8584b78971ebe065f)
2013-11-23 08:23:49 -05:00
Zev Benjamin bf8fb3c0df puppet: Add postgres2 to nagios monitoring
(imported from commit 799b1304eebe49cf6d8153fb2bfd0b11a3bcab00)
2013-11-23 08:10:44 -05:00
Zev Benjamin 658972dda3 [manual] puppet: Add postgres2 to munin monitoring
You must run
autossh -2 -fN -M 20018 -L 5009:localhost:4949 nagios@postgres2.zulip.net
as nagios on nagios.zulip.net after deploying this commit.

(imported from commit bd8a61f99555ccf0a0010d79dbd89017aaafbb8f)
2013-11-23 08:10:44 -05:00
Zev Benjamin d7d98aaacc puppet: Move /etc/iptables/rules to /etc/iptables/rules.v4
The /etc/init.d/iptables-persistent initfile changed to expect there to be two
files in /etc/iptables (rules.v4 and rules.v6) instead of a single rules file.
Several of our machines are currently running without iptables rules as a
result.

(imported from commit 266c2ff26b77f7c9ae793690b0d544ee4cfa5020)
2013-11-23 08:10:44 -05:00
Zev Benjamin c3f4ab6c94 puppet: Add replicator access from postgres2 to pg_hba.conf
(imported from commit 2a4f150c67d3136a5e97cb673cc7f14256ffae01)
2013-11-22 17:38:52 -05:00
Luke Faraone af02e45a17 [manual] Support authentication and profile prefilling via LDAP
The latter doesn't depend on the former; we can still fill in your full
name even if you didn't authenticate via LDAP.

This commit requires django_auth_ldap to be installed. On Debian
systems, you can do so via APT:
    sudo apt-get install python-django-auth-ldap

On OS X, use your favourite package manager. For pip, I believe this
will work:
    pip install django_auth_ldap

django_auth_ldap depends on the "ldap" Python package, which should be
installed automatically on your system.

(imported from commit 43967754285990b06b5a920abe95b8bce44e2053)
2013-11-22 16:51:26 -05:00
Tim Abbott 8919ebe6b2 puppet: Make sure prod0's future external IP has access to postgres.
(imported from commit 91523dc92fd15dc0cf19b7bca70513250c4da983)
2013-11-22 16:43:10 -05:00
Zev Benjamin 18fc8c2059 puppet: Do peer authentication for user zulip on the DB servers
(imported from commit dceed53990db64b3c345726b02bf0c25815c2b25)
2013-11-22 15:58:09 -05:00
Tim Abbott c0e951f843 Add user_activity_test worker that does nothing.
This should help us debug the source of our memory leak problems.

(imported from commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6)
2013-11-22 11:24:48 -05:00
Tim Abbott c31dbba9cc [puppet] Update pg_hba.conf to include staging's public IP.
This is for the interval while staging is running in VPC and postgres
is not; we can clean up these changes once that's no longer the case.

This also updates test1's IP, which apparently someone forgot to
commit previously.

We're currently running this on prod.

(imported from commit 3feced750f643bb218d4240e9a3d5cd7116963ee)
2013-11-21 11:27:16 -05:00
Tim Abbott 8bfbaab1d5 Fix typo in puppet directories for zulip_internal.
(imported from commit 52627a9e71dfc28bedd6c955069da46d3ef56e83)
2013-11-21 10:06:40 -05:00
Leo Franchi c6b862d26d Add desktop sso puppet folder dependencies
(imported from commit 63d8cad722cf015a5925a28ab860431be31175be)
2013-11-20 19:12:53 -05:00
Tim Abbott f0c6b63526 [puppet] Add cron job to restart our workers daily.
This is to ensure that if we have an interval where we're not doing
prod deploys, we don't have to worry about worker memory leaks killing
us.

(imported from commit 0b0180b0751f6c618d877b9c9ffc2b8287254e4d)
2013-11-20 18:34:16 -05:00
Tim Abbott 630cfd72f1 Deregister event queues when done in our Nagios scripts.
(imported from commit a1f73403163323e1dd9eda2f5269e94c60abdd1a)
2013-11-20 18:34:16 -05:00
Tim Abbott ca8225cf47 [manual] Add endpoint to cleanup a finished events queue.
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.

(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
2013-11-20 18:34:15 -05:00
Tim Abbott 8806ec698a puppet: Increase nginx worker connections limit to match open files.
I want these limits to be at least 40x our current scale.

(imported from commit f22fc40d45292788666e5079ca79bcae683de510)
2013-11-20 10:13:07 -05:00
Tim Abbott ec23996538 puppet: Increase our nginx open file limit.
(imported from commit a002e1df484dcdab8cdbfb2f37ca2281d6f4eecd)
2013-11-20 10:13:07 -05:00
Tim Abbott b50db26a18 puppet: Add monitoring for camo.
(imported from commit b3cf29b02de285cf860fc173183cb6f4f3a17c74)
2013-11-19 15:25:14 -05:00
Tim Abbott 1bcf37664f zulip-sso: Make our Apache service use SSL.
(imported from commit ebf8c9b01cd16f38203c9585514c0d0be108b729)
2013-11-19 15:25:14 -05:00
Kevin Mehall 3a4b576135 Add a directory for enterprise admins to add static files.
For things like custom default avatars that shouldn't be overwritten
on update.

(imported from commit 2487d2532a5255b91bff956fdfb0d885cb786701)
2013-11-18 11:48:53 -05:00
Tim Abbott 259dca9508 puppet: Merge the ports.conf into zulip-sso.example.
This makes us not blow away a customer's ports.conf configuration on
upgrade if they needed to change it while setting up their SSO.

Also we change the NameVirtualHost line to better match the
VirtualHost line.

(imported from commit fd52e00c35afa8982e0377859ad794085ec2af80)
2013-11-15 18:13:09 -05:00
Tim Abbott 1b009c47fc Clean up our nginx configuraiton to make better use of app.d.
Now app.d is something that any app frontend will read, and we just
have secondary manifests add additional files to the app.d directory
for custom stuff.

This fixes the issue that we were incorrectly including the
lb0-related app configuration in the enterprise version.

(imported from commit dec8dcdf2506b82e51186ff936c26dc1cd6cf61b)
2013-11-15 15:04:13 -05:00
Tim Abbott 6826ef4e9a puppet: Switch from nginx to nginx-full.
(imported from commit 38dd5966d75946842b39e4e619d82ebbb0fb041c)
2013-11-15 15:04:13 -05:00
Kevin Mehall fe0dcd4313 Disable camo on enterprise.
CUSTOMER13 doesn't want it, and there's currently no nginx config
or configurable Camo URI, so it wouldn't work if image preview
were enabled.

(imported from commit 615d4a32acbc4d4d590f88cf4e7d45d8f49db1d3)
2013-11-15 14:27:16 -05:00
Tim Abbott ccae6ef5ce Remove unpublished work comment about our SSO example.
(imported from commit 20585c5caa2e7019e3817d40ea2ab90c13b582e2)
2013-11-14 11:16:15 -05:00
Leo Franchi be3257de62 Add sso sparkle paths and files
(imported from commit f8a953f4262b170931792ed223f1ddc29c5fe5ed)
2013-11-14 10:22:35 -05:00
Tim Abbott f9ab464c82 puppet: Fix owner/group for our puppet.conf file.
(imported from commit c91dcd07646482d380cd36df1a0002a5d8d76241)
2013-11-13 21:47:22 -05:00
Tim Abbott 498cce8ef9 puppet-apt: Fix missing owner/group for created files.
(imported from commit a435f90593788034421d4817386215bc14e83d01)
2013-11-13 21:47:14 -05:00
acrefoot 0175440afc [manual] fixup nagios postmaster configuration
(imported from commit e3c00b31bbb0ced38e62d31ae80b58e8c6374c7f)
2013-11-13 17:37:54 -05:00
acrefoot 6d38285a2e fixup supervisor oops related to postmaster config
(imported from commit 8b5c39f0d13abb5e1def9f88a2ab82cfa67b42f6)
2013-11-13 17:15:55 -05:00
acrefoot eab6a1d190 [manual] add nagios checks for email_deliver
manual step: puppet apply, make sure that these nagios checks are working properly

(imported from commit abc75b8a5b153510243c14035b820fbc864b7776)
2013-11-13 16:41:36 -05:00
acrefoot 08069bf34e [manual] Add the new deliver_email management command to the supervisor config.
You will have to puppet apply on both staging and prod to get this worker running.

(imported from commit 92969371220f08142f510d3415e8611dcfecd91f)
2013-11-13 16:41:36 -05:00
Zev Benjamin 8aa307f438 enterprise: Restart all Zulip processes every week
(imported from commit e44167023e0ad49eaf14c314a7731b93289900e6)
2013-11-13 16:26:03 -05:00
Kevin Mehall f7f2ec0aca [puppet] Report enterprise and prod errors to staging.
Errors are sent to a queue processor that posts them to staging,
just like the feedback bot.

(imported from commit 4a8d099672a1b3e48a8bc94148d8b53db73d2c64)
2013-11-13 16:22:21 -05:00
Jessica McKellar 1cb339a23d Expose password-protected /enterprise/download through nginx.
(imported from commit 8fba915bf0d0c718138ac62bd4333aef4e845d55)
2013-11-13 16:20:43 -05:00
Tim Abbott a16a7a028c Fix internal postgres_common.pp.
We didn't remove python-argparse from the requirements when that was
removed, and we also still need python-pip to install wal-e :(.

(imported from commit b82d3b429cffe0a3993819358511e11268ee2fef)
2013-11-13 15:35:45 -05:00
Tim Abbott 4cb50d8f3b Fix class name for apache_sso.pp.
(imported from commit 25daa2d98a88866824203f2e7016a12b9f91a32b)
2013-11-13 12:07:15 -05:00
Tim Abbott 47682d47c9 Move our apache2site library into the public manifest.
(imported from commit 8cb6d0a01fc286ad1a98a7e2d27a80293667e9b8)
2013-11-13 12:07:15 -05:00
Tim Abbott e877536de5 process_fts_updates: Use peer authentication.
(imported from commit 329ec3c07c9cfb648c706e01aec7e8826c3f7737)
2013-11-13 12:02:50 -05:00
Tim Abbott a6af391125 Remove dependency on python-argparse -- it isn't need with Python 2.7.
(imported from commit 388f6a5b7ff2b20364d22fad0d9e7d992791a18b)
2013-11-13 12:02:50 -05:00
Tim Abbott e986f1e988 puppet: Don't create /home/zulip/deployments/current symlink.
This is now managed on enterprise systems via the unpack-tarball
system.

(imported from commit 8cc4ac0b47990ed1d5a02113a27bd126f4e3f011)
2013-11-13 12:02:50 -05:00
Tim Abbott f4a9e99498 puppet: Move the postgres dictionary symlink creation to puppet.
(imported from commit 823f6683e3d8f3604da68e55dd6761ecb38d4b63)
2013-11-13 12:02:50 -05:00
Tim Abbott 9b9c43fd50 Fix paths to nginx configuration for zulip-enterprise.
(imported from commit 4c0ec7ac752d2f4810fe089b85f733c7c21a1676)
2013-11-13 12:02:49 -05:00
Tim Abbott 5e0433009f Add enterprise apt repository to puppet.
(imported from commit e68964089adbd81580cdee6dfc2ee61e31835ce7)
2013-11-13 12:02:49 -05:00
Tim Abbott 4bf9594750 puppet: Fix dependency on postgres in enterprise.pp.
(imported from commit 09baae1a0d6adc86124a1518b14ff28ef71db6ca)
2013-11-13 12:02:49 -05:00
Tim Abbott 155d50f0cb Apply sysctls before restarting postgres.
(imported from commit bc1f1ce1b880962640b8ed5f516139eaf91cb1db)
2013-11-13 12:02:02 -05:00
Zev Benjamin b4403fda43 enterprise: Generate a 40-postgresql.conf sysctl file
(imported from commit ec64d283f88d1b2f4b13e3405d39cfbe8400adc7)
2013-11-13 12:02:02 -05:00
Zev Benjamin c1bdb0c7a8 puppet: Move the removal of 30-postgresql-shm.conf into zulip::postgres_appdb
(imported from commit 6a4d089fd47a8ba6ea92eeac321e3077fa0d8cc4)
2013-11-13 12:02:01 -05:00
Zev Benjamin cf7c468e74 enterprise: Specify half the system memory for pgtune
(imported from commit 287acb1eb7b03dddd60045c43e04af1fe61f354b)
2013-11-13 12:02:01 -05:00
Tim Abbott 317a6f308f Fix missing dependency on python-mock.
(imported from commit e82be819a7dd0f7c8683e7efc0a5095b9844b159)
2013-11-12 23:21:52 -05:00
Luke Faraone aac6ee308d puppet: apache configuration for an example SSO instance.
(imported from commit 0aa6c80e206a70efd481d710aa2a6c2431c662f3)
2013-11-12 19:18:35 -05:00
Luke Faraone dcb4487c56 puppet: generic localhost SSO manifest
(imported from commit 9398800ba7e26390d1bd6f691f39a45d2740870f)
2013-11-12 19:18:35 -05:00
Luke Faraone 61ce8ccd95 puppet: Add app.d configuration for a separate authentication server
(imported from commit 9f564bb513292e446ee4f7498ebeca5d167e5037)
2013-11-12 19:18:35 -05:00
Luke Faraone 950300d20f puppet: Include /etc/nginx/zulip-include/app.d/ configuration if defined
Some sites may want to have small modifications to the base app config.  This
directory helps support that, but does not need to exist if it is unused.

(imported from commit d23b19dc59bb56d00e69ff03af3279b66af9466d)
2013-11-12 19:18:35 -05:00
Tim Abbott c7d9bf41bc Rename SSL certificates for Zulip Enterprise.
(imported from commit 4f15f2aee2bcc7450953488c94e8b88734aeaff7)
2013-11-12 15:57:42 -05:00
Tim Abbott 966fde261a puppet: Rename local_server => enterprise.
(imported from commit 5faa269df5937f6db99098e44aaea7d0a4f2c14a)
2013-11-12 15:57:02 -05:00
Tim Abbott 1354d522cc puppet: Require postgres being installed before installing our config.
(imported from commit c871792d78de368f922586124725edfa6725246e)
2013-11-12 15:57:01 -05:00
Kevin Mehall 9b6c99c7af Copy localserver prod-static instead of linking so old files are retained.
(imported from commit 728cce9f1b3a47d62f2856a509ed65a3ac6bd451)
2013-11-12 15:50:08 -05:00
Tim Abbott 17a98b3afd puppet: Move embedly to the internal puppet configuration.
(imported from commit 9b5c2f2726f2cac5bba5619ee9b7371dada0ea35)
2013-11-12 09:34:25 -05:00
Tim Abbott 5de6b879b7 puppet: Move minify-js and statsd dependencies to internal manifest.
(imported from commit 3363fb962a4fde575591b42db888b92bb6edd0f5)
2013-11-12 09:34:25 -05:00
Tim Abbott bf0fda7426 Use the enterprise apt key when installing local server.
(imported from commit 24fa7474a77cac7b1d033c6732de6873d4aa9880)
2013-11-12 09:34:25 -05:00
Tim Abbott bbf8424b3e puppet: Move our ops repository apt configuration to internal module.
(imported from commit 2aca8e8c2edbd87a77fd5f00b3ae250484721fb4)
2013-11-12 09:34:25 -05:00
Tim Abbott 2b7e4d186a Document why we need python-dns.
(imported from commit 15f1ab77232db9d6667b98079b9a65436884dedf)
2013-11-12 09:34:25 -05:00
Tim Abbott 138d7053b7 puppet: Move the wal-e dependencies to the internal postgres config.
(imported from commit 67251263ec98e5b141f4c7587042b4db7aed36f2)
2013-11-09 07:28:19 -05:00
Tim Abbott 43e1c5e47d puppet: Move our camo installation setup to its own manifest.
(imported from commit 401018c3ff49fea485c3c8b4adb42574bb0b54b0)
2013-11-09 07:28:19 -05:00
Tim Abbott b902cc6eb1 puppet: Fix loadbalancer dependencies to include nodejs.
(imported from commit 9962e8ee774b13abdb95028d44cd687cee63aa13)
2013-11-09 07:28:19 -05:00
Tim Abbott a3b813e7af puppet: Comment the dependency lists for some service manifests.
(imported from commit 49817eb7daeab7cb83799d116ac829c7cd8e84e7)
2013-11-09 07:28:19 -05:00
Tim Abbott f552149772 puppet: Organize the puppet configuration for various internal services.
(imported from commit ad3525f608dbc5ceb04e6829fb1da0b3baba3258)
2013-11-09 07:28:19 -05:00