Commit Graph

535 Commits

Author SHA1 Message Date
Umair Khan 336a041ac0 Django 1.10: Use uWSGI.
Fixes: #1121

With some tweaks by tabbott to make the number of processes configurable.
2016-12-13 21:40:43 -08:00
Igor Tokarev c93f1d4eda Add oembed/Open Graph/Meta tags data retrieval from inline links.
This change adds support for displaying inline open graph previews for
links posted into Zulip.

It is designed to interact correctly with message editing.

This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.

By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.

Eventually, we may want to make this manageable via a (set of?)
per-realm settings.  E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
Jason Le 144d82305d mypy: Annotate puppet/zulip_ops. 2016-12-03 11:00:25 -08:00
bulat22101 adebc75740 pep8: Fix E502 violations 2016-12-03 10:56:36 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Bickio 6b0df43463 pep8: Fix E125. 2016-11-30 20:03:29 -08:00
Tommy Ip 46b7d54b3e pep8: Fix E701 violations. 2016-11-30 20:45:09 +00:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Anders Kaseorg 092fe4fecb puppet: Write rabbitmq-env.conf before installing rabbitmq-server
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 18:54:12 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Anders Kaseorg 1ea8abe493 Replace python -u with PYTHONUNBUFFERED=1
(Why is -u needed at all?  I’m not sure, but test-run-dev spins forever
“Polling run-dev...” without it.)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Anders Kaseorg d1dc2cf30e Mark scripts executable
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Anders Kaseorg 18e49bec3a puppet: Add another missing dependency on postgresql-common
The postgres group must exist before we give files to it.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 11:23:32 -08:00
Tim Abbott f78eb9d82d puppet: Add missing dependency on postgresql-common.
postgresql-$(version) depends on postgresql-common depends on ssl-cert,
which creates the ssl-cert group.
2016-11-21 07:38:45 -08:00
Tim Abbott 2e65dc1206 puppet: make check_send_receive_time target host configurable. 2016-11-02 23:40:53 -07:00
Tim Abbott 9578908601 check_send_receive_time: Stop hardcoding EXTERNAL_API_URI. 2016-11-02 23:35:08 -07:00
Tim Abbott eceaf36001 setup_disks: Fix postgres RAID setup to work correctly on Xenial.
Nobody's going to run this on Wheezy again.
2016-10-28 11:04:08 -07:00
Tim Abbott 9b7a3f040c Remove now-unused /json/get_events endpoint. 2016-10-27 21:34:58 -07:00
Tim Abbott 4fbe201187 puppet: Automate autossh process monitoring maintenance.
Previously, the Zulip Nagios configuration effectively hardcoded the
count for how many system should have autossh connections.
2016-10-26 00:49:03 -07:00
Tim Abbott 6bdb10b71b puppet: Update emacs dependency to emacs-nox metapackage.
This way, one doesn't need to keep updating the dependency every time
a new major emacs release comes out.
2016-10-26 00:42:22 -07:00
Tim Abbott 11b5d203f7 sshd_config: Increase MaxStartups.
This fixes connection problems when using the full Zulip recommended
Nagios configuration against a given server.
2016-10-26 00:41:03 -07:00
Tim Abbott 73f54dd0cb sshd_config: Add updates from Xenial upstream.
It seems worth updating this to match the Linux distro this
configuration targets.
2016-10-26 00:40:44 -07:00
Tim Abbott 0a5a2c4eda nagios: Automate authorized users list maintenance. 2016-10-26 00:37:29 -07:00
Tim Abbott fa4998db59 puppet: Add zulip_zephyr_mirror plugins. 2016-10-26 00:35:57 -07:00
Tim Abbott ac4f28050c zmirror: Remove unnecessary krb5-clients dependency.
I'm pretty sure krb5-clients isn't needed to run the Zephyr mirroring
service.
2016-10-26 00:35:11 -07:00
Tim Abbott d490e83645 puppet: Upgrade nagios cgi.cfg with modern defaults. 2016-10-26 00:31:41 -07:00
Tim Abbott 1159ad4857 puppet: Upgrade nagios.cfg with modern defaults. 2016-10-26 00:31:41 -07:00
Tim Abbott 73178e5e5a puppet: Run check_send_receive_time via a cron job.
This allows the actual nagios work involved with
check_send_receive_time nagios checks to be done by an unprivileged
"nagios" user rather than the "zulip" user.
2016-10-26 00:26:52 -07:00
Tim Abbott 96cf330649 puppet: ssh as the nagios user instead of zulip user.
This is a follow-up to 4f58fef54b,
touching services.cfg instead of commands.cfg.
2016-10-26 00:23:47 -07:00
Tim Abbott a350d43683 puppet: Add recovery.conf configuration to postgres_slave.pp.
This file is needed to run a valid postgres slave; it's not clear why
this wasn't installed in the original zulip.com configuration.
2016-10-26 00:22:57 -07:00
Tim Abbott c3727c9886 nagios: Remove old zulip.com trac/git/replica servers.
These are unlikely to be relevant to anyone.
2016-10-26 00:21:53 -07:00
Tim Abbott 383f39b543 nagios: Enable allow_empty_hostgroup_assignment.
This fixes the configuration being broken when we remove some of the
old zulip.com hosts that are unlikely to be of interest to anyone.
2016-10-26 00:19:21 -07:00
Tim Abbott 4f58fef54b zulip_ops: Use nagios user for all Nagios checks.
There's no reason these Nagios checks needs to run as the
semi-priviliged Zulip user.
2016-10-26 00:17:26 -07:00
Tim Abbott 32d244dbe5 puppet: Add Nagios checks for other consumers. 2016-10-26 00:11:08 -07:00
Tim Abbott f1fa4397f3 puppet: Fix package deps for zulip-ec2-configure-interfaces. 2016-10-26 00:11:08 -07:00
Tim Abbott 3448ab4c7a zulip-ec2-configure: Fix network IDs for Ubuntu Xenial. 2016-10-26 00:11:08 -07:00
Tim Abbott 91da4bd59b puppet: Add check_cron_file generic helper. 2016-10-26 00:11:08 -07:00
Tim Abbott 080dd8c987 nagios: Ignore kthreads in check_procs tests.
Modern Linux can have a lot of kernel threads not doing anything.
Since this isn't interesting from a monitoring perpsective, we ignore
these.
2016-10-26 00:10:40 -07:00
Tim Abbott 4c9a283542 puppet: Remove configuration for old builder host.
I don't think this configuration was ever even used; it's just
clutter.
2016-10-26 00:01:52 -07:00
Tim Abbott f9ad75f58e puppet: Remove configuration for old zulip.com bots host.
This configuration didn't do anything anyway and just clutters the
repo.
2016-10-26 00:01:29 -07:00
Tim Abbott 9d4f3f1e1b puppet: Replace zulip_ops postgres configs with postgres_appdb_tuned.
There's no longer a reason to have copies of forked postgres
configuration files in our repository, since some time ago we merged
the features of these configuration files into the main
postgres_appdb_tuned.pp.
2016-10-25 23:58:53 -07:00
Tim Abbott 105ea972f6 puppet: Remove now-unncessary kernel.shm sysctl values.
With modern Linux and postgres, these settings are not required.
2016-10-25 23:58:33 -07:00
Tim Abbott 2227e77cce puppet: Remove Dropbox usernames from Nagios config. 2016-10-25 23:55:42 -07:00
Tim Abbott 8584c05d80 zulip_ops: Remove unnecessary loadbalancer stanzas. 2016-10-25 23:52:37 -07:00
Tim Abbott 624ee3989f puppet: Remove old Dropbox certificates. 2016-10-25 23:52:30 -07:00
Tim Abbott f0bb78ba2d puppet: Fix iptables-persistent->netfilter-persistent rename. 2016-10-25 23:45:21 -07:00
Tim Abbott c4ca7ee6e1 puppet: Move Apache sites files to correct paths.
Apache now actually requires its configuration files have names ending
with .conf.
2016-10-25 23:44:28 -07:00
Tim Abbott 2b8324b778 emoji: Fix caching permissions issues.
Previously, you needed to be root to update the emoji cache, which
caused problems with how Zulip is upgraded in production.
2016-10-25 17:52:19 -07:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 56c0b80067 nginx: Fix JavaScript not being compressed properly.
Apparently, we weren't actually compressing our JavaScript being sent
over the wire due to incorrect nginx configuration.
2016-10-23 20:06:02 -07:00
Tim Abbott a5a03c2e0b zulip_ops: Include zulip::apt_repository.
This replaces the old wheezy configuration.
2016-10-16 20:13:35 -07:00
Tim Abbott 8c68c6f09b zulip_ops: Remove wheezy apt repo.
Nobody uses wheezy anymore, and the configuration wasn't even
conditional on the OS version.
2016-10-16 20:13:35 -07:00
Tim Abbott 5210b0a6a4 zulip_ops: Cleanup old redis configuration.
One can now just use the improved configuration we've merged into the
main Zulip repo.
2016-10-16 20:13:35 -07:00
Tim Abbott 869f0724ce zulip_ops: Remove humbughq.com nginx configuration.
The humbughq.com name hasn't been the product's name since 2013, and
it's nice to finish clearing it out of the repository.
2016-10-16 20:13:29 -07:00
Tim Abbott 29448fb47b zulip_ops: Remove old Zulip, Inc. trac configuration.
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
Tim Abbott 771e03cfa7 zulip_ops: Remove old Zulip, Inc. mediawiki configuration.
This isn't useful to anyone.
2016-10-16 19:23:47 -07:00
Tim Abbott 36e336edc3 puppet: Rename zulip_internal to zulip_ops.
The old "zulip_internal" name was from back when Zulip, Inc. had two
distributions of Zulip, the enterprise distribution in puppet/zulip/
and the "internal" SAAS distribution in puppet/zulip_internal.  I
think the name is a bit confusing in the new fully open-source Zulip
work, so we're replacing it with "zulip_ops".  I don't think the new
name is perfect, but it's better.

In the following commits, we'll delete a bunch of pieces of Zulip,
Inc.'s infrastructure that don't exist anymore and thus are no longer
useful (e.g. the old Trac configuration), with the goal of cleaning
the repository of as much unnecessary content as possible.
2016-10-16 19:23:27 -07:00
Tim Abbott 2e50dce9dd check_queue_worker_errors: Don't import zproject.settings directly. 2016-10-15 22:53:28 -07:00
Tim Abbott 913d913c90 puppet: Annotate Nagios plugins. 2016-10-15 22:44:15 -07:00
Tim Abbott 458e455b75 Annotate process_fts_updates. 2016-10-15 22:44:15 -07:00
Tim Abbott a4b2a6b6d4 puppet: Fix missing dependency for supervisor enabling.
You can't run `systemctl enable` on a service before the package
containing the service is installed.
2016-10-06 21:44:10 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
Tim Abbott 12ddb1a36f puppet: Fix buggy logrotate configuration. 2016-09-08 16:51:17 -07:00
Kouhei Sutou 683f49aa99 Support full text search for all languages using pgroonga.
This adds support for using PGroonga to back the Zulip full-text
search feature.  Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.

Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.

PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.

Fixes #615.

[docs and tests tweaked by tabbott]
2016-08-26 21:04:03 -07:00
Tim Abbott 546c23fd1f puppet: Activate pgroonga extension after installation.
The pgroonga extension needs to be activated using postgres root
access, so we make that happen from puppet.
2016-08-25 18:37:45 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
Tim Abbott 682d78bc30 puppet: Add default logrotate configuration.
This should prevent Zulip from eating all the disk now that we are
using the race-free WatchedFileHandler rather than
TimedRotatingFileHandler.
2016-08-24 14:30:54 -07:00
Tim Abbott 1c40df9363 static_asset_compiler: Use correct package name on trusty. 2016-08-22 23:47:03 -07:00
Tim Abbott 97b622dffa puppet: Fix postgres_appdb_tuned manifest on trusty. 2016-08-22 23:41:07 -07:00
Tim Abbott 750e43518f puppet: Ensure supervisord starts on boot with systemd.
This is important to ensure that Zulip comes up properly after a
reboot on Ubuntu Xenial.
2016-08-22 23:25:54 -07:00
Tim Abbott 2be597dd09 puppet: Move support for accepting a loadbalancer to zulip manifests.
Previously, we didn't have an easy process for how one should do this.

Now, it's an (undocumented) zulip.conf setting!
2016-08-22 17:19:03 -07:00
Tim Abbott 88a123d5e0 Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.

Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.

Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.

Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
Tim Abbott 852af83d3c check_postgres_replication_lag: Fix psql command line.
This allows the plugin to be run as users other than the "zulip" user,
where the "zulip" database would not be the default.
2016-08-12 13:19:08 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Tim Abbott 2581ac166c puppet: Automatically scale default memcached memory allocation.
Previously, we used a fixed memcached memory allocation of 512MB,
regardless of the size of the server.  While that is a good allocation
for a server with 4GB of RAM, for servers with less, we should
decrease the allocation, and for a large server with much more RAM, we
should increase it.  We still support the user overriding the
configuration setting, but this produces more sensible defaults.
2016-08-11 20:27:45 -07:00
Tim Abbott 2e0e8193d4 puppet: Make memcached memory allocation a variable. 2016-08-11 20:27:45 -07:00
Tim Abbott 3edf880c9b Remove moreutils from zulip_internal::base.
We recently added this to zulip::base, so it would be a duplicate.
2016-08-11 18:12:59 -07:00
Tim Abbott 9afb1c7a71 puppet: Stop wholesale replacing /etc/redis/redis.conf.
Zulip had only patches the redis configuration in one small way, which
resulted in unnecessary portability issues for using Redis on
different versions of Linux.  We replace this with just a adding an
include mechanism to the redis config.

While we're at it, we configure this to take advantage of the
new REDIS_PASSWORD secret to automatically configure redis passwords.
2016-08-11 17:26:04 -07:00
Tim Abbott cb21584ffe check_send_receive_time: Support being run in a cron job. 2016-08-11 14:48:21 -07:00
Tim Abbott 4dcbaf1e6b check_send_receive_time: Cleanup unnecessary print statement. 2016-08-11 14:48:21 -07:00
Tim Abbott 1d6ebd2b3d check_fts_update_log: Fix psycopg2 arguments.
* Fixes passing a string argument rather than an actual Python
  argument.
* Switches to hardcoding the database to connect to rather than the
  user, so this check can be run as an arbitrary user.
2016-08-03 14:58:46 -07:00
Tim Abbott 1c65508624 puppet: Add postgres user to Zulip group. 2016-08-02 14:59:25 -07:00
Tim Abbott 04fc3ff1e1 pg_backup_and_purge: Don't try to use a virtualenv.
This makes this actually work if the postgres server is different from
the Zulip app frontend.
2016-08-02 14:59:25 -07:00
Tim Abbott 4bdd9d3769 check_postgres_backup: Don't try to use a virtualenv.
The dependencies for this tool are installed using apt.
2016-08-02 14:59:25 -07:00
Tim Abbott a5115d54ee env-wal-e: Rename s3_backup_bucket to s3_backups_bucket.
This makes it consistent with the other variables in this file.
2016-08-02 14:59:15 -07:00
Tim Abbott ff80daef16 puppet: Move zulipsecret into its own file.
Apparently puppet doesn't support declaring multiple functions in the
same file.
2016-08-02 14:55:51 -07:00
Tim Abbott 75b5d021fa Remove unused puppet-common third-party module code. 2016-07-31 19:24:42 -07:00
Tim Abbott 6158acb41b nagios: Fix path to check_debian_packages. 2016-07-31 14:25:07 -07:00
Tim Abbott 6954dd84ab postgres_appdb_tuned: Add SSL certificate path configs. 2016-07-31 14:25:07 -07:00
Tim Abbott d66f6b8176 Fix postgresql configuration template for Ubuntu Xenial.
I incorrectly assumed that this was working because Travis CI passed
the build, whereas in fact Travis CI only tests the Trusty templates.
2016-07-31 14:24:34 -07:00
Tim Abbott 8b285ec0ff puppet: Read camo key from zulip-secrets.conf. 2016-07-31 00:23:24 -07:00
Tim Abbott 2b40309029 puppet: Add new zulipsecret function. 2016-07-31 00:23:24 -07:00
Tim Abbott 298e040bac puppet: Strip newlines in zulipconf function.
This avoids creating unnecessary extra newlines in config generated
using this.
2016-07-31 00:23:24 -07:00
Tim Abbott 16a4ce1bd2 puppet: Fix Nagios check_disk flagging special filesystems. 2016-07-30 23:57:31 -07:00
Tim Abbott c764b46cef puppet: Fix missing base include in nagios manifest. 2016-07-30 23:48:41 -07:00
Tim Abbott bc9bdd53aa puppet: Use hostgroup for postgres_other monitoring. 2016-07-30 23:46:05 -07:00
Tim Abbott 6b34bee806 puppet: Remove unused ancient humbug-default apache site. 2016-07-30 23:43:55 -07:00
Tim Abbott 217faed3b3 Update check_postgres plugin to version 2.22. 2016-07-30 23:19:40 -07:00
Tim Abbott 1f549dcfab check_fts_update_log: Fix use on non-appbd machines. 2016-07-30 21:48:17 -07:00