Subclasses of QueueProcessingWorker that don't override start() will
have their consume() functions wrapped by consume_wrapper(), which
will catch exceptions and log data from troublesome events to a log
file.
We need to do a puppet apply to create /var/log/zulip/queue_error.
(imported from commit 3bd7751da5fdef449eeec3f7dd29977df11e2b9c)
I somehow missed it yesterday before doing the deploy. This change
is already on postgres1 and does not need to be deployed anywhere
else.
(imported from commit 01b3f0b81ac5eeeb985ff844e7f2d5cb3c986d0f)
We need to do a puppet apply immediately after deploying this, or our
rabbitmq consumer state files will become stale.
(imported from commit 18696a5a8b1ff431425d1f71c208acc9bf0694f2)
This commit must be simultaneously deployed on both staging and
prod0. It also requires completely taking down the app.
To deploy these changes, do:
* check out this commit at /root/zulip on postgres0, postgres1, staging, and prod0
* stop the process_fts_updates job on postgres0 and postgres1
* stop the app on staging and prod0
* do a puppet apply on postgres0, postgres1, staging, and prod0
* move the new client certificates into place on staging and app
* move the new server certificates into place on postgres0 and postgres1
* reload the database config on postgres0 and postgres1 (this might
actually require a restart)
* run tools/migrate-db on postgres0 as root
* do a deploy through this commit on staging and prod0
* start the process_fts_updates job on postgres0 and postgres1
* do a puppet apply on nagios
(imported from commit 819bdd14326c1425e2d3041a491a8ca3b9716506)
Steps needed:
- puppet apply on staging/prod when deployed to respective sites
- puppet apply on bots.z.n when deployed to prod
- copy /var/tmp/.feedback-bot-ticket-number from bots to staging
(imported from commit 2c943dac8d871809b0997a4484f508ec5b078bcd)
We ran into the 1024 file descriptor limit today for tornado. The
limits.conf file descriptor limit isn't used for supervisor started as
we currently do it, but it's a good idea to have this in place so that
if we move supervisor to run as the zulip user, we don't experience a
nasty surprise.
(imported from commit 2eb4805b5c129bc2684d151b77f295c2eaa9fc3e)
To apply this change, we must not only do a puppet apply, but also
restart rabbitmq and epmd. Rabbitmq is easy to restart, but epmd is
a little more annoying. epmd is run as a side effect of starting up
rabbitmq-server, but is not stopped when rabbitmq-server is stopped.
Therefore, the correct procedure is to stop rabbitmq-server, kill
epmd (by running `epmd -kill`), and then start rabbitmq-server again.
(imported from commit a651e5363a8b9a04b713c31baef379c566d5dbfc)
New dependency: sockjs-tornado
One known limitation is that we don't clean up sessions for
non-websockets transports. This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47
(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
These parameters collectively determine when very old transaction ids
are replaced with the special FrozenXID transaction id, which is
older than any other transaction id. Old transaction ids must be
periodically replaced like this to prevent transaction id wraparound.
The only disadvantage of increasing autovacuum_freeze_max_age is
increased disk usage, but a value of 2 billion should only require
500MB, which is pretty trivial.
Since changing autovacuum_freeze_max_age requires a restart, we will
still have to issue a manual VACUUM, with vacuum_freeze_min_age and
vacuum_freeze_table_age temporarily set to lower values.
(imported from commit 22f9ecdfc5b6a07918771d541192aa6d6369878a)
The psql command was previously failing because it was trying to use
the non-existant DB user "zulip".
(imported from commit 68ad5c979ce33cc54e9d55f1822cba9fac648944)
make sure to run `apt-get update` and then `tools/zulip-puppet-apply`
on staging and prod
(imported from commit 8d1e6295c7c395333a05cd664aa28cab6496bdaa)
We need to make sure it's totally drop-in functional on the app servers
before I'd feel comfortable modifying their config.
(imported from commit e42d6e37732d65f827982aabaff9399ec1bda0f2)
This may require just doing an mv on the home directory, plus changing
the home directory in /etc/passwd. It should of course be done carefully.
(imported from commit 660997d897ee6d33563af74f0fc5d4267a911755)
This will probably require shutting down all the supervisor processes
and restarting them to deploy properly.
(imported from commit ce687078980b43be495c49f52ed6aa0e25cdad00)
This requires doing an `mv` and then `puppet apply` on each of staging
and prod as part of the deployment process.
(imported from commit 5d0be64a3846f7151d2036d2e0b31049bc1c2dd2)
Steve Howell
Kevin Mehall
Tim Abbott
Zev Benjamin
Leo Franchi
Jessica McKellar
Luke Faraone
Tim Abbott
acrefoot