Commit Graph

325 Commits

Author SHA1 Message Date
Tim Abbott b70986469c generate_secrets: Fix handling of missing trailing newlines.
When we added support for automatically adding new secrets in
generate_secrets.py, we failed to account for the possibility that a
human editor might have let the secrets file without a trailing
newline.

We address this by adding a leading newline before our new secret.

Fixes #5209.
2017-06-03 23:17:04 -07:00
Tim Abbott c37204c62a upgrade-zulip-stage-2: Remove an unused import. 2017-06-02 15:14:13 -07:00
Tim Abbott 9e0749ad82 email-mirror-postfix: Fix mypy errors. 2017-06-01 22:51:26 -07:00
K.Kanakhin 47ec9fbbe2 email-mirror: Rewrite email mirror script on pure python.
The Zulip email mirror script called by postfix had performance/load
issues, because it spent so much time on startup/import due to use of
the Zulip virtualenv.

The script was rewritten using pure python (no Django) to improve
performance.
2017-06-01 21:50:49 -07:00
Tim Abbott be814b940d setup_venv: Pin a version of setuptools for creating venvs.
This works around a bad recent setuptools release:

https://github.com/pypa/setuptools/issues/1042
2017-05-31 23:58:24 -07:00
Aditya Bansal ee369ceb0c pep8: Add compliance with rule E261 scripts/lib/email-mirror-postfix. 2017-05-31 17:07:15 -07:00
Aditya Bansal 807fee68d6 pep8: Add compliance with rule E261 nagios/check-rabbitmq-consumers. 2017-05-31 17:07:15 -07:00
Aditya Bansal dee726f234 pep8: Add compliance with rule E261 scripts/lib/pythonrc.py. 2017-05-31 17:07:15 -07:00
Elliott Jin 0ec9e54954 bots: Add queue and QueueProcessingWorker for embedded bots. 2017-05-25 15:00:51 -07:00
theopen-institute 7eaa1fa0d0 create-production-venv: Fix symlink creation.
The install script was failing on 2nd+ attempts if the first attempt
was interrupted.

This failure happened because zulip-venv already existed at
`current_venv_path`. Changing the `ln` command's flags from `-s` to
`-nsf` should make this part of the script idempotent.
2017-05-20 21:31:50 -07:00
Tim Abbott 11adbf5783 generate_secrets: Fix placement of mypy type: ignore. 2017-05-17 00:05:57 -07:00
Tim Abbott b01ba5f389 generate_secrets: Fix mypy errors.
I'm pretty sure these errors reflect a problem with Typeshed, but
don't have time to investigate.
2017-05-16 23:28:44 -07:00
Tim Abbott 45a4aeac64 scripts: Run generate_secrets.py during the upgrade process.
Now that generate_secrets.py is idempotent, this allows us to
conveniently add new secrets whenever they are required.
2017-05-16 22:15:26 -07:00
Tim Abbott 2c6a91e24a scripts: Make generate_secrets.py idempotent.
Now, generate_secrets.py will never overwrite existing secrets.  In
addition to being a safer model in generate, this fixes 2 significant
issues:

(1) It makes it much easier to preserve secrets like Oauth tokens in a
development environment (previously, provision would destroy them).
(2) It makes it possible to automatically add new secrets as part of
the upgrade process.  In particular, this is useful for the
zulip_org_id settings.

Fixes #4797.
2017-05-16 22:15:25 -07:00
Tim Abbott 03b5200d8b generate_secrets: Reformat list of autogenerated secrets. 2017-05-16 22:15:25 -07:00
rht 00e057bf44 install-node: bypass nvm wrapper for faster node startup.
This fixes a significant performance issue with LaTeX rendering (and
other things that invoked node) where starting up node took a few
hundred milliseconds due to nvm initialization.

Tweaked by tabbott to avoid copying the node binary itself, instead
using a tiny wrapper script.

This is important primarily because it's possible a future version of
node will expect to find libraries/dependencies/etc. installed via NVM
at some path related to the path of the node binary itself, and that's
more guaranteed with this new model.

Fixes #4618.
2017-05-09 09:17:54 -07:00
Aditya Bansal b3ad6a6d77 pep8: Add compliance with rule E261 to hash_reqs.py. 2017-05-07 23:21:50 -07:00
vaibhav 8881b5eb9f Outgoing Webhook System: Check for @-mentioned outgoing webhook bots.
Also puts them into a processing queue, though the queue processor
does nothing.

Rewritten by tabbott to avoid unnecessary database queries in
do_send_messages.
2017-05-02 09:22:04 -07:00
Alexander Trost 889547ff5e configure-rabbitmq: Add support for RABBITMQ_NODE flag.
This can potentially be used by things like a Docker configuration
that runs RabbitMQ on another server.
2017-04-29 15:03:05 -07:00
K.Kanakhin e3e52e7284 email-mirror: Move postfix email mirror integration to separate script.
This fixes a performance problem where we were previously starting up
a full Django process (~0.7s even on a fast machine) every time a new
email came in, potentially allowing users to accidentally DoS a Zulip
server.  Now, we just post over HTTPS, allowing the existing thread
pool support to do its job.

- Add script wrapper to communicate postfix pipe with django web server
  over HTTP(S). It uses shared_secret authentication mode.
- Add django view to process messages from email mirror server.
- Clean management command `email-mirror`. Left just functional
  for cron email processing.
- Add routes for new tornado view.
- Change pipe script in master process postfix config template
  based on updated script.
- Add tests.

Tweaked by tabbott to adjust the directory and set better defaults.

Fixes #2421.
2017-04-24 21:24:23 -07:00
Feorlen 6ff78ca0e8 Set umask 022 before starting prod upgrade.
Follow-on from #2373/ PR https://github.com/zulip/zulip/pull/4316, to set an
appropriate umask also when upgrading so files have appropriate permissions.

I've tested this by starting from a clean install, deleting /srv/* so new
files are downloaded, and then doing an upgrade. It worked starting with both
a current version from master and an older release installed with a less
restrictive umask and then the umask changed.

Fixes #2373.
2017-04-19 10:28:06 -07:00
Tim Abbott 55a9101573 settings: Add support for ZULIP_ORG secrets.
These can be used to authenticate the current Zulip server to
zulip.org.
2017-04-18 23:00:10 -07:00
Feorlen 10ccfcdc8e Set umask 022 before starting prod install.
Fixes #2372.
2017-03-25 23:59:44 -07:00
Tim Abbott 5bf01fb7d4 create-production-venv: Fix issues with api/ relative path.
Fixes #4313.
2017-03-25 19:49:26 -07:00
K.Kanakhin 6a801db1c2 missed-emails-sending: Move email sending to separate queue worker.
- Add new 'missedmessage_email_senders' queue for sending missed messages emails.
- Add the new worker to process 'missedmessage_email_senders' queue.
- Split aggregation missed messages and sending missed messages email
  to separate queue workers.
- Adapt tests for sending missed emails to the new logic.

Fixes #2607
2017-03-07 20:08:40 -08:00
Tim Abbott 0afe832fc7 check-rabbitmq-consumers: Fix typing import issue. 2017-03-04 15:35:26 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 2bbfdeeb7b Fix more errors caught by mypy 0.501.
Another set of relatively easy to review changes.
2017-03-03 14:15:38 -08:00
Rishi Gupta 28d3af0965 Fix several new errors caught by mypy 0.501.
Clear out a bunch of easy to review errors, so we can focus on the more
complicated ones.
2017-03-03 14:12:52 -08:00
Tim Abbott f6f2c62caf zulip-puppet-apply: Fix running it as ./zulip-puppet-apply. 2017-02-27 15:25:14 -08:00
Tim Abbott fe0c4cad85 check-rabbitmq-consumers: Go back to hardcoding for now.
This should fix the production test suite in Travis CI, so that we can
debug what's broken here offline.
2017-02-22 22:58:59 -08:00
Tim Abbott b81add60fe check-rabbitmq-consumers: Fix queue_workers call. 2017-02-22 00:48:43 -08:00
Tim Abbott aa6567ee34 queue_workers: Fix confusing --queue_type argument name. 2017-02-22 00:23:26 -08:00
Tim Abbott 19896460f0 nagios: Fix RabbitMQ Nagios checks running Django as root.
This can cause problems by making the /var/log/zulip files owned by
root (not zulip) and thus not writable by the Zulip user.
2017-02-22 00:20:57 -08:00
Tim Abbott 51d3ab1cb7 initialize-database: Clean up final instructions.
Fixes #3678.
2017-02-21 20:19:16 -08:00
Tim Abbott 620f1e444e travis: Fix various bugs in new queue worker test.
* Now queue_workers.py sorts queue names and prints them on their own
  line.  Previously it's output was nondeterministic.
* Simplified grep strategy for removing the "test" worker.
2017-02-19 21:17:42 -08:00
Tim Abbott 333062f08e nagios: Automate queue list in check-rabbitmq-consumers. 2017-02-19 16:19:55 -08:00
Tim Abbott d6bbcd2737 travis: Automate updates to production-helper Nagios test.
This list was likely to end up out of date quickly, since it wasn't
documented that you need to update it when adding a queue.  The best
solution is to just not require it to be updated.
2017-02-19 16:19:53 -08:00
Tim Abbott 34046c1f55 check-rabbitmq-consumers: Add missing embed_links consumer. 2017-02-19 13:12:00 -08:00
Tim Abbott 213af24e47 check-rabbitmq-consumers: Reformat worker_queues list. 2017-02-19 13:12:00 -08:00
Tim Abbott 473c0ee1fe mypy: Remove now-unused type: ignores. 2017-02-18 17:01:01 -08:00
Tim Abbott 609082c475 puppet: Remove hardcoding of /root/zulip from puppet path.
This is an important prerequisite to being able to remove dependence
on the /root/zulip symlink altogether.
2017-02-08 11:13:19 -08:00
Tim Abbott 31d4f99573 upgrade: Stop trying to copy node_modules out of tarballs.
Now that we no longer use node_modules at all in production (it's only
used to generate static assets), we don't include `node_modules` in
the production tarballs, and thus we shouldn't attempt to copy
`node_modules` out of the production tarballs when installing.

Fixes a regression introduced in
d71f2e7b9b.
2017-02-07 10:39:31 -08:00
Tim Abbott 126b1c4edc upgrade: Don't require authors updates deploying from Git.
Fixes #3392.
2017-01-23 23:00:09 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott de99f48ce7 lint: Clean up E401 PEP-8 rule. 2017-01-23 21:36:39 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Ayush Goyal a85b539c4a zulip_tools: Improve color and copy for run() errors.
Tweaks to the text are edited by tabbott.
2017-01-17 14:37:15 -08:00
Tim Abbott 747f66bfe1 docs: Add README.md files for scripts/ and tools/. 2017-01-17 11:01:27 -08:00
Tim Abbott 1d5edff927 setup_venv: Give up if virtualenv-clone isn't working.
virtualenv-clone can sometimes fail if the old virtualenv is
broken; in that case, we can just make a new one.
2017-01-09 11:21:42 -08:00