Commit Graph

1187 Commits

Author SHA1 Message Date
Tim Abbott 88a123d5e0 Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.

Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.

Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.

Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Tim Abbott c7059c9751 travis: Update success-http-headers to match current certs.
Travis CI seems to have changed the way the snakeoil SSL certs are
generated in their infrastructure, so we need to update our expected
"success" HTTP headers accordingly.
2016-08-12 09:35:41 -07:00
Tim Abbott a648513580 production-helper: Remove /root/zulip during setup process.
This fixes a problem that caused production-helper to not be
idempotent.
2016-08-11 22:21:13 -07:00
Tim Abbott 7011c94465 production-helper: Use ln -nsf to install snakeoil symlinks.
This fixes a problem where production-helper was not idempotent.
2016-08-11 22:20:24 -07:00
Tim Abbott 28a7d991f6 setup-production: Remove more postgres packages.
Travis CI seems to have added Postgres 9.5 to the default-installed
Postgres package list, and we also missed a few side packages
previously as well.
2016-08-11 22:19:57 -07:00
Rishi Gupta 0301347e86 provision.py: Fix typo.
Typo in the last line of the script; fixing since we'll see it every time
we provision.
2016-08-11 15:20:29 -07:00
Tim Abbott cb21584ffe check_send_receive_time: Support being run in a cron job. 2016-08-11 14:48:21 -07:00
Tim Abbott 4dcbaf1e6b check_send_receive_time: Cleanup unnecessary print statement. 2016-08-11 14:48:21 -07:00
Tim Abbott dca585fa75 travis: re-disable test-queue-worker-reload.
It seems to still fail sometimes in Travis CI.
2016-08-10 16:43:00 -07:00
Umair Khan 740b5634ac Fix and re-enable test-queue-worker-reload.
It seems that we no longer get the message, 'zerver/lib/actions.py
modified; restarting server', but the server reloads successfully
nonetheless.

Fixes: #1341.
2016-08-10 16:17:23 -07:00
Tomasz Kolek 4a25b92298 Cleanup result message code in test-js-with-node.
Remove duplicate code after quick fix in passing returned code.

Fixes: #1489.
2016-08-09 16:46:24 -07:00
Steve Howell a904222947 Add tools/find-add-class to tools/lint-all.
The find-add-class tool, when in lint mode, verifies that we can
understand all calls to addClass from our JS code.

When in non-lint mode, i.e. verbose mode, the tool prints out a
list of tuples of (fn, class) that we can use as we wish in other
tools.
2016-08-08 15:32:48 -07:00
Steve Howell 95b19f9765 Add test_validate_django(). 2016-08-08 15:20:29 -07:00
Steve Howell d29e745142 Add test_validate_handlebars(). 2016-08-08 15:20:29 -07:00
Steve Howell 70bc932913 Add test_html_tag_tree(). 2016-08-08 15:20:29 -07:00
Steve Howell 65147e7368 Add tests for template_parser.tokenize. 2016-08-08 15:20:29 -07:00
Steve Howell 184e45b774 Fix bug with tools/html-grep.
We were ignoring singleton tags like "input" tags in
html-grep.  This was an artifact of our tokenizer originally
being built to check indentation of templates, for which
singleton tags had been a distraction. This fix actually cleans up
the template checking logic as well, since it can now rely
on the tokenizer to classify special tags and singleton tags.
The tokenizer is more complete and more specific.
2016-08-08 15:20:29 -07:00
Tim Abbott b3a768f4b2 settings: Improve ALLOWED_HOSTS defaults logic and docs.
This removes the requirement for the user to put localhost/127.0.0.1
in their ALLOWED_HOSTS list, since it is now added automatically.

Fixes: #1358.
2016-08-05 21:25:29 -07:00
Tim Abbott 5bff72c385 Revert "Use apt-add-repository to setup Zulip PPA."
This reverts commit 3f95e567c1.

Apparently `apt-add-repository` fails periodically in CI.  I suspect
this is some sort of silly networking problem, but given that all
we're saving is a few lines of code, the old version was better if
this fails basically ever.
2016-08-05 13:29:17 -07:00
Tim Abbott 3f95e567c1 Use apt-add-repository to setup Zulip PPA. 2016-08-04 22:17:07 -07:00
Tim Abbott b50abc5131 npm: Split out production-only dependencies. 2016-08-04 22:17:07 -07:00
Tim Abbott 39f1024740 test_messages: Fix remaining mypy errors. 2016-08-04 15:57:03 -07:00
Tim Abbott 89267f926e Annotate tools/run-dev.py. 2016-08-04 15:57:02 -07:00
Tim Abbott 0689485666 Add lint check for malformed type annotations missing :.
Also fix the 2 annotations that weren't being checked because of this.
2016-08-04 15:53:23 -07:00
Tim Abbott 604e2821f8 Remove tools/deprecated/backup. 2016-08-04 11:34:53 -07:00
Steve Howell 7cc1b1ebc4 Add test coverage for parsers in tools/lib.
Now, `tools/test-all` calls a new program called `tools/tests-tools`
that runs unit tests in `test_css_parser.py` and 'test_template_parser.py`.

This puts 100% line coverage on tools/lib/css_parser.py.
This puts about 50% line coverage on tools/lib/template_parser.py.
2016-08-04 10:47:09 -07:00
Steve Howell 5af47e0eef Make fn param optional with template_parser.validate().
The caller can now pass in text directly.  This is mostly for
testing, but it could be useful in other situations.
2016-08-04 10:47:09 -07:00
Steve Howell 1f8ba1d1b5 Make minor changes to css_parser.py 2016-08-04 10:47:09 -07:00
Tim Abbott 09b677b605 test-js-with-node: Fix istanbul not being in PATH. 2016-08-04 09:29:30 -07:00
Tim Abbott 82b1218af9 test-js-with-node: Fix return status on failure. 2016-08-04 09:27:54 -07:00
Tomasz Kolek c96722b124 Add istanbul cover info to help in test-js-with-node. 2016-08-03 14:57:03 -07:00
Tomasz Kolek 87b554906d Add clear success message after node tests run.
Fixes: #1489.
2016-08-03 14:56:58 -07:00
Steve Howell 0f8d014096 Add tools/html-grep.
This tools greps for HTML nodes that have relevants words
pertaining to classes, ids, or tags in themselves or their
parents.
2016-08-03 11:29:37 -07:00
Steve Howell 3b5c187f55 Add tools/check-css and tools/lib/css_parser.py.
`tools/lint-all` now calls the new `tools/check-css`

The css_parser library parsers CSS into a data structure
that remembers line numbers and columns of semantically
meaningful tokens and adjoining white space/tokens.  It
is intended to be used for various linting tasks.

The file `tools/check-css` runs a few files through the
parser and makes sure they round trip.  This has some value
right away, as files that fail to parse will cause an
exception to be thrown and thus alert developers to syntax
errors.  We expect to grow this into more advanced linting
tasks eventually.
2016-08-02 16:22:46 -07:00
Taranjeet Singh b82836a901 tools/setup/emoji_dump/emoji_dump.py: Move bitmaps & *.ttx to var/.
Also update .gitignore to remove the old .gitignore markings for
these files' old locations.
2016-08-02 15:18:36 -07:00
Tim Abbott 0ee3178167 Move update-deployment to scripts/deploy-zulip-from-git. 2016-08-02 14:31:07 -07:00
Tim Abbott fef32af28c update-deployment: Switch to using update-zulip-part-2. 2016-08-02 14:31:07 -07:00
Steve Howell 9151ee42e9 Extract tools/lib/template_parser.py. 2016-08-01 16:27:50 -07:00
Steve Howell 9951e92b3b Extract tokenize() in tools/check-templates. 2016-08-01 16:05:07 -07:00
Steve Howell 7e772ed644 Clean up code in check-templates.
I extracted a few helper methods.
2016-08-01 16:05:07 -07:00
Tim Abbott 75b5d021fa Remove unused puppet-common third-party module code. 2016-07-31 19:24:42 -07:00
Tim Abbott 143575a5bd provision: Fetch tsearch_extras package from Zulip apt repository.
This fixes installing a Zulip development environment on Ubuntu Xenial.
2016-07-30 21:01:07 -07:00
Steve Howell 6a65b3482c Move index.js to frontend_tests/zjsunit. 2016-07-30 14:54:30 -07:00
Tim Abbott f0bf883772 provision: Retry node_modules setup on failures.
`npm install` fails nondeterministically occasionally, and this makes
such failures likely to be automatically resolved in most cases by
simple retrying.
2016-07-30 10:09:23 -07:00
Tim Abbott 315766ae02 provision: Provide a nice success message. 2016-07-30 10:09:23 -07:00
Tim Abbott ed95c34b83 check-templates: Remove now-removed terms.html. 2016-07-30 09:56:54 -07:00
Steve Howell 8b13530712 Add tools/analyze-url-coverage 2016-07-29 21:23:33 -07:00
Steve Howell 21f83afe3a Add --url-coverage option to ./tools/test-backend. 2016-07-29 21:23:33 -07:00
Steve Howell 4556bf528f Add --report-slow-tests option to tools/test-backend. 2016-07-29 16:41:20 -07:00