The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.
Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.
Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.
Further TODO items on this front include removing the hardcoded list
of queues.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.
We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.
To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:
[rabbitmq]
nodename = zulip@localhost
However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer. So for now, this just
fixes our Travis CI problems.
Fixes: #1579.
Travis CI seems to have changed the way the snakeoil SSL certs are
generated in their infrastructure, so we need to update our expected
"success" HTTP headers accordingly.
It seems that we no longer get the message, 'zerver/lib/actions.py
modified; restarting server', but the server reloads successfully
nonetheless.
Fixes: #1341.
The find-add-class tool, when in lint mode, verifies that we can
understand all calls to addClass from our JS code.
When in non-lint mode, i.e. verbose mode, the tool prints out a
list of tuples of (fn, class) that we can use as we wish in other
tools.
We were ignoring singleton tags like "input" tags in
html-grep. This was an artifact of our tokenizer originally
being built to check indentation of templates, for which
singleton tags had been a distraction. This fix actually cleans up
the template checking logic as well, since it can now rely
on the tokenizer to classify special tags and singleton tags.
The tokenizer is more complete and more specific.
This reverts commit 3f95e567c1.
Apparently `apt-add-repository` fails periodically in CI. I suspect
this is some sort of silly networking problem, but given that all
we're saving is a few lines of code, the old version was better if
this fails basically ever.
Now, `tools/test-all` calls a new program called `tools/tests-tools`
that runs unit tests in `test_css_parser.py` and 'test_template_parser.py`.
This puts 100% line coverage on tools/lib/css_parser.py.
This puts about 50% line coverage on tools/lib/template_parser.py.
`tools/lint-all` now calls the new `tools/check-css`
The css_parser library parsers CSS into a data structure
that remembers line numbers and columns of semantically
meaningful tokens and adjoining white space/tokens. It
is intended to be used for various linting tasks.
The file `tools/check-css` runs a few files through the
parser and makes sure they round trip. This has some value
right away, as files that fail to parse will cause an
exception to be thrown and thus alert developers to syntax
errors. We expect to grow this into more advanced linting
tasks eventually.
`npm install` fails nondeterministically occasionally, and this makes
such failures likely to be automatically resolved in most cases by
simple retrying.
When running ./tools/test-backend, the script to generate
fixtures, ./tools/setup/generate-fixtures, looks for a file
called migration-status to determine whether it can short
circuit doing database migrations. This file got moved as
part of the effort to put files in "var," but the existence
check was still looking for that file in its old location.
Run '/puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_send_receive_time'
script as 'zulip' user so that the connection to the database can be
made correctly.
Initialize Record by using __init__ instead of setting attributes
in validate. This is needed because mypy complains when we set
new attributes outside __init__.
Factor out the code in tools/provision.py which installs a python2
and python3 venv into a module (tools/setup/setup_venvs.py) which
can also be used as a script.
Twisted is not python 3 compatible. So for now create a python2
venv and install twisted in it when running provision.py in python3
mode and use twisted from the python2 venv.
Clear memcached when tools/run-dev.py is run. This prevents
errors on using a different python version because values are
pickled before being stored in memcached and different python
versions implement pickling differently.
Also provide a command-line option --no-clear-mc to prevent
memcached from being cleared.
tools/provision.py: Create directory var/uploads.
zproject/local_settings_template.py: Update Upload dir to var/uploads.
zproject/dev_settings.py: Update upload dir to var/uploads.
runtornado unbuffers its output using
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0).
This is not python 3 compatible since we can't specify
buffering on a text stream in python 3. So use the '-u'
option of python when calling runtornado.py to make output
unbuffered.
The purpose of this is to move a lot of the log and other generated
files used by the Zulip development environment into a consistent
hierarchy.
We also need to create this in tools/build-release-tarball as well,
since that runs a development environment out of a temporary
directory.
Instead of just checking .html files in the templates directory,
we now also check them everywhere, including .html files used
for Casper tests and some static files.
* get_realm returns None if no matching realm is present, but
create_stream.py assumed it raises Realm.DoesNotExist.
* encoded/decode strings properly.
* Replace filter by list comprehension.
* Add '# type: ignore' to statements which use attributes from
modeule `posix`, since stubs for posix are missing on python 3.
* get_realm returns None if no matching realm is present, but
create_stream.py assumed it raises Realm.DoesNotExist.
* encoded/decode strings properly.
The functions truncate_content, truncate_body and truncate_topic
are only meant to be used on text strings. So change its
parameter types from AnyStr to text_type.
Many stubs in xml.etree.ElementTree use Union[str, bytes] as
return type. Mypy wants us to correctly handle each case. This
is correct, but not useful for us since we know that we'll always
get str. So force the return value to text_type, to supress mypy
errors.
When provision.py is run using python 2, retain original behavior
of creating a python 2 venv using requirements/py2_dev.txt and
creating a python 3 venv using requirements/mypy.txt.
When provision.py is run using python 3, install a single python 3
venv with requirements/py3_dev.txt.
This works around a nasty problem with Webpack that you can't run two
copies of the Webpack development server on the same project at the
same time (even if on different ports). The second copy doesn't fail,
it just hangs waiting for some lock, which is confusing; but even if
that were to be solved, we don't actually need the webpack development
server running to run the Casper tests; we just need bundle.js built.
So the easy solution is to just run webpack manually and be sure to
include bundle.js in the JS_SPECS entry.
As a follow-up to this change, we should clean up how test_settings.py
is implemented to not require duplicating code from settings.py.
Fixes#878.
This is important for both ensuring the Nagios checks work correctly
in production, as well as making sure the `zulip` user can access the
virtualenv (owned by the `travis` user) in Travis CI.
Rather than looking at which venvs are used by this particular build,
we instead look at which venvs have a hash that is the hash_reqs value
of a current requirements.txt file.
Tim Abbott
Rishi Gupta
Umair Khan
Tomasz Kolek
Steve Howell
Taranjeet Singh
Eklavya Sharma
Steve Howell
Steve Howell
Vishnu Ks
acrefoot
Aakash Tyagi
rahuldeve