The problem this commit solves is related to how we search for
testcases in the test files. We use a simple 'search_key' in file_data.
This will return a false positive if there is a 'longer_search_key' in
file_data.
While searching for TestCases we should use a longer key to remove the
possibility of collision. Using 'class TestCase(' should be precise
enough.
Fixes#4983
We enable data_suffix option when creating Coverage instances which
causes the output files to include the hostname, pid, and random id.
Before each run erase is called which clears all existing coverage data
files. And then at the end of the test run use the combine method which
merges the reports.
We collect coverage in the main process which collects data from
imports and also when running in single process mode. In the workers we
collect coverage in run_subsuite. This creates more stats files than
strictly required but I don't see a better place to save the stats when
stopping workers.
Note that this has the side effect of enabling parallel testing in
Travis CI.
This commit changes the backend testing framework to run
in parallel mode which is same as --processes=4. If --coverage
is supplied, we enforce serial mode, --processes=1, because
coverage is not compatible with parallel mode at the moment.
This fixes the fact that our test suites would have trouble connecting
to the other parts of the Zulip service when run with a proxy
configuration (e.g. trying to send requests to localhost through the
proxy!).
Thanks for Vishnu Ks for his work on this.
Fixes#971.
In backend tests, only call generate-fixtures when --generate-fixtures
is explicitly passed or is_template_database_current() returns False.
We don't need to flush cache for backend tests because we bounce the key
prefix used to create cache keys before running every test
This adds the option '--rerun' to the `test-backend` infrastructure.
It runs the tests that failed during the last 'test-backend' run. It
works by stailing failed test info at var/last_test_failure.json
Cleaned up by Umair Khan and Tim Abbott.
This code was added as part of the Django 1.10 migration to make our
tests work with both Django 1.8 and 1.10. Now that we're on 1.10,
it's no longer required.
Like many rare-case code with new tests, it turns out that the logic
for handling null characters in our Zephyr postgres query escaping
never worked, in multiple ways. First, it always changed the second
character in s, not the current one being inspected, and second, the
value it replaced it with was no the correct postgres escape of the
null byte. We fix this and add tests.
This completes the effort to get zerver/views/messages.py to 100%
test coverage.
Fixes#1006.
This adds an assertion, when `test-backend` is run with `--coverage`,
that we have 100% test coverage on a list of files that we expect to.
There's a whitelist/blacklist, managed in tools/test-backend.
Fixes#3363.
This helps make the Zulip development environment somewhat more robust
to new contributors, since it will give them a nice warning if they
try running any of our development tools outside the Zulip virtualenv.
Fixes#3468.
We now instrument URL coverage whenever you run the back end tests,
and if you run the full suite and fail to test all endpoints, we
exit with a non-zero exit code and report failures to you.
If you are running just a subset of the test suite, you'll still
be able to see var/url_coverage.txt, which has some useful info.
With some tweaks to the output from tabbott.
Fixes#1441.
These are not particularly interesting to measure test coverage for,
since migrations are not run as the part of the test suite by
construction, and management commands aren't being tested by this test
suite.
Previously, running `tools/test-backend analytics/` (or any other test suite
name ending with a '/') would give a cryptic error about modules not
importing properly. This commit rstrip's the trailing slash from test suite
names given on the command line.
Previously, the generate-fixtures shell script by called into Django
multiple times in order to check whether the database was in a
reasonable state. Since there's a lot of overhead to starting up
Django, this resulted in `test-backend` and `test-js-with-casper`
being quite slow to run a single small test (2.8s or so) even on my
very fast laptop.
We fix this is by moving the checks into a new Python library, so that
we can avoid paying the Django startup overhead 3 times unnecessarily.
The result saves about 1.2s (~40%) from the time required to run a
single backend test.
Fixes#1221.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
aggregating user/stream level stats into realm level stats
* A management command for pulling the data
Note that counts.py was added to the linter exclude list due to errors
around %%s.
This optimizes the process of running individual or small groups of
backend tests (./tools/test-backend
zerver.tests.test_bugdown.FencedBlockPreprocessorTest.test_simple_quoting)
to allow the following syntaxes:
./tools/test-backend zerver/tests/test_bugdown.py
./tools/test-backend zerver.tests.test_bugdown.py
./tools/test-backend zerver/tests/test_bugdown
./tools/test-backend zerver.tests.test_bugdown
./tools/test-backend test_bugdown.py
./tools/test-backend test_bugdown
./tools/test-backend FencedBlockPreprocessorTest
./tools/test-backend FencedBlockPreprocessorTest.test_simple_quoting
Fixes#1670.
Since relatively few systems have the typing module, this makes the
checks for whether the user is properly running our test scripts in
the virtualenv more likely to trigger well.
This should make users much more likely to be able to debug issues
where they ran Zulip outside the Vagrant environment or virtualenv.
[error messages tweaked by tabbott]
Add two options to the `test-backend` script:
1. verbose
If given the `test-backend` script will give detailed output.
2. no-shallow
Default value is False. If given the `test-backend` script will
fail if it finds a template which is shallow tested.
Add call to tools/generate-fixtures in tools/test-backend before
starting the tests. Previously, test-backend could fail if called
after tools/test-js-with-casper had failed.
Fixes#501.
The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.
We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.
Thanks to Dan Fedele for the bug report and suggested solution!
This includes a hack to preserve humbug/backends.py as a symlink, so
that we don't need to regenerate all our old sessions.
(imported from commit b7918988b31c71ec01bbdc270db7017d4069221d)
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).
At the time it is deployed, we need to make a few changes directly in
the database:
(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';
(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
Does a test-all using /tmp.
Not safe to run with other tests simultaneously.
You use it just like tools/review, and on a successful test-all
it will submit the code review, without tying up your git working directory.
It depends on new patches to the acrefoot-bulk_create_with_id-1.5.1 branch of
django. Updating this is the [manual] step required. Since test-all doesn't happen
on staging or prod, this is not required for this commit to be deployed, but other
django patches will help with the SSL connection errors we've been seeing.
(imported from commit 3fd3ff00240a2d648c4c54748a8a70616067ef7d)
Currently our test database is backed by sqlite; this commit moves
us to using postgres for our all database needs. This, in conjunction
with the patched django on github, allow us to have fewer hacks and
more true-to-life tests. It also sets the stage for testing the bulk_create
and schema search_path patches made to django.
Developers will need to run:
./tools/postgres-init-test-db
./tools/do-destroy-rebuild-test-database
this is assuming that they have already run:
./tools/postgres-init-db
./tools/do-destroy-rebuild-database
at some point on this pg_cluster. (The ordering is important; it will other-
wise complain about the south_migration table).
(imported from commit c56c6f27e13df7ae10b2e643e65d669dde61af3d)
It doesn't save much time (maybe 0.5s out of 12+s).
I'm leaving the option in because I think it is still useful for
iteratively testing a single test case.
(imported from commit a0ac43f4c48eec101f05d731740394b30a15773b)
This now allows e.g.
./tools/test-backend zephyr.BugdownTest
or
./tools/test-backend zephyr.BugdownTest --verbosity=2
but unfortunately not
./tools/test-backend --verbosity=2 zephyr.BugdownTest
(imported from commit 95302db3efe45182d789c40c2a2899230a06b091)
Umair Khan
Pweaver (Paul Weaver)
Mehanig
Steve Howell
Jason Michalski
Tim Abbott
K.Kanakhin
Ayush Jain
Adarsh S
Elliott Jin
sinwar
Tomasz Kolek
reallyly
paxapy
sonali0901
Rishi Gupta
umkay
Taranjeet Singh
Tim Abbott
Eklavya Sharma
Aakash Tyagi
Alex Wilson
Ernesto Vargas
Vladislav Manchev
Steven Oud
Tim Abbott
Tim Abbott
acrefoot
Keegan McAllister