Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/tools/setup/generate-fixtures

45 lines
1.7 KiB
Plaintext
Raw Normal View History

Add support for running OpenBSD in development environment.
2016-01-12 13:08:43 +01:00
#!/usr/bin/env bash
set -e
generate-fixtures: Run after schema changes even without --force. (imported from commit 1daff2ee6746c4036c3cc71746ad59e18b5a331e)
2013-04-01 23:02:24 +02:00
function migration_status {
Optimize checks of test database state by moving into Python. Previously, the generate-fixtures shell script by called into Django multiple times in order to check whether the database was in a reasonable state. Since there's a lot of overhead to starting up Django, this resulted in `test-backend` and `test-js-with-casper` being quite slow to run a single small test (2.8s or so) even on my very fast laptop. We fix this is by moving the checks into a new Python library, so that we can avoid paying the Django startup overhead 3 times unnecessarily. The result saves about 1.2s (~40%) from the time required to run a single backend test. Fixes #1221.
2016-09-13 22:40:13 +02:00
./manage.py get_migration_status --settings=zproject.test_settings > $1
generate-fixtures: Run after schema changes even without --force. (imported from commit 1daff2ee6746c4036c3cc71746ad59e18b5a331e)
2013-04-01 23:02:24 +02:00
}
tests: Don't regenerate the test database every time we run tests This cuts about 6 seconds off of the test startup time on my laptop. The other startup costs are about 1 second for the server to come up and about half a second for casperjs to begin executing tests. generate-fixtures now takes a '--force' option that can be use to force a regeneration of the test database. (imported from commit 1f473507502f0edf159b2638abb392d9357eb46f)
2013-03-05 23:08:55 +01:00
Optimize checks of test database state by moving into Python. Previously, the generate-fixtures shell script by called into Django multiple times in order to check whether the database was in a reasonable state. Since there's a lot of overhead to starting up Django, this resulted in `test-backend` and `test-js-with-casper` being quite slow to run a single small test (2.8s or so) even on my very fast laptop. We fix this is by moving the checks into a new Python library, so that we can avoid paying the Django startup overhead 3 times unnecessarily. The result saves about 1.2s (~40%) from the time required to run a single backend test. Fixes #1221.
2016-09-13 22:40:13 +02:00
if [ "$1" != "--force" ]; then
"$(dirname "$0")/../../scripts/setup/terminate-psql-sessions" zulip zulip_test zulip_test_base zulip_test_template
psql -h localhost postgres zulip_test << EOF
generate-fixtures: Be more robust to databases being missing (imported from commit f42d673c433b8d3df4af79159831e57f5db1ee1c)
2014-01-30 23:40:07 +01:00
DROP DATABASE IF EXISTS zulip_test;
[manual] Change references to the humbug_test user, schema, and database to zulip_test This requires no changes in production, but is tagged as manual to remind developers that they need to edit and run the tools/migrate-db script to fix up their local database instances. (imported from commit fbf764fb61592ef994d6d2ad56edad65ff01f14b)
2013-10-26 04:24:40 +02:00
CREATE DATABASE zulip_test TEMPLATE zulip_test_template;
[manual] Humbug testing databases now backed by postgres Currently our test database is backed by sqlite; this commit moves us to using postgres for our all database needs. This, in conjunction with the patched django on github, allow us to have fewer hacks and more true-to-life tests. It also sets the stage for testing the bulk_create and schema search_path patches made to django. Developers will need to run: ./tools/postgres-init-test-db ./tools/do-destroy-rebuild-test-database this is assuming that they have already run: ./tools/postgres-init-db ./tools/do-destroy-rebuild-database at some point on this pg_cluster. (The ordering is important; it will other- wise complain about the south_migration table). (imported from commit c56c6f27e13df7ae10b2e643e65d669dde61af3d)
2013-05-08 05:30:42 +02:00
EOF
Optimize checks of test database state by moving into Python. Previously, the generate-fixtures shell script by called into Django multiple times in order to check whether the database was in a reasonable state. Since there's a lot of overhead to starting up Django, this resulted in `test-backend` and `test-js-with-casper` being quite slow to run a single small test (2.8s or so) even on my very fast laptop. We fix this is by moving the checks into a new Python library, so that we can avoid paying the Django startup overhead 3 times unnecessarily. The result saves about 1.2s (~40%) from the time required to run a single backend test. Fixes #1221.
2016-09-13 22:40:13 +02:00
sh "$(dirname "$0")/../../scripts/setup/flush-memcached"
exit 0
tests: Don't regenerate the test database every time we run tests This cuts about 6 seconds off of the test startup time on my laptop. The other startup costs are about 1 second for the server to come up and about half a second for casperjs to begin executing tests. generate-fixtures now takes a '--force' option that can be use to force a regeneration of the test database. (imported from commit 1f473507502f0edf159b2638abb392d9357eb46f)
2013-03-05 23:08:55 +01:00
fi
[manual] Rename Django app from zephyr to zerver. This needs to be deployed to both staging and prod at the same off-peak time (and the schema migration run). At the time it is deployed, we need to make a few changes directly in the database: (1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr'; (2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr'; (imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-07-29 23:03:31 +02:00
mkdir -p zerver/fixtures
Add hack to clean up zephyr/lib/time.pyc -- it will break tests. (imported from commit f649b31010adc38e3820879fcdd543e81d271d0f)
2013-01-10 22:58:01 +01:00
Move tools/generate-fixtures to tools/setup/.
2016-05-26 12:40:41 +02:00
"$(dirname "$0")/../../scripts/setup/terminate-psql-sessions" zulip zulip_test zulip_test_base zulip_test_template
Drop database users prior to DROP/CREATE database. This fixes an annoying issue where one tries to rebuild the database, and it fails due to there being existing connections. The one thing that is potentially scary about this implementation is that it means it's now a lot easier to accidentally drop your production database by running the wrong script; might be worth adding a "--force" flag controlling this behavior or something. Thanks to Nemanja Stanarevic and Neeraj Wahi for prototypes of this implementation! They did most of the work and testing for this.
2015-09-28 19:45:35 +02:00
Persist the tsearch_extras extension across rebuilding of the database We do this by creating a new zulip{_test}_base database that only has the zulip schema and the tsearch_extras extension. We then use that as a template when creating zulip{_test}. (imported from commit 8adb4b98410e4042a0187902e89c99561eac8c8f)
2014-01-30 23:40:37 +01:00
psql -h localhost postgres zulip_test <<EOF
DROP DATABASE IF EXISTS zulip_test;
CREATE DATABASE zulip_test TEMPLATE zulip_test_base;
EOF
Move tools/generate-fixtures to tools/setup/.
2016-05-26 12:40:41 +02:00
sh "$(dirname "$0")/../../scripts/setup/flush-memcached"
Persist the tsearch_extras extension across rebuilding of the database We do this by creating a new zulip{_test}_base database that only has the zulip schema and the tsearch_extras extension. We then use that as a template when creating zulip{_test}. (imported from commit 8adb4b98410e4042a0187902e89c99561eac8c8f)
2014-01-30 23:40:37 +01:00
Always start python via shebang lines. This is preparation for supporting using Python 3 in production. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-22 01:44:16 +01:00
./manage.py migrate --noinput --settings=zproject.test_settings
text_fixtures: Rename migration status file to have test in name. This is preparation for making things clear as we move towards not always rebuilding the test database in provision.
2017-06-02 21:48:30 +02:00
migration_status "var/migration_status_test"
generate-fixtures: Run after schema changes even without --force. (imported from commit 1daff2ee6746c4036c3cc71746ad59e18b5a331e)
2013-04-01 23:02:24 +02:00
Generate more test messages to avoid test failures Sometimes I get the error "Selected message id not in MessageList" when running the casper tests. I think it's probably when the test user's home view does not contain any messages. Ideally we would fix this in a way that guarantees that we generate whatever messages the test suite needs... (imported from commit 51a02da612dda88d60681b9e09cd6e6a2c39a470)
2015-08-20 23:47:05 +02:00
# This next line can be simplified to "-n0" once we fix our app (and tests) with 0 messages.
Always start python via shebang lines. This is preparation for supporting using Python 3 in production. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-22 01:44:16 +01:00
./manage.py populate_db --settings=zproject.test_settings --test-suite -n30 \
Make the Django-based test suite much faster. So, I got annoyed that our test suite was taking forever to run: real 2m13.443s user 1m32.630s sys 0m3.748s Some quick profiling determined that the test suite is spending all of its time loading the fixtures files (zephyr/fixtures/messages.json) that it loads for each test case (3s to load that for each test case). To improve this situation, I cut out from the test database used by the test suite most of the users, subscriptions, etc. that aren't being used directly by the test cases. The impact is a quite significant speedup: real 0m15.176s user 0m9.161s sys 0m0.508s We're still spending over a quarter of a second per test, which isn't great -- but this is at least no longer unbearable. This commit doesn't make any changes to the populate_db output if you don't pass the new --test-suite option. (imported from commit 2334ba5399b33edab3d29ff269fde4ea77ccd48e)
2013-01-08 21:59:52 +01:00
--threads=1 --huddles=0 --personals=0 --percent-huddles=0 --percent-personals=0
Always start python via shebang lines. This is preparation for supporting using Python 3 in production. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-22 01:44:16 +01:00
./manage.py dumpdata --settings=zproject.test_settings \
[manual] Rename Django app from zephyr to zerver. This needs to be deployed to both staging and prod at the same off-peak time (and the schema migration run). At the time it is deployed, we need to make a few changes directly in the database: (1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr'; (2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr'; (imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-07-29 23:03:31 +02:00
zerver.UserProfile zerver.Stream zerver.Recipient \
zerver.Subscription zerver.Message zerver.Huddle zerver.Realm \
zerver.UserMessage zerver.Client \
zerver.DefaultStream > zerver/fixtures/messages.json
tests: Don't regenerate the test database every time we run tests This cuts about 6 seconds off of the test startup time on my laptop. The other startup costs are about 1 second for the server to come up and about half a second for casperjs to begin executing tests. generate-fixtures now takes a '--force' option that can be use to force a regeneration of the test database. (imported from commit 1f473507502f0edf159b2638abb392d9357eb46f)
2013-03-05 23:08:55 +01:00
[manual] Humbug testing databases now backed by postgres Currently our test database is backed by sqlite; this commit moves us to using postgres for our all database needs. This, in conjunction with the patched django on github, allow us to have fewer hacks and more true-to-life tests. It also sets the stage for testing the bulk_create and schema search_path patches made to django. Developers will need to run: ./tools/postgres-init-test-db ./tools/do-destroy-rebuild-test-database this is assuming that they have already run: ./tools/postgres-init-db ./tools/do-destroy-rebuild-database at some point on this pg_cluster. (The ordering is important; it will other- wise complain about the south_migration table). (imported from commit c56c6f27e13df7ae10b2e643e65d669dde61af3d)
2013-05-08 05:30:42 +02:00
# create pristine template database, for fast fixture restoration after tests are run.
[manual] Change references to the humbug_test user, schema, and database to zulip_test This requires no changes in production, but is tagged as manual to remind developers that they need to edit and run the tools/migrate-db script to fix up their local database instances. (imported from commit fbf764fb61592ef994d6d2ad56edad65ff01f14b)
2013-10-26 04:24:40 +02:00
psql -h localhost postgres zulip_test << EOF
generate-fixtures: Be more robust to databases being missing (imported from commit f42d673c433b8d3df4af79159831e57f5db1ee1c)
2014-01-30 23:40:07 +01:00
DROP DATABASE IF EXISTS zulip_test_template;
[manual] Change references to the humbug_test user, schema, and database to zulip_test This requires no changes in production, but is tagged as manual to remind developers that they need to edit and run the tools/migrate-db script to fix up their local database instances. (imported from commit fbf764fb61592ef994d6d2ad56edad65ff01f14b)
2013-10-26 04:24:40 +02:00
CREATE DATABASE zulip_test_template TEMPLATE zulip_test;
[manual] Humbug testing databases now backed by postgres Currently our test database is backed by sqlite; this commit moves us to using postgres for our all database needs. This, in conjunction with the patched django on github, allow us to have fewer hacks and more true-to-life tests. It also sets the stage for testing the bulk_create and schema search_path patches made to django. Developers will need to run: ./tools/postgres-init-test-db ./tools/do-destroy-rebuild-test-database this is assuming that they have already run: ./tools/postgres-init-db ./tools/do-destroy-rebuild-database at some point on this pg_cluster. (The ordering is important; it will other- wise complain about the south_migration table). (imported from commit c56c6f27e13df7ae10b2e643e65d669dde61af3d)
2013-05-08 05:30:42 +02:00
EOF