Commit Graph

819 Commits

Author SHA1 Message Date
Tim Abbott 6528b18ad3 Switch all urllib/urlparse usage to six.moves.urllib.
This provides Python 2+3 compatibility for our use of urllib.

Also add a test to avoid future regressions.
2016-01-26 21:09:43 -08:00
Tim Abbott 700055c194 Apply modernize transform libmodernize.fixes.fix_file.
This replaces use of file() with open() which is python 3 compatible,
and also adds it to our python 3 support test suite.
2016-01-26 21:09:42 -08:00
Tim Abbott a5d4d0aae0 test_backend: Add option to profile the backend test suite. 2016-01-26 20:41:01 -08:00
Tim Abbott f9791558e9 test_runner: Support continuing running tests after a failure. 2016-01-26 20:41:01 -08:00
Tim Abbott b43aadad8b test-backend: Rewrite in python to support computing test coverage.
The code for doing test coverage is just a lot cleaner this way over
adding it to the shell script version.

Based on the basic test runner code here:
https://docs.djangoproject.com/en/1.9/topics/testing/advanced/
2016-01-26 20:41:01 -08:00
Tim Abbott 24fd3bbf55 travis: Test whether migrations are consistent with models.
This should automatically catch mistakes where someone updates the
database models but forgets to generate migrations afterwards.
2016-01-26 20:38:46 -08:00
Tim Abbott c6d06b0c4e Add zulip distribution tarballs to backups. 2016-01-23 11:44:15 -08:00
Tim Abbott d3588cb7d0 api: Include get_subscribers endpoint in public release.
It's possible we should just eliminate this mechanism, but this fixes
a proximal problem where the multi-line get_subscribers endpoint
description was being handled wrong.
2016-01-23 11:38:42 -08:00
Vladislav Manchev df4d1b3c14 Add linting code for detecting shebang bashisms.
This will prevent regressions in OpenBSD compatibility, since OpenBSD
doesn't support passing arguments in the #! line.
2016-01-21 22:33:55 -08:00
Vladislav Manchev dfbea01c8f Add support for running OpenBSD in development environment. 2016-01-21 22:33:55 -08:00
Tim Abbott 6943a142ea Fix postgres errors in Travis CI again.
Travis CI's model of installing every version of postgres on the test
VM and then shutting all the versions other than the one requested
down seems to not work very well with doing apt upgrades.  It seems
the best way to resolve this is to just uninstall the versions we
don't need.
2016-01-21 22:07:10 -08:00
Tim Abbott 26e9d55e16 deployments: Refactor locking libraries into zulip_tools.py.
The code in update-deployment and upgrade-zulip for managing the
deployment lock was nearly identical.
2016-01-11 21:36:42 -08:00
Tim Abbott 186f563176 Fix deployment locks being leaked when a deployment fails.
The point of the lock is to prevent two deployments happening at the
same time and racing with each other, not to prevent doing any future
deployments after an error happens (which is what the current
implementation does in practice).

Addresses part of #208.
2016-01-11 21:36:41 -08:00
Tim Abbott a98b0cf35d travis: Workaround postgres 9.1 conflict issues on trusty.
We ran into a bug with the Travis CI infrastructure where it postgres
9.1 is installed on the system, and so when we'd do an apt upgrade
with a new version of 9.1, the 9.1 daemon would end up getting started
and conflict with the 9.3 daemon we were trying to run.
2016-01-09 16:59:43 -08:00
Tim Abbott 24ebc10ec8 Travis: Display all errors, not just first one, in py3k testing. 2016-01-09 15:43:26 -08:00
Josh Mandel bdb9535251 Don't require link tags to close when checking templates. 2016-01-07 23:08:49 -08:00
Tim Abbott c5f08022f9 tools/clean-repo: Don't print out the list of .pyc files deleted.
This was never useful output and was pretty spammy.
2015-12-25 16:23:57 -08:00
Tim Abbott dfaf45b2b6 Wrap Django runserver to prevent spammy logging.
Django's `manage.py runserver` prints a relatively low-information log
line for every request of the form:

[14/Dec/2015 00:43:06]"GET /static/js/message_list.js HTTP/1.0" 200 21969

This is pretty spammy, especially given that we already have our own
middleware printing a more detailed version of the same log lines:

2015-12-14 00:43:06,935 INFO     127.0.0.1       GET     200   0ms /static/js/message_list.js (unauth via ?)

Since runserver doesn't have support controlling whether these log
lines are printed, we wrap it with a small bit of code that silences
the log lines for 200/304 requests (aka the uninteresting ones).
2015-12-25 16:23:57 -08:00
Tim Abbott 39e80b351d Add simple tools for fetching content of a pull request by ID. 2015-12-25 16:22:00 -08:00
Allie Jones 1bd1291f3c Replace json/send_message endpoint usage with json/message. 2015-12-12 18:14:08 -08:00
Kara McNair c947f3ed3c Add Zulip-specific details in jslint used_before_a error message.
Previously, it wasn't clear why you were getting an error when trying
to reference a newly added global variable.
2015-12-07 20:33:36 -08:00
Tim Abbott 2be7ac8d70 travis: Fix prompting for user input in production-helper. 2015-12-07 20:33:36 -08:00
Tim Abbott 0fe819eb57 lint: Tighten lint rules around whitespace around '%' comprehensions. 2015-12-05 15:29:42 -08:00
Tim Abbott 69b6b60017 lint: Clean up whitespace_rules.
Several of these rules only apply to one of Python and Javascript, and
this simplifies the logic and should make our linter code more readable.

In the process, we add support for per-rule/file pair exclusions to
handle the tab exception for codehilite.py.
2015-12-05 15:29:42 -08:00
Tim Abbott a712954c59 lint: Rewrite custom checks to use a more consistent framework. 2015-12-05 15:29:42 -08:00
Tim Abbott 8a18e78a65 Add lint rules checking for our % comprehension style. 2015-12-05 15:29:42 -08:00
Tim Abbott a79e89b28f Cleanup remaining usage of % comprehensions without explicit tuples. 2015-12-05 15:29:42 -08:00
Tim Abbott 7395003e6a Fix buggy #! lines using "/usr/bin/env python2.7 -u".
The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.

We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.

Thanks to Dan Fedele for the bug report and suggested solution!
2015-11-16 08:46:48 -08:00
Tim Abbott 9d75fd33d9 Add new test for management commands running with --help.
This test caught a few bugs where refactoring had made management
commands fail (and would have caught a few more recent ones).

Ideally we'd replace this with a more advanced test that actually
tests that the management command do something useful, but it's a
start.
2015-11-11 21:34:39 -08:00
Tim Abbott b81ecc4064 Add whitespace lint rules checking for missing spaces before {. 2015-11-10 10:06:47 -08:00
Allie Jones 85809e6140 Add webpack build process. 2015-11-06 09:13:25 -08:00
Allie Jones 4de0325a9d Install node dependencies using npm.
The node packages 'jQuery' and 'jquery' are different--'jQuery' is the
legacy support package that is needed for Zulip so the require statements
in the tests were updated.

Travis uses node 4.0 by default and we are using 0.10, so the command to
install the correct version had to be added to the .travis.yml file.
2015-11-06 09:08:59 -08:00
Tim Abbott f6f8f1fe36 Use new-style classes consistently for Python 3 support.
Also add the fixer for this to our list of fixers we check.
2015-11-04 08:01:52 -08:00
Tim Abbott f52ffa7923 travis: Add Python 3 compatibility test.
This tests whether a new patch introduces any regressions related to
any of the Python 3 compatibility fixers we've run in the past, so
that we can make continuous forward progress on our path towards
Python 3 compatibility.

This produces error output that looks like this:
"""
Testing for additions of Python 2 patterns we've removed as part of moving towards Python 3 compatibility.

Running Python 3 compatibility test lib2to3.fixes.fix_apply
Running Python 3 compatibility test lib2to3.fixes.fix_except
diff --git a/zerver/views/__init__.py b/zerver/views/__init__.py
index b5c0102..2defd46 100644
--- a/zerver/views/__init__.py
+++ b/zerver/views/__init__.py
@@ -296,7 +296,7 @@ def accounts_register(request):
                 do_activate_user(user_profile)
                 do_change_password(user_profile, password)
                 do_change_full_name(user_profile, full_name)
-            except UserProfile.DoesNotExist, e:
+            except UserProfile.DoesNotExist as e:
                 user_profile = do_create_user(email, password, realm, full_name, short_name,
                                               prereg_user=prereg_user,
                                               newsletter_data={"IP": request.META['REMOTE_ADDR']})

Python 3 compatibility error(s) detected!  See diff above for what you need to change.
"""
2015-11-04 08:00:25 -08:00
Ahmed Shibani 47d8d784a2 Add 'blocktrans' to tools/check-templates.
Running check-templates test fails when there are 'blocktrans' tags in
django templates. The fix is to add 'blocktrans' to
is_django_block_tag function in check-templates.
2015-11-03 08:06:48 -08:00
Tim Abbott 6eb670097c Expand testing done via Travis CI to cover production pipeline.
With this change, we are now testing the production static asset
pipeline and installation process in a new testing job (and also run
the frontend/backend tests separately).

This means that changes that break the Zulip static asset pipeline or
production installation process are more likely to fail tests.  The
testing is imperfect in that it does not have proper isolation -- we
build a complete Zulip development environment and then install a
Zulip production environment on top of it, so e.g. any apt
dependencies installed for Zulip development will still be available
for the Zulip production environment.  But, it's better than nothing!

A good v2 of this would be to have the production setup process just
install the minimum stuff needed to run `build-release-tarball` and
then uninstall it / clean it up so that we can do a more clear
production installation, but that's more work.
2015-11-01 18:11:39 -08:00
Tim Abbott 3c31f9a2e3 Drop database users prior to DROP/CREATE database.
This fixes an annoying issue where one tries to rebuild the database,
and it fails due to there being existing connections.

The one thing that is potentially scary about this implementation is
that it means it's now a lot easier to accidentally drop your
production database by running the wrong script; might be worth adding
a "--force" flag controlling this behavior or something.

Thanks to Nemanja Stanarevic and Neeraj Wahi for prototypes of this
implementation!  They did most of the work and testing for this.
2015-11-01 18:11:39 -08:00
Tim Abbott f7878a61e1 Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six. 2015-11-01 09:35:06 -08:00
Tim Abbott b3ac668779 Apply Python 3 futurize transform libmodernize.fixes.fix_filter. 2015-11-01 09:26:16 -08:00
Tim Abbott f3783fb4a1 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import. 2015-11-01 09:26:16 -08:00
Tim Abbott 43abd83d1c Apply Python 3 futurize transform lib2to3.fixes.fix_ws_comma. 2015-11-01 09:26:14 -08:00
Tim Abbott 10657c1d53 Move node tests to node_tests/. 2015-10-28 10:11:47 -07:00
Tim Abbott 81f32f4aa1 casper: Rename frontend_tests/run to clarify it uses casper. 2015-10-28 10:11:47 -07:00
Tim Abbott 5aa89fd6af casper: Move common.js and test-credentials.js to casper-lib/. 2015-10-28 10:11:47 -07:00
Tim Abbott 01f0d362d9 Move test_credentials.js into casper_tests/. 2015-10-28 10:11:47 -07:00
Tim Abbott f1074aa491 Move frontend tests out of zerver/tests/.
This fixes an unfortunate bug where the backend tests in
zerver/tests.py were not being run automatically, and also makes these
a bit easier to find.
2015-10-28 10:11:47 -07:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Tim Abbott 758baca01a run-dev.py: Report a nice error if you run it as root.
Fixes #172.
2015-10-15 12:45:38 -04:00
Tim Abbott bf694fa832 Flush memcached whenever we drop the databases.
This fixes some issues that we've had where commands will fail is
confusing ways after the database is rebuilt because data from before
the database was dropped is still in the memcached cache.
2015-10-15 12:18:41 -04:00
Tim Abbott 3e735d36d1 Rename tools/postgres-init-db to tools/postgres-init-dev-db.
The previous name was confusing because we also have
scripts/setup/postgres-init-db.
2015-10-15 09:14:21 -04:00
Tim Abbott eac6ea75dd emoji_dump: Exit with nonzero status when there are failures.
Previously, emoji_dump would happily exit successfully even if it
wasn't able to generate all the emoji.
2015-10-14 21:48:13 -04:00
Tim Abbott 4309f92062 Pass worker errors through to the run-dev.py console. 2015-10-14 16:08:32 -04:00
Tim Abbott 9748780192 Remove unnecessary puppet.conf configuration.
Fixes #23.
2015-09-26 21:34:19 -07:00
Anders Kaseorg 0d12dfd06f Improve shell quoting hygiene
Most of these problems were found by ShellCheck
(http://www.shellcheck.net).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2015-09-25 23:25:08 -04:00
Leo Franchi 974a9bd0f3 Use -O option to wget when fetching zxcvbn
Otherwise the downloaded zxcvbn.js also includes garbage we don't want
2015-09-25 15:24:44 -07:00
Tim Abbott 9c0c5c57a7 Download zxcvbn.js rather than vendoring it.
(imported from commit 8d55b6b903dc2292852636dfb444219b643b5f64)
2015-09-25 02:24:07 -07:00
Anders Kaseorg 0ace7fe502 zanitizer: Remove -- argument that confuses some Git verisons
(imported from commit 42555ac7ca17c1568507ba665a8133a4db3375e6)
2015-09-25 00:46:00 -07:00
Tim Abbott cc33b68d73 Remove pre-built emoji images from version control.
Instead, build them automatically when provision the development
environment and in update-prod-static.

(imported from commit aac8dfeaafbe872c113e5f2b6bd8f655a1af36f2)
2015-09-25 00:22:15 -07:00
Tim Abbott d7618ff8dc Don't include zulip-dropbox.png in open source project.
(imported from commit 3b0068521818534ce34f039a3e16c418c448ed9c)
2015-09-24 22:40:59 -07:00
Anders Kaseorg c0bf02b4c2 zanitizer: Support renaming files
(imported from commit 6a094177d652a267b53aa5042110b39fcbf21b42)
2015-09-24 18:10:02 -07:00
Anders Kaseorg 49cf7abaed Tool for removing secrets from Git history
(imported from commit 1345658dc83eb9a98178d8c8ca8f37d19e281549)
2015-09-24 18:10:02 -07:00
Tim Abbott b6e2fd6e83 Rename build-voyager-tarball to build-release-tarball and document.
(imported from commit 06f102f4f4c5caf187f4930fadff06147f095401)
2015-09-22 21:22:11 -07:00
Tim Abbott 99fd2924cf Add nice logging output for build-voyager-tarball.
(imported from commit 6d11550aac6e8d6c123398b137987f93b31fbece)
2015-09-20 15:21:17 -07:00
Tim Abbott efa7a90ecc Install closure-compiler directly for use on the Dev VM.
(imported from commit 0dae11b46364026acf29e1fc953401162db68746)
2015-09-20 15:21:16 -07:00
Tim Abbott bfcae58cac Deprecate old Zulip internal server installation script.
(imported from commit 5d6aeaff6126aae1aa5bf610fc94cfdb0eda595e)
2015-09-20 00:39:53 -07:00
Tim Abbott 5e0b6b809a Clean up tools/ by deleting files and moving others to deprecated/
(imported from commit 57a8e6c3bd746e83e2d22e941a8321ead1396aef)
2015-09-20 00:18:48 -07:00
Tim Abbott 602077f75b We're no longer using the review script.
(imported from commit 9840ebdfdacee6e3cb32934b81502c70c133e2b0)
2015-09-20 00:10:54 -07:00
Tim Abbott 94d7ed006f inject-messages: Document and remove hardcoded API keys.
(imported from commit 9152ef85937b707157800da9efed068b151a438c)
2015-09-19 23:53:42 -07:00
Tim Abbott 232de3015a Add lint checks for tab-based whitespace.
We exclude the one file we have with issues that comes from a
third-party vendor.

(imported from commit 47e097b39440aea8e4d7468892176ffce4818b5f)
2015-09-19 23:23:02 -07:00
Tim Abbott b3dfacae19 Change API example usernames to use example.com domain.
(imported from commit 41b7be58735df0ea3d3a3daf23bc0f78606abc0d)
2015-09-19 18:12:02 -07:00
Tim Abbott 87d2d543a5 Remove some tools unlikely to be relevant externally.
(imported from commit 557f5590d87b3267a2e37e9be72a77815ca392d9)
2015-09-19 17:45:31 -07:00
Tim Abbott 155958871d Include Zulip financial projections tool in open source release.
It doesn't have any sensitive data since that lives in a separate
configuration file, and it's potentially useful.

(imported from commit 094e315439f8bd23ad07a8c2bc7d9776c8c7f096)
2015-09-19 17:45:31 -07:00
Reid Barton 3239ca6728 Update build-voyager-tarball local settings hack
The tarball build process runs in DEVELOPMENT mode, assuming it is run
on a dev VM (since then there is no /etc/zulip directory). Commit
d067bcfe9d71 made settings.py import local_settings_template.py in
DEVELOPMENT mode (then "not DEPLOYED"), not local_settings.py.

(imported from commit 9a08138d748dfca9c4ab8b366bee5c2fb96c25af)
2015-08-23 23:08:45 -07:00
Reid Barton 1f07cbed1e Remove tree forcefully when finished in build-voyager-tarball
Just importing zerver.lib.cache creates a file memcached_prefix that
is mode 0444, so we need to use -f or rm will prompt about whether to
remove it. Not sure why this is apparently a new issue.

(imported from commit 93c5140b66992339859e2b204c200d1dd7a35f2d)
2015-08-23 23:00:32 -07:00
Reid Barton 3cfebbd9c0 Fix peer authentication error for process_fts_updates in run-dev.py
(imported from commit 1c73877a1c437ec51d909034460c3c1b54178183)
2015-08-21 14:10:52 -07:00
David Roe 3f7cb34b00 enterprise => voyager
(imported from commit 04be792bb480d5e5db1c91d296d1000cf1682571)
2015-08-21 10:33:35 -07:00
David Roe edf7e732a2 ENTERPRISE => VOYAGER.
(imported from commit 4f8080b9f506a87ca40bef32e39de5218cba916a)
2015-08-21 10:33:35 -07:00
David Roe d702ddb3cf DEPLOYED => PRODUCTION
(imported from commit a7d365c07916c5103fc721c712db94f9af977640)
2015-08-21 10:33:35 -07:00
Reid Barton 362f9c6c5f Django 1.7: syncdb and migrate are now synonyms
(imported from commit b8a1b2476892bab2418240b04deea6aa40ff1b03)
2015-08-20 23:01:26 -07:00
Reid Barton 599742536b Switch from South to native Django migrations
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.

(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
2015-08-20 23:01:26 -07:00
David Roe 13f3b448e5 Kill Embed.ly integration.
(imported from commit dfe42a26e57100108c3c50298041bd520e0d8b1b)
2015-08-20 19:46:48 -07:00
Tim Abbott e79447131f Make build-enterprise-tarball run again.
(imported from commit e3dd8d75811458d0cee8a2d05a4b47abd6eda7fd)
2015-08-20 17:33:16 -07:00
Reid Barton 53c2b3f6f4 Clean up vagrant postgres user creation code
We don't need to check whether the user exists before creating it:
CREATE USER failing is fine.

(imported from commit e8b2bc5495e328ee30d15445a566c0edff2f069d)
2015-08-20 16:30:21 -07:00
Reid Barton 748f931999 Fix postgres users when re-provisioning
If we run provision.py a second time, there will already be
zulip/zulip_test users, so the CREATE USER will fail and the password
won't get updated to the newly generated value. By creating the user
and setting the password in two commands, we allow the creation to
fail without affecting whether the password is set.

Also the quoting for updating .pgpass was wrong.

(imported from commit 5e249813c17cb4829e4e4958e92aaa30563c5f96)
2015-08-20 16:30:21 -07:00
Anders Kaseorg dd8eda4edc run-dev: Run compile-handlebars-templates inside the pgrp
(imported from commit bb13f8804b7e9e930ecd2542ea8772c76370d32e)
2015-08-20 15:54:53 -07:00
Anders Kaseorg bd66b2139b run-dev: Eschew useless shells
(imported from commit 34dab15941f5c63f381f5b2b7b78482dbc2b02ea)
2015-08-20 15:54:53 -07:00
Reid Barton 664db6b622 Generate more test messages to avoid test failures
Sometimes I get the error "Selected message id not in MessageList"
when running the casper tests. I think it's probably when the test
user's home view does not contain any messages.

Ideally we would fix this in a way that guarantees that we generate
whatever messages the test suite needs...

(imported from commit 51a02da612dda88d60681b9e09cd6e6a2c39a470)
2015-08-20 15:43:43 -07:00
Reid Barton 5a747ab690 Make the CasperJS frontend tests work again
(At least when PhantomJS doesn't crash...)

(imported from commit 62936b12883e4c6bbd3701da80783b4bffe46025)
2015-08-20 13:33:07 -07:00
Cat Miller 0a20f168a7 Auto-generate dev-secrets file.
Source LOCAL_DATABASE_PASSWORD and INITIAL_PASSWORD_SALT from the secrets file.
Fix the creation of pgpass file.

Tim's note: This will definitely break the original purpose of the
tool but it should be pretty easy to add that back as an option.

(imported from commit 8ab31ea2b7cbc80a4ad2e843a2529313fad8f5cf)
2015-08-20 00:20:44 -07:00
Jonathan Dahan 795f191fb3 Remove git commit domain restrictions.
(imported from commit a756428d0b95db95fb093e78a182ad8873914f54)
2015-08-19 21:40:04 -07:00
Luke Faraone 1e7d7c0b71 Install a supervisor job for run-dev.
The development server will automatically start after provisioning finishes.

(imported from commit 6be3d954d9049acd0360189b07b1481756b69eef)
2015-08-19 21:07:23 -07:00
Luke Faraone 04d3eb33b1 run-dev.py: Reword --interface help text.
The old language was confusing because "the interface" could refer to something
like eth0, but in actuality refers to the IP/hostname to listen on.

(imported from commit 4f77d72a4dfcdbe7e7747c6228975aa68dfbe6ac)
2015-08-19 21:07:22 -07:00
Reid Barton 54adbb1737 Create vagrant postgres user and grant it rights to zulip{,_test}
(imported from commit bb57a3f5ef8bc48553f1eff95c47894aed4ef975)
2015-08-18 22:41:31 -07:00
Yoyo Zhou c25d9679a5 Utility for unescaping HTML entities leftover from MediaWiki->rST conversion
(imported from commit 2f2507f299c3d09c67bfd26d6c11fab47431d935)
2015-08-18 17:31:33 -07:00
Reid Barton 9c310c6cdd Update review email address
(imported from commit 2642b9fb91ab4ca0445063bccc0d46738adb64eb)
2015-08-17 15:24:33 -07:00
Luke Faraone 4702d8e9b3 Fix broken shebang in postgres-init-test-db
This was causing failures when execing outside of a shell

(imported from commit b63f8269d121b2bb8be5bcd84b3f868c9bedc9cd)
2015-08-16 21:36:36 -07:00
Luke Faraone 81a2a20bbf Disable sync_api_key if no zuliprc
(imported from commit f9e7f924c00230137ca45c204158b709cd311ebb)
2015-08-16 16:45:15 -07:00
Zev Benjamin 466a678c2c Remove "More messages below" indicator
It's been very buggy for a while, has limited usefulness compared with
unread counts, and profiling over the weekend indicates that it's very
slow.

(imported from commit 716fe47f2bbec1bd8a6e4d265ded5c64efe2ad5c)
2015-03-30 15:44:36 -07:00
Jason Michalski 53b02a694a Update the review script to use @dropbox.com
(imported from commit 30c8c516d0542c1319cb2faf5a9da8e9c8b26041)
2015-01-30 23:39:47 -08:00
Zev Benjamin 2c760ae735 Remove mixpanel
(imported from commit 9b6cc58ee9be483db8bf2d2eaaaecabc14f821e4)
2015-02-06 13:59:30 -08:00