Commit Graph

10054 Commits

Author SHA1 Message Date
Zev Benjamin 4887a79d21 Make the nginx log directory owned by zulip
This is required to make log2zulip not error out when reading the nginx
error.log.
2016-02-02 23:05:44 -08:00
Zev Benjamin e780f5dab5 Make log2zulip error message more accurate.
In particular, in the case of a permissions issue reading the log
file, it would claim the log file doesn't exist.
2016-02-02 23:04:48 -08:00
Tim Abbott 206dc3aafc Add python 3 compatibility check for libmodernize.fixes.fix_dict_six.
It's not clear whether this will end up being net negative in value in
the long term since it's kinda hard to understand the output, but in
the short term it should prevent regressions.
2016-01-26 21:11:25 -08:00
Tim Abbott 5bacda3662 python3: Fix usage of .keys()/.values() to handle iterators.
This fixes the places where we use the result of .keys(), .items(),
and .values() that wouldn't work with an iterator to wrap them with
list().
2016-01-26 21:11:25 -08:00
Tim Abbott f5de149976 python3: specify explicit sorting algorithm for subscriptions. 2016-01-26 21:11:25 -08:00
Tim Abbott 05a827c520 need_to_render_content: Fix comparison with None.
If the content wasn't rendered, both rendered_content and
rendered_content_version would be None.  In addition to being
confusing, in Python 3, `None < 2` is an error and this code breaks.
2016-01-26 21:11:25 -08:00
Tim Abbott bd0918cd5a python3: Use zip from the six.moves package. 2016-01-26 21:11:24 -08:00
Tim Abbott 757e89260e Migrate use of StringIO to Python 2+3 compatible six.moves.cStringIO.
And add a check for additional usage of the original StringIO module.
2016-01-26 21:09:43 -08:00
Tim Abbott 1f44417fc1 Switch to using Python 3 style division everywhere.
Also add testing for this to our Python 3 compatibility test suite.
2016-01-26 21:09:43 -08:00
Tim Abbott 6528b18ad3 Switch all urllib/urlparse usage to six.moves.urllib.
This provides Python 2+3 compatibility for our use of urllib.

Also add a test to avoid future regressions.
2016-01-26 21:09:43 -08:00
Tim Abbott 52f9574047 Fix missing python-six dependency for Zulip API. 2016-01-26 21:09:42 -08:00
Tim Abbott 700055c194 Apply modernize transform libmodernize.fixes.fix_file.
This replaces use of file() with open() which is python 3 compatible,
and also adds it to our python 3 support test suite.
2016-01-26 21:09:42 -08:00
Tim Abbott 83dd51dcd6 Remove now-obsolete get_message_by_id_dbwarn transition code. 2016-01-26 21:05:11 -08:00
Tim Abbott eecd1513b3 Don't access/store full Client objects in Tornado queue servers.
At present, we only do a few simple checks on the client type inside
the event system, and this saves database/memcached queries.

Note that this preserves the structure of the marshalled name in
to_dict/from_dict as client_type to avoid an unnecessary migration.
2016-01-26 21:04:32 -08:00
Tim Abbott e3b6bfa3ca Remove code for pickle event queue store.
Since we've had the JSON store in all environments for some time, we
no longer need this legacy code.
2016-01-26 20:59:34 -08:00
Tim Abbott f6073d1708 Move HTTPResponse construction out of event_queue.py. 2016-01-26 20:59:19 -08:00
Tim Abbott a9bf4b4cc7 Store the client name for the current handler in ClientDescriptor. 2016-01-26 20:58:01 -08:00
Tim Abbott c7e3c3ce38 Look up client descriptors by handler_id.
Previously, client descriptors were referenced directly from the
handler object.  Once we split the Tornado process into separate queue
and connection servers, these will no longer be in the same process,
so we need to reference them by ID instead.
2016-01-26 20:57:25 -08:00
Tim Abbott ea6211c041 Allocate handler ids in AsyncDjangoHandler setup process. 2016-01-26 20:56:53 -08:00
Tim Abbott ae760a351e Move most of ClientDescriptor.finish_current_handler into library. 2016-01-26 20:56:08 -08:00
Tim Abbott 7df61fccbd Add user_profile_email field to ClientDescriptor. 2016-01-26 20:55:59 -08:00
Tim Abbott 2ea0daab19 Track Tornado handlers by uniquely assigned IDs rather than objects.
This is early preparation for splitting apart Tornado into a queue
server and a frontend server.
2016-01-26 20:55:41 -08:00
Tim Abbott 8b42fdd0d7 Move get_events logic into a backend function in event_queue.py.
This commit is somewhat ugly, but its purpose is to be early
preparation for splitting Tornado into a queue server and a frontend
server, and this code belongs, by and large, in the queue server
component.
2016-01-26 20:51:54 -08:00
Tim Abbott 5a6154c8ba test_helpers: Fetch streams from cache in subscribe_to_stream. 2016-01-26 20:41:01 -08:00
Tim Abbott a5d4d0aae0 test_backend: Add option to profile the backend test suite. 2016-01-26 20:41:01 -08:00
Tim Abbott f9791558e9 test_runner: Support continuing running tests after a failure. 2016-01-26 20:41:01 -08:00
Tim Abbott b43aadad8b test-backend: Rewrite in python to support computing test coverage.
The code for doing test coverage is just a lot cleaner this way over
adding it to the shell script version.

Based on the basic test runner code here:
https://docs.djangoproject.com/en/1.9/topics/testing/advanced/
2016-01-26 20:41:01 -08:00
Tim Abbott 24fd3bbf55 travis: Test whether migrations are consistent with models.
This should automatically catch mistakes where someone updates the
database models but forgets to generate migrations afterwards.
2016-01-26 20:38:46 -08:00
Tim Abbott 5ef57a07e1 Add missing migrations present in models.py.
89a2765553 didn't include the database
migration corresponding to the change, which means it didn't take full
effect when it was merged.

I noticed this because `manage.py makemigrations` would generate these
migrations; that suggests a good idea for a test to add.
2016-01-26 20:38:46 -08:00
Tim Abbott 1c73c992dd Fix missing puppet dependencies on postgres package. 2016-01-26 20:32:33 -08:00
Tim Abbott 2e16b44b24 puppet: Use $postgres_version in postgres template.
This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.
2016-01-26 20:32:33 -08:00
Tim Abbott 806aa986b7 puppet: Use a variable to configure the postgres version. 2016-01-26 20:32:33 -08:00
Tim Abbott a3ac56efe2 puppet: Make apt repository conditional on the Ubuntu version.
We still will need to address this in the install script as well.
2016-01-26 20:32:33 -08:00
Tim Abbott f6c59feb05 Document the puppet configuration somewhat in zulip::voyager. 2016-01-26 20:32:33 -08:00
Tim Abbott 345b5254d7 puppet: Move default nginx configuration out of voyager.pp. 2016-01-26 20:32:33 -08:00
Tim Abbott dd61e3f97d puppet: Move memcached and rabbitmq include out of app_frontend_base.pp. 2016-01-26 20:32:33 -08:00
Tim Abbott c3153274c1 puppet: Move memcached into its own puppet module. 2016-01-26 20:32:33 -08:00
Tim Abbott 8a0e07fe1a puppet: Rename app_frontend.pp to app_frontend_base.pp.
This will enable us to move the remaining app-frontend related content
out of voyager.pp.
2016-01-26 20:32:33 -08:00
Tim Abbott 91286d00aa puppet: Move Zulip apt repository to its own manifest. 2016-01-26 20:32:33 -08:00
Tim Abbott 69dd17dfb6 puppet: Move prod-static creation from voyager.pp to app_frontend.pp.
Every app frontend will need this directory and this should help
enable more modular puppet rules.
2016-01-26 20:32:33 -08:00
Tim Abbott 702f501638 puppet: Move tuned postgres configuration out of voyager.pp.
This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.
2016-01-26 20:32:28 -08:00
Tim Abbott d5f04bd20b Rename zulip::postgres_appdb to zulip::postgres_appdb_base.
The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.
2016-01-26 20:30:12 -08:00
Tim Abbott 3f27573cb2 puppet: Move several debugging tool dependencies out of base.pp. 2016-01-26 20:30:11 -08:00
Tim Abbott fdc7f5b86a Vagrantfile: Default to using LXC when Virtualbox is also available.
This solves the problem reported in #331 with needing to specify
--provider=lxc to use the LXC provider in an Ubuntu Linux environment;
additionally, it adds the LXC option needed to run LXC on Ubuntu
15.10, but not on 14.04 where that option is unavailable and would
totally break LXC.
2016-01-23 11:45:47 -08:00
Tim Abbott 50bc32dc95 integrations: Document server-side setup for twitter integration. 2016-01-23 11:44:53 -08:00
Tim Abbott c6d06b0c4e Add zulip distribution tarballs to backups. 2016-01-23 11:44:15 -08:00
Tim Abbott 529d7a2877 Release API version 0.2.5. 2016-01-23 11:38:47 -08:00
Tim Abbott d3588cb7d0 api: Include get_subscribers endpoint in public release.
It's possible we should just eliminate this mechanism, but this fixes
a proximal problem where the multi-line get_subscribers endpoint
description was being handled wrong.
2016-01-23 11:38:42 -08:00
Tim Abbott fdf708039b Disable empty-stream notifications for email gateway bot. 2016-01-23 11:37:52 -08:00
Vladislav Manchev df4d1b3c14 Add linting code for detecting shebang bashisms.
This will prevent regressions in OpenBSD compatibility, since OpenBSD
doesn't support passing arguments in the #! line.
2016-01-21 22:33:55 -08:00