Commit Graph

1893 Commits

Author SHA1 Message Date
Tim Abbott 59e2be2f5f Notify clients when an event queue is garbage collected. 2016-03-30 23:09:14 -07:00
Tim Abbott bf43db0dad Send responses from fetch_events in dict format. 2016-03-30 22:04:09 -07:00
Tim Abbott 485e46f136 Wrap most of fetch_events in a try/except JsonableError.
This is a no-op right now, but we'll want the new structure for the
next commit, and splitting this out makes it a lot easier to read what
is actually changed in the next commit.
2016-03-30 22:02:32 -07:00
Tim Abbott ad1494f8e0 Use dictionaries for passing data into event server subsystem.
This marshalls data in a portable format in preparation for splitting
the event queue server from the Tornado server.
2016-03-30 22:01:08 -07:00
Tim Abbott 6cd14af18f events: Migrate client_type => client_type_name in marshalling code. 2016-03-30 22:00:50 -07:00
Tim Abbott e9637a545f Rename zerver.handlers to zerver.logging_handlers for clarity. 2016-03-30 21:48:38 -07:00
Tim Abbott 06b33da709 process_queue: Fix missing worker.setup() in single-threaded codepath. 2016-03-27 23:17:16 -07:00
Tim Abbott cd2348e9ae Run queue processers multithreaded in development.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.

On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.".  Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.

Progress towards #32.
2016-03-20 18:04:24 -07:00
Tim Abbott 06e68d52ce Remove dead AsyncDjangoHandler field. 2016-03-20 16:53:13 -07:00
Tim Abbott 0419430000 Enhance Tornado logging with Handler stats.
This was useful data for debugging handler/memory leaks.
2016-03-20 16:53:13 -07:00
Tim Abbott e51811aa9e Add comment documenting logic flow of fetch_events. 2016-03-20 16:53:13 -07:00
Tim Abbott 7fabfe9cb9 Add __repr__s for ClientDescriptor and AsyncDjangoHandler. 2016-03-20 16:53:13 -07:00
Tim Abbott 0b96e5e43f Fix connection not being closed cleanly on event queue GC.
Apparently, our event queue garbage collection logic never actually
disconnected any existing handler objects.

We fix this by disconnecting the handler inside cleanup(), adding a
special check to avoid creating a pointless timeout object.
2016-03-20 16:53:13 -07:00
Tim Abbott 3f55e26a9f Fix Tornado memory leak with synchronously handled requests.
The new Tornado handler tracking logic properly handled requests that
threw an exception or followed the RespondAsynchronously code path,
but did not properly de-allocated the handler in the syncronous case.

An easy reproducer for this is to load a new Zulip browser window;
that will leak 2 handler objects for the 2 synchronous requests made
from Django to Tornado as part of initial state fetching.
2016-03-20 16:53:13 -07:00
Tim Abbott 12a5a3a6e1 Actually fix main Tornado memory leak of handler objects.
This line appears to have been lost in rebasing from the original
implementation of 1396eb7022faec4c2d91553800a35781a96dd5bd; so the
previous fix actually only addressed the issue in a rare exception
case.
2016-03-20 16:53:12 -07:00
Tim Abbott 320428052a Fix AsyncDjangoHandler view exception code path.
The recent Tornado memory leak fix
(1396eb7022) didn't use the correct
variable name for the current handler ID, causing this cleanup code to
fail in the event that a view raised an exception.
2016-03-19 22:50:35 -07:00
Eklavya Sharma 9e3c3e14f5 Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
2016-03-19 15:52:58 -07:00
Eklavya Sharma 176c507b0a Removed calls to ifilterfalse.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3.  Also changed some lists to sets
for faster lookup.

Refer to #256.
2016-03-19 15:46:31 -07:00
Eklavya Sharma 851b0a871d Fix python 3 decode error in zerver/tests.py.
Refer to #256.
2016-03-19 15:46:18 -07:00
Eklavya Sharma 186efc6a6d Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
2016-03-19 15:46:07 -07:00
Tim Abbott 1396eb7022 Fix Tornado memory leak of handler objects.
In 2ea0daab19, handlers were moved to
being tracked via the handlers_by_id dict, but nothing cleared this
dict, resulting in every handler object being leaked.  Since a Tornado
process uses a different handler object for every request, this
resulted in a significant memory leak.  We fix this by clearing the
handlers_by_id dict in the two code paths that would result in a
Tornado handler being de-allocated: the exception codepath and the
handler disconnect codepath.

Fixes #463.
2016-03-17 18:33:59 -07:00
Tomasz Kolek 3e3a224607 Moved pagerduty webhook into its own file pagerduty.py 2016-03-14 20:44:50 -07:00
Tomasz Kolek 05dce01cee Moved travis webhook into its own file travis.py 2016-03-14 20:44:45 -07:00
Tomasz Kolek f640470fa4 Moved zendesk webhook into its own file zendesk.py 2016-03-14 20:44:42 -07:00
Tomasz Kolek b3e5a256f5 Moved freshdesk webhook into its own file freshdesk.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 021c66fd9a Moved stash webhook into its own file stash.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 7a4c9d243f Moved deskdotcom webhook into its own file deskdotcom.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 087bd72814 Moved bitbucket webhook into its own file bitbucket.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 93b52f6f8e Moved newrelic webhook into its own file newrelic.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek a2b31da045 Moved pivotal webhook into its own file pivotal.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 5ade895936 Moved jira webhook into its own file jira.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek a0512244b3 Moved beanstalk webhook into its own file beanstalk.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 6a3ab0605d Moved github webhook into its own file github.py 2016-03-14 20:44:41 -07:00
Tomasz Kolek 8a0ed47751 moved webhooks to python package 2016-03-14 20:44:41 -07:00
Eklavya Sharma c59185e119 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import
Refer #256
2016-03-10 22:02:17 -08:00
Tim Abbott df0d2a726d python3: Add missing utf-8 encoding/decoding in various places. 2016-03-08 09:14:15 -08:00
Tim Abbott a46647a87a python3: Mark certain strings as unicode strings.
This is required in Python 3 since these strings are combined with
other unicode strings.
2016-03-08 09:14:11 -08:00
Tim Abbott cbf9b7605a Add test that all functions defined in urls.py actually exist.
This would have caught the create_user_backend issue introduced recently.
2016-02-07 19:21:53 -08:00
Reid Barton 6c6dc1d81d Allow create user API to create any user in an open realm. 2016-02-07 19:19:19 -08:00
Reid Barton 9735025167 Refactor logic around restricted_to_domain.
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.

This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
2016-02-07 10:54:52 -05:00
Reid Barton 0755b51c2e Move create_user_backend into zerver.views.users.
Commit aa33a0da moved users views into their own file, but missed this one.
2016-02-07 10:54:48 -05:00
Reid Barton 4e5f18407d Add comment in create_user_backend about not needing to invite users first. 2016-02-07 10:52:50 -05:00
Tim Abbott 1af7cbfd64 runtornado: Move more imports to the top of the file.
This is needed for adding more specific type annotations but is
otherwise counterproductive since it increases the diff from the
original.
2016-02-03 19:47:14 -08:00
Tim Abbott 2259ce62f8 tornado: Fix AsyncDjangoHandler get() and friends missing args/kwargs. 2016-02-03 19:47:14 -08:00
Tim Abbott 1d008576f2 QueueProcessingWorker: Add stub consume function and queue_name.
QueueProcessingWorker will now throw errors if there's a misconfigured
queue processor.
2016-02-03 19:47:14 -08:00
Tim Abbott 3469fd4bb2 Fix last remaining use of file() -> open(). 2016-02-03 19:31:46 -08:00
Tim Abbott d7b7ae2d0f rundjango: Fix confusing return line in log_message_monkey. 2016-02-03 19:31:45 -08:00
Tim Abbott 05a40f11b3 runtornado: Add explicit return None. 2016-02-03 19:31:45 -08:00
Tim Abbott 37e987e250 Make realm_filters --show's empty list output more consistent. 2016-02-03 19:29:44 -08:00
Tim Abbott 3475a5c1ed Fix test_runner.py fast_test_only return type. 2016-02-03 19:29:44 -08:00