Commit Graph

207 Commits

Author SHA1 Message Date
Tim Abbott 06b33da709 process_queue: Fix missing worker.setup() in single-threaded codepath. 2016-03-27 23:17:16 -07:00
Tim Abbott cd2348e9ae Run queue processers multithreaded in development.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.

On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.".  Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.

Progress towards #32.
2016-03-20 18:04:24 -07:00
Tim Abbott 06e68d52ce Remove dead AsyncDjangoHandler field. 2016-03-20 16:53:13 -07:00
Tim Abbott 7fabfe9cb9 Add __repr__s for ClientDescriptor and AsyncDjangoHandler. 2016-03-20 16:53:13 -07:00
Tim Abbott 3f55e26a9f Fix Tornado memory leak with synchronously handled requests.
The new Tornado handler tracking logic properly handled requests that
threw an exception or followed the RespondAsynchronously code path,
but did not properly de-allocated the handler in the syncronous case.

An easy reproducer for this is to load a new Zulip browser window;
that will leak 2 handler objects for the 2 synchronous requests made
from Django to Tornado as part of initial state fetching.
2016-03-20 16:53:13 -07:00
Tim Abbott 320428052a Fix AsyncDjangoHandler view exception code path.
The recent Tornado memory leak fix
(1396eb7022) didn't use the correct
variable name for the current handler ID, causing this cleanup code to
fail in the event that a view raised an exception.
2016-03-19 22:50:35 -07:00
Eklavya Sharma 9e3c3e14f5 Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
2016-03-19 15:52:58 -07:00
Tim Abbott 1396eb7022 Fix Tornado memory leak of handler objects.
In 2ea0daab19, handlers were moved to
being tracked via the handlers_by_id dict, but nothing cleared this
dict, resulting in every handler object being leaked.  Since a Tornado
process uses a different handler object for every request, this
resulted in a significant memory leak.  We fix this by clearing the
handlers_by_id dict in the two code paths that would result in a
Tornado handler being de-allocated: the exception codepath and the
handler disconnect codepath.

Fixes #463.
2016-03-17 18:33:59 -07:00
Reid Barton 9735025167 Refactor logic around restricted_to_domain.
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.

This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
2016-02-07 10:54:52 -05:00
Tim Abbott 1af7cbfd64 runtornado: Move more imports to the top of the file.
This is needed for adding more specific type annotations but is
otherwise counterproductive since it increases the diff from the
original.
2016-02-03 19:47:14 -08:00
Tim Abbott 2259ce62f8 tornado: Fix AsyncDjangoHandler get() and friends missing args/kwargs. 2016-02-03 19:47:14 -08:00
Tim Abbott d7b7ae2d0f rundjango: Fix confusing return line in log_message_monkey. 2016-02-03 19:31:45 -08:00
Tim Abbott 05a40f11b3 runtornado: Add explicit return None. 2016-02-03 19:31:45 -08:00
Tim Abbott 37e987e250 Make realm_filters --show's empty list output more consistent. 2016-02-03 19:29:44 -08:00
Tim Abbott fcc32b1093 Fix check_redis abuse of setting self.trim global. 2016-02-03 19:29:44 -08:00
Tim Abbott 7595e4b05f process_queue: Fix worker variable being accessed before initialization. 2016-02-03 19:29:44 -08:00
Tim Abbott aad3bff193 Harden style rule for % comprehensions and fix existing errors. 2016-02-02 23:08:19 -08:00
Tim Abbott 6528b18ad3 Switch all urllib/urlparse usage to six.moves.urllib.
This provides Python 2+3 compatibility for our use of urllib.

Also add a test to avoid future regressions.
2016-01-26 21:09:43 -08:00
Tim Abbott c7e3c3ce38 Look up client descriptors by handler_id.
Previously, client descriptors were referenced directly from the
handler object.  Once we split the Tornado process into separate queue
and connection servers, these will no longer be in the same process,
so we need to reference them by ID instead.
2016-01-26 20:57:25 -08:00
Tim Abbott ea6211c041 Allocate handler ids in AsyncDjangoHandler setup process. 2016-01-26 20:56:53 -08:00
Tim Abbott dfaf45b2b6 Wrap Django runserver to prevent spammy logging.
Django's `manage.py runserver` prints a relatively low-information log
line for every request of the form:

[14/Dec/2015 00:43:06]"GET /static/js/message_list.js HTTP/1.0" 200 21969

This is pretty spammy, especially given that we already have our own
middleware printing a more detailed version of the same log lines:

2015-12-14 00:43:06,935 INFO     127.0.0.1       GET     200   0ms /static/js/message_list.js (unauth via ?)

Since runserver doesn't have support controlling whether these log
lines are printed, we wrap it with a small bit of code that silences
the log lines for 200/304 requests (aka the uninteresting ones).
2015-12-25 16:23:57 -08:00
Tim Abbott a79e89b28f Cleanup remaining usage of % comprehensions without explicit tuples. 2015-12-05 15:29:42 -08:00
Tim Abbott 3e794351ab Remove expunge_* management commands.
These management commands never worked and given that we removed the
retention_policy code that they call, it makes sense to remove them as
well.
2015-11-11 21:34:39 -08:00
Walter Heck f145b01d91 Update create_user management command.
notify_new_user was recently moved to zerver.lib.actions from
zerver.views and this wasn't properly updated. This would give an
error when doing a `manage.py create_user` from the command line.
2015-11-08 19:41:00 -08:00
Tim Abbott 3b185ad4de Apply Python 3 futurize transform libmodernize.fixes.fix_input_six. 2015-11-01 09:26:17 -08:00
Tim Abbott f3783fb4a1 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import. 2015-11-01 09:26:16 -08:00
Tim Abbott 43abd83d1c Apply Python 3 futurize transform lib2to3.fixes.fix_ws_comma. 2015-11-01 09:26:14 -08:00
Tim Abbott 06f6ee6566 Apply Python 3 futurize transform lib2to3.fixes.fix_idioms. 2015-11-01 09:25:47 -08:00
Tim Abbott 5ce6a3c8f9 Apply Python 3 futurize transform lib2to3.fixes.fix_funcattrs. 2015-11-01 08:09:54 -08:00
Tim Abbott 8c34c40924 Apply Python 3 futurize transform lib2to3.fixes.fix_except. 2015-11-01 08:08:33 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Tim Abbott 71a06d58de Convert uses of Realm.objects.get() to get_realm().
get_realm is better in two key ways:
* It uses memcached to fetch the data from the cache and thus is faster.
* It does a case-insensitive query and thus is more safe.
2015-10-15 09:16:58 -04:00
Tim Abbott 419d31a007 Expand documentation for the LDAP auth integration.
Fixes #134, #173.
2015-10-15 09:16:58 -04:00
Tim Abbott 858d0a984b Move API super users configuration into the database.
(imported from commit 3cc702f93e7252b42930dba4bde93a915b6dbf44)
2015-09-20 11:13:20 -07:00
Tim Abbott 3b84a9a010 Modernize management command for deactivating a realm.
(imported from commit 9ef681193ff04e447c0afbba66187357624d98f3)
2015-09-19 19:10:27 -07:00
Tim Abbott a5c1fc6d76 Remove unnecessary import_realm_filters.py management command.
It's superceded by realm_filters.py.

(imported from commit b53c8bc4daf965387f5ba88711dd7ece2db1dfd9)
2015-09-19 19:05:16 -07:00
Tim Abbott 930c64df8a Move zulip.com-related statsd configuration out of main settings.py.
This also removes the convenient way to run statsd in the Dev VM,
because we don't anticipate anyone doing that.  It's just 2 lines of
config to configure it anyway:

STATSD_HOST = 'localhost'
STATSD_PREFIX = 'user'

(imported from commit 5b09422ee0e956bc7f336dd1e575634380b8bfa2)
2015-08-22 13:49:43 -07:00
Tim Abbott 83a653efce Change semantics of ZULIP_COM to include ZULIP_COM_STAGING machines.
(imported from commit 2e84c756b4ac41c13d619ab6ceb99e347d1f48cb)
2015-08-21 10:41:14 -07:00
Tim Abbott a8261dd6ac Remove Deployment initialization from Development.
(imported from commit 4596fcf480c10c88496cd6506edb0a6e702d823c)
2015-08-21 10:40:57 -07:00
Tim Abbott 16e0f21687 Fix a bunch of Zulip Voyager related comments.
(imported from commit 176e875cbc310ba00d6137930ddb718bd68f1866)
2015-08-21 10:40:54 -07:00
David Roe e3f38acbce Enterprise => Voyager.
(imported from commit 41b9a67301aeaf5fd40bbbb8f34a326ca98431fd)
2015-08-21 10:33:35 -07:00
David Roe 3f7cb34b00 enterprise => voyager
(imported from commit 04be792bb480d5e5db1c91d296d1000cf1682571)
2015-08-21 10:33:35 -07:00
David Roe edf7e732a2 ENTERPRISE => VOYAGER.
(imported from commit 4f8080b9f506a87ca40bef32e39de5218cba916a)
2015-08-21 10:33:35 -07:00
David Roe d702ddb3cf DEPLOYED => PRODUCTION
(imported from commit a7d365c07916c5103fc721c712db94f9af977640)
2015-08-21 10:33:35 -07:00
Reid Barton ae0ae3dde8 Django 1.8: declare positional arguments in management commands
(imported from commit d9efca1376de92c8187d25f546c79fece8d2d8c6)
2015-08-20 23:35:40 -07:00
Reid Barton 9ded218950 Django 1.7 compatibility: miscellaneous changes
(imported from commit 1f63d936c3d195acd033bb42fddacd240af3889b)
2015-08-20 23:01:26 -07:00
Nemanja Stanarevic d9f111ec28 Added import_dump command to manage.py to import JSON database dumps
(imported from commit abd410e5299f15c1a5851e87d18664b59837fbb0)
2015-08-20 21:19:57 -07:00
David Roe 809efc4f2b Add realm option to show or not show digest e-mail.
(imported from commit e9cfe519aa9f2857a1dcc7c75f5e1889834fcb86)
2015-08-20 15:29:45 -07:00
Tim Abbott 4f1388706c Disable digests and Missed Message notifications for internal realm.
(imported from commit 24a25bcf6653c674ac8a34324a4674e438f0af38)
2015-02-05 18:40:13 -08:00
Jason Michalski 2a5ec20295 Add support for one time use email address
The one time use address are a unique token which maps to stored stated
in redis. We store the user_id, recipient_id, and subject. When an email
is received at this address it is sent to the stored recipient by the
stored user. Anyone with this address can send a single message as this
user.

(imported from commit 4219417bdc30c033a6cf7a0c7c0939f7d0308144)
2014-10-06 15:27:54 -07:00