Commit Graph

273 Commits

Author SHA1 Message Date
Steve Howell f5edeb01ae Calculate idle users more efficiently when sending messages.
Usually a small minority of users are eligible to receive missed
message emails or mobile notifications.

We now filter users first before hitting UserPresence to find idle
users.  We also simply check for the existence of recent activity
rather than borrowing the more complicated data structures that we
use for the buddy list.
2017-09-07 06:59:44 -07:00
Steve Howell 981f557422 Extract receiver_is_off_zulip().
We are splitting out this logic from the more complicated
UserPresence-related logic, so that we can simplify the latter.
2017-09-07 06:59:44 -07:00
Steve Howell 776bdc59db Avoid unnecessary steps in process_message_event().
There is no reason to compute receiver_is_idle() unless a user
is actually PM'ed or mentioned.
2017-09-07 06:59:44 -07:00
Greg Price b63e995e82 tornado: Fix a Redis-caused str/bytes bug in `status_inquiries`.
Because the Redis client returns exclusively bytes -- even for
hash keys -- even on Python 3, the test `'response' in status`
was always returning false, and the line that tries to decode
as JSON was never running, so we were passing `response`
through as a `bytes` object encoding some JSON.

I'm not sure what the impact of this bug was, and in particular
whether something downstream would have fudged it to make up for
this error.
2017-08-25 16:14:33 -07:00
Tim Abbott b8e7369dee mypy: Remove type: ignores not needed in Python 3. 2017-08-25 11:04:20 -07:00
Tim Abbott 80caa5c66f socket: Use get_user_profile_by_id.
This should have a slight performance benefit.
2017-08-24 23:16:42 -07:00
Tim Abbott c8f53e55db websocket_client: Use get_system_bot.
This messy code is used only for our websocket Nagios tests, so using
get_system_bot makes sense.
2017-08-24 23:16:18 -07:00
neiljp (Neil Pilgrim) b782db48e1 mypy: Remove superfluous older 'type: ignore' annotations. 2017-08-08 11:27:51 -07:00
neiljp (Neil Pilgrim) d9e36c8156 mypy: strict-optional fixes in zerver/tornado/socket.py. 2017-08-07 21:27:50 -07:00
Greg Price 0d2f7e8bcd errors: Eliminate redundant `json_response_to_error` calls. 2017-07-25 15:33:11 -07:00
Greg Price 709c3b50fc tornado: Use a machine-readable error code when an event queue is gone.
This fixes the original issue that #5598 was the root cause of; when
the user returns to a Zulip browser tab after they've been idle past
the timeout (10 min, per IDLE_EVENT_QUEUE_TIMEOUT_SECS), we now
correctly reload the page even if they're using Zulip in German or
another non-English language where we have a translation for the
relevant error message.
2017-07-24 16:41:22 -07:00
Greg Price 45b718a2a2 tornado: Use the new structured JsonableError API for /json/events errors. 2017-07-24 16:41:22 -07:00
Greg Price 124d623d05 tornado/socket: Replace an ad-hoc error class with JsonableError.
The one purpose this exception was serving was to carry a message
in `msg`.  We can do that with `JsonableError`, and as a bonus replace
a repetition of the familiar "'result': 'error', ..." JSON pattern
with a call to a common implementation.

Also wrap the error messages for translation -- we hadn't been doing
that, oops.  Our linter notices that issue now that it's the familiar
JsonableError class.

There's one other potential change in behavior here: this
except-clause might now catch a JsonableError raised from some other
code.  That seems like a bonus, if so; the handler isn't doing
anything actually specific to this code, and the more exceptions it
successfully turns into proper error responses to the client and lines
in the log, the better.
2017-07-24 16:41:22 -07:00
Greg Price 9faa44af60 JsonableError: Optionally carry error codes and structured data.
This provides the main infrastructure for fixing #5598.  From here,
it's a matter of on the one hand upgrading exception handlers -- the
many except-blocks in the codebase that look for JsonableError -- to
look beyond the string `msg` and pass on the machine-readable full
error information to their various downstream recipients, and on the
other hand adjusting places where we raise errors to take advantage
of this mechanism to give the errors structured details.

In an ideal future, I think all exception handlers that look (or
should look) for a JsonableError would use its contents in structured
form, never mentioning `msg`; but the majority of error sites might
continue to just instantiate JsonableError with a string message.  The
latter is the simplest thing to do, and probably most error types will
never have code looking for them specifically.

Because the new API refactors the `to_json_error_msg` method which was
designed for subclasses to override, update the 4 subclasses that did
so to take full advantage of the new API instead.
2017-07-24 16:41:22 -07:00
Greg Price ff5013c619 JsonableError: Add types, and eliminate duck-typing.
In order to benefit from the modern conveniences of type-checking,
add concrete, non-Any types to the interface for JsonableError.

Relatedly, there's no need at this point to duck-type things at
the places where we receive a JsonableError and try to use it.
Simplify those by using straightforward standard typing.
2017-07-24 16:41:22 -07:00
Umair Khan b4d49720b0 tornado: Close queue connection on reload.
Tornado reloads the app whenever there is a change in code. Due to this,
new connection is created to the client which also results in a new
channel. To avoid creating two channels for the queue in the RabbitMQ
broker we should close the old channel. Otherwise messages sent to the
queue will be distributed among these two channels in a round robin
scheme and we will end up losing one message since one of the channels
doesn't have an active consumer.

This commit closes the connection to the queue whenever Tornado reloads
the application using add_reload_hook().

Fixes #5824.
2017-07-21 11:19:26 -07:00
Umair Khan 758dbec9e2 tornado: Move setup_tornado_rabbitmq to application.py 2017-07-21 09:55:25 +05:00
Steve Howell 475eb21a5e Revert commits related to client_message_id.
I pushed a bunch of commits that attempted to introduce
the concept of `client_message_id` into our server, as
part of cleaning up our codepaths related to messages you
sent (both for the locally echoed case and for the host
case).

When we deployed this, we had some strange failures involving
double-echoed messages and issues advancing the pointer that appeared
related to #5779.  We didn't get to the bottom of exactly why the PR
caused havoc, but I decided there was a cleaner approach, anyway.
2017-07-14 12:13:35 -07:00
Steve Howell 8fbb55df85 Introduce client_message_id on the server.
We are deprecating local_id/local_message_id on the Python server.
Instead of the server knowing about the client's implementation of
local id, with the message id = 9999.01 scheme, we just send the
server an opaque id to send back to us.

This commit changes the name from local_id -> client_message_id,
but it doesn't change the actual values passed yet.

The goal for client_key in future commits will be to:
    * Have it for all messages, not just locally rendered messages
    * Not have it overlap with server-side message ids.

The history behind local_id having numbers like 9999.01 is that
they are actually interim message ids and the numerical value is
used for rendering the message list when we do client-side rendering.
2017-07-13 23:42:27 -04:00
Aditya Bansal 1e3b57570e pep8: Add compliance with rule E261 to tornado/websocket_client.py. 2017-07-11 11:55:02 -07:00
Aditya Bansal 94b790c28d pep8: Add compliance with rule E261 to tornado/socket.py. 2017-07-11 11:55:02 -07:00
Vaida Plankyte 9461f490c8 event_queue.py: Use the singular 'they' pronoun. 2017-07-05 09:27:44 -07:00
Umair Khan 95fc16d90d Django 1.11: MIDDLEWARE_CLASSES setting is deprecated.
Django provides MiddlewareMixin to upgrade old-style middlewares. See
https://docs.djangoproject.com/en/1.11/topics/http/middleware/#upgrading-middleware
2017-06-13 15:04:04 -07:00
Aditya Bansal 5b2dec0845 pep8: Add compliance with rule E261 event_queue.py. 2017-06-04 09:18:23 -07:00
K.Kanakhin 2434f2d96c messages: Add support for admins deleting messages.
This makes it possible for Zulip administrators to delete messages.
This is primarily intended for use in deleting early test messages,
but it can solve other problems as well.

Later we'll want to play with the permissions model for this, but for
now, the goal is just to integrate the feature.

Note that it saves the deleted messages for some time using the same
approach as Zulip's message retention policy feature.

Fixes #135.
2017-05-29 21:59:38 -07:00
Tim Abbott 315a9ee72e handlers: Fix type of zulip_finish. 2017-05-23 17:36:19 -07:00
umkay a67e427293 mypy: Add comment for future use of Deque.
In the future, the type annotation should use Deque in order to be
compatible with the latest mypy version. See
https://github.com/python/mypy/pull/2845 for more info.
2017-05-23 10:42:44 -07:00
Aditya Bansal 27a1b23392 pep8: Add compliance with rule E261 to ioloop_logging.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 5c82319101 pep8: Add compliance with rule E261 to handlers.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 72d5760d7c pep8: Add compliance with rule E261 to descriptors.py. 2017-05-07 23:21:50 -07:00
Tim Abbott cbc731963f decorator: Add support for Django internal_notify_view. 2017-04-18 09:59:07 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
Steve Howell 2718bd0b5d Extract presence.js to track presence info.
Most of this code was simply moved from activity.js with some
minor renaming of functions like set_presence_info -> set_info.

Some functions were slightly nontrivial extractions:

    is_not_offline:
        came from activity.huddle_fraction_present

    get_status/get_mobile:
        simple getters

    set_user_status:
        partial extraction from activity.set_user_status

    last_active_date:
        pulled out of admin.js code

We also fixed activity.filter_and_sort to take user_ids.
2017-04-04 15:57:10 -07:00
Tim Abbott 2b62c4fa38 mypy: Fix missing annotation in Tornado handlers.
This was accidentally removed in
9866124b78, but is needed on the current
mypy version.
2017-03-19 22:26:25 -07:00
Tim Abbott 8fa60cf77c py3: Remove use of typing.Deque.
Apparently this is only available in ultra-modern versions of Python 3?
2017-03-19 21:39:54 -07:00
Tim Abbott 9866124b78 mypy: Fix some new errors flagged by latest mypy master.
Mostly list -> List bugs in annotations.
2017-03-19 21:03:45 -07:00
Tim Abbott 25488b550f casper: Fix issues with server autoreloading on save.
This fixes an issue where if you saved a Python file (even just
changing whitespace) while casper tests were running, the Tornado
server being used would restart, triggering a confusing error like
this:

ReferenceError: Can't find variable: $
Traceback:
  undefined:2
  :4
Suite explicitly interrupted without any message given.
2017-03-17 20:45:07 -07:00
Tim Abbott fe237118b0 Rename zerver/lib/session_user.py to sessions.py. 2017-03-08 02:43:35 -08:00
sinwar 6f0564e9f4 python: Fix remaining bare excepts in codebase.
Fixes #2862.
2017-03-05 16:17:04 -08:00
Tim Abbott 75e81253f2 mypy: Work around several new mypy bugs in 0.501. 2017-03-04 15:33:39 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 2bbfdeeb7b Fix more errors caught by mypy 0.501.
Another set of relatively easy to review changes.
2017-03-03 14:15:38 -08:00
Rishi Gupta 15d60fa7ed Change now() to timezone.now() throughout codebase.
Change `from django.utils.timezone import now` to
`from django.utils import timezone`.

This is both because now() is ambiguous (could be datetime.datetime.now),
and more importantly to make it easier to write a lint rule against
datetime.datetime.now().
2017-02-28 16:03:37 -08:00
Tim Abbott 8487a24a6c socket: Hardcode website message sender for fake messages. 2017-02-17 15:20:32 -08:00
Tim Abbott 680d30adc2 docs: Add comments linking to new events doc. 2017-02-11 16:59:28 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Tim Abbott bde2da7dfd lint: clean up PEP-8 W391 rule. 2017-01-23 20:39:02 -08:00
Tim Abbott 42612161df tornado: Remove unused caching code. 2017-01-19 16:36:31 -08:00
Tim Abbott 007d4becfd Revert "update-sockjs: Update sockjs from version 0.3.4 to 1.1.1."
This reverts commit 7bf10ec74f.

Apparently, SockJS 1.1.1 is broken with the browser used in our legacy
desktop app, resulting in messages being silently not sent.
2017-01-10 11:46:15 -08:00
Rishi Gupta cf762eaf84 Change X.realm.id to X.realm_id across codebase.
This makes it more clearly the pattern in the Zulip codebase, and thus
decreases the risk of accidentally doing database queries.
2017-01-03 16:46:26 -08:00
K.Kanakhin 0d8c18a6dd nagios-plugins: Add websocket checking to nagios message sending test.
- Add websocket client to create connection with SockJS websocket server.
  It contains callback method to launch after connection setup.
- Add '--websocket' parameter to 'check_send_receive_time' script to
  check websocket connection.
- Add testing  websocket connection to production installation checking.
- Add cronjob to launch websocket connection nagios test.

This makes it possible for Zulip Nagios monitoring to check for
problems impacting the websockets sending code path, which is what all
web users use.
2016-12-30 15:36:37 -08:00
Juan Verhook cfa9c2eaf2 mypy: Update zerver directory to use Text 2016-12-29 09:12:15 -08:00
Rafid Aslam e49bd6a6c7 event_queue: Improve error message for ConnectionError exception
Improve error message for "ConnectionError:
HTTPConnectionPool(host='localhost', port=9993): Max retries
exceeded" exception.

Fixes #215.
2016-12-20 11:49:12 -08:00
Rafid Aslam 78f3fd7999 API: Expose `is_mentioned` in `message` dictionary.
Expose `is_mentioned` in `message` dict which contains
boolean value about our account is mentioned in the message
content or not.

This is already available via `flags`, but it seems worth making this
data point more explicit, given its importance in writing bots.

Fixes #2667.
2016-12-14 10:52:13 -08:00
K.Kanakhin 7bf10ec74f update-sockjs: Update sockjs from version 0.3.4 to 1.1.1.
- Add browserify to npm dependencies.
- Add SockJS-client to npm dependencies.
- Add npm postinstall script to generate browser version of SockJS-client
  from npm package.
- Change deprecated SockJS object property 'protocol_whitelist' to
  'transports'.
- Fix settings.
2016-12-05 21:36:10 -08:00
Reid Barton ecfa397567 Fix several type annotations for mypy 0.4.6 compatibility.
A few functions had arguments removed without having their type
annotations updated accordingly. As a result mypy version 0.4.6
thinks that the first type in the annotation is supposed to be
the type of `self`, a new mypy feature which we are not intending
to use here.
2016-12-03 13:45:52 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
bulat22101 a6f91064a2 pep8: Fix E129 violations 2016-12-03 10:56:36 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
AZtheAsian 1ba150fa85 pep8: Fix E203 violations 2016-12-01 20:37:57 -08:00
AZtheAsian f7582f0050 pep8: Fix E124 violations 2016-12-01 14:18:11 -08:00
Tommy Ip 46b7d54b3e pep8: Fix E701 violations. 2016-11-30 20:45:09 +00:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Tim Abbott 3d1bcb05e1 tornado: Move event_queue.py to zerver/tornado/.
Fixes #729.
2016-11-26 22:29:28 -08:00
Tim Abbott 1fcf2ff525 tornado: Move zerver.tornadoviews to zerver.tornado.views.
This furthers the overall goal of moving all the Tornado-specific code
to zerver/tornado/.
2016-11-26 22:29:28 -08:00
Tim Abbott 169d404579 tornado: Move zerver.lib.handlers into zerver.tornado.handlers.
This cleans up the confusingly duplicated file names, while also
moving more of the Tornado-specific code under zerver/tornado/.
2016-11-26 22:29:28 -08:00
Tim Abbott d75f49b119 tornado: Move descriptor dict management code to handlers.py. 2016-11-26 22:29:28 -08:00
Tim Abbott 282f74609c tornado: Move socket code to zerver/tornado/. 2016-11-26 22:29:27 -08:00
Tim Abbott f2782971e0 tornado: Clean up AsyncDjangoHandler Python style. 2016-11-26 22:05:04 -08:00
Tim Abbott b83361bcfc tornado: Extract create_tornado_application helper. 2016-11-26 21:43:13 -08:00
Tim Abbott 9e9066f77a tornado: Extract AsyncDjangoHandler to its own file. 2016-11-26 21:43:11 -08:00
Tim Abbott 8739f4d9f1 tornado: Move tornado_ioloop_logging to new zerver/tornado tree. 2016-11-26 21:24:05 -08:00