Commit Graph

5597 Commits

Author SHA1 Message Date
akshatdalton 52c411df8a emoji: Add padding around the gif on GIF emoji upload.
Replaced ImageOps.fit by ImageOps.pad, in zerver/lib/upload.py, which
returns a sized and padded version of the image, expanded to fill the
requested aspect ratio and size.
Fixes part of #16370.
2020-10-06 17:28:02 -07:00
Alex Vandiver baf882a133 queue: Only ACK drain_queue once it has completed work on the list.
Currently, drain_queue and json_drain_queue ack every message as it is
pulled off of the queue, until the queue is empty.  This means that if
the consumer crashes between pulling a batch of messages off the
queue, and actually processing them, those messages will be
permanently lost.  Sending an ACK on every message also results in a
significant amount lot of traffic to rabbitmq, with notable
performance implications.

Send a singular ACK after the processing has completed, by making
`drain_queue` into a contextmanager.  Additionally, use the `multiple`
flag to ACK all of the messages at once -- or explicitly NACK the
messages if processing failed.  Sending a NACK will re-queue them at
the front of the queue.

Performance of a no-op dequeue before this change:
```
$ ./manage.py queue_rate --count 50000 --batch
Purging queue...
Enqueue rate: 10847 / sec
Dequeue rate: 2479 / sec
```
Performance of a no-op dequeue after this change (a 25% increase):
```
$ ./manage.py queue_rate --count 50000 --batch
Purging queue...
Enqueue rate: 10752 / sec
Dequeue rate: 3079 / sec
```
2020-10-06 17:26:14 -07:00
Abhijeet Prasad Bodas a20d22de43 i18n: Move locale select logic in home to i18n.py.
Part of #16094.
Moved the language selection preference logic from home.py to a new
function in i18n.py to avoid repetition in analytics views and home
views.
2020-10-02 14:56:20 -07:00
Tim Abbott 8c8f3ee13b test_classes: Extract home view helpers for reuse. 2020-10-01 15:14:25 -07:00
Tim Abbott 6d041a3b34 home: Include is_web_public_guest in page_params. 2020-10-01 15:07:19 -07:00
Tim Abbott 351d73ac5a home: Pass realm to build_page_params_for_home_page_load.
This is preparation for this needing to be a separate parameter from
the user.
2020-10-01 15:00:36 -07:00
Aman Agrawal 18d852de49 unreads: Add support for web public guests.
This handles the case of web public guests by returning
RawUnreadMessagesResult with empty initalized values.
2020-10-01 14:53:43 -07:00
Aman Agrawal e02f0fb1c4 get_raw_unread_data: Extract func to get unreads from ums. 2020-10-01 14:46:46 -07:00
Tim Abbott e8e876d54c events: Deduplicate anonymous user code.
This approach lets us deduplicate much of the fetch_initial_state_data
logic around logged-out users.
2020-10-01 14:43:51 -07:00
Aman Agrawal f46f251688 post_process_state: Allow web public guests.
Because the logic already works correctly, we just need to change mypy
types.
2020-10-01 14:41:49 -07:00
Aman Agrawal 190f481f49 stream_subscription: Mark notifications disabled for web public users.
Users without an account can't get notifications, so we might as well
ensure any UI displays them appropriately.
2020-10-01 14:40:48 -07:00
Anders Kaseorg 46babbe9e1 import_realm: Close the memcached connection before forking.
This prevents the memcached connection from being shared across
multiple processes, and hopefully addresses unexpected behavior from
cached functions like get_user_profile_by_id invoked inside the worker
processes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-01 11:20:39 -07:00
sahil839 6c473ed75f message: Call build_message_send_dict from check_message.
We call build_message_send_dict from check_message instead of
do_send_messages.

This is a prep commit for adding a new setting for handling
wildcard mentions in large streams.
2020-09-29 17:18:04 -07:00
sahil839 f1a5fbaeb0 message: Extract build_message_send_dict function.
We extract the loop for building message dict in
do_send_messages in a separate function named
build_message_send_dict.

This is a prep commit for moving the code for building
of message dict in check_message.
2020-09-29 16:50:47 -07:00
sahil839 0514ba7ecb message: Add 'links_for_embed' to message_dict.
There is a bug where we send event for even
those messages which do not have embedded links
as we are using single set 'links_for_embed' to
check whether we have to send event for
embedded links or not.

This commit fixes the bug by adding 'links_for_embed'
in message dict itself and send the event only
if that message has embedded links.
2020-09-29 16:50:47 -07:00
Steve Howell c199571112 mypy: Add StreamDict.
This requires us to rework the view code a little
bit to explicitly assign fields.
2020-09-29 16:49:10 -07:00
Steve Howell bee18c70f0 mypy: Use str in statsd_key. 2020-09-29 16:49:10 -07:00
Steve Howell 2c496d9afd mypy: Fix do_send_user_group_update_event. 2020-09-29 16:49:10 -07:00
Steve Howell a37ef208dc mypy: Add RawReactionRow. 2020-09-29 16:49:10 -07:00
Vishnu KS 367c792968 actions: Downgrade realm before scrubbing. 2020-09-28 15:37:49 -07:00
Vishnu KS 0d30f59c97 billing: downgrade_now -> downgrade_now_without_creating_additional_invoice. 2020-09-28 15:37:49 -07:00
Steve Howell def3dac6ae event_schema: Add comments to top of the file.
The comments basically explain the common coding
patterns for making the checkers.
2020-09-28 12:19:28 -07:00
Tim Abbott 3242fc7388 soft_deactivation: Fix typo in logging output. 2020-09-28 12:12:04 -07:00
palash 0c18113910 soft_deactivation: Change root logger to zulip.soft_deactivation.
Update logger in the following files using this logger:
test_soft_deactivation, test_home, test_push_notifications
2020-09-28 12:12:00 -07:00
Tim Abbott 899cb41857 MessageDict: Remove _finalize_payload for simplicity.
finalize_payload already has a few options; there's little benefit to
this one being implemented as a separate helper function.
2020-09-28 12:00:18 -07:00
Tim Abbott 99396b25a6 MessageDict: Add a bit of docstring documentation. 2020-09-28 11:50:02 -07:00
Tim Abbott 90ff62aabc actions: Rename message local variable to message_dict.
This is a preparatory refactor to make it easy to see the changes
using `git show` in the next commit.
2020-09-28 11:14:59 -07:00
sahil839 ae74f8aafb actions: Remove unnecessary comment in do_send_messages function.
This commit removes the unnecessary comment which was added in
9454683108, when we were using message.get() for keys which
were also passed as args in do_send_messages, but there are no
such keys in the current code.
2020-09-28 10:58:35 -07:00
sahil839 76c75fea92 actions: Remove unnecessary line from do_send_messages.
This commit removes the unnecessary line of code to get
rendered_content from message dict sent by check_message
when it actually does not inlcude 'rendered_content' key.

This line was added in 9454683108, but now we do not send
rendered_content in the message dict as we render the message
in do_send_messages itself.
2020-09-28 10:58:35 -07:00
Dinesh acca870480 tests: Add a dummy request to self.client.login().
A later commit alters `authenticate` of EmailAuthBackend to
add a store `needs_to_change_password` variable to session
which is useful to insist users on changing their weak password.

The tests start failing with that change because client.login()
runs `authenticate` without a `request` object. So, this commit
sends a request object with `request.session=self.client.session`
to self.client.login() in tests wherever needed.
2020-09-25 16:24:18 -07:00
Abhijeet Prasad Bodas d9d51e32c1 i18n: Simplify logic for translation data in page_params.
This refactors the get_translation_data function to return an empty
dict when 'en' language is passed, to avoid repetition of code in
stats and home.
2020-09-25 16:21:37 -07:00
Clara Dantas 8674287192 digest: Support digest of web public streams for guest users.
This change requires some basic plumbing for test code creating
web-public streams.
2020-09-25 16:11:04 -07:00
Tim Abbott 94a9fa1891 event_schema: Add documentation and rename a few functions.
This should help make this revised subsystem readable for more new
contributors.  We still need to make updates to the high-level
documentation.
2020-09-25 12:53:00 -07:00
Steve Howell 5b7c9c4714 test_events: Add check_realm_user_remove. 2020-09-25 11:43:20 -07:00
Steve Howell 7bb7f2943f event_schema: Finish extraction with realm_emoji/update.
We now no longer define any schemas in test_events--all
of them are in event_schema, which helps our tooling
cross-check schemas for openapi and node tests.
2020-09-25 11:43:20 -07:00
Steve Howell ae4d083a5a event_schema: Extract check_realm_domains_*. 2020-09-25 11:43:20 -07:00
Steve Howell 298bed9fa1 event_schema: Split check_update_message_flags. 2020-09-25 11:43:20 -07:00
Steve Howell f6e0171d02 event_schema: Split check_reaction into add/remove.
It happens that whether you add a reaction or remove
a reaction, we send the exact same fields, just using
a different op code.

This sort of symmetry is actually kind of rare, as
usually "add" events have more fields, and "remove" events
might just send an id of something to remove.

Our openapi schema treats these as two seperate events,
so we are more consistent with it, and it helps our
schema-checking tooling for node fixtures, too.

Note that we now have to exempt the two events from
our openapi checks, due to the is_mirror_dummy field
in the deprecated user block.  We can decide how to
handle this later--one possibility is to just add it
as an optional field on the event_schema side.
2020-09-25 11:43:20 -07:00
Steve Howell b7b2546f44 event_schema: Extract check_subscription_update.
Note that we use value_type for value instead of
bool, since properties can be non-bool things
like color, which we just don't test now.  We
should test them.

We more than compensate for this by checking
the actual value of the value in
check_subscription_update.
2020-09-25 11:43:20 -07:00
Steve Howell b920ebce81 event_schema: Extract check_has_zoom_token. 2020-09-25 11:43:20 -07:00
Steve Howell 0c4286222f event_schema: Extract check_realm_update_dict. 2020-09-25 11:43:20 -07:00
Steve Howell 6ec6525624 event_schema: Extract check_delete_message.
There is a legacy format where we send
singular "message_id" instead of plural
"message_ids".

Then there are different fields for "private"
and "stream" message types.
2020-09-25 11:43:20 -07:00
Steve Howell 88165aee6b event_schema: Extract check_user_group_update. 2020-09-25 11:43:20 -07:00
Steve Howell aaaac11661 event_schema: Extract check_user_group_remove. 2020-09-25 11:43:20 -07:00
Steve Howell 1b7af13f37 event_schema: Extract check_user_group_remove_members. 2020-09-25 11:43:20 -07:00
Steve Howell 19b7739065 event_schema: Extract check_user_group_add_members. 2020-09-25 11:43:20 -07:00
Steve Howell 4084f0b949 event_schema: Extract check_realm_user_add.
Note that we make the schema for profile_data
slightly more realistic, but it doesn't actually get
exercised by our current tests (apart from
making sure it's a dict), since we don't have
profile data for our test realm.

We also don't have the optional fields for bots,
since our tests don't exercise that, nor
delivery_email.

So we exempt realm_user_add_event from openapi
checks for now.

When we try to match the openapi specs better, we
will probably want to add a few tests to test_events.

Obviously getting good coverage for adding users
would be nice for all these scenarios:

    * delivery_email matters
    * bots
    * realm has profile fields
2020-09-25 11:43:19 -07:00
Steve Howell dc2176a965 event_schema: Extract check_presence. 2020-09-25 11:43:19 -07:00
Steve Howell 6c74a44697 data_types: Generalize StringDictType.
This is a prep commit for supporting "presence"
events, where the key of the dictionary is some
arbitrary string like "website" but the value
of the dictionary is another dictionary itself
with keys that are more like variable names.
2020-09-25 11:43:19 -07:00
Steve Howell 4f3d5f2d87 event_schema: Extract check_realm_filters.
We have some known issues with representing
tuples in openapi, so we exempt realm_filters
from the relevant check.
2020-09-25 11:43:19 -07:00