Commit Graph

43367 Commits

Author SHA1 Message Date
Abhijeet Prasad Bodas 226ac4bd3b tornado: Use `UserMessageNotificationsData` for update_message.
This lets us simplify what had previously been somewhat duplicated
logic between the send and edit code paths.
2021-06-22 10:27:59 -07:00
Abhijeet Prasad Bodas c3fb413119 message edit: Don't send mentioned user_ids in event dict.
We already have this data in the `flags` for each user, so no need to
send this set/list in the event dictionary.

The `flags` in the event dict represent the after-message-update state,
so we can't avoid sending `prior_mention_user_ids`.
2021-06-22 10:27:55 -07:00
Tim Abbott 71d5d64086 node_tests: Use module_path as local variable name. 2021-06-22 10:21:07 -07:00
Anders Kaseorg 6dbe03c54b zjsunit: Undo mock_cjs abstraction violation.
There is still no need for mock_template to reach into the internals
of mock_cjs.  Make it a normal caller of mock_cjs.

This is basically the same as 60f5a00c09
(#18804), because the same abstraction violation was since
reintroduced in the same way.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-22 10:19:56 -07:00
Mateusz Mandera 85e19b2bde upload: Use URL manipulation for get_public_upload_url logic.
This is much faster than calling generate_presigned_url each time.

```
In [3]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
0.0010945796966552734
```
2021-06-22 09:35:56 -07:00
Mateusz Mandera e883ab057f upload: Cache the boto client to improve performance.
Fixes #18915

This was very slow, causing performance issues. After investigating,
generate_presigned_url is the cheap part of this, but the
session.client() call is expensive - so that's what we should cache.

Before the change:
```
In [4]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
6.408717393875122
```

After:
```
In [4]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
0.48990607261657715
```

This is not good enough to avoid doing something ugly like replacing
generate_presigned_url with some manual URL manipulation, but it's a
helpful structure that we may find useful with further refactoring.
2021-06-22 09:35:19 -07:00
Priyank Patel 043b0c6ef3 ts: Convert spoilers module to TypeScript. 2021-06-22 09:14:09 -07:00
Priyank Patel 91573531bb ts: Convert color_data module to TypeScript. 2021-06-22 09:14:09 -07:00
Priyank Patel adf83c6eb7 ts: Convert color_class module to TypeScript. 2021-06-22 09:14:09 -07:00
Priyank Patel d660f512fa ts: Convert list_util module to TypeScript. 2021-06-22 09:14:09 -07:00
Priyank Patel 6ed635b5ed ts: Convert night_mode module to TypeScript. 2021-06-22 09:14:09 -07:00
Priyank Patel 31abcaafb8 ts: Convert ready module to TypeScript. 2021-06-22 09:14:09 -07:00
Priyank Patel c862c673af ts: Convert blueslip module to TypeScript.
We turn off the eslint no-use-before-define for TypeScript files
because it does not work correctly for types. There is already
a typescript-eslint version of it that is enabled for TS.

We also update the error handler on window to use instanceof check
for ErrorEvent instead of checking the error property.
2021-06-22 09:09:36 -07:00
Priyank Patel f7235912ff ts: Convert ui_report module to TypeScript. 2021-06-22 09:09:36 -07:00
Priyank Patel 8a6f57d750 ts: Convert common module to TypeScript.
We also update the tippy.js mock in tests to expect a element
(zjquery FakeElement function) instead of zjquery instance, and get
rid of setting _tippy property itself to a noop function.
2021-06-22 09:09:36 -07:00
Priyank Patel 9418ae51fa ts: Convert i18n module to TypeScript.
We update our tsconfig target to ES2020 so that we don't get type
errors related to some properties not being in Intl that is used by
@formatjs/intl.
2021-06-22 09:09:36 -07:00
Priyank Patel f1a0a790fe ts: Convert page_params module to TypeScript.
The plan for type annotating the page_params is to set it to
Record<string, unknown> for now and then annotate individual
properties on it as we use it in typescript modules.
2021-06-22 09:09:36 -07:00
Priyank Patel ec2ab1eb8d lint: Correctly detect the use of style attribute.
We add a exclude pattern that makes sure we don't catch two edge
cases: a variable declaration `const style =` and setting a
variable ending in style such as `require_cmd_style =`. We don't add
and exclude pattern for let declaration because it will catch lines
that modify it later in the code.

(Removed other files from exclude list that no longer needed to be
excluded from this lint rule.)
2021-06-22 09:09:36 -07:00
Priyank Patel 43da43701b refactor: Use variables to get TypeScript type narrowing.
We also add an explicit undefined check for the these return value
of the Map.get calls. This makes the TypeScript conversion diff
cleaner.
2021-06-22 09:09:36 -07:00
Priyank Patel 2a21f626a7 refactor: Use object destructuring for function parameter.
This make type annotating easier.
2021-06-22 09:09:36 -07:00
Priyank Patel fd45ef57fa ui_report: Do not pass jQuery element to the html method.
Although it works, this feature is not documented.
2021-06-22 09:09:36 -07:00
Priyank Patel 9e08c6db93 ui_report: Correctly check if remove_after is passed.
We use a simpler approach here which is to let it be undefined if
nothing was passed and use that in the if condition. This has two
benefits. There is no edge case where the condition will evaluate to
false if 0 was passed and it is easier to type annotate.
2021-06-22 09:09:36 -07:00
Priyank Patel 5ae523891b ui_report: Use numerical check for request status code. 2021-06-22 09:09:36 -07:00
Priyank Patel 97c8987e02 blueslip: Declare log properties directly on the Logger class.
This makes TypeScript type annotation easier (and maybe looks
cleaner.)
2021-06-22 09:09:36 -07:00
Tim Abbott 67fd5585f9 help: Briefly document new tabs in full profile. 2021-06-21 17:21:24 -07:00
Tim Abbott fcb2f66725 popovers: Show user profile popover for all humans. 2021-06-21 17:16:50 -07:00
Anders Kaseorg b4a26bcafd lightbox_overlay: “Pan and Zoom” → “Pan & zoom”.
The ‘t’ helper operates on text strings, not HTML.  The ‘&’ works
fine, and is correctly escaped for output by Handlebars, like any text
string interpolated with ‘{{}}’.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-21 16:38:11 -07:00
Tim Abbott e231a03eff message_edit: Fix non-alternating resolve topic notifications.
Previously, it was possible for an unusual series of topic-edit
actions to result in Notification Bot reporting that a topic was
marked as resolved that had already been marked as resolved, etc.
2021-06-21 12:45:19 -07:00
Tim Abbott b2dd15fd86 message_edit: Reject buggy noop topic edit requests.
A buggy client might send a message_edit request to change the topic
field, sending the current topic as the new value. Previously, we
would treat that as a normal request to edit the topic; now we act as
though the API request had not requested a topic change.  In the
common case that only the topic was in the edit request, this now
results in an error that should help client implementations identify
their bug.

This fixes a bad interaction with the "unresolve topic" logic, which
assumed that upstream logic had verified that the topic was actually
changing.
2021-06-21 12:16:00 -07:00
sahil839 5a47e1f289 api: Remove incorrect realm setting in 'realm/update_dict' schema.
We incorrectly include many realm settings in the data section of
'realm/update_dict' schema. It should only contain the settings
related to message edit, realm icon, realm logo and authentication
methods and not other settings, becausea all the other settings send
'realm/update' event and not 'realm/update_dict' event.

This commit only removes 'default_twenty_four_hour_time' and
'default_language', others will be removed separately.
2021-06-21 11:30:57 -07:00
sahil839 c42031bc05 signup: Do not translate signup messages sent to "Zulip internal" realm. 2021-06-21 11:30:56 -07:00
Suyash Vardhan Mathur e5221a8434 curl_examples: Fix curl testing for multiple configs.
Now, the markdown extension of curl_examples generates
all examples of all possible configurations with
their descriptions, and so we need to separate
executable curl commands from the rest of the raw
HTML.

This commit simply changes the indentation of the
block and replaces the command being tested
with each element of the commands array. This
was split for an easier review.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur acbae4b6cf curl_examples: Fix curl testing for multiple configs.
Now, the markdown extension of curl_examples generates
all examples of all possible configurations with
their descriptions, and so we need to separate
executable curl commands from the rest of the raw
HTML.

This commit adds a commands variable to store
all the curl commands in HTML using regex.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur 6bd11285c1 openapi: Fix minor assertion in curl examples.
Now, include and exclude configuration
are fetched from openapi data, and only
one type can be encoded for every example.

This removes the need for the assertion to
test if both include and exclude are present
since at a time, only one can be present.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur a2be9a0e2d apidocs: Modify curl examples in templates.
Now, all possible configurations for curl examples
are rendered directly from openapi data, and
the templates should just call the markdown
function once.

This commit adjusts all the templates by removing
hardcoded configurations and descriptions for
individual curl examples.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur 246262fb57 openapi: Modify curl example generation logic.
This commit adds support for using the
x-curl-examples-parameters parameter in OpenAPI
data to fetch curl examples configuration. This
also contains any descriptions necessary for each
example, and directly generates all possible
curl examples directly.

A follow-up commit is needed to modify the templates
accordingly.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur a56ad2a26a openapi: Add curl examples' configurations in OpenAPI data.
As a goal of moving towards a common template,
we need to fetch curl examples' configurations
directly from openapi data instead of having them
hardcoded in templates. This commit introduces
x-curl-examples-parameters to store the configs
for the same.
2021-06-21 11:06:05 -07:00
Abhijeet Prasad Bodas c6d696cf61 process_message_event: Deduplicate `is_notifiable` logic.
We use the `UserMessageNotificationsData` methods to check if the
message is notifiable, and use a cleaner early continue pattern.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas 5c483e3b58 get_active_presence_idle_user_ids: Check notifiability more thoroughly.
* Have the `get_active_presence_idle_user_ids` function look at all the
user data, not just `private_message` and `mentioned`.
* Fix a couple of incorrect `missedmessage_hook` tests, which did not
catch the earlier behaviour.
* Add some comments to the tests for this function for clarity.
* Add a helper to create `UserMessageNotificationsData` objects from the
user ID lists. This will later help us deduplicate code in the event_queue
logic.

This fixes a bug which earlier existed, that if a user turned on stream
notifications, and received a message in that stream which did not mention
them, they wouldn't be in the `presence_idle_users` list, and hence would
never get notifications for that message.

Note that, after this commit, users might still not get notifications in
the above scenarios in some cases, because the downstream logic in the
notification queue consumers sometimes erroneously skips sending
notifications for stream messages.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas aeb2ad5f46 notification_data: Annotate `flags` with `Collection`.
Since `flags` here could be iterated through multiple times
(to check for push/email notifiability), we use `Collection`.
Inspired by 871e73ab8f.

The other change here in the `event_queue` code is prep for using
the `UserMessageNotificationsData` class there.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas 8182632d7e notification_data: Add methods to determine notifiability.
We will later consistently use these functions to check for notifiable
messages in the message send and event_queue code.

We have these functions accept the `sender_id` so that we can avoid the
`private_message = message["type"] == "private" and user_id != sender_id`
wizardy.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas ed58393079 message send: Optimize how user data is sent to Tornado.
Before this commit, we used to pre-calculate flags for user data and send
it to Tornado, like so:
```
{
    "id": 10,
    "flags": ["mentioned"],
    "mentioned": true,
    "online_push_enabled": false,
    "stream_push_notify": false,
    "stream_email_notify": false,
    "wildcard_mention_notify": false,
    "sender_is_muted": false,
}
```

This has the benefit of simplifying the logic in the event_queue code a bit.

However, because we sent such an object for each user receiving the event,
the string keys (like "stream_email_notify") get duplicated in the JSON
blob that is sent to Tornado.

For 1000 users, this data may take up upto ~190KB of space, which can
cause performance degradation in large organisations.

Hence, as an alternative, we send just the list of user_ids fitting
each notification criteria, and then calculate the flags in Tornado.
This brings down the space to ~60KB for 1000 users.

This commit reverts parts of following commits:
- 2179275
- 40cd6b5

We will in the future, add helpers to create `UserMessageNotificationsData`
objects from these lists, so as to avoid code duplication.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas 6167c36adc event_queue: Translation code for user data migration.
This is separate from the next commit for ease of testing.
To verify that the compatibility code works correctly, all message send
and event_queue tests from our test suite should pass on just this commit.
2021-06-21 10:52:59 -07:00
aryanshridhar 477a066a4c delete_message: Move and rename the delete_message modal.
Moved the `delete_message` user-confirmation modal to
the `confirm_dialog` folder and renamed the modal to `confirm_delete_message.hbs`
to follow the common naming convention.
2021-06-21 10:36:35 -07:00
aryanshridhar df556a199a settings_invites: Move and rename the revoke_invite modal.
Moved `revoke_invite` user-confirmation modal to the
`confirm_dialog` folder and renamed the modal to
`confirm_revoke_invite.hbs` to follow the common naming convention.
2021-06-21 10:36:35 -07:00
aryanshridhar d0c683a8d7 settings_invites: Move and rename the resend_invite modal.
Moved `resend_invite` user-confirmation modal to the
`confirm_dialog` folder and renamed the modal to
to `confirm_resend_invite.hbs` to follow the common naming convention.
2021-06-21 10:36:35 -07:00
aryanshridhar 83473ba647 settings_emoji: Move and rename the emoji_settings_warning modal.
Moved `emoji_settings_warning` modal to the `confirm_dialog`
folder and renamed the modal to `confirm_emoji_settings_warning.hbs`
to follow the common naming convention.
2021-06-21 10:36:35 -07:00
aryanshridhar 4549cbe48c settings_users: Move and rename the deactivation_user modal.
Moved `deactivation_user` modal to the `confirm_dialog`
folder within `/static/templates` and renamed the modal to
to `confirm_deactivate_user.hbs` to follow the
common naming convention.
2021-06-21 10:36:35 -07:00
aryanshridhar c5d65555fa deactivation_stream: Move and rename the deactivation_stream modal.
Moved `deactivation_stream` user-confirmation modal to
the `confirm_dialog` folder and renamed the modal to
`confirm_deactivate_stream.hbs` to follow the common naming convention.
2021-06-21 10:36:35 -07:00
aryanshridhar 803a6dd60a deactivate_realm: Move and rename the deactivate_realm modal.
Moved `deactivate_realm` modal to the `confirm_dialog` folder
within `/static/templates` and renamed the modal to
`confirm_deactivate_realm.hbs` to follow the naming convention.
2021-06-21 10:36:35 -07:00