Commit Graph

13843 Commits

Author SHA1 Message Date
Abhijeet Prasad Bodas b6806fbf9d message send: Calculate `presence_idle_user_ids` after building user data.
This is a direct code move and a prep change for having this function look
at all the user data variables, not just `private_message` and `mentioned`.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 42e4fa7952 message send: Refactor how user data is sent for the event.
This is a prep change for calling `get_active_presence_idle_user_ids`
after we have collected all user data variables, so that that function
does not erroneously skip some user IDs from not having the complete
data.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 3ff3ded1ae process_message_event: Always calculate `online_push_enabled`.
Unlike `receiver_is_off_zulip`, fetching from a dict is pretty cheap,
so we can calculate `online_push_enabled` along with the other
variables.

This is a prep change to start using the dataclass introduced in the
earlier commit in this code.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 951b49c048 message send: Introduce dataclass to wrap user-notifications variables.
We will in later commits, extend this class to contain methods
to determine if a message is notifiable or not, but for now
we only turn it into a dict and pass it on.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 2179275020 event_queue: Deduplicate `mentioned` flag calculation.
This gives us a single place where all user data for the message
send event is calculated, and is a prep change for introducing
a TypedDict or dataclass to keep this data toghether.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 40cd6b5440 message send: Calculate `wildcard_mention_notify` pre-send_event.
This deduplicates some logic, and makes it so that
`wildcard_mention_notify` always contains what its name says it does.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 919f0a5964 message send: Extract local variables for user event data. 2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 56f6656d8b message send: Don't create user data list with inline `for`.
This will make the code more readable, and will allow creating local
variables within the loop for reuse.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 5a642cea11 missedmessage_hook: Don't enqueue notificationss if sender is muted.
This is a follow up to 71742dce24 to handle
muted senders in the missedmessage_hook too.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 86a20c5b3c event queue: Always set internal_data flags.
We have already calculated these values, so storing them should not cause
significant performance degradation.

This is a prep chenge for sending a few more flags through internal_data,
namely if `sender_is_muted`.
2021-06-15 12:30:31 -07:00
Riken Shah b2ab448b71 gear_menu: Extract the gear menu as a handlebars template.
For this extraction, we need to move some context
parameter (from home_real in `views/home.py`) to extra
page_params parameter (of
build_page_params_for_home_page_load in
`lib/home.py`) so handlebars template can access them.

While moving I confirmed that these parameters are not
used elsewhere if some parameter is used elsewhere
(like `apps_page_url`) then I didn't remove it from the
context list, I just added it to the page_params list.

Fixes: #18795.
2021-06-15 12:17:45 -07:00
Riken Shah d2f7aaf6f6 refactor: Rename `enable_marketing_emails_enabled` param of page_params.
This is a prep commit to extract the gear menu as a
handlebars template.

We are renaming `enable_marketing_emails_enabled` to
`corporate_enabled` as it will be also used in the
handlebars template of the gear menu.
2021-06-15 12:05:37 -07:00
Riken Shah 5fe21ae08f refactor: Make `UserProfile` field optional for `get_billing_info`.
This is preparatory work for extracting the gear menu
as a handlebars template.
2021-06-15 12:05:37 -07:00
Riken Shah 61b30e61ad home: Extract `get_apps_page_url` function from `zulip_default_context`.
This is a prep commit to extract the gear menu as a
handlebars template.

We are extracting it as a function so we can also use it as
a parameter in `page_params`.
2021-06-15 12:05:37 -07:00
Ganesh Pawar 1d4b6c1320 about_zulip: Migrate to handlebars.
This results in moving the `zulip_merge_base` parameter to
page_params, so that it's available to JavaScript.

Since this is technically a tiny overlay, it needs to be initialized
before hashchange.js.
2021-06-15 11:55:20 -07:00
Steve Howell e739bee00a poll widget: Add server validation. 2021-06-14 17:46:16 -07:00
Steve Howell 24ed5a2bb4 submessages: Add verify_submessage_sender.
Before this change a rogue actor could try to
widgetize another person's message. (The
rogue actor would already have access to read
the message.)
2021-06-14 17:46:16 -07:00
Vishnu KS 876af17dd8 models: Change return type of get_human_admin_users to QuerySet.
I would be accessing some methods of QuerySet in the subsequent
commits. So marking this as Sequence results in mypy errors.
2021-06-14 16:55:43 -07:00
Vishnu KS 6e3d4e7e75 billing: Create is_free_trial_offer_enabled function. 2021-06-14 16:49:53 -07:00
Vishnu KS 5da3a69505 plans: Mention whether realm is on free trial. 2021-06-14 16:49:53 -07:00
Vishnu KS 44d8368003 plans: Don't hardcode plan_type values in template. 2021-06-14 16:49:53 -07:00
sahil839 aa5d2199b1 invite: Use handlebar template for invite-user overlay.
Fixes a part of #18794.
2021-06-14 13:46:57 -07:00
akshatdalton c507931ac8 refactor: Export non-markdown logic in mention.py. 2021-06-14 13:26:30 -07:00
akshatdalton 1a76d06add test_push_notifications: Use responses module to mock HTTP responses. 2021-06-12 07:31:12 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Mateusz Mandera b1da5d9d76 ldap: Prevent creation of users with malformed email value. 2021-06-11 17:30:46 -07:00
Mateusz Mandera 18dab5619c ldap: Fix ldap access control tests.
These tests didn't configure ldap settings correctly and as a result,
the user involved in these tests wasn't actually hamlet@zulip.com, but a
new, malformed user with email "hamlet" that was being created by the
ldap auto-registration codepath. This wasn't caught because the codepath
didn't validate the email address and thus created such a malformed user
silently.
2021-06-11 17:30:46 -07:00
Abhijeet Prasad Bodas 10dd5f784b event_queue: Don't check for "read" flag when processing events.
* In `event_queue.py`, only the sender and recipient users who have muted
the sender will have the "read" flag set.

* We already skip enqueueing notifications for users who've muted the sender
after 58da384da3.

* The queue consume functions for email and push notifications already
check filter messages which have been read before sending notifications.

* So, the "read" logic in `event_queue.py` is unnecessary, and the
processing power saved from not enqueueing notifications for a single
user should be insignificant, so we remove these checks all toghether.
2021-06-11 08:07:37 -07:00
Abhijeet Prasad Bodas 3fd53d6f48 test_email_notifications: Add test for "read" flag return.
This logic was peviously untested. This is a prep change for us
to completely depend on the logic here for the "read" flag, and
not on the `event_queue` code.
2021-06-11 08:07:36 -07:00
Abhijeet Prasad Bodas 2b438fd7ce test_push_notifications: Fix incorrect "read" flag test.
It was unclear what the original test was testing, and more
importantly, the test passed even if one removed the `read` flag
check in the `handle_push_notifications` function, so we fix it
to be more comprehensive.
2021-06-11 08:05:27 -07:00
Abhijeet Prasad Bodas c6a31dcd9f event_queue: Extract local variables. 2021-06-11 08:05:27 -07:00
Suyash Vardhan Mathur 001435a297 refactor: Rename and move app_filters.py.
Moved the code in zerver/templatetags/app_filters.py
to more intuitive location and name at
zerver/lib/templates.py.
2021-06-11 07:43:22 -07:00
Suyash Vardhan Mathur 274f73d117 apidocs: Automatically add javascript example tabs.
The javascript tab in .md templates can be
generated along with the line that adds js
example.

Further, as a part of the effort of moving
towards a single template, the markdown extension
for javascrit examples is modified to return empty
string if javascript example doesn't exist for that
endpoint. This would make it possible to cover more
endpoints with a single template.

The js example tabs are now automatically generated
during generation of javascript code, and so need to
be removed. Also, the markdown function to render js
examples can be added in all templates, since it is
parses and returns an empty string if the examples
don't exist, and allows us to move towards a common
template.

The pages have been verified to be correct
by using diff between old and new pages' raw HTML.
2021-06-11 07:43:22 -07:00
Vishnu KS 4e26746b98 tests: Move tests of /activity pages from zerver to analytics module.
This module deals with the testing of /activity, /realm_activity
and /user_activity. All these pages reside in analytics module.
Keeping these tests in zerver/tests is kind is not appropriate
since person who makes changes to /activity pages would not think
it is necessary to run tests in zerver. So better to keep them
in the analytics module.
2021-06-11 07:41:02 -07:00
Vishnu KS 46c902686d tests: Specify the update_fields in test_activity.
This is a prep commit. Else the lint will complain during the
refactoring of this function to a new location that update_fields
is not specified.
2021-06-11 07:41:02 -07:00
Vishnu KS 40ab415005 activity: Show effective rate of realms in /activity page. 2021-06-11 07:41:02 -07:00
Tim Abbott 17fe09bb6d webhooks: Update link to BuildBot documentation. 2021-06-10 17:15:55 -07:00
Suyash Vardhan Mathur 5be92ad30d openapi: Auto-render heading with return values.
The headings for return values were currently hardcoded
in cases where they occur, but they can be rendered directly
in the markdown extension if the return values exist.
2021-06-10 11:25:13 -07:00
Eeshan Garg 50a6887d94 plans_page: Compute sponsorship link for subdomain pages correctly.
Currently, in the FAQ on our /plans page, when the user clicks on
the sponsorship link in the answer for the first question, they
are always taken to /accounts/go, causing them to have to input
their organization URL even if they are on a subdomain page.

This commit makes it so that when the user is on a subdomain page,
they are taken to /upgrade#sponsorship directly. On the other
hand, when they are on a root domain (/) page, they have to go
through /accounts/go and specify their organization's name.
2021-06-10 09:10:38 -07:00
Vishnu KS 1938076f67 billing: Enforce license limit for plans on manual license management. 2021-06-09 17:42:38 -07:00
Vishnu KS 8c055107d9 exceptions: Move InvitationError to zerver/lib/exceptions. 2021-06-09 17:42:38 -07:00
Vishnu KS 7197c8ae89 test_classes: Create helper function for subscribing realm to manual plan. 2021-06-09 17:42:38 -07:00
PIG208 e8b92b79e5 integrations: Add V3 support for PagerDuty. 2021-06-09 16:34:34 -07:00
PIG208 32ca263914 integrations: Change format of templates for PagerDuty V3.
Because the payload of V3 will no longer include the description,
We replace the ":" by "." in the message and create the new string
template for trigger messages.
2021-06-09 16:34:34 -07:00
Anders Kaseorg 342834ee9c python: Simplify stdio flushing using print(…, flush=True).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:05:31 -07:00
Anders Kaseorg d8adeb66ca docs: Deemphasize Hubot as an integration bridge.
The Hubot project looks to be abandoned; it hasn’t been updated in
years and its own installation instructions don’t work anymore.
Remove our special placement of Hubot alongside Zapier and IFTTT.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:04:01 -07:00
Tim Abbott d041bb0674 import: Fix propagation of subdomain error messages.
The previous logic would provide a very confusing error message if the
subdomain was already in use.
2021-06-09 12:58:05 -07:00
Suyash Vardhan Mathur 6692250858 openapi: Enhance markdown extension for responses.
Currently, the descriptions for API responses were
hardcoded in the templates. However, they now exist
in YAML data as well, and so can be fetched from there.

Also, as a part of moving towards a common template, it is
beneficial to show return response description along with
the code example directly. Also, for the same reason, the
need for mentioning subschemas for the response has been
removed, and ALL responses for that response code should
be returned in the proper format.

This also includes returning empty string if that response
code doesn't exist. This allows us to directly mention both
200 and 400 responses in all templates, and they are parsed
automatically.

A followup is necessary to remove the descriptions in the template
as they are duplicated just by this commit
2021-06-09 12:43:12 -07:00
Suyash Vardhan Mathur 38b63a85e2 openapi: Migrate response descriptions into schemas.
Currently, the descriptions of API responses were
hardcoded in the template. However, as a part of the
goal of moving towards a common template, these should
be auto-generated.

This commit copies these descriptions into the `description`
parameter in the OpenAPI data for that effect.
2021-06-09 12:43:12 -07:00
akshatdalton 3cc6ff57db test_link_embed: Use responses module to mock HTTP responses. 2021-06-09 12:17:26 -07:00
akshatdalton bef7120390 test_slack_importer: Use responses module to mock HTTP responses. 2021-06-09 12:17:26 -07:00
akshatdalton 1259f5a70e minor: Properly unpack `kwargs`. 2021-06-09 12:17:26 -07:00
Alex Vandiver 721546dfc0 subdomains: Extend "static" to include resources hosted on S3.
This causes avatars and emoji which are hosted by Zulip in S3 (or
compatible) servers to no longer go through camo.  Routing these
requests through camo does not add any privacy benefit (as the request
logs there go to the Zulip admins regardless), and may break emoji
imported from Slack before 1bf385e35f,
which have `application/octet-stream` as their stored Content-Type.
2021-06-08 15:28:10 -07:00
Abhijeet Prasad Bodas 58da384da3 muting: Don't enqueue notifications for messages from muted senders.
Earlier, the notification-blocking for messages from muted senders
was a side-effect of we never sending notifications for messages
with the "read" flag.

This commit decouples these two things, as a prep for having new
settings which will allow users to **always** receive email
notifications, including when/if they read the message during the
time the notifications is in the queue.

We still mark muted-sender messages as "read" when they are sent,
because that's desirable anyways.
2021-06-08 14:58:14 -07:00
Abhijeet Prasad Bodas 006b92ed6d message send: Plumb sender muters through `get_recipient_info`.
This will make it possible to share this code with the message
update codepath.
2021-06-08 14:58:14 -07:00
Mateusz Mandera c54b48452d saml: Support syncing custom profile fields with SAML attributes.
Fixes #17277.

The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.

We detail that limitation in the documentation.
2021-06-08 14:47:33 -07:00
Mateusz Mandera 00c7ac15df saml: Define a TypedDict for the type of IdP settings dict.
The old type in default_settings wasn't right - limit_to_subdomains is a
List[str]. We define a TypeDict for capturing the typing of the settings
dict more correctly and to allow future addition of configurable
attributes of other non-str types.
2021-06-08 14:30:21 -07:00
Abhijeet Prasad Bodas 30c993451d test_message_edit_notifications: Cleanup irrelevant parameters.
This is in the same spirit as the previous commit.
2021-06-08 11:10:18 -07:00
Abhijeet Prasad Bodas 2bbdd42e1a test_event_queue: Fill-up default data in missedmessage_hook tests.
This allows us to skip sending parameters which are irrelevant
to what we are testing, and only send the specific changed data.
2021-06-08 11:10:18 -07:00
Abhijeet Prasad Bodas d9395e7b52 test_event_queue: Introduce helper to fill-up default values.
This allows us to only mention the values that are relevant
to the behavior being tested by the `check_notify` function
in the current assertion.
2021-06-08 11:10:18 -07:00
Abhijeet Prasad Bodas 8eec7b4718 test_event_queue: Extract common data setup in `check_will_notify`.
The `message_id` and `user_profile_id` values don't really matter for
our testing here, so we might as well set these dummy values in the
main function.
2021-06-08 11:10:18 -07:00
Anders Kaseorg ad7466a2bf dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 10:15:52 -07:00
Tim Abbott 3eee7c067c test_example: Add discussion of other common pitfalls.
The new test_example file is great; here I document more pitfalls that
I've seen very often with tests contributed by folks new to the
project.
2021-06-07 16:02:12 -07:00
Steve Howell 47da28429b tests: Add test_example.py.
The idea here is to quickly learn basic testing
idioms with code that's executed (unlike docs
which are prone to bitrot).
2021-06-07 15:29:49 -07:00
Abhijeet Prasad Bodas b7fcb0275c cache: Use `id`s instead of `UserProfile`s for get_muting_users.
This will make it easier to call this function in the message
send codepath.
2021-06-07 13:41:37 -07:00
Tim Abbott d84d61f718 mypy: Fix user_id typo in maybe_enqueue_notifications_for_message_update.
This also lets us remove importing UserProfile at all in this module.
2021-06-07 13:31:54 -07:00
Abhijeet Prasad Bodas 5f5323be7d event_queue: Pass `bool`s to `maybe_enqueue_for_message_update`.
This simplifies the logic in the function itself and makes this function
consistent with how we pass data to `maybe_enqueue_notifications`.
2021-06-07 13:29:37 -07:00
Abhijeet Prasad Bodas 22d19375f2 event_queue: Reorder `maybe_enqueue_for_update` parameters.
This will make `maybe_enqueue_notifications_for_message_update`'s
arguments more consistent with `maybe_enqueue_notifications`.
2021-06-07 19:54:04 +05:30
Abhijeet Prasad Bodas 99b181629d actions: Create dataclass for message read events.
This allows us to deduplicate the common fields.
2021-06-07 19:51:45 +05:30
Abhijeet Prasad Bodas e33c4105b4 actions: Use `custom_profile_field` as variable name, not `field`.
This is a prep change for importing (and using) `dataclasses.field`
elsewhere in the same file, because pyflakes would throw "Import
module shodowed" errors otherwise.
2021-06-07 19:50:36 +05:30
Gaurav Pandey 9b696cf212 api: Expose event_queue_longpoll_timeout_seconds in /register.
Rename poll_timeout to event_queue_longpoll_timeout_seconds
and change its value from 90000 ms to 90 sec. Expose its
value in register api response when realm data is fetched.
Bump API_FEATURE_LEVEL to 74.
2021-06-05 07:37:19 -07:00
Gaurav Pandey 18ef0c3bc1 api: Expose server_needs_upgrade in register response.
Expose the boolean value server_needs_upgrade in the
responses for register api so that it can be used
by mobile and terminal clients as well.
Highlighted in api changelog as part of
feature level 74 in commit fb93c96
(next commit).
2021-06-05 07:33:25 -07:00
Gaurav Pandey f82aba5a3d compatibility: Shift functions to new module.
Shift functions used for compatibility from
zerver.lib.home (is_outdated_server) and
zerver.view.compatibility (pop_numerals,
version_lt, find_mobile_os,
is_outdated_desktop_app, is_unsupported_browser)
to zerver.lib.compatibility module.
2021-06-05 07:33:25 -07:00
Suyash Vardhan Mathur 44c25619d7 curl_examples: Add testing for template endpoints.
This commit adds support for testing of
those endpoints whose .md files would
be deleted in favour of their pages
to be generated automatically by the template.

curl examples for such endpoints must exist
in accordance to the pattern of template, which
can be used to run the tests for them.
2021-06-04 09:43:24 -07:00
Suyash Vardhan Mathur 93cbe6f175 curl_examples: Refactor testing code.
The list curl_commands_to_test gets filled
already and so, the code to test each command
in the list can be moved out of the block that
opens the file. The only change in this commit
is reducing an indentation for the entire block.

This has been done to reuse the whole block
in case the file does not exist.
2021-06-04 09:43:24 -07:00
Suyash Vardhan Mathur 2359ddd402 curl_examples: Refactor testing code.
This commit refactors the code to find the
lines to be first, and then test all the
lines that contained commands.

This was done to avoid duplication of
the code for the other case, where the
.md file won't exist, as those are soon
to be deleted in favour of a common template.
2021-06-04 09:43:24 -07:00
Suyash Vardhan Mathur 64df949f7a curl_examples: Use endpoint list from rest-endpoints.
Soon, the .md files of each endpoint would
be removed and be auto-generated from OpenAPI
data.
So, instead of using the files directly,
we should check from the list of endpoints and
open the files from there.

A follow-up to change logic when the .md files
get deleted should be done.
2021-06-04 09:43:24 -07:00
Abhijeet Prasad Bodas 86d6872a80 onboarding: Select Message FOR UPDATE when adding reaction.
Although race conditions are super-unlikely for these onboarding
messages, it's better to write correct code since it might be copied.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 90b6fa7881 reactions: Select Message FOR UPDATE when adding/removing.
This locks the message row while a reaction is being added/removed,
which will handle race conditions caused by deleting the message
at the same time.

We make sure that events work happens outside the transaction,
so that in case there's some problem with the queue processor, the
locks aren't held for too long.

As a nice side-effect, we also handle race conditions from double
adding reactions, because once the message is locked, a duplicate
request will wait till the earlier transaction commits, and hence
will not throw `IntegrityErrors`s (rather, will be handled in our
safety check in the /views code itself), which earlier had to be
handled explicitly.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas efc2f49e3c submessage: Select Message FOR UPDATE when creating.
This locks the message while creating a submessage, which
will handle race conditions caused by deleting the message
simultaneously.

We make sure that events work happens outside the transaction,
so that in case there's a problem with the queue processor,
the locks aren't held for too long.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 5f4113cf60 message delete: Select Message FOR UPDATE when archiving.
Further commits will start locking the message rows while
adding related fields like reactions or submessages,
to handle races caused by deleting the message itself at the
same time.

The message locking implemented then will create a possibility
of deadlocks, where the related field transaction holds a lock
on the message row, and the message-delete transaction holds a
lock on the database row of the related field (which will also
need to be deleted when the message is deleted), and both
transactions wait for each other.

To prevent such a deadlock, we lock the message itself while
it is being deleted, so that the message-delete transaction
will have to wait till the other transaction (which is about
to delete the related field, and also holds a lock on the
message row) commits.

https://chat.zulip.org/#narrow/near/1185943 has more details.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 1a9f385e17 access_message: Allow selecting message row FOR UPDATE.
This is a prep change to start using `SELECT FOR UPDATE` queries
when there is a chance of race conditions.
2021-06-04 08:18:16 -07:00
Abhijeet Prasad Bodas 51f5bbcd57 test_events: Run on_commit send_event hooks immediately.
Further commits will hook some `send_event` calls to `on_commit`.
With those changes, these will never be executed in tests, because
transactions never get commited with `TestCase`, which the
`ZulipTestCase` is a subclass of.

We want to make sure that these events are actually sent for testing
purposes, hence this change.

There's no need to actually capture the callbacks, because the
events are already thoroughly tested.
2021-06-04 18:12:39 +05:30
Abhijeet Prasad Bodas 60464a4cea tests: Run on_commit hooks immediately in tornado_redirected_to_list.
Further commits will hook `send_event` calls to `on_commit`
in some cases. This change will make it easier to test such
situations.

We don't need to actually capture the callbacks, because the
events sent are already tested via the list in which they are
captured by `tornado_redirected_to_list`.
2021-06-04 18:12:39 +05:30
Abhijeet Prasad Bodas 8b14704d35 test_classes: Fix misplaced `tornado_redirected_to_list` comment. 2021-06-04 18:12:21 +05:30
Tim Abbott b920929182 lint: Fix line-wrapping for a recently merged test change. 2021-06-03 18:52:39 -07:00
sahil839 bf9c17e8a8 message: Fix moving messages between streams for non-admins.
This commit fixes a bug where moving messages between streams was
not allowed for non-admins when allow_community_topic_editing was
set to false and move_messages_between_streams_policy was set to
Realm.POLICY_MEMBERS_ONLY.

The bug is fixed by calling can_edit_content_or_topic only when
topic or content edit is there and not in the case where only
message is moved from one stream to another.
2021-06-03 17:53:59 -07:00
sahil839 db294cb386 message: Extract logic to check message edit permissions in a function.
This commit extracts the logic of checking the message edit permissions,
like whether the sender is same as user, whether it is a (no topic)
message or whether community topic editing is allowed, into a separate
function.

This is a prep commit for fixing a bug where permission to move messages
between streams is affected by permission of editing topics.
2021-06-03 17:53:55 -07:00
sahil839 6be232aa42 message_edit: Check 'topic_name is not None' and not 'content is None'.
Previously when enforcing the check to do not allow editing topics
after a certain time, we were checking whether 'content is None' and
considering it as that if content is None then there must be topic
edit.

But after adding support for moving messages between streams it can be
the case that we are neither changing topic nor content and just moving
streams, and the original code raises error if this is done after the
time limit of editing topics, which is wrong.

This commit fixes this by actually checking 'topic_name is not None'.
2021-06-03 17:13:32 -07:00
sahil839 e2835d3f4f message_edit: Modify the error message for topic edit deadline.
This commit modifies the error message shown when topic edit
deadline is passed to make it more clear that the limit is for
editing message's topic.
2021-06-03 17:13:32 -07:00
Suyash Vardhan Mathur 94a22f9323 apidocs: Modify tests to reflect new templates.
Soon, each endpoint won't necessarily have a .md
file, but would generate API doc directly from
OpenAPI data using a template.

So, the lists of endpoints to be tested should not
be taken from the .md files, but from the REST
endpoints available in the sidebar.

This commit also adds a missing test for an invalid
article being accessed in the URL of an API page.
2021-06-03 10:52:22 -07:00
Suyash Vardhan Mathur d1ccf15cd3 apidocs: Fix invalid API page bug.
The current logic to get API pages' title using
OperationID should be used when the first line
of the file explicitly mentions so.

In cases where the files didn't begin with `#` but also
didn't need to get title from OpenAPI summary,
this logic fails and causes Server error.
This particularly happens when the article is invalid,
and the `missing.md` file doesn't need title to be
generated, but doesn't start with `#` either.

This commit fixes the logic of using the generated title and covers the bug.
2021-06-03 10:47:47 -07:00
Vishnu KS 5db53029a5 api: Include is_billing_admin as an attribute in user response.
This is sufficiently useful that it should be made available to clients.
2021-06-03 10:27:07 -07:00
Vishnu KS 3d5ee69b21 stripe: Create make_user_billing_admin helper function. 2021-06-03 10:13:59 -07:00
Wesley Aptekar-Cassels d5ba94082a markdown: Increase max rendered message length to 1MB.
This should help with #17425, where messages with lots of LaTeX are
lost, due to the large expansion factor.

This isn't a total fix for this - large messages with lots of LaTeX
can still end up larger than 1MB, and rendering could timeout, but
this fix should help significantly.

1MB is still small enough that I don't expect we'll run into any DOS
problems - my testing didn't show any problems rendering messages that
contain ~1MB of LaTeX.
2021-06-03 10:10:35 -07:00
seiwailai b584790541 validator: Add generic check_or.
Fixes part of #17914. Added generic check_or function and tests.
2021-06-03 09:49:50 -07:00
Alex Vandiver 4525543ec6 delete_realm: Disallow deletion of realms we've had billing plans with. 2021-06-03 09:38:12 -07:00
akshatdalton 7df62ebbaf settings: Make `MAX_MESSAGE_LENGTH` a server-level setting.
This will offer users who are self-hosting to adjust
this value. Moreover, this will help to reduce the
overall time taken to test `test_markdown.py` (since
this can be now overridden with `override_settings`
Django decorator).

This is done as a prep commit for #18641.
2021-06-03 09:26:28 -07:00
akshatdalton 6143cb6e73 test_markdown: Use assertTrue/assertFalse instead of assertEqual. 2021-06-02 17:20:45 -07:00
Gilbert Bishop-White e4d9e58c7d emails: Make it obvious when registering creates new realm.
Checked the email looked OK in `/emails` for both creating realm and
registering within an existing one.

Not sure zerver/tests/test_i18n.py test has been suppressed correctly.

Fixes #17786.
2021-06-02 15:18:03 -07:00
Gaurav Pandey f67fc37f7d settings: Rename MAX_EMOJI_FILE_SIZE.
Rename MAX_EMOJI_FILE_SIZE to
MAX_EMOJI_FILE_SIZE_MIB reflecting
the size in mebibytes.
2021-06-02 13:55:15 -07:00
Gaurav Pandey 73059dd0c7 settings: Rename MAX_LOGO_FILE_SIZE.
Rename MAX_LOGO_FILE_SIZE to
MAX_LOGO_FILE_SIZE_MIB reflecting
the size in mebibytes.
2021-06-02 13:55:15 -07:00
Gaurav Pandey 2ddc62e9b3 settings: Rename MAX_ICON_FILE_SIZE.
Rename MAX_ICON_FILE_SIZE to
MAX_ICON_FILE_SIZE_MIB reflecting
size in mebibytes.
2021-06-02 13:55:15 -07:00
Gaurav Pandey 0f6bb210a6 settings: Rename MAX_AVATAR_FILE_SIZE.
Rename MAX_AVATAR_FILE_SIZE to
MAX_AVATAR_FILE_SIZE_MIB reflecting
size in mebibytes.
2021-06-02 13:55:15 -07:00
Alex Vandiver 66f77d3b4b send_test_email: Capture and show SMTP log on errors. 2021-06-02 13:18:11 -07:00
PIG208 2ee00972a0 doc: Change supported extension type to reflect the change. 2021-06-02 09:31:38 -07:00
Tim Abbott 05d4c070f4 api docs: Clarify topic and stream_id fields of message edit API. 2021-06-02 08:53:02 -07:00
Suyash Vardhan Mathur db4b674270 apidocs: Fix description of topic parameter in update_message.
The information of topic being required for stream messages
in update_message OperationID wasn't true. Fixed the
incorrect description.
2021-06-02 08:38:00 -07:00
Abhijeet Prasad Bodas 2489a1bb4c test_event_queue: Don't use lists for call_args assertions.
This is a bit hacky, but will make these tests more readable,
in that the reader would not have to remember the order or parameter
names.

Python 3.8 introduced `mock.call_args.kwargs`, and once we upgrade,
we can use those to assert actual dictionaries instead of this hack.
2021-06-01 15:26:49 -07:00
Abhijeet Prasad Bodas f88b943ff4 email_notifications: Remove unnecessary `online_emails` check.
We already check this in `handle_missedmessage_emails` through
`receives_offline_email_notifications`.
2021-06-01 15:26:49 -07:00
Abhijeet Prasad Bodas 44534ca47e refactor: Move receives_email_notifications tests to designated file. 2021-06-01 15:26:49 -07:00
Abhijeet Prasad Bodas 3990b183ce models: Remove unused `receives_stream_notifications` function.
This was introduced in c3a8138f74, but
doesn't have any callers, apart from it's own tests.
2021-06-01 15:26:49 -07:00
Abhijeet Prasad Bodas 518deb7b9e models: Rename `receives_online_notifications` function.
Prep for later when we will have a similar setting for
online email notifications.
2021-06-01 15:26:49 -07:00
Mateusz Mandera 932c846454 custom_fields: Fix confusing comments for do_update_..._if_changed.
d66cbd2832 added these mentioning
"always_notify" for some reason, but always_notify clearly isn't a real
thing in this context so the comments need to be fixed to eliminate this
potential source of confusion.
2021-05-28 15:51:28 -07:00
Mateusz Mandera e17758f8ad auth: Implement a generic OpenID Connect backend.
Fixes #11939.
2021-05-28 09:43:06 -07:00
Abhijeet Prasad Bodas 7a4c212dff test_subs: Add comments about num_events.
This is a bit complex, because many events not directly related
to what we are testing are also sent during these operations, hence
these comments.
2021-05-28 09:42:14 -07:00
Abhijeet Prasad Bodas aa2fd1593e tornado_redirected_to_list: Fix event list leaks in tests.
Ideally this should have been part of
481a890ec5.
2021-05-28 09:42:14 -07:00
Abhijeet Prasad Bodas 3039a01b31 tornado_redirected_to_list: Make expected_num_events required argument.
Follow up to 481a890ec5.
This will make this more explicit and readable.
2021-05-28 09:42:14 -07:00
Adam Birds ee375e2999 integrations: Add Freshstatus webhook integration. 2021-05-27 23:38:11 -07:00
Tim Abbott 9f2daeee45 upload: Use get_public_upload_url for export tarballs too.
This deduplicates the code so that we now just have one function for
constructing S3 URLs.
2021-05-27 23:26:45 -07:00
ryanreh99 5a4aecfc40 s3 uploads: Refactor to access objects via `get_public_upload_url`.
Our current logic only allows S3 block storage providers whose
upload URL matches with the format used by AWS. This also allows
other styles such as the "virtual host" format used by Oracle cloud.

Fixes #17762.
2021-05-27 23:26:42 -07:00
Abhijeet Prasad Bodas 15f78abd68 message edit: Handle topic edit tries for private messages.
Fixes #18604.
2021-05-27 23:09:33 -07:00
Abhijeet Prasad Bodas 4b30fc01e4 message edit: Extract data validity checks from check_update_message.
These checks are more related to the API than the editability
or permissions logic, so it makes sense to handle them first
before further processing the request.
Also split the main test class to separate out the tests for
this logic.

This also simplifies some tests by reducing the data setup
required to reach failure.

Tweaked by tabbott to avoid losing the topic_name.strip().
2021-05-27 23:07:59 -07:00
Abhijeet Prasad Bodas 10be798a3c message edit: Extract test helpers in their own class.
This is mostly a direct code move and a prep for splitting the
main MessageEdit test class into smaller chunks.
2021-05-27 22:52:55 -07:00
Abhijeet Prasad Bodas 3d99292b4c message edit: Fix `stream_id` description in API docs.
Since caa08d76b5, we no longer have
a common component for stream IDs in zulip.yaml, so we might as
well change the description to be specific and clear.
2021-05-27 22:52:55 -07:00
Alex Vandiver 4ffda1be87 send_email: Fix sleep logic.
This was broken in the refactor in 1e67e0f218.
2021-05-27 22:49:28 -07:00
Alex Vandiver b9948994d7 send_email: Increase loglevel of stuck ScheduledEmail messages to error.
Since these stay in the queue indefinitely, they deserve to be raised
at a higher error than just warning.
2021-05-27 22:49:28 -07:00
Alex Vandiver ff9126ac1e data_import: Protect better against bad Slack tokens.
An invalid token would be treated the same as a token with no scopes;
differentiate these better.
2021-05-27 22:46:58 -07:00
Alex Vandiver 94e4f33b29 data_import: Support importing from Slack conversions in a directory.
Sometimes the Slack import zip file we get isn't quite the canonical
form that Slack produces -- often because the user has unzip'd it,
looked at it, and re-zip'd it, resulting in extra nested directories
and the like.

For such cases, support passing in a path to an unpacked Slack export
tree.
2021-05-27 22:46:58 -07:00
Alex Vandiver 8228ea2a17 import_data: Do some quick verification of Slack import formats. 2021-05-27 22:46:58 -07:00
Suyash Vardhan Mathur 6107679f61 minor: Fix typo in summary.
The summary for add_code_playground wrongly mentions
'Remove' instead of 'Add' in the summary. Fixed the typo.
2021-05-27 16:56:53 -07:00
Abhijeet Prasad Bodas c19b7b4db3 tests: Use tornado_redirected_to_list instead of mocking send_event. 2021-05-27 13:16:22 -07:00
Abhijeet Prasad Bodas 090f2d6664 test_helpers: Remove capture_event helper.
We now consistently use the `tornado_redirected_to_list`
helper, which asserts the number of events sent too.
2021-05-27 13:16:22 -07:00
Abhijeet Prasad Bodas d2bc93f6dd tests: Extract helper in test_user_status.
This deduplicates some common code for all these scenarios.
2021-05-27 13:16:22 -07:00
Abhijeet Prasad Bodas 481a890ec5 tests: Assert num_events in tornado_redirected_to_list. 2021-05-27 13:16:22 -07:00
Anders Kaseorg 0ce015348e actions: Fix incorrect audit logging in bulk_remove_subscriptions.
modified_user=sub_info.user and modified_stream=sub_info.stream, added
by commit 6d1f9de7d3 (#16553), were
always coming from the last entry in the loop above, not from the
enclosing list comprehension.

Found by the Pylint rule undefined-loop-variable.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-26 18:12:03 -07:00
Suyash Vardhan Mathur 52b9c96c5d apidocs: Add support for using OpenAPI title for /api pages.
The current logic of geneerating HTML titles requires the title to be
present as a heading in the first line of .md file. However, this will
shortly be no longer true for /api pages where these are
auto-generated from OpenAPI data.  Modified the code to fetch the
title from OpenAPI data in case of such pages.
2021-05-26 17:38:01 -07:00
Suyash Vardhan Mathur 58a8009786 apidocs: Add markdown processor for title.
Currently, the title of each endpoint are hardcoded in .md
files, but these are also added in summary parameter of
openapi. Added a markdown processor to insert the title
of a given endpoint.
2021-05-26 17:38:01 -07:00
Suyash Vardhan Mathur 94c4e22898 apidocs: Add summary field to endpoints.
Some pages did not have .md files, and so the summaries for these
was newly written. Commit split for easy review.
2021-05-26 17:38:01 -07:00
Suyash Vardhan Mathur e15af5ae14 apidocs: Add summary field to endpoints.
As a part of effort of removing .md files for /api pages, the
title of each page can be added into the summary OpenAPI
parameter. This also adds a nice summary visible in Swagger
and enhances the documentation. Added the parameter for all endpoints.
2021-05-26 17:38:01 -07:00
Suyash Vardhan Mathur b41be98c42 apidocs: Fix playground URLs and operationIDs.
We standardized on "Code playground" for this feature, so use that
in the URLs and API documentation.
2021-05-26 17:38:00 -07:00
akshatdalton 832c763c38 minor: Remove unnecessary `__init__` method in `InlineInterestingLinkProcessor`.
Subclass `Treeprocessor` takes care of the `__init__` method.
2021-05-26 17:13:03 -07:00
Adam Birds 303bb61f9c integrations: Add Sonarr Integration. 2021-05-26 17:12:28 -07:00
Adam Birds 17fe2e4a08 integrations: Add Radarr inegration. 2021-05-26 17:11:57 -07:00
Adam Birds 11e7f27031 integrations: Add Lidarr integration. 2021-05-26 17:11:24 -07:00
Abhijeet Prasad Bodas f236a0d10d message send: Rename `always_push_notify` -> `online_push_enabled`.
This is a better name, since it clearly denotes a user
configured setting.
2021-05-26 15:19:32 -07:00
Abhijeet Prasad Bodas ddd123f133 message send: Rename `push_notify_user_ids` -> `online_push_user_ids`.
The old name `push_notify_user_ids` was misleading, because
it does not contain user ids which should be notified for
the current message, but rather user ids who have the online
push notifications setting enabled.

When the Tornado server is restarted during an upgrade, if
server has old events with the `push_notify_user_ids` fields,
the server will throw error after this rename. Hence, we need
to explicitly handle such cases while processing the event.
2021-05-26 15:13:08 -07:00
sahil839 8ec9987999 emails: Remove referrer name from 'From' header in invitation emails.
We should only show the referrer name in subject of invitation emails,
and show only 'Zulip' in the 'From' header. This helps in preventing
the email from being marked as suspicious by the detection systems
when they see an employee's name as sender of an email sent from an
unrelated domain.

The behavior is already the same for reminder invitation emails where
we do not show name and only 'Zulip' in the 'From' header.

Fixes #18256.
2021-05-26 11:41:13 -07:00
Gaurav Pandey 64bd461bad register api: Rename realm_upload_quota.
Rename `realm_upload_quota` to `realm_upload_quota_mib`
reflecting file size in mebibytes. Update frontend and bump
API_FEATURE_LEVEL.
2021-05-26 11:38:24 -07:00
Gaurav Pandey 1da818f848 register api: Rename max_logo_file_size.
Renaming `max_logo_file_size` to `max_logo_file_size_mib`
reflecting the file size in mebibyte and update the frontend.
2021-05-26 11:38:24 -07:00
Gaurav Pandey 531cf041f1 register api: Rename max_icon_file_size.
Renaming `max_icon_file_size` to `max_icon_file_size_mib`
reflecting the file size in mebibyte and update the frontend.
2021-05-26 11:38:24 -07:00
Gaurav Pandey 38614bbaec events api: Delete incorrect parameter in response.
max_logo_file_size isn't included in the responses
of GET /events api but is mentioned in the api
documentation. Hence it is deleted.
2021-05-26 11:38:24 -07:00
Mateusz Mandera 496e744053 queue_processors: Log more detailed info when marking messages as read. 2021-05-26 11:17:21 -07:00
Adam Birds c7421cd69c integrations: Add `label_create_activity` to unsupported pivotal events.
Fixes #18580.
2021-05-25 20:56:52 -07:00
Mateusz Mandera 0e42a3f117 tests: Rename helper method to generate_access_token_url_payload.
generate_access_url_payload was not a good name - there's no such thing
as "access_url" in this context - access_token_url is what was meant.
2021-05-25 14:22:46 -07:00
Mateusz Mandera 2e2bcd9619 apple: Rewrite comment in tests in generate_access_url_payload.
The original comment is worded rather unclearly, we should explain these
details better.
2021-05-25 14:22:46 -07:00
Anders Kaseorg bac96cae80 markdown: Fix Dropbox image previews.
?dl=1 causes Dropbox to send Content-Type: application/binary, which
can’t be interpreted by Camo.  Use ?raw=1 instead.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-25 13:42:29 -07:00
Sumanth V Rao 98253f261e models: Fix bug in unique_together condition on RealmPlayground.
We don't need to worry about breaking already configured playgrounds
since this tweak makes the condition less strict.
2021-05-24 23:11:41 -07:00
Anders Kaseorg 276d95b58b actions: Fix “Fix UserActivityInterval overlap bug” bug.
Commit 1a7ddd9ea3 “Fix
UserActivityInterval overlap bug” introduced a mathematically
incorrect assertion about how intervals work.  There’s a third way two
intervals could overlap: both the start and end of the old interval
could be inside the new interval.  This probably can’t happen here
because the old interval should be at least as long as the new
interval.  However, a correct overlap test can be formulated in a
simpler way anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-24 18:06:05 -07:00
Abhijeet Prasad Bodas 8bcdbc720e tests: Move tornado_redirected_list to ZulipTestCase.
This is will make it easier to systematically use Django's
`capturOnCommitCallbacks` in tests outside of the main
`test_events` file which involve assertions on events.
2021-05-24 17:45:13 -07:00
AdamVB a9e73358bf
integrations: Enhance Grafana integration with alert state.
Having the alert state in the message body is useful when alert topics 
are not defined by alert description but encoded in the url.

E.g. in large environments having a topic for each alert [alerting] and [ok] would 
make it harder to properly track if an alert has been resolved.

When each alert is in a single topic, so far, the alert state has been missing.

This change will add the current alert state and a fitting icon in front
of the alert name.(Similar to the prometheus alertmanager integration)

The test cases have been amended to cover all possible alert states, even
though realistically grafana only fires the ok and alerting states via
webhook.
2021-05-24 14:25:33 -07:00
akshatdalton 503247ebfa refactor: Add class `CompiledInlineProcessor` to de-duplicate code. 2021-05-23 14:30:22 -07:00
akshatdalton 78f26b6031 minor: Use `super` to initialize subclass. 2021-05-23 14:30:22 -07:00
sahil839 8424a82dec setting: Add moderator option in email visibility setting in backend. 2021-05-23 14:20:20 -07:00
sahil839 43cdbdbf9d tests: Move a part of email_address_visibility tests to test_users.py.
We should only keep tests for changing email_address_visibility in
test_realm.py. The tests for checking the value of delivery_email
and email in the user dicts returned by 'GET users/{user_id}'
endpoint according to email_address_visibility should be in
test_users.py and not test_realm.py.

The tests of other realm settings are also arranged in the same way.

This commit makes the test more robust by checking for all
possible values of email_address_visibility and checking
emails and delivery emails values received by different
user roles.
2021-05-23 14:20:20 -07:00
Mateusz Mandera cd88a6e6fb tests: Rename APPLE_JWK to a generic name for re-use in other tests.
This can be re-used for the JWK in OIDC implementation in the follow-up
commits.
2021-05-23 13:30:17 -07:00
Mateusz Mandera d17a4f64d6 registration: Return status code 404 if invalid confirmation key passed.
We were returning 200 before - 404 is more appropriate.
2021-05-20 17:29:09 -07:00
Mateusz Mandera 40169353ad auth: Validate multiuse invite link confirmation key in social auth. 2021-05-20 17:29:09 -07:00
Aman Agrawal cba766f66f do_make_stream_web_public: Update all affected fields.
Convert this function that absolutely makes a stream web public.
We already have do_change_stream_invite_only to convert
streams to public and private streams.

We also update all the fields that should be set when a stream
is made web public.
2021-05-20 15:23:19 -07:00
Aman Agrawal 70c0abc2e5 do_change_stream_invite_only: Ensure stream is not web public.
When changing stream permissions to invite_only or public, ensure
that stream doesn't have is_web_public set to True.
2021-05-20 15:23:19 -07:00
Tim Abbott af2b1ef3a1 page_params: Remove duplicate debug_mode field.
This had the same value as development_environment, I believe in a way
likely to persist long term, and was less clear about its meaning.
2021-05-20 14:57:34 -07:00
Tim Abbott 2e67b879ed api: Add server_timestamp to register response.
Since this is currently only useful to interpret presence data, we
send this only if presence is requested.

I'm not sure that server_timestamp is the right name for this field,
but ultimately it should match the main presence API format.
2021-05-20 14:57:34 -07:00
Abhijeet Prasad Bodas ec8a931761 message send: Pass individual parameters instead of single Dict.
This will allow for stronger type checking and better readability.
2021-05-20 11:06:19 -07:00
Abhijeet Prasad Bodas d6871dbae5 message send: Do not re-assign widget_content variable.
After re-assignment, mypy will still think the type of
`widget_content` to be `str`, not `Dict`. So we need to
create a new variable.
This is a prep change for stronger type checking in this
code.
2021-05-20 11:06:19 -07:00
Abhijeet Prasad Bodas 334db60a07 message send: Fix incorrect field types in SendMessageRequest class.
The `local_id` and `queue_id` parameters are actually accepted as strings,
not integers, by the API.
This was introduced in 2fa33be683.
2021-05-20 11:06:19 -07:00
Tim Abbott 6edd78a9e5 events: Reorganize realm fields code in page_params.
This also adds a few comments on things that look to be bugs.
2021-05-19 15:35:30 -07:00
Tim Abbott ea7fee11b2 events: Add block comment documenting naming conventions.
Fixes part of #14166.
2021-05-19 15:28:46 -07:00
Tim Abbott 323152c5fd send_custom_email: Never email users who haven't agreed to ToS.
The `create_user` API and data import tools can result in our having
active users in the database who haven't intentionally created a Zulip
account or agreed to the ToS; we should never email such users.

The check for `TOS_VERSION is not None` is necessary for the
development environment, which has `TERMS_OF_SERVICE` set but not
`TOS_VERSION`.

It's likely that we will want this check in other places as well.
2021-05-19 13:47:36 -07:00
Abhijeet Prasad Bodas 352634a851 tests: Consistently use assert_length helper.
This helper does some nice things like printing out
the data structure incase of failure.
2021-05-19 11:55:56 -07:00
Abhijeet Prasad Bodas 4d45b45640 test_classes: Generalize assert_length helper.
Make it so that `assert_length` can be used for not
just lists but all `Collections`.

This is prep for using this helper consistently for
all tests.
2021-05-19 11:55:56 -07:00
Tim Abbott 3b181968aa settings: Fix setting JITSI_SERVER_URL to None.
This fixes a bug introduced in
55a23754c3, that resulted in Zulip
crashing on startup if JITSI_SERVER_URL=None.

Fixes #18512.
2021-05-18 19:16:47 -07:00
Tim Abbott 80a30c83e5 home: Remove unused root_domain_uri parameter. 2021-05-18 17:36:55 -07:00
Tim Abbott e1c885aeb9 home: Remove unused has_mobile_devices parameter.
This parameter has never been used, and causes an unnecessary database
query.

We keep the num_push_devices_for_user function, since we may have uses
for it down the line.

Fixes part of #14166.
2021-05-18 17:36:55 -07:00
rht aa3cdfa2e9 Fix typos using codespell.
The command:
codespell --skip='./locale,*.svg,./docs/translating,postgresql.conf.template.erb,.*fixtures,./yarn.lock,./docs/THIRDPARTY,./tools/setup/emoji/emoji_names.py,./tools/setup/emoji/emoji_map.json,./zerver/management/data/unified_reactions.json' --ignore-words=codespell_ignore_words.txt .

The content of codespell_ignore_words:
```
te
ans
pullrequest
ist
cros
wit
nwe
circularly
ned
ba
ressemble
ser
sur
hel
fpr
alls
nd
ot
```
2021-05-18 17:33:51 -07:00
akshatdalton 18203d8af3 markdown: Silence user group mention inside blockquotes. 2021-05-18 17:31:25 -07:00
akshatdalton 0245b590e9 markdown: Add support for user group silent mention.
Prior to this, we only supported direct mention to
the user groups. This commit extends that support
to silent mention for the user groups.
A related test case is also added.

Fixes: #11711.
2021-05-18 17:31:25 -07:00
akshatdalton f56fca308a mention: Refactor `USER_GROUP_MENTIONS_RE` and simplify its related code path.
Earlier, USER_GROUP_MENTIONS_RE was:
r"(?<![^\s\'\"\(,:<])@(\*[^\*]+\*)"

For the syntax: *foo*, this was unnecessarily capturing it as
*foo* and the extraction of `foo` was done using another helper
function: `extract_user_group`.

This is now changed as:
r"(?<![^\s\'\"\(,:<])@(\*(?P<match>[^\*]+)\*)"

and extraction of `foo` can be done just by using the named capture
group `match`.

This change also helps to simplify its related code path.
2021-05-18 17:31:25 -07:00
akshatdalton d5a36ac5e2 mention: Refactor `MENTIONS_RE` and simplify its related code path.
Earlier, MENTIONS_RE was:
r"(?<![^\s\'\"\(,:<])@(?P<silent>_?)(?P<match>\*\*[^\*]+\*\*)"

For the syntax: **foo**, this was unnecessarily capturing it as
**foo** and adding extra operation for the extraction of `foo`.

This is now changed as:
r"(?<![^\s\'\"\(,:<])@(?P<silent>_?)(\*\*(?P<match>[^\*]+)\*\*)"

and extraction of `foo` can be done just by using the named capture
group `match`.

This change also helps to simplify its related code path.
2021-05-18 17:31:25 -07:00
akshatdalton 738a676534 mention: Remove deprecated wildcard mention pattern from `MENTIONS_RE`.
Earlier wildcard mentions were used as: @all, @everyone, @stream.
This syntax is deprecated and we will no longer support
this syntax in future. See the commits:

1. 7a4c3c1a5c
2. b650b6b38c

When we started to use these syntaxes for wildcard mentions.
2021-05-18 17:31:25 -07:00
akshatdalton a9d89b3c56 minor: Convert `unicode_emoji_regex` to uppercase.
Following the convention, we use uppercase for
regex. Also, `unicode_emoji_regex` is given a
conventional name ending with `*_RE`: `UNICODE_EMOJI_RE`.
2021-05-18 17:31:25 -07:00
akshatdalton ffc4724287 minor: Convert `emoticon_regex` to uppercase.
Following the convention, we use uppercase for
regex. Also, `emoticon_regex` is given a
conventional name ending with `*_RE`: `EMOTICON_RE`.
2021-05-18 17:31:25 -07:00
akshatdalton 9f6e6709d3 minor: Convert `user_group_mentions` to uppercase.
Following the convention, we use uppercase for
regex. Also, `user_group_mentions` is given a
conventional name ending with `*_RE`: `USER_GROUP_MENTIONS_RE`.
2021-05-18 17:31:25 -07:00
akshatdalton 0a01b1b28e minor: Convert `find_mentions` to uppercase.
Following the convention, we use uppercase for
regex. Also, `find_mentions` is given a
conventional name ending with `*_RE`: `MENTIONS_RE`.
2021-05-18 17:31:25 -07:00
Alex Vandiver 670c7e7ba4 settings: Remove now-unnecessary EMAIL_DELIVERER_DISABLED setting. 2021-05-18 12:39:28 -07:00
Alex Vandiver 1e67e0f218 deliver_scheduled_*: SELECT FOR UPDATE the relevant rows.
`deliver_scheduled_emails` and `deliver_scheduled_messages` use their
respective tables like a queue, but do not have guarantees that there
was only one consumer (besides the EMAIL_DELIVERER_DISABLED setting),
and could send duplicate messages if multiple consumers raced in
reading rows.

Use database locking to ensure that the database only feeds a given
ScheduledMessage or ScheduledEmail row to a single consumer.  A second
consumer, if it exists, will block until the first consumer commits
the transaction.
2021-05-18 12:39:28 -07:00
Alex Vandiver 82797dd53c settings: Standardize the name of the deliver_scheduled_messages logs.
This makes it match its command name, and other logfile name.
2021-05-18 12:39:28 -07:00
Vishnu KS e019045abe actions: Create send_message_to_signup_notification_stream. 2021-05-14 15:10:02 -07:00
Anders Kaseorg e015f3ed7d docs: Correct “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-14 13:15:38 -07:00
Tim Abbott 73123300e3 models: Improve documentation for UserMessage. 2021-05-14 10:48:00 -07:00
Tim Abbott 6e9683fe6f test_compatibility: Mock desktop minimum versions.
This fixes the tests failing since
454eced466, in a way that won't require
us to change these tests for future version bumps.
2021-05-13 13:03:42 -07:00
Anders Kaseorg 668b5137b0 version: Display Zulip version in About Zulip dialog.
We record Git details about the merge-base with upstream branches in
the zulip-git-version file, if the upstream repository is available.

Note that the first Git upgrade after merging the parent commit will
not include the merge-base details, since the upstream repository will
not have been available.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 11:36:12 -07:00
Tim Abbott b829b95108 email_mirror: Remove unnecessary EMAIL_GATEWAY_PATTERN logging.
These logs were pretty spammy, and there have long been much better
ways to communicate to system administrators that the incoming email
gateway is great, including, most importantly, in the section of the
emails themselves that explains how replying works.
2021-05-13 11:17:01 -07:00
sahil839 213eda1f32 message: Check stream_post_policy when moving messages between streams.
Previously only admins were allowed to move messages between streams
and admins are allowed to post in any stream irresepctive of stream
post policy, so there was no need to check for stream post policy.

But as we now allow other members to also move messages, we need
to check whether the user who is moving the message is allowed
to post to the target stream (i.e. stream to which the messages
are being moved) and thus we allow moving messages only if the
user is allowed to post in target stream.
2021-05-13 08:42:24 -07:00
Mateusz Mandera f1cf9f740d tests: Add test for bulk_access_messages scenarios in message moving. 2021-05-13 08:42:24 -07:00
sahil839 b52ad3e536 message_edit: Allow moving message to stream based on setting value.
We allow the users to move message between streams according to the
value of 'move_messages_between_streams_policy'.
2021-05-13 08:40:57 -07:00
Mateusz Mandera de6bd22ee9 outgoing_webhook: Treat "" json in response as response_not_required.
b7b1ec0aeb made our checks of the response
format stronger, to enforce that the json translates to a valid dict.
However, old client code (zulip_botserver) was using "" as equivalent to
response_not_required - so we need to keep backward-compatibility to not
break things built on it.
2021-05-13 08:07:38 -07:00
Tim Abbott 41d499d44c message_edit: Require access to messages to move between streams.
Currently, moving messages between streams is an action limited to
organization administrators. A big part of the motivation for that
restriction was to prevent users from moving messages from a private
stream without shared history as a way to access messages they should
not have access to.

Organization administrators can already just make the stream have
shared history if they want to access its messages, but allowing
non-administrators to move messages between would have
introduced a security bug without this change.
2021-05-12 16:23:22 -07:00
Tim Abbott 0bfef96543 bulk_access_messages: Bulk fetch Subscription details.
This completes the effort to make it possible to use
bulk_access_message in contexts where there are more than a handful of
messages without creating performance issues.
2021-05-12 16:23:22 -07:00
Tim Abbott c6e1702335 bulk_access_messages: Support passing in a stream for all messages.
This lets us avoid fetching the Stream in a loop.
2021-05-12 16:23:22 -07:00
Tim Abbott 4a8dcf14ef bulk_access_messages: Use a bulk UserMessage query.
This removes one of the three queries that runs in a loop with this
bulk access function.
2021-05-12 16:23:22 -07:00
Tim Abbott ee780e1f08 message: Test queries used by bulk_access_messages. 2021-05-12 16:23:22 -07:00
Tim Abbott c84ea01869 message: Refactor has_message_access parameters. 2021-05-12 16:23:22 -07:00
Tim Abbott 7ef0d21fc2 message_edit: Pass old_stream to update_messages_for_topic_edit.
We'll need this for checking access to moved messages.
2021-05-12 16:23:22 -07:00
Tim Abbott f78e604868 message_edit: Pass acting_user to update_messages_for_topic_edit.
We'll need for checking access if non-administrators can move topics.
2021-05-12 16:23:22 -07:00
Gaurav Pandey dada7bb4bb tests: Clean up tests for realm_domain and update_streams endpoint.
Cleaning up test_realm_domains.RealmDomainTest.test_list_realm_domains,
test_subs.StreamAdminTest.test_private_stream_live_updates,
test_subs.StreamAdminTest.test_realm_admin_can_update_unsub_private_stream
and test_subs.StreamAdminTest.test_non_admin_cannot_access_unsub_private_stream.
2021-05-12 12:38:16 -07:00
Steve Howell a518cea65d performance: Use values_list for message edits. 2021-05-12 10:52:09 -07:00
Vishnu KS bbcf396303 tests: Create assertNotInHomePage helper function. 2021-05-12 10:34:12 -07:00
Vishnu KS 75a3059b57 tests: Create assertInHomePage helper function. 2021-05-12 10:34:12 -07:00
Tim Abbott 06f1ea657e navbar: Adjust label for usage statistics. 2021-05-12 10:34:12 -07:00
Vishnu KS 051dab58ea navbar: Add gear menu advertisement for sponsoring zulip. 2021-05-12 10:21:48 -07:00
Steve Howell b4470ac8e1 performance: Add get_subscriptions_for_send_message.
This new function optimizes how we fetch subscriptions
for streams.  Basically, it excludes most long-term-idle
users from the query.

With 8k users, of which all but 400 are long term idle,
this speeds up get_recipient_info from about 150ms
to 50ms.

Overall this change appears to save a factor of 2-3 in the backend
processing time for sending or editing a message in large, public
streams in chat.zulip.org (at 18K users today).
2021-05-12 08:10:57 -07:00
Tim Abbott b15941610d message: Support avoiding database queries in has_message_access.
If the caller has already fetched the Stream or subscription details
for the user, those can be passed to has_message_access to avoid extra
database queries.
2021-05-11 20:46:49 -07:00
Tim Abbott 44fddcc9c1 message: Reorder checks in has_message_access.
This refactor makes this function easier to read and understand.
2021-05-11 20:46:49 -07:00
Alex Vandiver 237efdee6c send_email: Show more information about messages which failed to send. 2021-05-11 14:27:45 -10:00
Mateusz Mandera c95cac2dd7 outgoing_webhook: Send response payload to bot owner if it was invalid.
When the format of the response received from the outgoing webhook
server is invalid (unparsable json, or just wrong format that doesn't
translate into a dictionary etc.), a message with the error is sent to
the bot owner. We should include the actual payload to make reasonable
debugging possible.

In notify_bot_owner we have to move the `if response_content` block to
append the payload to the message whenever it was specified as an
argument to the function. It shouldn't be nested inside
`elif status_code` as before.
2021-05-11 14:27:00 -07:00
Alex Vandiver 0f1611286d management: Rename the deliver_email command to deliver_scheduled_email.
This makes it parallel with deliver_scheduled_messages, and clarifies
that it is not used for simply sending outgoing emails (e.g. the
`email_senders` queue).

This also renames the supervisor job to match.
2021-05-11 13:07:29 -07:00
Steve Howell 1e63282cc2 tests: Test email/push more thoroughly for idle users. 2021-05-11 14:33:12 +00:00
Steve Howell 3161b5d892 tests: Simplify test_user_message_filter. 2021-05-11 13:53:55 +00:00
PIG208 d8db797798 doc: Fix configurations for images in the doc of newrelic.
The screenshot generating mechanism doesn't work for newrelic and
causes error because its configuration file doesn't exist. This
commit fixes the configuration and re-generate the screenshots.
2021-05-10 17:50:32 -07:00
Abhijeet Prasad Bodas 67cd385cec mute user: Add user documentation page.
Also link to it from the API documentation page,
other help pages, and the confirmation dialog for
muting a user.

With substantial edits by tabbott and alya.
2021-05-10 17:10:40 -07:00
Mateusz Mandera 20f99f429d actions: Extract get_active_bots_owned_by_user function. 2021-05-10 15:38:24 -07:00
Mateusz Mandera 8261f7e801 commands: Add delete_user management command and document it. 2021-05-10 15:38:14 -07:00
Ganesh Pawar 529f72fa3f markdown: Add support for sms and tel links.
Fixes #18390
2021-05-10 15:15:34 -07:00
Mateusz Mandera 6b056ca332 retention: Change default behavior of restore_messages command to noop.
Having "restore everything" as the default is a very bad idea. An
argument should explicitly be given for that behavior.
2021-05-10 15:13:10 -07:00
Dinesh 7413966255 tests: Remove duplicate verify_backend() calls. 2021-05-10 13:56:06 -07:00
Alex Vandiver 27012e343b edit_linkifiers: Show the right command name in help text. 2021-05-10 13:51:22 -07:00
Siddharth Asthana 8f864c5107 commands: Add change_realm_subdomain management command.
Add command to change realm's subdomain. This command takes two
arguments, realm's string_id and new subdomain name.
2021-05-10 12:30:58 -07:00
Tim Abbott 2c01354569 management: Use required kwargs in add_realm_args.
This makes management commands more readable, since one doesn't need
to know details of how the library works to read based code.
2021-05-10 12:30:58 -07:00
akshatdalton 55f4996f16 markdown: Fix silent wildcard mentions bug.
A message containing wildcard mention when quoted (which
is turned into a silent mention) or message with silent
wildcard mention notifies the users by sending desktop,
sound, and missed message email notifications. This
is clearly a bug which is fixed by this commit.

Fixes: #18354.
2021-05-10 12:19:40 -07:00
Tim Abbott 70244896e5 test_subs: Remove now invalid JSON encoding of parameters.
All of these were missed in the previous commit.
2021-05-10 10:42:11 -07:00
Gaurav Pandey 96e035a2f0 api: Fix encoding of strings in streams endpoint.
* Remove unnecessary json_validator for string parameters.
* Update frontend to pass right parameter.

Bump api feature level and highlight the fix for `emojiset`
parameter of `settings/display` endpoint in zulip.yaml file.

Fixes part of #18035.
2021-05-10 10:29:22 -07:00
Gaurav Pandey 9abe1cafbe api: Fix encoding of strings in display settings endpoint.
* Remove unnecessary json validator for string validator.
* Update frontend to pass right validator.
* Update zulip.yaml to pass right parameter for curl request
  in openapi.
* Update python_examples to pass right paramater.

Fixes part of #18035.
2021-05-10 10:03:32 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Tim Abbott 8b75b6f14f message_edit: Use target_message as local variable name.
This name helps emphasize the fact that there's a single targeted
message, even though multiple messages may be affected by the edit.
2021-05-09 21:10:32 -07:00
Aman Agrawal c70541c905 do_update_message: Reuse extracted message IDs from changed_msgs. 2021-05-09 21:05:27 -07:00
PIG208 7150fe5dc5 backend: Extract check_update_message from update_message_backend. 2021-05-09 20:44:04 -07:00
Adam Birds 589a7799dc integrations: Add Uptime Robot integration.
Note that the documentation cannot fully use our macros, because
Uptime Robot requires an & of the end of the URL, because of how it
passes its payload.

Fixes #13854. Fixes #13939.
2021-05-09 20:34:19 -07:00
Adam Birds 49b920d33a integrations: Fix bug with github multiple reviews requested on PR.
Fixes a bug in which multiple of the same messages are posted when
multiple reviewers are added to a PR.

Fixes #18393.
2021-05-08 07:56:48 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Steve Howell 91599df0e2 send performance: Optimize create_user_messages.
We combine the two loops into one, so that we
can check our flags before creating the
UserMessageList object.

And we lift a few calculations out of the loop.

For 8k users, with 95% long-term-idle, this was
about a 10x speedup for me.  (~30ms -> 3ms)
2021-05-07 18:40:30 -04:00