Commit Graph

14056 Commits

Author SHA1 Message Date
Hemanth V. Alluri f1f0a26c37 drafts: Extract all of the real work to lib functions.
The main reason why this is needed is because this seems to be
convention and because we can't easily test event creation without
doing this.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2021-07-28 16:33:01 -07:00
Priyansh Garg 044fe547d3 data_import: Add huddle import support for Rocket.Chat. 2021-07-28 15:45:54 -07:00
Priyansh Garg 24dd0ff96c data_import: Add rocket chat import tool.
This commit allows to import the following from rocketchat:
* All users
* All public/private channels
* All teams and its public/private channels
* All discussion rooms as topics in their parent channel
* All the messages in all the channels
* All private conversations
* Reactions on messages (except for custom emojis)
* Mentions in messages (except @all, @here mentions)
2021-07-28 15:28:56 -07:00
Mateusz Mandera 4161e0caeb rate_limit: Show html page when rate limited at /new/ endpoint.
Previously this showed a json error, but this is an endpoint that human
users use in the browser, so a proper HTML page is more appropriate.
2021-07-28 15:24:04 -07:00
Abhijeet Prasad Bodas de04f0ad67 notifications: Calculate PMs/mentions settings like other settings.
Previously, we checked for the `enable_offline_email_notifications` and
`enable_offline_push_notifications` settings (which determine whether the
user will receive notifications for PMs and mentions) just before sending
notifications. This has a few problem:

1. We do not have access to all the user settings in the notification
handlers (`handle_missedmessage_emails` and `handle_push_notifications`),
and therefore, we cannot correctly determine whether the notification should
be sent. Checks like the following which existed previously, will, for
example, incorrectly not send notifications even when stream email
notifications are enabled-
```
if not receives_offline_email_notifications(user_profile):
    return
```
With this commit, we simply do not enqueue notifications if the "offline"
settings are disabled, which fixes that bug.

Additionally, this also fixes a bug with the "online push notifications"
feature, which was, if someone were to:
* turn off notifications for PMs and mentions (`enable_offline_push_notifications`)
* turn on stream push notifications (`enable_stream_push_notifications`)
* turn on "online push" (`enable_online_push_notifications`)

then, they would still receive notifications for PMs when online.
This isn't how the "online push enabled" feature is supposed to work;
it should only act as a wrapper around the other notification settings.

The buggy code was this in `handle_push_notifications`:
```
if not (
    receives_offline_push_notifications(user_profile)
    or receives_online_push_notifications(user_profile)
):
    return

    // send notifications
```

This commit removes that code, and extends our `notification_data.py` logic
to cover this case, along with tests.

2. The name for these settings is slightly misleading. They essentially
talk about "what to send notifications for" (PMs and mentions), and not
"when to send notifications" (offline). This commit improves this condition
by restricting the use of this term only to the database field, and using
clearer names everywhere else. This distinction will be important to have
non-confusing code when we implement multiple options for notifications
in the future as dropdown (never/when offline/when offline or online, etc).

3. We should ideally re-check all notification settings just before the
notifications are sent. This is especially important for email notifications,
which may be sent after a long time after the message was sent. We will
in the future add code to thoroughly re-check settings before sending
notifications in a clean manner, but temporarily not re-checking isn't
a terrible scenario either.
2021-07-28 13:55:25 -07:00
Abhijeet Prasad Bodas 683c8507e4 models: Remove MutedTopic alias for UserTopic.
Part of #19272
2021-07-28 10:25:54 -07:00
Abhijeet Prasad Bodas 798defc046 models: Rename MutedTopic to UserTopic.
Part of #19272

We still keep refering to this model with "MutedTopic" to reduce the
diff size of this commit. The alias will be removed in the next commit.

This commit skips on renaming the `date_muted` field to something more
general. That will be done in further commits, along with the code and
API changes.
2021-07-28 10:25:53 -07:00
Riken Shah 9fadd43830 user_status: Add backend changes to support status emoji.
In this commit:

* We update the `UserStatus` model to accept
`AbstractReaction` as a base class so, we can get all the
fields related to store status emoji.

* We update the user status endpoint
(`users/me/status`) to accept status emoji fields.

* We update the user status event to add status emoji
fields.

Co-authored-by: Yash Rathore <33805964+YashRE42@users.noreply.github.com>
2021-07-27 17:57:15 -07:00
Tim Abbott ed01ffadba tests: Fix leading slash in test exclusion as well. 2021-07-27 17:01:18 -07:00
sahil839 0d6f378ad7 help: Update documentation for add_custom_emoji_policy. 2021-07-27 16:41:23 -07:00
sahil839 1ce09d9db6 realm: Extend add_custom_emoji_policy to use COMMON_POLICY_TYPES.
This commit adds full members and moderators option for
add_custom_emoji_policy by using COMMON_POLICY_TYPES.
2021-07-27 16:41:23 -07:00
sahil839 5efc854f66 models: Add can_add_custom_emoji helper.
This commit adds can_add_custom_emoji
helper to check whether the user can
add custom emoji or not.

This function will be used further when
add_custom_emoji_policy will be extended
to include all COMMON_POLICY_VALUES.
2021-07-27 16:41:23 -07:00
sahil839 4b1313a92b models: Replace add_emoji_by_admins_only with add_custom_emoji_policy.
This commit replaces boolean field add_emoji_by_admins_only with an
integer field add_custom_emoji_policy as we would also add full members
and moderators option for this setting in further commits.
2021-07-27 16:41:22 -07:00
Anders Kaseorg 17749cb608 archive: Remove non-functional archive code.
This removes a bunch of non-functional duplicate JavaScript, HTML, and
CSS that was interfering with maintenance on the functional originals,
because it was never clear how to update the duplicates or how to
check that you’d updated the duplicates correctly.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-27 16:25:24 -07:00
Sahil Batra 79b0ff5b23 events: Send enter_sends if update_display_settings in fetch_event_types.
We now send 'enter_sends' in response of '/register' endpoint if
'update_display_settings' is present in 'fetch_event_types' instead
of 'realm_user'.
2021-07-27 15:54:16 -07:00
Sahil Batra 355f6e9b53 settings: Move 'enter_sends' setting to property_types dict.
This commit moves "enter_sends" setting to property_types dict.
With this change, changing enter_sends setting also sends an
event of type "update_display_settings" and thus enables us
to live-update the UI.
2021-07-27 15:52:53 -07:00
Mateusz Mandera 7af789c48f models: Make realm_id arg of get_system_bot required. 2021-07-27 15:51:59 -07:00
Mateusz Mandera 013d6fa5e6 bots: Assume STAFF_SUBDOMAIN to be the realm where ERROR_BOT lives.
Now that get_system_bot requires a realm argument, a subdomain
needs to be made explicit to know in which organization to look up the
bot.
2021-07-27 15:51:59 -07:00
PIG208 e517f967b8 typing: Amend inaccurate type annotations.
This amend some type annotations that turn out to be inaccurate with
django-stubs.
2021-07-27 12:17:00 -07:00
PIG208 8121d2d58d typing: Fix misuse of HttpResponse.
Amend usage of HttpResponse when appropriate.
2021-07-27 14:31:19 +08:00
Tim Abbott 9ada794ce1 test_openapi: Fix mis-resolved merge conflict. 2021-07-26 18:22:09 -07:00
Suyash Vardhan Mathur ec6739ff29 openapi: Document get-subscribers endpoint. 2021-07-26 18:12:12 -07:00
Suyash Vardhan Mathur 2d2b089066 openapi: Document delete-topic endpoint. 2021-07-26 17:56:04 -07:00
sahil839 c7e756f984 setting: Allow only owners to change "Who can invite users" setting. 2021-07-26 17:48:01 -07:00
sahil839 8ced74192a invite: Add 'Nobody' option to invite_to_realm_policy in backend. 2021-07-26 17:48:01 -07:00
sahil839 ace527af0a api: Remove incorrect realm setting in 'realm/update_dict' schema.
We incorrectly include many realm settings in the data section of
'realm/update_dict' schema. It should only contain the settings
related to message edit, realm icon, realm logo and authentication
methods and not other settings, becausea all the other settings send
'realm/update' event and not 'realm/update_dict' event.

This commit only removes 'invite_to_realm_policy' and others will
be removed separately.
2021-07-26 17:43:51 -07:00
Mateusz Mandera 8f588dcbab models: Pass realm to get_user_including_cross_realm calls. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 43329b6a34 bots: Pass realm to get_system_bot calls in export/import. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 7a36ac6eca bots: Pass realm to get_system_bot call in server_initialization.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 0416f0385a bots: Pass realm to get_system_bot calls in onboarding.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera c34260426a bots: Pass realm to remaining get_system_bot calls in tests. 2021-07-26 15:33:13 -07:00
Mateusz Mandera a6924e23fa bots: Pass realm to get_system_bot calls in streams.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera a124a4dfcf bots: Pass realm to get_system_bot calls in actions.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 6f6be04f2e bots: Pass realm to get_system_bot call in test_users. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 994ee70497 bots: Pass realm to self.notification_bot test helper. 2021-07-26 15:33:13 -07:00
Mateusz Mandera a01594e72b bots: Pass realm to get_system_bot call in DeferredWorker. 2021-07-26 15:33:13 -07:00
Mateusz Mandera f790a32d13 bots: Pass realm to get_system_bot calls in test_cache.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 104139657b bots: Pass realm to get_system_bot calls in email_mirror. 2021-07-26 15:33:13 -07:00
Mateusz Mandera d45f3eecaa models: Add optional realm_id argument to get_system_bot. 2021-07-26 15:31:10 -07:00
PIG208 22c84e69ac tests: Fix inappropriate assertEqual.
We do not give a second argument to `self.assertEqual` here. We only need
to invoke the function.
2021-07-26 15:09:07 -07:00
PIG208 91de2cbe03 typing: Avoid scoped redefinition of different types.
Mypy doesn't allow redefinition of a variable using a different type
within the same scope.
https://github.com/python/mypy/issues/1174
2021-07-26 15:09:07 -07:00
PIG208 178a4b4eff tests: Initialize requests with different post data.
Instead of directly changing the `POST` attribute of a request, we
utilize the `HostRequestMock` initializer to produce requests with
different post data.
2021-07-26 15:09:07 -07:00
PIG208 c3abc0b472 tests: Correctly mock and invoke view functions.
The decorators require the decorated function to be a valid view
function. This changes the way the mocked view functions and requests
are implemented such that we can invoke view functions without future
type errors.
2021-07-26 15:09:07 -07:00
PIG208 57f46db09f tests: Pass HostRequestMock to export_realm.
`export_realm` accepts an HttpRequest as the first argument,
while `self.client_post` conflicts with it. Though the argument is
unused in `export_realm`, we keep it to be compliant with the
view function type.
2021-07-26 15:09:07 -07:00
PIG208 f280ebe4b5 decorator: Type zulip_login_required with overload.
As we only return the actual decorator as-is only if `function` is
`None`, we can use `@overload` to accurately annotate the return type
for the decorator.
2021-07-26 15:09:07 -07:00
PIG208 d2af20eb2a typing: Do explicit type conversion when appropriate.
When calling some functions or assigning values to certain attributes,
the arguments/right operand do not match the exact type that the
functions/attributes expect, and thus we fix that by converting types
beforehand.
2021-07-26 15:09:07 -07:00
PIG208 7d1c475f69 typing: Use assertions for function arguments.
Utilize the assert_is_not_None helper to eliminate errors of
'Argument x to "Foo" has incompatible type "Optional[Bar]"...'
2021-07-26 14:48:45 -07:00
PIG208 c4179428ac caching: Make sender type optional for flush_cache.
That's the correct Django interface for this function.
2021-07-26 14:48:07 -07:00
PIG208 2972e921b7 utils: Add a helper function for None-checks. 2021-07-26 14:46:45 -07:00
PIG208 cf8687662f confirmation: Use the correct type hints for create_confirmation_link.
Previously we annotate the first argument as `ContentType`, which
is wrong as suggested by django-stubs.
2021-07-26 14:46:45 -07:00
Dinesh beadb5ec7f test_middleware: Replace logging mocks with assertLogs. 2021-07-26 14:46:01 -07:00
Dinesh 2846d18854 test_decorators: Replace webhook logger mock with assertLogs.
Of the two other logging mocks left in this file, one checks
a logging call isn't made and another makes sure errors
aren't allowed by raising an exception as a side_effect
to the logger.
2021-07-26 14:46:01 -07:00
Dinesh 199f522808 test_classes.py: Replace markdown_logger mock with assertLogs.
Set level to 'ERROR' since exceptions create logs with that level.
2021-07-26 14:46:01 -07:00
Dinesh b195cc3635 test_push_notifications.py: Replace logging mocks with assertLogs.
Left the mocks which are used to assert a logging call isn't made.
2021-07-26 14:46:01 -07:00
Aman Agrawal 7d8f62e7a0 portico: Add a developer-community page.
This is intended to replace the ReadTheDocs page of this form, but
this commit does not yet update any links.
2021-07-25 17:01:26 -07:00
Tim Abbott 09b5bb7930 export: Improve error message for missing registrations. 2021-07-24 17:36:15 -07:00
Mateusz Mandera 1c64bed8e4 rate_limiter: Rate limit the /new/ endpoint. 2021-07-24 15:52:06 -07:00
Mateusz Mandera 4418aefde4 decorators: Extract rate_limit_request_by_ip function. 2021-07-24 15:52:06 -07:00
Mateusz Mandera 119f1da04a bots: Rename is_cross_realm_bot API field to is_system_bot.
Cross realm bots will soon stop being a thing. This param is responsible
for displaying "System Bot" in the user info popover - so this rename is the
right way to handle the situation.

We will likely want to rename the `cross_realm_bots` section as well,
but that is a more involved API migration.
2021-07-24 15:46:40 -07:00
Mateusz Mandera 1652e5c031 test_webhooks_common: Compare users by id not email.
Comparing by email can be incorrect, as different UserProfile can share
the same .email if they're in different realm.
2021-07-24 15:39:00 -07:00
Mateusz Mandera 86c330b752 social_auth: Fix handling of user errors in the authentication process.
The code didn't account for existence of SOCIAL_AUTH_SUBDOMAIN. So the
redirects would happen to endpoints on the SOCIAL_AUTH_SUBDOMAIN, which
is incorrect. The redirects should happen to the realm from which the
user came.
2021-07-24 15:17:52 -07:00
PIG208 5359d20a13 webhooks: Add support to event filtering system for webhooks.
This add support to event filtering system for most webhooks that
require trivial changes to adapt this feature.
2021-07-24 15:10:09 -07:00
PIG208 987f859681 webhooks: Detect view function with default naming convention. 2021-07-24 15:10:09 -07:00
PIG208 66b1a4e7ca backend: Add None-checks with assertions and if-elses.
This fixes a batch of mypy errors of the following format:
'Item "None" of "Optional[Something]" has no attribute "abc"'
2021-07-24 15:00:21 -07:00
PIG208 fffd4ed8d5 invitations: Correctly compare realms for revoke_user_invite. 2021-07-24 14:59:19 -07:00
PIG208 de2678a319 tests: Fix missed mypy errors in tests.
This fixes a batch of mypy errors of the following format:
'Item "None" of "Optional[Something]" has no attribute "abc"
as a follow-up for 9892951703.
2021-07-24 14:59:19 -07:00
Tim Abbott 01ce58319d mypy: Fix most AnonymousUser type errors.
This commit fixes several mypy errors with Django stubs, by telling
mypy that we know in a given code path that the user is authenticated.
2021-07-24 14:55:46 -07:00
Tim Abbott d603b0641e mypy: Extend type for post_data in send_to_push_bouncer. 2021-07-24 12:55:15 -07:00
Tim Abbott bd134019e4 mypy: Fix return type for num_push_devices_for_user. 2021-07-24 12:55:15 -07:00
Tim Abbott 4aba6c9833 mypy: Use Sequence in push notifications types.
This is important for variance reasons.
2021-07-24 12:55:15 -07:00
Tim Abbott 6e26c8912e actions: Fix stream_dict/sub_dict types and variable names. 2021-07-24 11:02:08 -07:00
Tim Abbott 3790146c75 actions: Fix typing for raw stream dictionaries. 2021-07-24 11:02:08 -07:00
PIG208 495a8476be tests: Use assertion to enforce None-checks in tests.
This fixes a batch of mypy errors of the following format:
'Item "None" of "Optional[Something]" has no attribute "abc"

Since we have already been recklessly using these attritbutes
in the tests, adding assertions beforehand is justified presuming
that they oughtn't to be None.
2021-07-24 09:54:21 -07:00
PIG208 442adfaff3 sessions: Use SessionBase instead of Session. 2021-07-24 09:54:21 -07:00
Tim Abbott 12737fdc02 models: Add models for emoji in user statuses. 2021-07-23 13:11:21 -07:00
Hemanth V. Alluri b5cd232a2e drafts: Use enable_drafts_synchronization for access control.
If a user doesn't have enable_drafts_synchronization set to True, then
don't let them access the drafts API.  This will help protect us
against client bugs accidentally sending drafts to the server when the
feature is disabled.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2021-07-23 12:34:15 -07:00
Hemanth V. Alluri 08e9e48205 models: Add a new field controlling drafts synchronization.
This field will control whether or not a user wants to sync their
drafts between different clients.  Defaults to enabled.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2021-07-23 12:31:00 -07:00
Abhijeet Prasad Bodas 5db4fe8652 api: Allow setting email_notifications_batching_period_seconds.
We allow a maximum value of one week to make sure there aren't a huge
number of rows in the table for any user (this could happen if stream
notifications are enabled).

This commit also fixes a small error in the user_settings test.
2021-07-23 12:22:05 -07:00
Abhijeet Prasad Bodas dd5e12d112 MissedMessageWorker: Use custom batching periods from UserProfile. 2021-07-23 12:13:46 -07:00
Abhijeet Prasad Bodas d5d3f14eee models: Add UserProfile field for custom email notifs batching periods.
The default is 2 minutes (120 seconds), which was the hardcoded value
earlier.
2021-07-23 12:13:46 -07:00
Abhijeet Prasad Bodas 9fcb6e51ce MissedMessageWorker: Handle deleted messages.
The test for the try-except block is hacky. See the comment for
explaination.
2021-07-23 12:13:46 -07:00
Abhijeet Prasad Bodas de78b015d9 MissedMessageWorker: Remove unnecessary transaction.atomic.
We only have one query which will change database state in this function,
and we already have a lock on the process itself, so there's no need for
a transaction.

This was added in ebb4eab0f9.
2021-07-23 12:13:46 -07:00
Anders Kaseorg 162e9d6c0b fenced_code: Optimize FENCE_RE to fix cubic worst-case complexity.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-22 16:40:44 -07:00
Anders Kaseorg dea935f26f fenced_code: Write FENCE_RE with a raw string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-22 16:40:43 -07:00
Sahil Batra 318d71469b api: Remove '/users/me/enter-sends' endpoint.
We remove the '/users/me/enter-sends' endpoint
and 'enter_sends' setting will now be edited
using the '/settings' endpoint.
2021-07-22 11:01:43 -07:00
akshatdalton 7d9e71be21 fenced_code: Add `process_contents` flag to de-duplicate code. 2021-07-22 10:57:23 -07:00
Anders Kaseorg f9430674d2 github: Update macOS dmg filename pattern.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-21 19:03:39 -07:00
Tim Abbott 1f497bf8c8 test_docs: Update tests for recent plans page changes.
This should have been in b03cab8ced.
2021-07-21 18:03:55 -07:00
Aman Agrawal 56ce1f9f63 portico: Add pricing widget from /plans to for-education page. 2021-07-21 12:13:25 -07:00
Aman Agrawal 3c024b2fba portico: Add /for/education, /for/events, /for/research pages.
These modern landing pages cover use cases previously not detailed on
our website. Technically, we had a /for/research page before, but it
wasn't finished or linked everywhere.

Removed "function-url-quotes" stylelint rule
since I need to use quotes in url to use an
svg as list bullet point. There are spacing issues
using it as an image. Also, using quotes in url
is actually the recommended way to do it otherwise
there could be issue with escaping.
2021-07-20 22:37:52 -07:00
Erik Tews 1ecb87ec80 auth: show _OR_ during login only when other methods are available.
There might be good reasons to have other external authentication
methods such as SAML configured, but none of them is available.

This happens, for example, when you have enabled SAML so that Zulip is
able to generate the metadata in XML format, but you haven't
configured an IdP yet. This commit makes sure that the phrase _OR_ is
only shown on the login/account page when there are actually other
authentication methods available. When they are just configured, but
not available yet, the page looks like as if no external
authentication methods are be configured.

We achieve this by deleting any_social_backend_enabled, which was very
similar to page_params.external_authentication_methods, which
correctly has one entry per configured SAML IdP.
2021-07-20 14:29:48 -07:00
Anders Kaseorg fd0ab7c4ec tornado: Call close() on Django HttpResponse objects.
This is necessary to break the uncollectable reference cycle created
by our ‘request_notes.saved_response = json_response(…)’, Django’s
‘response._resource_closers.append(request.close)’, and Python’s
https://bugs.python.org/issue44680.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-20 11:07:36 -07:00
PIG208 7cf859882d rate_limiter: Avoid strong reference to user.
This prevents a memory leak caused by the `SimpleLazyObject` instance of
`UserProfile` that create a reference loop with the request object
via `ZulipRequestNotes`.
2021-07-20 10:32:29 -07:00
Tim Abbott 256091dc15 settings: Merge settings API endpoints.
This API change removes unnecessary complexity from a client that
wants to change a user's personal settings, and also saves developers
from needing to make decisions about what sort of setting something is
at the API level.

We preserve the old settings endpoints as mapping to the same function
as the new one for backwards-compatibility. We delete the
documentation for the old endpoints, though the documentation for the
merged /settings endpoint mentions how to use the old endpoints when
needed.

We migrate all backend tests to the new endpoints, except for
individual tests for each legacy endpoint to verify they still work.

Co-authored-by: sahil839 <sahilbatra839@gmail.com>
2021-07-19 21:14:32 -07:00
Anders Kaseorg 6564b258f1 request: Weaken ZulipRequestNotes.tornado_handler reference.
This prevents a memory leak arising from Python’s inability to collect
a reference cycle from a WeakKeyDictionary value to its key
(https://bugs.python.org/issue44680).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-19 16:48:23 -07:00
Anders Kaseorg 7c32134fb5 Revert "Revert "request: Refactor to record rate limit data using ZulipRequestNotes.""
This reverts commit 49eab4efef.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-19 16:48:23 -07:00
Tim Abbott b73356e891 portico: Add UCSD case study. 2021-07-18 21:55:57 -07:00
Tim Abbott d7ef0c7232 billing: Fix options and help text for sponsorship requests.
This makes several changes:

* Fixes a bug where the help text explaining our policies was not displayed.
* No help text was defined for many organization types.
* Copy-edits the help text somewhat.
* Offers all of the organization type options.
* Removes the 100% coverage requirement because it's annoying to test
  the e.currentTarget click handler.
2021-07-18 20:48:27 -07:00
Alya Abbott 844c62c64e portico: Create new "Zulip for communities" page. 2021-07-18 12:10:33 -07:00
Suyash Vardhan Mathur 989d14299c openapi: Make base class for Markdown Preprocessors.
Most of the Markdown Preprocessors followed a common
template, and the `run` and `init` code was duplicated
multiple times for different preprocessors.

This commit adds a base class from which the preprocessors
following the pattern can inherit, and can override the
`render` and `generate_text` functions to execute the code.
2021-07-18 11:42:27 -07:00