Commit Graph

14098 Commits

Author SHA1 Message Date
Alex Vandiver e94b6afb00 nagios: Remove broken check_email_deliverer_* checks and related code.
These checks suffer from a couple notable problems:
 - They are only enabled on staging hosts -- where they should never
   be run.  Since ef6d0ec5ca, these supervisor processes are only
   run on one host, and never on the staging host.
 - They run as the `nagios` user, which does not have appropriate
   permissions, and thus the checks always fail.  Specifically,
   `nagios` does not have permissions to run `supervisorctl`, since
   the socket is owned by the `zulip` user, and mode 0700; and the
   `nagios` user does not have permission to access Zulip secrets to
   run `./manage.py print_email_delivery_backlog`.

Rather than rewrite these checks to run on a cron as zulip, and check
those file contents as the nagios user, drop these checks -- they can
be rewritten at a later point, or replaced with Prometheus alerting,
and currently serve only to cause always-failing Nagios checks, which
normalizes alert failures.

Leave the files installed if they currently exist, rather than
cluttering puppet with `ensure => absent`; they do no harm if they are
left installed.
2021-08-03 16:07:13 -07:00
Ganesh Pawar 1965584eec time_widget: Make the generated time string more readable.
Before: <time:2021-07-14T00:14:00-07:00>
After: <time:2021-07-14|00:14:00|UTC-07:00>

Fixes #19205
2021-08-02 23:17:01 -07:00
Anders Kaseorg 498d2b48d9 fenced_code: Use find_lexer_class_by_name.
This is more efficient than get_lexer_by_name, since we don’t need to
instantiate the class just to get its name.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 04feadd917 mypy: Add pika-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg f87a0e912b test_queue: Fix ConnectionClosed usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 9f8ba913fd queue: Fix _on_connection_open_error type to accept reason: str.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg f7e2426fc5 queue: Fix ensure_queue type to accept a callback returning any object.
channel.basic_consume actually returns str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 5e355abe2e queue: Add missing imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 87799177b5 queue: Fix channel type for TornadoQueueClient.
The BlockingChannel annotations in TornadoQueueClient were flat-out
wrong.  BlockingChannel and Channel have no common base classes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 5751479932 queue: Switch TornadoQueueClient to the new base QueueClient.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg bd6a2b149c queue: Split common part of SimpleQueueClient into new base class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Alex Vandiver 6fe67f0143 delete_realm: Allow deletion of realms with empty customers.
This is effectively a step closer to what was proposed in
https://github.com/zulip/zulip/pull/18678#discussion_r644490540 when
this code was written in #18678.

If the Customer object has neither of a Stripe id, nor any historical
plans, then there's no real billing association contained in the
existence of the Customer object, and it's safe to delete.
2021-08-02 22:29:16 -07:00
Tim Abbott 9968fb5081 send_custom_email: Fix emailing single users with TOS_VERSION set.
This code path previously threw an exception.
2021-08-02 17:57:16 -07:00
Anders Kaseorg fae92f2e3f actions: Convert TypedDict declaration to Python 3.6 style.
Generated by pyupgrade, with manual indentation adjustment for
https://github.com/asottile/pyupgrade/issues/507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 5483ebae37 python: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 96234016f5 python: Remove obsolete directives.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg ad5f0c05b5 python: Remove default "utf8" argument for encode(), decode().
Partially generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 1760897a8c python: Remove default "r" mode for open().
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 47f795ae64 python: Remove redundant parentheses.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
rht 1bbd36d181 slack_import: Remove obsolete SlackImportAttachment placeholder.
This was introduced in f4ad464d82, and
incompletely removed in e037c2f93e649c28a71c02559b5ae7a3333f42a8; here
we finish removing it.
2021-08-02 13:13:28 -07:00
rht 9441bca282 import_realm: Explain the usage of the relative_path var of attachment_path. 2021-08-02 13:13:19 -07:00
Tim Abbott 24b1439e93 email_notifications: Avoid trying to email to bot users.
This fixes a regression in de04f0ad67.

We'll do a proper test in a follow-up commit; this is a quick fix to
make sure master works.

The emails will bounce, but it'll create all sorts of infrastructure
headaches.
2021-08-02 13:13:01 -07:00
Sahil Batra 93b51088c0 api: Fix "Changes" entry for enter_sends in zulip.yaml.
This commit fixes the feature level for enter_sends in
`/regiser` response and also modifies the "Changes"
entry to match with other settings.
2021-08-02 09:23:56 -07:00
Sahil Batra c3f31bd507 api: Fix feature level for zulip_merge_base in zulip.yaml. 2021-08-02 09:23:56 -07:00
Tim Abbott d1dd34d7e0 send_custom_email: Add option for sending marketing emails. 2021-08-01 21:45:34 -07:00
Aman Agrawal c1c6674809 emails: Retheme to purple colors to match present logo. 2021-08-01 21:27:42 -07:00
Tim Abbott 65d60f4a25 api docs: Improve documentation for user_settings transition. 2021-08-01 15:30:40 -07:00
Sahil Batra 998d710275 frontend: Add new user_settings module for user's settings.
We add a new user_settings module similar to page_params
module in frontend and use it to access user's personal
settings instead of page_params.
2021-08-01 15:30:17 -07:00
Sahil Batra e6f828a8e2 register: Send user_settings in response based on client capabilities.
We added "user_settings" object containing all the user settings in
previous commit. This commit modifies the code to send the existing
setting fields in the top-level object only if user_settings_object
client_capabilities field is False.
2021-08-01 15:30:17 -07:00
Sahil Batra 84742a040e api: Add "user_settings" object to '/register' response.
We add a "user_settings" object, containing all the user
settings, in the '/register' endpoint response for
better readability.
2021-08-01 15:30:17 -07:00
Sahil Batra 0364d0c8ca events: Add "user_settings_object" to client_capabilities.
This commit adds "user_settings_object" field to
client_capabilities which will be used to determine
if the client needs 'update_display_settings' and
'update_global_notifications' event.
2021-08-01 15:30:17 -07:00
Sahil Batra 7959ae3fab events: Add new event type 'user_settings' for updating user settings.
We send a event with type 'user_settings' on updating user's display
and notification settings.

The old event types - 'update_global_notifications' and
'update_display_settings', are still supported for backwards
compatibility.
2021-08-01 15:30:17 -07:00
Sahil Batra e73d2fff97 tests: Remove separate event tests for "enable_drafts_synchronization".
We do not require separate tests for checking events when changing
"enable_drafts_synchronization" as we already do this in the display
settings test because this setting is included in property_types.
2021-08-01 15:30:17 -07:00
Anders Kaseorg aeb0dc7269 video_calls: Support separate testing credentials for Zoom.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-01 14:26:46 -07:00
Gaurav Pandey d89b4dcd0b api: Return zulip_merge_base alongside zulip_version.
Return zulip_merge_base alongside zulip_version
in `/register`, `/event` and `/server_settings`
endpoint so that the value can be used by other
clients.
2021-07-30 12:03:41 -07:00
Hemanth V. Alluri 6a3e98d14b drafts: Add API documentation for the core drafts endpoints.
These were added at some point in the past, but were not complete, and
it makes sense to document the current feature level as and when they
become available, since clients should not use the drafts endpoints on
older feature levels.
2021-07-30 10:41:52 -07:00
Tim Abbott 472c55a1ff populate_db: Create some drafts in the initial database. 2021-07-30 10:41:52 -07:00
Hemanth V. Alluri 6fee946a43 drafts: Send events to clients when drafts change.
With this, the core of the new drafts system is complete.
2021-07-30 10:41:52 -07:00
Hemanth V. Alluri c00089ac28 drafts: Add support for toggling drafts synchronization.
With changes mostly to the API documentation by tabbott.
2021-07-30 10:00:27 -07:00
Anders Kaseorg 1f5fc12989 video_calls: Remove call to deprecated Zoom Data Compliance API.
https://marketplace.zoom.us/docs/guides/stay-up-to-date/announcements#data-compliance

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-30 09:19:22 -07:00
Suyash Vardhan Mathur ea6b21cd8c openapi: Document update-status endpoint. 2021-07-29 11:50:25 -07:00
Hemanth V. Alluri f1f0a26c37 drafts: Extract all of the real work to lib functions.
The main reason why this is needed is because this seems to be
convention and because we can't easily test event creation without
doing this.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2021-07-28 16:33:01 -07:00
Priyansh Garg 044fe547d3 data_import: Add huddle import support for Rocket.Chat. 2021-07-28 15:45:54 -07:00
Priyansh Garg 24dd0ff96c data_import: Add rocket chat import tool.
This commit allows to import the following from rocketchat:
* All users
* All public/private channels
* All teams and its public/private channels
* All discussion rooms as topics in their parent channel
* All the messages in all the channels
* All private conversations
* Reactions on messages (except for custom emojis)
* Mentions in messages (except @all, @here mentions)
2021-07-28 15:28:56 -07:00
Mateusz Mandera 4161e0caeb rate_limit: Show html page when rate limited at /new/ endpoint.
Previously this showed a json error, but this is an endpoint that human
users use in the browser, so a proper HTML page is more appropriate.
2021-07-28 15:24:04 -07:00
Abhijeet Prasad Bodas de04f0ad67 notifications: Calculate PMs/mentions settings like other settings.
Previously, we checked for the `enable_offline_email_notifications` and
`enable_offline_push_notifications` settings (which determine whether the
user will receive notifications for PMs and mentions) just before sending
notifications. This has a few problem:

1. We do not have access to all the user settings in the notification
handlers (`handle_missedmessage_emails` and `handle_push_notifications`),
and therefore, we cannot correctly determine whether the notification should
be sent. Checks like the following which existed previously, will, for
example, incorrectly not send notifications even when stream email
notifications are enabled-
```
if not receives_offline_email_notifications(user_profile):
    return
```
With this commit, we simply do not enqueue notifications if the "offline"
settings are disabled, which fixes that bug.

Additionally, this also fixes a bug with the "online push notifications"
feature, which was, if someone were to:
* turn off notifications for PMs and mentions (`enable_offline_push_notifications`)
* turn on stream push notifications (`enable_stream_push_notifications`)
* turn on "online push" (`enable_online_push_notifications`)

then, they would still receive notifications for PMs when online.
This isn't how the "online push enabled" feature is supposed to work;
it should only act as a wrapper around the other notification settings.

The buggy code was this in `handle_push_notifications`:
```
if not (
    receives_offline_push_notifications(user_profile)
    or receives_online_push_notifications(user_profile)
):
    return

    // send notifications
```

This commit removes that code, and extends our `notification_data.py` logic
to cover this case, along with tests.

2. The name for these settings is slightly misleading. They essentially
talk about "what to send notifications for" (PMs and mentions), and not
"when to send notifications" (offline). This commit improves this condition
by restricting the use of this term only to the database field, and using
clearer names everywhere else. This distinction will be important to have
non-confusing code when we implement multiple options for notifications
in the future as dropdown (never/when offline/when offline or online, etc).

3. We should ideally re-check all notification settings just before the
notifications are sent. This is especially important for email notifications,
which may be sent after a long time after the message was sent. We will
in the future add code to thoroughly re-check settings before sending
notifications in a clean manner, but temporarily not re-checking isn't
a terrible scenario either.
2021-07-28 13:55:25 -07:00
Abhijeet Prasad Bodas 683c8507e4 models: Remove MutedTopic alias for UserTopic.
Part of #19272
2021-07-28 10:25:54 -07:00
Abhijeet Prasad Bodas 798defc046 models: Rename MutedTopic to UserTopic.
Part of #19272

We still keep refering to this model with "MutedTopic" to reduce the
diff size of this commit. The alias will be removed in the next commit.

This commit skips on renaming the `date_muted` field to something more
general. That will be done in further commits, along with the code and
API changes.
2021-07-28 10:25:53 -07:00
Riken Shah 9fadd43830 user_status: Add backend changes to support status emoji.
In this commit:

* We update the `UserStatus` model to accept
`AbstractReaction` as a base class so, we can get all the
fields related to store status emoji.

* We update the user status endpoint
(`users/me/status`) to accept status emoji fields.

* We update the user status event to add status emoji
fields.

Co-authored-by: Yash Rathore <33805964+YashRE42@users.noreply.github.com>
2021-07-27 17:57:15 -07:00
Tim Abbott ed01ffadba tests: Fix leading slash in test exclusion as well. 2021-07-27 17:01:18 -07:00
sahil839 0d6f378ad7 help: Update documentation for add_custom_emoji_policy. 2021-07-27 16:41:23 -07:00
sahil839 1ce09d9db6 realm: Extend add_custom_emoji_policy to use COMMON_POLICY_TYPES.
This commit adds full members and moderators option for
add_custom_emoji_policy by using COMMON_POLICY_TYPES.
2021-07-27 16:41:23 -07:00
sahil839 5efc854f66 models: Add can_add_custom_emoji helper.
This commit adds can_add_custom_emoji
helper to check whether the user can
add custom emoji or not.

This function will be used further when
add_custom_emoji_policy will be extended
to include all COMMON_POLICY_VALUES.
2021-07-27 16:41:23 -07:00
sahil839 4b1313a92b models: Replace add_emoji_by_admins_only with add_custom_emoji_policy.
This commit replaces boolean field add_emoji_by_admins_only with an
integer field add_custom_emoji_policy as we would also add full members
and moderators option for this setting in further commits.
2021-07-27 16:41:22 -07:00
Anders Kaseorg 17749cb608 archive: Remove non-functional archive code.
This removes a bunch of non-functional duplicate JavaScript, HTML, and
CSS that was interfering with maintenance on the functional originals,
because it was never clear how to update the duplicates or how to
check that you’d updated the duplicates correctly.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-27 16:25:24 -07:00
Sahil Batra 79b0ff5b23 events: Send enter_sends if update_display_settings in fetch_event_types.
We now send 'enter_sends' in response of '/register' endpoint if
'update_display_settings' is present in 'fetch_event_types' instead
of 'realm_user'.
2021-07-27 15:54:16 -07:00
Sahil Batra 355f6e9b53 settings: Move 'enter_sends' setting to property_types dict.
This commit moves "enter_sends" setting to property_types dict.
With this change, changing enter_sends setting also sends an
event of type "update_display_settings" and thus enables us
to live-update the UI.
2021-07-27 15:52:53 -07:00
Mateusz Mandera 7af789c48f models: Make realm_id arg of get_system_bot required. 2021-07-27 15:51:59 -07:00
Mateusz Mandera 013d6fa5e6 bots: Assume STAFF_SUBDOMAIN to be the realm where ERROR_BOT lives.
Now that get_system_bot requires a realm argument, a subdomain
needs to be made explicit to know in which organization to look up the
bot.
2021-07-27 15:51:59 -07:00
PIG208 e517f967b8 typing: Amend inaccurate type annotations.
This amend some type annotations that turn out to be inaccurate with
django-stubs.
2021-07-27 12:17:00 -07:00
PIG208 8121d2d58d typing: Fix misuse of HttpResponse.
Amend usage of HttpResponse when appropriate.
2021-07-27 14:31:19 +08:00
Tim Abbott 9ada794ce1 test_openapi: Fix mis-resolved merge conflict. 2021-07-26 18:22:09 -07:00
Suyash Vardhan Mathur ec6739ff29 openapi: Document get-subscribers endpoint. 2021-07-26 18:12:12 -07:00
Suyash Vardhan Mathur 2d2b089066 openapi: Document delete-topic endpoint. 2021-07-26 17:56:04 -07:00
sahil839 c7e756f984 setting: Allow only owners to change "Who can invite users" setting. 2021-07-26 17:48:01 -07:00
sahil839 8ced74192a invite: Add 'Nobody' option to invite_to_realm_policy in backend. 2021-07-26 17:48:01 -07:00
sahil839 ace527af0a api: Remove incorrect realm setting in 'realm/update_dict' schema.
We incorrectly include many realm settings in the data section of
'realm/update_dict' schema. It should only contain the settings
related to message edit, realm icon, realm logo and authentication
methods and not other settings, becausea all the other settings send
'realm/update' event and not 'realm/update_dict' event.

This commit only removes 'invite_to_realm_policy' and others will
be removed separately.
2021-07-26 17:43:51 -07:00
Mateusz Mandera 8f588dcbab models: Pass realm to get_user_including_cross_realm calls. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 43329b6a34 bots: Pass realm to get_system_bot calls in export/import. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 7a36ac6eca bots: Pass realm to get_system_bot call in server_initialization.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 0416f0385a bots: Pass realm to get_system_bot calls in onboarding.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera c34260426a bots: Pass realm to remaining get_system_bot calls in tests. 2021-07-26 15:33:13 -07:00
Mateusz Mandera a6924e23fa bots: Pass realm to get_system_bot calls in streams.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera a124a4dfcf bots: Pass realm to get_system_bot calls in actions.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 6f6be04f2e bots: Pass realm to get_system_bot call in test_users. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 994ee70497 bots: Pass realm to self.notification_bot test helper. 2021-07-26 15:33:13 -07:00
Mateusz Mandera a01594e72b bots: Pass realm to get_system_bot call in DeferredWorker. 2021-07-26 15:33:13 -07:00
Mateusz Mandera f790a32d13 bots: Pass realm to get_system_bot calls in test_cache.py. 2021-07-26 15:33:13 -07:00
Mateusz Mandera 104139657b bots: Pass realm to get_system_bot calls in email_mirror. 2021-07-26 15:33:13 -07:00
Mateusz Mandera d45f3eecaa models: Add optional realm_id argument to get_system_bot. 2021-07-26 15:31:10 -07:00
PIG208 22c84e69ac tests: Fix inappropriate assertEqual.
We do not give a second argument to `self.assertEqual` here. We only need
to invoke the function.
2021-07-26 15:09:07 -07:00
PIG208 91de2cbe03 typing: Avoid scoped redefinition of different types.
Mypy doesn't allow redefinition of a variable using a different type
within the same scope.
https://github.com/python/mypy/issues/1174
2021-07-26 15:09:07 -07:00
PIG208 178a4b4eff tests: Initialize requests with different post data.
Instead of directly changing the `POST` attribute of a request, we
utilize the `HostRequestMock` initializer to produce requests with
different post data.
2021-07-26 15:09:07 -07:00
PIG208 c3abc0b472 tests: Correctly mock and invoke view functions.
The decorators require the decorated function to be a valid view
function. This changes the way the mocked view functions and requests
are implemented such that we can invoke view functions without future
type errors.
2021-07-26 15:09:07 -07:00
PIG208 57f46db09f tests: Pass HostRequestMock to export_realm.
`export_realm` accepts an HttpRequest as the first argument,
while `self.client_post` conflicts with it. Though the argument is
unused in `export_realm`, we keep it to be compliant with the
view function type.
2021-07-26 15:09:07 -07:00
PIG208 f280ebe4b5 decorator: Type zulip_login_required with overload.
As we only return the actual decorator as-is only if `function` is
`None`, we can use `@overload` to accurately annotate the return type
for the decorator.
2021-07-26 15:09:07 -07:00
PIG208 d2af20eb2a typing: Do explicit type conversion when appropriate.
When calling some functions or assigning values to certain attributes,
the arguments/right operand do not match the exact type that the
functions/attributes expect, and thus we fix that by converting types
beforehand.
2021-07-26 15:09:07 -07:00
PIG208 7d1c475f69 typing: Use assertions for function arguments.
Utilize the assert_is_not_None helper to eliminate errors of
'Argument x to "Foo" has incompatible type "Optional[Bar]"...'
2021-07-26 14:48:45 -07:00
PIG208 c4179428ac caching: Make sender type optional for flush_cache.
That's the correct Django interface for this function.
2021-07-26 14:48:07 -07:00
PIG208 2972e921b7 utils: Add a helper function for None-checks. 2021-07-26 14:46:45 -07:00
PIG208 cf8687662f confirmation: Use the correct type hints for create_confirmation_link.
Previously we annotate the first argument as `ContentType`, which
is wrong as suggested by django-stubs.
2021-07-26 14:46:45 -07:00
Dinesh beadb5ec7f test_middleware: Replace logging mocks with assertLogs. 2021-07-26 14:46:01 -07:00
Dinesh 2846d18854 test_decorators: Replace webhook logger mock with assertLogs.
Of the two other logging mocks left in this file, one checks
a logging call isn't made and another makes sure errors
aren't allowed by raising an exception as a side_effect
to the logger.
2021-07-26 14:46:01 -07:00
Dinesh 199f522808 test_classes.py: Replace markdown_logger mock with assertLogs.
Set level to 'ERROR' since exceptions create logs with that level.
2021-07-26 14:46:01 -07:00
Dinesh b195cc3635 test_push_notifications.py: Replace logging mocks with assertLogs.
Left the mocks which are used to assert a logging call isn't made.
2021-07-26 14:46:01 -07:00
Aman Agrawal 7d8f62e7a0 portico: Add a developer-community page.
This is intended to replace the ReadTheDocs page of this form, but
this commit does not yet update any links.
2021-07-25 17:01:26 -07:00
Tim Abbott 09b5bb7930 export: Improve error message for missing registrations. 2021-07-24 17:36:15 -07:00
Mateusz Mandera 1c64bed8e4 rate_limiter: Rate limit the /new/ endpoint. 2021-07-24 15:52:06 -07:00
Mateusz Mandera 4418aefde4 decorators: Extract rate_limit_request_by_ip function. 2021-07-24 15:52:06 -07:00