Commit Graph

13956 Commits

Author SHA1 Message Date
Mateusz Mandera d9ab70bdde queue_processors: Make timer_expired receive list of events as argument.
This will give queue workers more flexibility when defining their own
override of the method.
2021-07-06 13:46:48 -07:00
Mateusz Mandera c101f3acd6 queue_processors: Make timer_expired() a method.
This allows specific queue workers to override the defaut behavior and
implement their own response to the timer expiring. We will want to use
this for embed_links queue at least.
2021-07-06 13:46:48 -07:00
sahil839 c13f0626e3 tests: Use the list in models.py directly in do_test_realm_update_api.
The list of possible values of all settings was re-defined in
do_test_realm_update_api and we can instead use the list defined
in models.py which is used to validate values in views/realm.py.

There is no problem in order of values as we always initialize the
first value of the list.

This has also added some more values to test for a couple of
settings as a result.
2021-07-06 13:45:01 -07:00
sahil839 b4a0a33f4c tests: Test all values in the list in do_test_realm_update_api.
This commit changes do_test_realm_update_api to test all values
in the list of a given setting instead of checking only two
values.
2021-07-06 13:45:01 -07:00
sahil839 8314620314 tests: Change test_change_realm_default_language to test invalid value.
We change the test_change_realm_default_language to test only invalid
value and rename it to test_invalid_realm_default_language, because
we already test whether the value is changed correctly or not in
do_test_realm_update_api.
2021-07-06 13:45:01 -07:00
sahil839 6a3a07a359 tests: Remove test_change_bot_creation_policy from test_realm.py.
This commit removes test_change_bot_creation_policy which is used
to test changing bot_creation_policy using 'PATCH /realm' endpoint
as we already do this in do_test_realm_update_api and invalid value
is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 77471a5b44 tests: Remove test_change_email_address_visibility from test_realm.py.
This commit removes test_change_email_address_visibility which is used
to test changing email_address_visibility using 'PATCH /realm' endpoint
as we already do this in do_test_realm_update_api and invalid value is
also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 74023bfacf tests: Remove test_change_invite_to_stream_policy from test_realm.py.
This commit removes test_change_invite_to_stream_policy which is used
to test changing invite_to_stream_policy using 'PATCH /realm' endpoint
as we already do this in do_test_realm_update_api and invalid value
is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 99e5058bd6 tests: Remove test_change_invite_to_realm_policy from test_realm.py.
This commit removes test_change_invite_to_realm_policy which is used
to test changing invite_to_realm_policy using 'PATCH /realm' endpoint
as we already do this in do_test_realm_update_api and invalid value
is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 1c0e67e7c4 tests: Remove test_change_move_messages_between_streams_policy.
This commit removes test_change_move_messages_between_streams_policy
which is used to test changing move_messages_between_streams_policy
using 'PATCH /realm' endpoint as we already do this in
do_test_realm_update_api and invalid value is also tested in
test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 8831fb22b1 tests: Remove test_user_group_edit_policy from test_realm.py.
This commit removes test_user_group_edit_policy which is used
to test changing user_group_edit_policy using 'PATCH /realm'
endpoint as we already do this in do_test_realm_update_api and
invalid value is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 a1e6cbdc80 tests: Remove test_private_message_policy from test_realm.py.
This commit removes test_private_message_policy which is used to
test changing private_message_policy using 'PATCH /realm' endpoint
as we already do this in do_test_realm_update_api and invalid
value is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 f95faaf732 tests: Remove test_change_wildcard_mention_policy from test_realm.py.
This commit removes test_change_wildcard_mention_policy which is
used to test changing wildcard_mention_policy using 'PATCH /realm'
endpoint as we already do this in do_test_realm_update_api and
invalid value is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
sahil839 8e26f920ab tests: Remove test_change_stream_creation_policy from test_realm.py.
This commit removes test_change_stream_creation_policy which is
used to test changing create_stream_policy using 'PATCH /realm'
endpoint as we already do this in do_test_realm_update_api and
invalid value is also tested in test_invalid_integer_attribute_values.
2021-07-06 13:45:01 -07:00
Eeshan Garg 5a94bfcb88 docs: Add proper backlink to the homepage from /help and /api.
Currently, the "Home" link at the top takes one to the doc root,
i.e., /help or /api. This is a little misleading since "Home"
seems to be more synonymous with the Zulip homepage.

This commit adds a proper backlink to the top logo that takes you to
the homepage and renames "Home" to be more specific. The text after
"|" will now take you to the doc root instead (/help or /api). Note
that this allows us to link the /help and /api pages from the
homepage while ensuring that backlinks allow the visitor to get back
to the homepage.
2021-07-06 10:35:41 -07:00
Suyash Vardhan Mathur acb1244f93 api docs: Move stray get_events Python example.
We also remove the call_on_each_event note, which has been misplaced
ever since we created the "real-time-events" page.
2021-07-06 10:17:03 -07:00
Suyash Vardhan Mathur 7bbafbc881 python_examples: Add auto-rendering of extra imports.
Currently, there is no provision of rendering
additional imports automatically, and those examples
were hardcoded in the templates.

This commit adds an openapi parameter
to store the additional imports required for the endpoint.
Further, it changes code to replace `import zulip`
with the modified imports.
2021-07-06 10:14:46 -07:00
PIG208 43cca07b4b exceptions: Add ValidationFailureError. 2021-07-06 09:44:45 -07:00
Riken Shah a96c614420 message: Fix the edit topic bug by removing leading \n from msg content.
The reason for this bug is because of different striping
processes in the backend and frontend, i.e The frontend
checks if the message's `raw_content` has changed to
decide if the `content` of the message should be sent in
the request to the backend, or not. So, it removes the
leading new line ('\n') from the message `raw_content`
when checking it, which is causing the "Error saving edit:
You don't have permission to edit this message" error.

This commit fixes it by removing the leading new line
when cleaning message content.

The bug was explained by @punchagan and its solution
by @timabbott.
2021-07-06 09:41:54 -07:00
Anders Kaseorg 08592c9731 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-06 08:40:32 -07:00
Anders Kaseorg 3853285241 push_notifications: Replace PyAPNs2 with aioapns.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 20:16:50 -07:00
Anders Kaseorg 0bc002270c push_notifications: Use lru_cache decorator.
This does the same thing in a simpler way.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 20:16:50 -07:00
PIG208 83ea6c37c5 webhook: Add expect_noop option for check_webhook.
This change allow check_webhook to raise an error when a message is
sent and vice versa. This is useful when one payload is not expecting
any output messages.
2021-07-05 19:17:11 -07:00
PIG208 2f9c586af5 webhooks: Allow event registration using webhook_view decorator.
In addition to event filtering, we add support for registering supported
events for a webhook integration using the webhook_view decorator.

The event types are stored in the view function directly as a function
attribute, and can be later accessed via the module path and the view
function name are given (which is already specified the integrations.py)

Note that the WebhookTestCase doesn't know the name of the view function
and the module of the webhook. WEBHOOK_DIR_NAME needs to be overridden
if we want exceptions to raised when one of our test functions triggered
a unspecified event, but this practice is not enforced.

all_event_type does not need to be given even if event filters are used
in the webhook. But if a list of event types is given, it will be possible
for us to include it in the documentation while ensuring that all the
tested events are included (but not vice versa at the current stage, as
we yet not required all the events included in the list to be tested)

This guarantees that we can always access the list of all the tested
events of a webhook. This feature will be later plumbed to marcos to
display all event types dynamically in doc.md.
2021-07-05 19:10:31 -07:00
aryanshridhar 13399833b0 right_sidebar: Migrate to handlebars.
This commit migrates the `right_sidebar.html` Django template
to handlebars by creating a new file as `right_sidebar.hbs`
which is then rendered using `ui_init` module.

It also removes the tests in `test_home` due to the template
migration, since these elements aren't rendered on the backend
anymore.

We also remove `test_compute_show_invites_and_add_streams*`.

Fixes part of #18792.
2021-07-05 18:58:24 -07:00
aryanshridhar 8fb7ff7637 left_sidebar: Migrate to handlebars.
This commit migrates the `left_sidebar.html` Django template
to handlebars by creating a new file as `left_sidebar.hbs`
which is then rendered using `ui_init` module.

These are the minor changes introduced by virtue of template
migration -
 - The `compute_show_invites_and_add_streams` function now
   only concerns with the invite_to_realm_policy.
 - Renamed the `compute_show_invites_and_add_streams` function
   to `compute_show_invites` due to the above change.
 - Fixes relevant `test_home.py` tests due to the above
   changes.

Fixes part of #18792.
2021-07-05 18:49:43 -07:00
Anders Kaseorg ee8724e436 docs: Correct Big Blue Button to BigBlueButton.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 16:59:00 -07:00
Hashir Sarwar 87d90f47ee email_notifications: Show mentioned user group in email notifications.
Fixes #13080.
Cherry-picked from #15011 with edits.

Co-authored-by: Abhijeet Bodas <abhijeetbodas2001@gmail.com>
2021-07-05 14:23:59 -07:00
Hashir Sarwar 83399e2e72 push_notifications: Show mentioned user group in mobile notifications.
Part of #13080.
Cherry-picked from #15011 with edits.

Co-authored-by: Abhijeet Bodas <abhijeetbodas2001@gmail.com>
2021-07-05 14:23:59 -07:00
Abhijeet Prasad Bodas 167be7dbdc mentions: Send user group mention data to notification notices.
We will later use this data to include text like:
`<sender> mentioned @<user_group>` instead of the current
`<sender> mentioned you` when someone mentions a user group
the current user is a part of in email/push notification.

Part of #13080.
2021-07-05 14:23:59 -07:00
Abhijeet Prasad Bodas 07d6ab9753 notification_data: Add `get_user_group_mentions_data` function.
We will use this later to display which user group was mentioned
in push and email notifications.

`mentioned_user_group_ids` is kept as a List (not Set) to ensure proper
test coverage of the function, since it depends on the order of iteration,
and we cannot change the order of iteration for a set (which we'll need
to do for proper testing).

Part of #13080.
2021-07-05 14:23:59 -07:00
Abhijeet Prasad Bodas ee424c1f76 push_notifications: Assert observation written in comment.
This will ensure the comment does not go out of date.
2021-07-05 14:23:59 -07:00
Anders Kaseorg c56440ded0 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:23:06 -07:00
Anders Kaseorg 325cb89cbf check_redis: Fix for key format change and Python 3.
Commit 81d7dd1fda broke this nearly
eight years ago, so probably nobody cares except the ever-watchful eye
of mypy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:23:06 -07:00
Anders Kaseorg 24c239d991 openapi: Add missing __init__.py; fix type errors hidden by its absence.
The absence of __init__.py was preventing mypy from following any of
the zerver.openapi imports.  These errors were being silenced by
ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:20:39 -07:00
PIG208 6a04648fd7 exceptions: Add ResourceNotFoundError. 2021-07-05 11:02:12 -07:00
PIG208 a6e88a5a76 exceptions: Add AccessDeniedError. 2021-07-05 11:02:09 -07:00
Vishnu KS e0f5fadb79 billing: Downgrade small realms that are behind on payments.
An organization with at most 5 users that is behind on payments isn't
worth spending time on investigating the situation.

For larger organizations, we likely want somewhat different logic that
at least does not void invoices.
2021-07-02 13:19:12 -07:00
Vishnu KS cb64a19edf models: Rename get_human_billing_admin_users to be more explicit. 2021-07-02 12:04:41 -07:00
Priyansh Garg 94a2be06f3 markdown: Use a shared variable for IMAGE_EXTENSION. 2021-07-02 11:22:55 -07:00
Priyansh Garg 5b2e21965c data_import: Add import attachments support for Mattermost.
Add support for importing message attachments from Mattermost.

Fixes: #18959
2021-07-02 11:19:45 -07:00
Mateusz Mandera b9a8fb4453 upload: Deduplicate logic for public upload url creation.
get_public_upload_root_url and construct_public_upload_url_base were
both doing basically the same thing in the same. We deduplicate this,
making them share the same code, using the approach from
get_public_upload_root_url of using urljoin.

Using a format string is not a great idea, as it doesn't handle the case
of the URL already having parts that will be interpreted as format
string metacharacters. On the downside, this approach negatively affects
performance:

```
...: s = time.time()
...: for i in range(0, 250):
...:     r = u.get_public_upload_url("foo")
...: print(time.time()-s)
0.020366191864013672
```

up from 0.001 before this change.
2021-07-02 08:05:53 -07:00
Tim Abbott 64aa8f80a0 events: Add heartbeat events to tests and documentation.
Heartbeat events are an important part of the API, even though they
are noops, so it's important to document them.
2021-07-01 17:14:32 -07:00
sahil839 437f4c2288 settings: Rename 'Your bots' to 'Bots' in settings sidebar.
As we have changed the tab selector above from "Settings" to "Personal
settings", we can simply change "Your bots" to "Bots" as "Bots" is
clear enough given the personal settings context.

We also need to update the API documentation for bots accordingly.
2021-07-01 12:14:48 -07:00
sahil839 37620ffef0 gear_menu: Rename "Settings" option to "Personal settings".
Also includes change in markdown used in docs.
2021-07-01 11:47:14 -07:00
Suyash Vardhan Mathur c0615a00e6 openapi: Fix response description in outgoing-webhooks.
A response description was hardcoded in the template,
and the OpenAPI description was wrongly written. This commit
fixes the response description.
2021-07-01 11:20:37 -07:00
Suyash Vardhan Mathur ba36d46658 openapi: Add response descriptions for rest-error-handling.
Some response descriptions weren't migrated into
OpenAPI data, which was done for other API pages.
This commit migrates the response descriptions for
error-handling page.
2021-07-01 11:20:37 -07:00
sahil839 706ec9714c events: Set state["emojiset_choices"] only once.
There was a minor typo where state["emojiset_choices"]
was part of the loop for UserProfile.property_types.
2021-06-30 17:03:44 -07:00
sahil839 0c80d23857 help: Update help pages according to profile page redesign.
This commit fixes the documentation of settings as we have
replaced "Your account" section into two new sections -
"Profile" and "Account & privacy".

This commit also fixes a comment in the test for settings
documentation in test_middleware.py.
2021-06-30 16:56:03 -07:00
PIG208 dcbb2a78ca python: Migrate most json_error => JsonableError.
JsonableError has two major benefits over json_error:
* It can be raised from anywhere in the codebase, rather than
  being a return value, which is much more convenient for refactoring,
  as one doesn't potentially need to change error handling style when
  extracting a bit of view code to a function.
* It is guaranteed to contain the `code` property, which is helpful
  for API consistency.

Various stragglers are not updated because JsonableError requires
subclassing in order to specify custom data or HTTP status codes.
2021-06-30 16:22:38 -07:00
Tim Abbott 0be35f530b push_notifications: Replace 'REDACTED' string.
The previous string was bold, potentially confusing, and doesn't
explain clearly what's happening. We replace this with a string that's
more or less copied from what we do in email notifications with the
similar setting enabled.
2021-06-30 15:15:22 -07:00
Shelly c958a023fe i18n: Add i18n to 'REDACTED' message in push notifications.
When a user has disabled message content in mobile push notifications,
we send a fixed string (currently "REDACTED") as the content of the
notification. Previously, this string was not tagged for translation;
we fix that here.

Additionally, because mobile push notifications are generated in a
queue worker, they do not have the user's language set by the Django
middleware. Our email notifications solve that problem using
`override_language`; we do the same here.

We choose to do override_language in get_message_payload_apns and
get_message_payload_gcm, rather than the caller, in order to be
consistent with tests.

Tested end-to-end by tabbott by setting a translation for "REDACTED"
manually in German.

Fixes #18713.
2021-06-30 15:15:13 -07:00
Tim Abbott 0997eeae9e api docs: Document rate limiting rules.
Unfortunately, the example doesn't work due to some sort of bug with
the fixture generation system, which I've reported as #19072.
2021-06-30 13:40:07 -07:00
PIG208 0740c94517 webhooks: Support configuring destination stream by id. 2021-06-30 09:31:07 -07:00
sahil839 2bf19d5651 api: Remove incorrect realm setting in 'realm/update_dict' schema.
We incorrectly include many realm settings in the data section of
'realm/update_dict' schema. It should only contain the settings
related to message edit, realm icon, realm logo and authentication
methods and not other settings, becausea all the other settings send
'realm/update' event and not 'realm/update_dict' event.

This commit only removes 'allow_message_deleting' and others will
be removed separately.
2021-06-29 17:37:52 -07:00
sahil839 ce133dc762 models: Add can_edit_user_group helper.
This helper will be used to check whether
the user is allowed to edit user groups or
not. Currently it is not used, but will
be used in next commit where we will
refactor the user_group_edit_policy to use
COMMON_POLICY_TYPES.
2021-06-29 17:25:42 -07:00
PIG208 542516b07b integrations: Support params for events filtering.
We modify check_send_webhook_message to make it accept three new
parameters: only_events and exclude_events that are retrieved using REQ,
and complete_event_type, which is passed by the incoming webhook view
that is filtered according to the former two parameters.

Part of #18525.
2021-06-29 17:07:46 -07:00
PIG208 5ecbfecd77 webhook: Rename FIXTURE_DIR_NAME to WEBHOOK_DIR_NAME.
Since FIXTURE_DIR_NAME is the name of the folder that contains the view
and tests modules of the webhook and another folder called "fixtures" that
store the fixtures, it is more appropriate to call it WEBHOOK_DIR_NAME,
especially when we want to refer to the view module using this variable.
2021-06-29 17:01:54 -07:00
PIG208 b720e10d88 integrations: Reformat expected messages for readability. 2021-06-29 17:01:54 -07:00
Damian Parrino caea7f167d help: Reorganize articles on editing topics.
* Move content on moving topics between streams to a dedicated
  article. We advertise it as "move content" to hint that one can move
  messages or split topics, and link to it.

* This deletes change-the-topic-of-a-message, because the same content
  is already covered in rename-a-topic.

* This commit mostly just moves content between articles. Most of that
  content was redundant with the first few paragraphs of the surviving
  "rename a topic" article. The former "This is useful for" se ntence
  was adapted to the remaining article.

* This commit also adds a redirect for the removed article, and
  updates related links.
2021-06-29 13:52:06 -07:00
sahil839 c8849f8fe3 logo: Compute realm-logo url in frontend only.
This commit fixes the bug of always showing
day-mode realm logo when color scheme display
setting is set to automatic but the OS setting
is dark theme. This is because we cannot check
the OS setting on backend and we need to set
the logo url accordingly in frontend only.
So, we remove the logo url computation from
backend completely and instead compute it in
the frontend only.

Fixes #18778.
2021-06-29 13:42:44 -07:00
Steve Howell 9e1d98a512 widgets: Add range checks on backend for indexes. 2021-06-29 13:40:33 -07:00
Steve Howell c25dbf7020 widgets: Validate todo data on the backend. 2021-06-29 13:40:33 -07:00
aryanshridhar 8140a1a6ee info_overlay: Render search operators using info_overlay.
This commits ports the `search_operators.html` file from
./templates to handlebars, essentially creating a new file
as `search_operators.hbs` within /static/templates which is
then rendered using info_overlays.js.

As part of this migration, we rewrote the way internationalization was
done, since the previous implementation incorrectly did not support
languages with a different word order than English.

We also not consistently use periods at the end of the descriptions.

Co-authored-by: Tim Abbott <tabbott@zulip.com>

Fixes #18504.
2021-06-28 09:14:09 -07:00
aryanshridhar 942f4372f0 info_overlay: Render keyboard shortcuts using info_overlay.
This commits ports the `keyboard_shortcuts.html` file from
using the Django template to handlebars, essentially creating
a new file as `keyboard_shortcuts.hbs` within /static/templates
which is then rendered using info_overlays.js.

Fixes part of #18792.
2021-06-27 18:29:52 -07:00
akshatdalton 44a298b671 minor: Use `OUTER_CAPTURE_GROUP` variable instead of string value. 2021-06-25 17:43:27 -07:00
akshatdalton 490f6b6880 markdown: Extract regex in local variables. 2021-06-25 17:43:01 -07:00
Gaurav Pandey 1d12fcd168 management: Delete fix_unreads command.
This command was part of the complex migration to introduce the
`unread_msgs` data structure as the source of truth for unreads.
Effectively, it's a migration to remove anomalies that we ran several
times before turning it into the final 0104_fix_unreads.py migration.

Fixes part of #18898.
2021-06-25 09:14:45 -07:00
Gaurav Pandey af08bcdb3f management: Delete send_stats command.
This command is part of a statsd infrastructure that we stopped
supporting years ago. Its only purpose for some time has been to
provide sample code for how the restart script might trigger a
notification to a graphing system, which doesn't justify maintaining
it.

Fixes part of #18898.
2021-06-25 09:13:48 -07:00
Gaurav Pandey 7db3d76ecd management: Delete dump_messages command.
This is basically a hacky version of `manage.py export`, implemented
primarily for development work.

Fixes part of #18898.
2021-06-25 09:13:19 -07:00
Gaurav Pandey 785895c265 management: Delete turn_off_digests command.
This command was part of early prototyping of the digests feature, and
in particular its purpose is better served via the organization-level
setting to control digest emails for the organization.

Fixes part of #18898.
2021-06-25 09:12:50 -07:00
Gaurav Pandey 461c6bd791 management: Delete generate_multiuse_invite_link command.
This command was written to allow generating multiuse invite links
before the "Invite a user" UI supported them. It no longer has a
purpose and can be safely deleted.

Fixes part of #18898.
2021-06-25 09:12:17 -07:00
Gaurav Pandey ef7ec4dbf2 management: Delete generate_invite_links command.
This command predates there being a normal UI for inviting users to
Zulip. It no longer has a role for which it's a better way to do
things. (Especially with upcoming API documentation for the endpoint).

Fixes part of #18898.
2021-06-25 09:11:27 -07:00
Gaurav Pandey 331ba801ae management: Delete set_message_flags command.
This command was introduced in 2013 via
6d6c3364dc as part of implementing
marking messages as read in a separate process for performance reasons.

We fixed the performance issues and removed that pipeline years ago,
but forgot to delete this.

Fixes part of #18898.
2021-06-25 09:09:53 -07:00
Abhijeet Prasad Bodas e196ea7e64 event_queue: Consistently user `user_notifications_data` as variable name.
This disambiguates this object from the those sent via the `send_event`
calls, which are also called "user_data".
2021-06-25 08:54:01 -07:00
Abhijeet Prasad Bodas 733e0ae75e notification_data: Rename `sender_id` -> `acting_user_id`.
This better shows the situation for message edits, where we use the same
class.
2021-06-25 08:54:00 -07:00
PIG208 87fc2bbe50 markdown: Remove do_render_markdown.
The function becomes unnecessary as we start to use a dataclass to
store the extra fields that it used to patch into the Messasge object.
2021-06-24 18:14:53 -07:00
PIG208 75cea329b4 markdown: Refactor out additional properties added to Message.
This adds a new class called MessageRenderingResult to contain the
additional properties we added to the Message object (like alert_words)
as well as the rendered content to ensure typesafe reference. No
behavioral change is made except changes in typing.

This is a preparatory change for adding django-stubs to the backend.

Related: #18777
2021-06-24 18:14:53 -07:00
PIG208 c5e5814242 message_edit: Refactor do_update_mobile_push_notification for clarity.
This is a prep commit of #18917.
2021-06-24 18:14:53 -07:00
Vishnu KS 4e0dac9e6c actions: Update default streams state when user role is changed. 2021-06-24 17:02:05 -07:00
Tim Abbott d7df1f208a test_signup: Add temporary nocoverage marker.
This was introduced in 23530ac81e and
will be used soon, but isn't used yet.
2021-06-24 15:24:15 -07:00
Tim Abbott abf2571133 test_signup: Support passing no password in verify_signup. 2021-06-24 14:55:05 -07:00
Gilbert Bishop-White 23530ac81e test_signup: Refactor test_signup_existing_email to use verify_signup.
Add a new `verify_signup` helper function, which currently implements
enough functionality to be used by `test_signup_existing_email`.

This is the first step towards #7564.
2021-06-24 14:52:25 -07:00
Ganesh Pawar eca2e10c4a test_realm: Extract out test_change_stream_creation_policy.
This is a prep commit in preparation of splitting
create_stream_policy into create_private_stream_policy
and create_public_stream_policy.

This extracts it in a way to make it possible to easily test
different stream policies in the upcoming stream policy split.
2021-06-24 14:08:49 -07:00
Ganesh Pawar 113e27615d test_subs: Extract out test_user_settings_for_adding_streams.
This is a prep commit in preparation of splitting
create_stream_policy into create_private_stream_policy
and create_public_stream_policy.

This extracts it in a way to make it possible to easily test
different stream policies in the upcoming stream policy split.
2021-06-24 14:08:49 -07:00
Ganesh Pawar a1ab79992c test_subs: Extract out test_user_settings_for_creating_streams.
This is a prep commit in preparation of splitting
create_stream_policy into create_private_stream_policy
and create_public_stream_policy.

This extracts it in a way to make it possible to easily test
different stream policies in the upcoming stream policy split.
2021-06-24 14:08:49 -07:00
Ganesh Pawar e74ad23091 test_subs: Remove test_create_stream_policy_setting.
test_create_stream_policy_setting (in class StreamAdminTest) and
test_user_settings_for_creating_streams (in class SubscriptionAPITest)
test essentially the same thing.

So, remove one of them.

Removing test_create_stream_policy_setting makes sense,
since class StreamAdminTest tests things admins can do, whereas
non-admin users can create streams.
2021-06-24 14:08:49 -07:00
Ganesh Pawar aec493a8d3 test_subs: Remove test_invite_to_stream_by_invite_period_threshold.
test_invite_to_stream_by_invite_period_threshold (in class StreamAdminTest)
and test_user_settings_for_subscribing_other_users
(in class SubscriptionAPITest) test essentially the same thing.

So, remove one of them.

Removing test_invite_to_stream_by_invite_period_threshold makes sense,
since class StreamAdminTest tests things admins can do, whereas
non-admin users can invite other users.
2021-06-24 14:08:49 -07:00
Ganesh Pawar d45e7274f9 test_subs: Avoid direct usage of list_to_streams.
This was used to test can_create_stream property of a guest user.
There are better ways to test it, which are already implemented in
test_can_create_streams.
2021-06-24 14:08:48 -07:00
Tim Abbott 8aa05620d2 help: Document resolving topics. 2021-06-24 13:57:02 -07:00
Suyash Vardhan Mathur 23b991a801 api docs: Replace most markdown files with a common template.
This PR adds a basic .md template that is followed by lot of /api
pages. Since we have recently done the migration work to ensure that
our REST API documentation pages for individual endpoints are almost
all identical files following a common pattern, we can now get the
payoff of deleting them all in favor of a shared template.

This removes 2000 lines of somewhat finicky configuration from the
codebase, and thus should save significant effort when documenting new
API endpoints in the future.

The markdown files for endpoints or other pages which deviate from the
standard template remain, and the docs are instead generated from
those files using the existing system.
2021-06-24 10:42:08 -07:00
Suyash Vardhan Mathur 15123bca68 openapi: Refactor return value of get_path function.
The ArticlePath dataclass added can now be used
for the return value of get_path function to make
it more extensible and robust.
2021-06-24 10:38:44 -07:00
Suyash Vardhan Mathur 1375e99ae7 openapi: Add dataclass for holding article path fields.
The returned values of get_path function would be
expanded soon, and defining a dataclass would make
the code cleaner for returning and using the fields.
2021-06-24 10:38:44 -07:00
Suyash Vardhan Mathur 0c4330f926 openapi: Automatically add python tab whenever required.
As a part of goal of moving towards a common template,
the hardcoded python tabs need to be removed to ensure
that endpoints which don't have python examples can be
covered by the common template as well.

This commit also modifies the markdown extension for python
examples to render empty string in case the examples don't
exist, which would allow it to be called whether the endpoint
has python examples or not.
2021-06-24 10:38:44 -07:00
Suyash Vardhan Mathur fab6a5192c openapi: Fix 'no parameters found' with x-parameter-description.
Currently, the message that no parameters are accepted by
the endpoint is displayed if there are no parameters in
OpenAPI data, but it is possible that information is
encoded in x-parameter-description (example in upload-file
endpoint), and we want to display that information rather
than the message.

Added an if condition to check the same.
2021-06-24 10:38:33 -07:00
Suyash Vardhan Mathur ad9d1c0f80 openapi: Fix return values in delete-queue endpoint.
The check for whether to do the special GET /events logic was
incorrectly also covering DELETE /events.
2021-06-24 10:37:56 -07:00
Suyash Vardhan Mathur fe07f67360 apidocs: Standardize parameters text for get-own-user.
We use the "does not accept any parameters" language in the common
template that we'll be migrating to shortly, so we remove this
variance (And adjust its test).
2021-06-24 10:37:56 -07:00
Tim Abbott 6b67b18c4b api docs: Clarify limits on use of propagate_mode. 2021-06-24 09:57:13 -07:00
akshatdalton 8d158852fd webhooks/slack_incoming: Update regex to correctly convert emphasis.
This was a bug where we were converting _text_ to **text**.
For emphasis we use the *text* format.
2021-06-24 09:43:32 -07:00
Abhijeet Prasad Bodas 1cf1d147aa event_queue: Move notification trigger logic to notification_data.
This removes some complexity from the event_queue module.
To avoid code duplication, we reduce the `is_notifiable` methods to
internally just call the `trigger` methods and check their return value.
2021-06-24 09:35:18 -07:00
Abhijeet Prasad Bodas 66192825c0 maybe_enqueue_notifications: Take in notification_data dataclass.
* Modify `maybe_enqueue_notifications` to take in an instance of the
dataclass introduced in 951b49c048.

* The `check_notify` tests tested the "when to notify" logic in a way
which involved `maybe_enqueue_notifications`. To simplify things, we've
earlier extracted this logic in 8182632d7e.
So, we just kill off the `check_notify` test, and keep only those parts
which verify the queueing and return value behavior of that funtion.

* We retain the the missedmessage_hook and message
message_edit_notifications since they are more integration-style.

* There's a slightly subtle change with the missedmessage_hook tests.
Before this commit, we short-circuited the hook if the sender was muted
(5a642cea11).
With this commit, we delegate the check to our dataclass methods.
So, `maybe_enqueue_notifications` will be called even if the sender was
muted, and the test needs to be updated.

* In our test helper `get_maybe_enqueue_notifications_parameters` which
generates default values for testing `maybe_enqueue_notifications` calls,
we keep `message_id`, `sender_id`, and `user_id` as required arguments,
so that the tests are super-clear and avoid accidental false positives.

* Because `do_update_embedded_data` also sends `update_message` events,
we deal with that case with some hacky code for now. See the comment
there.

This mostly completes the extraction of the "when to notify" logic into
our new `notification_data` module.
2021-06-24 09:35:17 -07:00
Abhijeet Prasad Bodas dedc39f139 notifications_data: Rename `id` -> `user_id`.
We also make this a mandatory named argument for our test helper
for clarity.
2021-06-24 17:34:50 +05:30
Abhijeet Prasad Bodas f6e705d477 maybe_enqueue_notifications: Require all keyword arguments.
This is a more readable way to call the function.
2021-06-24 17:34:50 +05:30
Suyash Vardhan Mathur e892a02bb8 curl_examples: Fix curl testing for template's examples.
AUTHENTICATION LINE variable needs to be set after each
line executed, but in the current code, it wasn't being
set in endpoints whose files were removed in favour of
the pages being generated directly from OpenAPI data.

Moved the block to set AUTHENTICATION LINE in the loop
which executes each command, which fixes the bug.
2021-06-23 16:12:18 -07:00
Suyash Vardhan Mathur d0e44eaaae openapi: Move stray register-queue description into OpenAPI. 2021-06-23 16:12:18 -07:00
Suyash Vardhan Mathur dd7c3de36f openapi: Move stray get-user-groups description into OpenAPI. 2021-06-23 16:09:36 -07:00
Suyash Vardhan Mathur 7f8d8e0a9c openapi: Move stray update-notification-settings description into OpenAPI. 2021-06-23 16:05:47 -07:00
Priyansh Garg 6d39dd9d01 import_realm: Fix stream `rendered_description` not being set.
While importing a realm, the stream dictionaries in data['zerver_stream']
already contains the field named `rendered_description`, which is set to
`""`. This lead the code to assume that the stream rendered_description
was already set, due to which, it was not setting the rendered_description
field for any stream.
2021-06-23 08:03:43 -07:00
Suyash Vardhan Mathur 3129895818 openapi: Move stray custom emoji description into OpenAPI. 2021-06-22 17:39:29 -07:00
Suyash Vardhan Mathur 3f104c378b openapi: Move stray get messages description into OpenAPI. 2021-06-22 17:39:22 -07:00
Suyash Vardhan Mathur 570f3d350c apidocs: Move stray update message description to OpenAPI. 2021-06-22 17:39:17 -07:00
Suyash Vardhan Mathur f90b23b265 apidocs: Move stray upload file description to OpenAPI. 2021-06-22 17:39:12 -07:00
Suyash Vardhan Mathur 987b7206aa apidocs: Move stray display settings description to OpenAPI. 2021-06-22 17:39:06 -07:00
Suyash Vardhan Mathur 78910ec16a apidocs: Move stray message flags description to OpenAPI. 2021-06-22 17:38:51 -07:00
Suyash Vardhan Mathur 550e43bb9f apidocs: Move stray events description text to OpenAPI. 2021-06-22 17:38:34 -07:00
Suyash Vardhan Mathur d6ed38e0c9 apidocs: Move stray server settings description to OpenAPI. 2021-06-22 17:38:30 -07:00
Suyash Vardhan Mathur aedca6d6c6 apidocs: Move stray message history description to OpenAPI. 2021-06-22 17:38:10 -07:00
Suyash Vardhan Mathur 8d7634a572 apidocs: Move stray events description to OpenAPI. 2021-06-22 17:38:06 -07:00
Suyash Vardhan Mathur 8da27b8a5b apidocs: Move stray delete description reference to OpenAPI. 2021-06-22 17:37:44 -07:00
sahil839 37bf160298 queue_processor: Add langauge to the events added to invites queue.
This is a prep commit for adding realm-level default for various
user settings. We add the language, in which the invite email will
be sent, to the dict added to queue itself to avoid making queries
in a loop when sending multiple emails from queue.

We also handle the case for old events in the queue.
2021-06-22 16:55:32 -07:00
sahil839 6a191a7bf0 queue_processor: Remove email_body from the invite email events.
We removed the use of email_body field in 47fcb27e39, but was
still passed in events from do_resend_user_invite_email and
in tests. So this commit removes the email_body field from
these places.
2021-06-22 16:55:32 -07:00
Suyash Vardhan Mathur 35ba5b8d6f openapi: Add markdown extension for parameter descriptions.
As a goal of a common template, there
is a need for a tool to auto-generate
general description for all parameters
directly from OpenAPI data.

This description is to be stored in
x-response-description field, and this
commit adds a markdown extesion to process the same.
2021-06-22 10:58:43 -07:00
Suyash Vardhan Mathur dbcffebf2c openapi: Add markdown extension for response descriptions.
As a goal of a common template, there
is a need for a tool to auto-generate
general description for all responses
directly from OpenAPI data.

This description is to be stored in
x-response-description field, and this
commit adds a markdown extesion to process the same.
2021-06-22 10:58:43 -07:00
Abhijeet Prasad Bodas 226ac4bd3b tornado: Use `UserMessageNotificationsData` for update_message.
This lets us simplify what had previously been somewhat duplicated
logic between the send and edit code paths.
2021-06-22 10:27:59 -07:00
Abhijeet Prasad Bodas c3fb413119 message edit: Don't send mentioned user_ids in event dict.
We already have this data in the `flags` for each user, so no need to
send this set/list in the event dictionary.

The `flags` in the event dict represent the after-message-update state,
so we can't avoid sending `prior_mention_user_ids`.
2021-06-22 10:27:55 -07:00
Mateusz Mandera 85e19b2bde upload: Use URL manipulation for get_public_upload_url logic.
This is much faster than calling generate_presigned_url each time.

```
In [3]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
0.0010945796966552734
```
2021-06-22 09:35:56 -07:00
Mateusz Mandera e883ab057f upload: Cache the boto client to improve performance.
Fixes #18915

This was very slow, causing performance issues. After investigating,
generate_presigned_url is the cheap part of this, but the
session.client() call is expensive - so that's what we should cache.

Before the change:
```
In [4]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
6.408717393875122
```

After:
```
In [4]: t = time.time()
   ...: for i in range(250):
   ...:     x = u.get_public_upload_url("foo")
   ...: print(time.time()-t)
0.48990607261657715
```

This is not good enough to avoid doing something ugly like replacing
generate_presigned_url with some manual URL manipulation, but it's a
helpful structure that we may find useful with further refactoring.
2021-06-22 09:35:19 -07:00
Tim Abbott e231a03eff message_edit: Fix non-alternating resolve topic notifications.
Previously, it was possible for an unusual series of topic-edit
actions to result in Notification Bot reporting that a topic was
marked as resolved that had already been marked as resolved, etc.
2021-06-21 12:45:19 -07:00
Tim Abbott b2dd15fd86 message_edit: Reject buggy noop topic edit requests.
A buggy client might send a message_edit request to change the topic
field, sending the current topic as the new value. Previously, we
would treat that as a normal request to edit the topic; now we act as
though the API request had not requested a topic change.  In the
common case that only the topic was in the edit request, this now
results in an error that should help client implementations identify
their bug.

This fixes a bad interaction with the "unresolve topic" logic, which
assumed that upstream logic had verified that the topic was actually
changing.
2021-06-21 12:16:00 -07:00
sahil839 5a47e1f289 api: Remove incorrect realm setting in 'realm/update_dict' schema.
We incorrectly include many realm settings in the data section of
'realm/update_dict' schema. It should only contain the settings
related to message edit, realm icon, realm logo and authentication
methods and not other settings, becausea all the other settings send
'realm/update' event and not 'realm/update_dict' event.

This commit only removes 'default_twenty_four_hour_time' and
'default_language', others will be removed separately.
2021-06-21 11:30:57 -07:00
sahil839 c42031bc05 signup: Do not translate signup messages sent to "Zulip internal" realm. 2021-06-21 11:30:56 -07:00
Suyash Vardhan Mathur e5221a8434 curl_examples: Fix curl testing for multiple configs.
Now, the markdown extension of curl_examples generates
all examples of all possible configurations with
their descriptions, and so we need to separate
executable curl commands from the rest of the raw
HTML.

This commit simply changes the indentation of the
block and replaces the command being tested
with each element of the commands array. This
was split for an easier review.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur acbae4b6cf curl_examples: Fix curl testing for multiple configs.
Now, the markdown extension of curl_examples generates
all examples of all possible configurations with
their descriptions, and so we need to separate
executable curl commands from the rest of the raw
HTML.

This commit adds a commands variable to store
all the curl commands in HTML using regex.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur 6bd11285c1 openapi: Fix minor assertion in curl examples.
Now, include and exclude configuration
are fetched from openapi data, and only
one type can be encoded for every example.

This removes the need for the assertion to
test if both include and exclude are present
since at a time, only one can be present.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur 246262fb57 openapi: Modify curl example generation logic.
This commit adds support for using the
x-curl-examples-parameters parameter in OpenAPI
data to fetch curl examples configuration. This
also contains any descriptions necessary for each
example, and directly generates all possible
curl examples directly.

A follow-up commit is needed to modify the templates
accordingly.
2021-06-21 11:06:05 -07:00
Suyash Vardhan Mathur a56ad2a26a openapi: Add curl examples' configurations in OpenAPI data.
As a goal of moving towards a common template,
we need to fetch curl examples' configurations
directly from openapi data instead of having them
hardcoded in templates. This commit introduces
x-curl-examples-parameters to store the configs
for the same.
2021-06-21 11:06:05 -07:00
Abhijeet Prasad Bodas c6d696cf61 process_message_event: Deduplicate `is_notifiable` logic.
We use the `UserMessageNotificationsData` methods to check if the
message is notifiable, and use a cleaner early continue pattern.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas 5c483e3b58 get_active_presence_idle_user_ids: Check notifiability more thoroughly.
* Have the `get_active_presence_idle_user_ids` function look at all the
user data, not just `private_message` and `mentioned`.
* Fix a couple of incorrect `missedmessage_hook` tests, which did not
catch the earlier behaviour.
* Add some comments to the tests for this function for clarity.
* Add a helper to create `UserMessageNotificationsData` objects from the
user ID lists. This will later help us deduplicate code in the event_queue
logic.

This fixes a bug which earlier existed, that if a user turned on stream
notifications, and received a message in that stream which did not mention
them, they wouldn't be in the `presence_idle_users` list, and hence would
never get notifications for that message.

Note that, after this commit, users might still not get notifications in
the above scenarios in some cases, because the downstream logic in the
notification queue consumers sometimes erroneously skips sending
notifications for stream messages.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas aeb2ad5f46 notification_data: Annotate `flags` with `Collection`.
Since `flags` here could be iterated through multiple times
(to check for push/email notifiability), we use `Collection`.
Inspired by 871e73ab8f.

The other change here in the `event_queue` code is prep for using
the `UserMessageNotificationsData` class there.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas 8182632d7e notification_data: Add methods to determine notifiability.
We will later consistently use these functions to check for notifiable
messages in the message send and event_queue code.

We have these functions accept the `sender_id` so that we can avoid the
`private_message = message["type"] == "private" and user_id != sender_id`
wizardy.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas ed58393079 message send: Optimize how user data is sent to Tornado.
Before this commit, we used to pre-calculate flags for user data and send
it to Tornado, like so:
```
{
    "id": 10,
    "flags": ["mentioned"],
    "mentioned": true,
    "online_push_enabled": false,
    "stream_push_notify": false,
    "stream_email_notify": false,
    "wildcard_mention_notify": false,
    "sender_is_muted": false,
}
```

This has the benefit of simplifying the logic in the event_queue code a bit.

However, because we sent such an object for each user receiving the event,
the string keys (like "stream_email_notify") get duplicated in the JSON
blob that is sent to Tornado.

For 1000 users, this data may take up upto ~190KB of space, which can
cause performance degradation in large organisations.

Hence, as an alternative, we send just the list of user_ids fitting
each notification criteria, and then calculate the flags in Tornado.
This brings down the space to ~60KB for 1000 users.

This commit reverts parts of following commits:
- 2179275
- 40cd6b5

We will in the future, add helpers to create `UserMessageNotificationsData`
objects from these lists, so as to avoid code duplication.
2021-06-21 10:52:59 -07:00
Abhijeet Prasad Bodas 6167c36adc event_queue: Translation code for user data migration.
This is separate from the next commit for ease of testing.
To verify that the compatibility code works correctly, all message send
and event_queue tests from our test suite should pass on just this commit.
2021-06-21 10:52:59 -07:00
Dinesh a71ee405e4 test_example: Add a few more common testing methods.
Adds an example to illustrate the use of:
* self.settings()
* self.assertLogs()
* self.assert_in_success_response()
2021-06-20 07:16:35 -04:00
Tim Abbott bfb6ff44d1 message_edit: Fix unicode encoding of resolve topic check.
We now encode resolved topics with just:

U+2714 HEAVY CHECK MARK, SPACE

Previously, the encoding was unintentionally this:

U+2714 HEAVY CHECK MARK, U+FE0F VARIATION SELECTOR-16, SPACE
2021-06-18 17:46:28 -07:00
akshatdalton 4cff56a47c minor: Rename variable from `filters` to `linkifiers`. 2021-06-18 22:15:58 +00:00
Tim Abbott 696236b6fc left sidebar: Implement basic resolve topic option.
Fixes part of #18751.
2021-06-18 09:24:48 -07:00
Gaurav Pandey 8fc3715ea8 settings: Remove language_list_dbl_col from page_params.
The language_list_dbl_col parameter in the page_params
is used by only the web client frontend. The value is
calculated in the backend and then passed as a page_param
which is unnecessary considering that the whole process
is beneficial for the front_end only. Hence move the entire
calculation code to the frontend.

Fixes part of #18673.
2021-06-17 17:02:27 -07:00
Gaurav Pandey 56d85fb833 settings: Remove default_language_name from page_params.
default_language_name was a part of page_params which is actually
redundant considering that we already have language_list and
default_language available to frontend which can be used to
get the default_language_name and hence prevents the backend
from sending an additional parameter.

Fixes part of #18673.
2021-06-17 16:56:21 -07:00
akshatdalton 0da00018ad models: Update `group_match_regex` to correctly detect parameters in `url_format_string`.
This is a follow-up for 98f8d94b25.
For cases when url_format_string is like https://example.com/%%(foo)s/%(bar)s
group_match_regex should only detect `bar` as the intended
parameter and not `foo`.
2021-06-17 15:33:36 -07:00
Carlos Bederian 1b51792459 email_mirror: Improve filter_footer delimiter detection 2021-06-17 09:11:28 -07:00
Tim Abbott 8ce1fd1c50 linkifiers: Sort validation errors in alphabetical order.
This prevents nondeterministic failures in the test suite when there's
more than one error.
2021-06-16 18:12:53 -07:00
Rohitt Vashishtha 98f8d94b25 linkifiers: Add joint validation for linkifier urls and patterns.
We now validate the linkifier urls and patterns together, and add
the following additional checks:

1. All groups in the pattern must be used in the URL format string.
2. All groups in the URL format string must be declared in the pattern.

Linkifier pattern is now validated inside the `clean` method.
`filter_pattern_validator` is moved from `clean_fields` to `clean`
method as a safe check. As a result of this, a Puppeteer test case
is updated.

NOTE: The changes here are IN ADDITION to the existing validations.

Fixes #16482.

Co-authored-by: akshatdalton <akshat.dak@students.iiit.ac.in>
2021-06-16 17:51:33 -07:00
Tim Abbott b345b492db tests: Exclude another redirect from URL testing. 2021-06-16 16:25:49 -07:00
sahil839 38fac6c359 settings: Add moderators and members options in edit_topic_policy.
This commit adds moderators, full members and members options for
edit_topic_policy in both the backend and frontend.
2021-06-16 15:04:29 -07:00
sahil839 02db407398 tests: Pass acting user name to helper functions in topic edit tests.
This is a prep commit for adding members, full members and moderator
options to edit_topic_policy. As we will be adding tests for these
options, we will need to add a login statment repeatedly and this
helps us in avoiding that.
2021-06-16 14:59:47 -07:00
sahil839 611de0faa9 tests: Login as iago in set_message_editing_params.
This is a prep commit for adding moderators, full
members and member roles in edit_topic_policy.

As we add these new options, we will add tests with
user with all these roles and thus we would need to
login as iago repeatedly when changing parameters.
So, to avoid this we instead login as Iago in
set_message_editing_params itself.
2021-06-16 14:59:47 -07:00
sahil839 828759d2ba models: Replace allow_community_topic_editing with edit_topic_policy.
This commit replaces the allow_community_topic_editing boolean with
integer field edit_topic_policy and includes both frontend and
backend changes.

We also update settings_ui.disable_sub_settings_onchange to not
change the color of label as we did previously when the setting
was a checkbox. But now as the setting is dropdown we keep the
label as it is and we don't do anything with label when disabling
dropdowns. Also, this function was used only here so we can safely
change this.
2021-06-16 14:59:36 -07:00
sahil839 283a3a5c28 actions: Remove 'topic_name is None' check from can_edit_content_or_topic.
We do not need the 'topic_name is None' check in this function as this is
called only when atleast one of the content and topic_name is not None,
and this condition cannot be true as there is 'content is not None'
check just before it.

Thus, 'if topic_name is None' condition being true means that both content
and topic_name are None which is not possible as this function itself will
not be called in such case. An assert statement is added to check that
topic_name is not None to make sure that it is handled when the function
is called in some other way later.
2021-06-16 14:37:44 -07:00
Suyash Vardhan Mathur 1acee9f516 apidocs: Migrate admin_config out of templates.
Currently, the `admin_config` configuration was
hardcoded in the templates, but as a goal of creating
a common template, we need to move all configurations
outside.

Moved the checking for function and language which need
admin_config out of templates into the code and added a
boolean `x-admin-config` to store whehter the operation
requires admin config.

Also, added the banner for admin access to auto-generate
if admin_config is present, and fixed the admin_config
for endpoints that were earlier missing it in the templates.
2021-06-16 09:52:43 -07:00
Tim Abbott 9aca1dffcd test_auth_backends: Add types for subclass fields of SocialAuthBase.
These are used in the tests on this base class, so they need to be
defined for mypy to be happy.  Discovered when we enabled Django
stubs.
2021-06-16 08:28:10 -07:00
Riken Shah e91054ef19 context_processors: Add test for `get_apps_page_url`.
This is a follow-up commit to 61b30e61ad.
2021-06-16 07:49:03 -07:00
Aman Agrawal 64f04e9913 web_public_view: Replace `web_public_visitor` with `spectator`.
We use `spectator` term instead of `web_public_visitor` as it is
more concise and better represents this type of users.
2021-06-16 07:46:34 -07:00
Eeshan Garg 512229cf7d plans_page: Fix broken redirection to /accounts/login/.
When an unauthenticated user tries to access the /plans page, we
redirect to /accounts/login/?next=plans (note the missing slash
before "plans"). After the user is authenticated, they are then
redirected to /accounts/login/plans, which is an invalid URL. The
correct URL should be just /plans.

This commit solves this by prefixing the "plans" in the query
parameter with a forward slash, which results in the correct
redirect URL, i.e., /plans.
2021-06-15 16:49:01 -07:00
Abhijeet Prasad Bodas b6806fbf9d message send: Calculate `presence_idle_user_ids` after building user data.
This is a direct code move and a prep change for having this function look
at all the user data variables, not just `private_message` and `mentioned`.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 42e4fa7952 message send: Refactor how user data is sent for the event.
This is a prep change for calling `get_active_presence_idle_user_ids`
after we have collected all user data variables, so that that function
does not erroneously skip some user IDs from not having the complete
data.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 3ff3ded1ae process_message_event: Always calculate `online_push_enabled`.
Unlike `receiver_is_off_zulip`, fetching from a dict is pretty cheap,
so we can calculate `online_push_enabled` along with the other
variables.

This is a prep change to start using the dataclass introduced in the
earlier commit in this code.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 951b49c048 message send: Introduce dataclass to wrap user-notifications variables.
We will in later commits, extend this class to contain methods
to determine if a message is notifiable or not, but for now
we only turn it into a dict and pass it on.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 2179275020 event_queue: Deduplicate `mentioned` flag calculation.
This gives us a single place where all user data for the message
send event is calculated, and is a prep change for introducing
a TypedDict or dataclass to keep this data toghether.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 40cd6b5440 message send: Calculate `wildcard_mention_notify` pre-send_event.
This deduplicates some logic, and makes it so that
`wildcard_mention_notify` always contains what its name says it does.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 919f0a5964 message send: Extract local variables for user event data. 2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 56f6656d8b message send: Don't create user data list with inline `for`.
This will make the code more readable, and will allow creating local
variables within the loop for reuse.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 5a642cea11 missedmessage_hook: Don't enqueue notificationss if sender is muted.
This is a follow up to 71742dce24 to handle
muted senders in the missedmessage_hook too.
2021-06-15 12:30:31 -07:00
Abhijeet Prasad Bodas 86a20c5b3c event queue: Always set internal_data flags.
We have already calculated these values, so storing them should not cause
significant performance degradation.

This is a prep chenge for sending a few more flags through internal_data,
namely if `sender_is_muted`.
2021-06-15 12:30:31 -07:00
Riken Shah b2ab448b71 gear_menu: Extract the gear menu as a handlebars template.
For this extraction, we need to move some context
parameter (from home_real in `views/home.py`) to extra
page_params parameter (of
build_page_params_for_home_page_load in
`lib/home.py`) so handlebars template can access them.

While moving I confirmed that these parameters are not
used elsewhere if some parameter is used elsewhere
(like `apps_page_url`) then I didn't remove it from the
context list, I just added it to the page_params list.

Fixes: #18795.
2021-06-15 12:17:45 -07:00
Riken Shah d2f7aaf6f6 refactor: Rename `enable_marketing_emails_enabled` param of page_params.
This is a prep commit to extract the gear menu as a
handlebars template.

We are renaming `enable_marketing_emails_enabled` to
`corporate_enabled` as it will be also used in the
handlebars template of the gear menu.
2021-06-15 12:05:37 -07:00
Riken Shah 5fe21ae08f refactor: Make `UserProfile` field optional for `get_billing_info`.
This is preparatory work for extracting the gear menu
as a handlebars template.
2021-06-15 12:05:37 -07:00
Riken Shah 61b30e61ad home: Extract `get_apps_page_url` function from `zulip_default_context`.
This is a prep commit to extract the gear menu as a
handlebars template.

We are extracting it as a function so we can also use it as
a parameter in `page_params`.
2021-06-15 12:05:37 -07:00
Ganesh Pawar 1d4b6c1320 about_zulip: Migrate to handlebars.
This results in moving the `zulip_merge_base` parameter to
page_params, so that it's available to JavaScript.

Since this is technically a tiny overlay, it needs to be initialized
before hashchange.js.
2021-06-15 11:55:20 -07:00
Steve Howell e739bee00a poll widget: Add server validation. 2021-06-14 17:46:16 -07:00
Steve Howell 24ed5a2bb4 submessages: Add verify_submessage_sender.
Before this change a rogue actor could try to
widgetize another person's message. (The
rogue actor would already have access to read
the message.)
2021-06-14 17:46:16 -07:00
Vishnu KS 876af17dd8 models: Change return type of get_human_admin_users to QuerySet.
I would be accessing some methods of QuerySet in the subsequent
commits. So marking this as Sequence results in mypy errors.
2021-06-14 16:55:43 -07:00
Vishnu KS 6e3d4e7e75 billing: Create is_free_trial_offer_enabled function. 2021-06-14 16:49:53 -07:00
Vishnu KS 5da3a69505 plans: Mention whether realm is on free trial. 2021-06-14 16:49:53 -07:00
Vishnu KS 44d8368003 plans: Don't hardcode plan_type values in template. 2021-06-14 16:49:53 -07:00
sahil839 aa5d2199b1 invite: Use handlebar template for invite-user overlay.
Fixes a part of #18794.
2021-06-14 13:46:57 -07:00
akshatdalton c507931ac8 refactor: Export non-markdown logic in mention.py. 2021-06-14 13:26:30 -07:00
akshatdalton 1a76d06add test_push_notifications: Use responses module to mock HTTP responses. 2021-06-12 07:31:12 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Mateusz Mandera b1da5d9d76 ldap: Prevent creation of users with malformed email value. 2021-06-11 17:30:46 -07:00
Mateusz Mandera 18dab5619c ldap: Fix ldap access control tests.
These tests didn't configure ldap settings correctly and as a result,
the user involved in these tests wasn't actually hamlet@zulip.com, but a
new, malformed user with email "hamlet" that was being created by the
ldap auto-registration codepath. This wasn't caught because the codepath
didn't validate the email address and thus created such a malformed user
silently.
2021-06-11 17:30:46 -07:00
Abhijeet Prasad Bodas 10dd5f784b event_queue: Don't check for "read" flag when processing events.
* In `event_queue.py`, only the sender and recipient users who have muted
the sender will have the "read" flag set.

* We already skip enqueueing notifications for users who've muted the sender
after 58da384da3.

* The queue consume functions for email and push notifications already
check filter messages which have been read before sending notifications.

* So, the "read" logic in `event_queue.py` is unnecessary, and the
processing power saved from not enqueueing notifications for a single
user should be insignificant, so we remove these checks all toghether.
2021-06-11 08:07:37 -07:00
Abhijeet Prasad Bodas 3fd53d6f48 test_email_notifications: Add test for "read" flag return.
This logic was peviously untested. This is a prep change for us
to completely depend on the logic here for the "read" flag, and
not on the `event_queue` code.
2021-06-11 08:07:36 -07:00
Abhijeet Prasad Bodas 2b438fd7ce test_push_notifications: Fix incorrect "read" flag test.
It was unclear what the original test was testing, and more
importantly, the test passed even if one removed the `read` flag
check in the `handle_push_notifications` function, so we fix it
to be more comprehensive.
2021-06-11 08:05:27 -07:00
Abhijeet Prasad Bodas c6a31dcd9f event_queue: Extract local variables. 2021-06-11 08:05:27 -07:00
Suyash Vardhan Mathur 001435a297 refactor: Rename and move app_filters.py.
Moved the code in zerver/templatetags/app_filters.py
to more intuitive location and name at
zerver/lib/templates.py.
2021-06-11 07:43:22 -07:00
Suyash Vardhan Mathur 274f73d117 apidocs: Automatically add javascript example tabs.
The javascript tab in .md templates can be
generated along with the line that adds js
example.

Further, as a part of the effort of moving
towards a single template, the markdown extension
for javascrit examples is modified to return empty
string if javascript example doesn't exist for that
endpoint. This would make it possible to cover more
endpoints with a single template.

The js example tabs are now automatically generated
during generation of javascript code, and so need to
be removed. Also, the markdown function to render js
examples can be added in all templates, since it is
parses and returns an empty string if the examples
don't exist, and allows us to move towards a common
template.

The pages have been verified to be correct
by using diff between old and new pages' raw HTML.
2021-06-11 07:43:22 -07:00
Vishnu KS 4e26746b98 tests: Move tests of /activity pages from zerver to analytics module.
This module deals with the testing of /activity, /realm_activity
and /user_activity. All these pages reside in analytics module.
Keeping these tests in zerver/tests is kind is not appropriate
since person who makes changes to /activity pages would not think
it is necessary to run tests in zerver. So better to keep them
in the analytics module.
2021-06-11 07:41:02 -07:00
Vishnu KS 46c902686d tests: Specify the update_fields in test_activity.
This is a prep commit. Else the lint will complain during the
refactoring of this function to a new location that update_fields
is not specified.
2021-06-11 07:41:02 -07:00
Vishnu KS 40ab415005 activity: Show effective rate of realms in /activity page. 2021-06-11 07:41:02 -07:00
Tim Abbott 17fe09bb6d webhooks: Update link to BuildBot documentation. 2021-06-10 17:15:55 -07:00