Commit Graph

7258 Commits

Author SHA1 Message Date
Yogesh Sirsat 1a426fa6be i18n: Add translation tags to default external accounts name. 2022-09-20 10:42:33 -07:00
Zixuan James Li 3193afffe8 test_runner: Remove monkey-patched SubSuiteList.
This monkey-patching approach is not meaningful when what we really need
is just the names of the test, that can already be done in
get_test_names.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-20 10:18:22 -07:00
Anders Kaseorg 23d9dd3163 streams: Fix can_remove_subscribers_from_stream type.
Added by commit c3fe8420fd (#22589).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-19 13:53:44 -07:00
Mateusz Mandera 29589a69eb bots: Replace some redundant logic with is_cross_realm_bot_email.
is_cross_realm_bot_email is just
`email.lower() in settings.CROSS_REALM_BOT_EMAILS` which is the same,
aside of looking at .lower() - which is actually more correct.
2022-09-17 17:52:01 -07:00
Anders Kaseorg 403837e52d python: Use ‘not in’ for negated membership tests
Fixes “E713 Test for membership should be `not in`” found by
ruff (https://github.com/charliermarsh/ruff).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-17 11:48:33 -07:00
Jason Hildebrand 7efd59b6d7 email_mirror: Remove limits (expiry, max uses) to improve usability.
These limits don't appear to provide useful security benefits, and
they do impact usability because they prevented email-based users from
replying more than once, or from replying to message more than 5 days
old.

Fixes #2755.
Fixes #19994.
2022-09-16 18:07:28 -07:00
Sahil Batra c3fe8420fd streams: Allow unsubscribing others based on can_remove_subscribers_group.
Current value of can_remove_subscribers_group field is admins system group
only so behavior is not changed. We would provide support to change this
setting from API and UI in further commits.
2022-09-14 16:14:15 -07:00
Sahil Batra b9248c75f4 stream: Add do_change_can_remove_subscribers_group and field to objects.
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.

This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
2022-09-14 16:03:11 -07:00
Sahil Batra b0de5c0f36 streams: Set can_remove_subscribers_group while creating streams.
This commit sets can_remove_subscribers_group to admins system
group while creating streams as it will be the default value
of this setting. In further we would provide an option to set
value of this setting to any user group while creating streams
using API or UI.
2022-09-14 16:03:11 -07:00
Sahil Batra 941bb5a372 import: Fix import related code for user groups.
We change the import order to import UserGroup objects before
Stream such that we can set can_remove_subscribers_group correctly.

We do not import UserGroupMembership objects here along with
UserGroup since UserProfile objects are not imported and
GroupGroupMembership are also imported later as these are not
required before.
2022-09-14 16:03:11 -07:00
Anders Kaseorg 61e149dbb5 subdomain: Remove impossible None case from user_matches_subdomain.
The only two callers pass get_subdomain(request) which can’t be None.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-14 09:35:36 -07:00
Lauryn Menard 09d94c62a5 integrations: Revise integration CATEGORY dict structure.
Adds a META_CATEGORY dict for categories that are not best described
as groups of 'tools', so that in the subsequent commit the PAGE_TITLE
can be set accordingly.

Also, removes 'tools' from the 'Miscellaneous' category text and
spells out 'Human resources' instead of using 'HR'.
2022-09-13 11:36:52 -07:00
Sahil Batra 1e55e7641e export: Do not export direct_members and direct_subgroups field.
We do not need direct_members and direct_subgroups field of
UserGroup objects in the export data since we already have
UserGroupMembership and GroupGroupMembership object data.

While importing we keep these fields empty when creating
UserGroup objects and direct_members and direct_subgroups
fields will get set when UserGroupMembership and
GroupGroupMembership objects are created.

This change will also help us in further changes when we
will change the order of importing to import UserGroup
objects just after Realm objects.
2022-09-13 11:07:09 -07:00
Yogesh Sirsat c355934179 custom_profile_fields: Change data of default external account type.
change the names of "github" and "twitter" external account fields to
"GitHub username" and "Twitter username" respectively and remove the
hints of them.
2022-09-12 14:29:11 -07:00
David Rosa 217915a50c integrations: Remove "Yo".
The Yo company shut down in 2016.
https://en.wikipedia.org/wiki/Yo_(app)#History

Removes `yo` instances from `zerver/lib/integrations.py`.
Removes `zerver/webhooks/yo`.
Removes `static/images/integrations/yo-app`.
2022-09-09 16:31:33 -07:00
Mateusz Mandera 95bc77ae4f help center: Update "SAML authentication" to follow help center guidelines.
Co-authored-by: Alya Abbott <alya@zulip.com>
2022-09-09 10:30:05 -07:00
Zixuan James Li 6b527dfa61 integrations: Add documentation for the event filtering system.
We create "event-filter-instruction.md" and add it to
"create-bot-construct-url.md". This allows the user to keep track of the
supported event types for most of the integrations that implement this
feature. Note that not all integrations use "create-bot-construct-url.md".

We also need to rename "function" to "view_function" to make this change
type-check.

This is relevant to #18392.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-08 13:37:35 -07:00
Lauryn Menard 7f738803f0 help-docs: Rename organization language help center file.
Renames article about organization language used for automated
messages and invitation emails. Creates URL redirect and updates
links in repository (web app, help center and api documentation).

Prior to this change, the article was named:
'change-the-default-language-for-your-organization'.

Fixes #21949.
2022-09-07 09:45:37 -07:00
Christopher Chong 28173cafc8 message_flags: Fix deadlocks when updating message flags.
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.

This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.

Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.

We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.

Fixes #19054.
2022-09-06 16:06:58 -07:00
Christopher Chong f1a3cdc4c6 integrations: Add Azure DevOps webhook integration. 2022-09-01 14:01:13 -07:00
Lauryn Menard 5a7aa8228a urls: Extend documentation URL redirects system to corporate landing pages.
Extends the URL redirect system used for documentation pages to corporate
landing pages. This makes it easier and consistent for contributors who
work on both areas to create new URL redirects when needed.
2022-08-31 15:01:29 -07:00
Lauryn Menard 878b46e758 urls: Simplify process to add URL redirects for documentation pages.
Creates `zerver.lib.url_redirects.py` to record old and new URLs
for documentation pages that have been renamed/moved and need URL
redirects.

This file is then used by `zproject.urls.py` to redirect links and
by `zerver.test.test_urls.py` to test that all of the old URLs
return a success response with a common page header/text depending
on the type of redirect (help center, policy, or API).

Adds a section to contributor docs on writing documentation for
how to use this redirect system when renaming a help center or api
documentation page.

Fixes #21946. Fixes #17897.
2022-08-31 15:01:29 -07:00
Mateusz Mandera 10a1596d96 send_analytics_to_remote_server: Log connection errors with traceback.
It seems helpful for this to get logged with the traceback rather than
just the general
"<exception name>  while trying to connect to push notification bouncer."
2022-08-31 14:43:15 -07:00
Khushiyant 8609c441ff images: Move Giphy images into proper subdirectories.
The logo were only used in the integration documentation and belong in
static/images/integrations/giphy/; the in-app image is given its own
directory.

Fixes #22464.
2022-08-30 22:40:59 -07:00
Aman Agrawal 09572fe09b templates: Provide proper error message if entrypoint is not defined. 2022-08-30 16:02:06 -07:00
Mateusz Mandera d5b450c1e0 import: Fix the import of authentication_methods BitField.
The ordering of bits that fix_realm_authentication_bitfield was making
was incorrect, it needs to be reversed.
2022-08-29 11:10:18 -07:00
Zixuan James Li bb9e80d7a2 i18n: Deal with lazy strings more carefully.
This uses a more specific type `_StrPromise` to replace `Promise`
providing typing information for lazy translation strings.

In places where the callee evaluates the `_StrPromise` object in all
cases we simply force the evaluation with `str()`. This includes
`JsonableError` that ends up handled by the error handler middleware,
and `internal_send_stream_message` that depends on `check_stream_topic`,
requiring the `topic` to be evaluated anyway. In other siuations, the
callee is expected to be able to handle `StrPromise` explicitly.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-29 10:52:45 -07:00
Zixuan James Li ab9279aabe models: Fix type annotation with StrPromise.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-29 10:52:45 -07:00
Mateusz Mandera ac9e927989 import_realm: Validate emails of users being imported.
Having users with malformed email addresses in our database can lead to
hard to debug issues and we really should try to prevent that.
2022-08-29 10:39:14 -07:00
evykassirer efee77b41f emoji: Add which emoji are supported to the /register call.
Fixes #21037.

This is part of fixing #19371. To bulk-add new emoji regularly,
mobile needs to know which servers support which emoji.
`staticfiles_storage.url` generates a unique URL with a hash
based on the file content, which lets mobile know if it needs
to update its locally stored data.
2022-08-26 17:58:31 -07:00
sahil839 decc8d41fa emails: Add quote prefix ">" in plain text missed message emails.
We add quote prefix ">" to each line of the message in the plain text
missed message emails, which are then rendered as quotes by email
clients. We also move the message content in the next line after sender.

This helps us in clearly showing the message authors in missed message
emails especially in emails with multiple messages and senders.

Fixes #15836.
2022-08-26 11:48:29 -07:00
Alya Abbott ff82c69480 help center: Rewrite Import from Slack documentation. 2022-08-25 15:17:43 -07:00
Tim Abbott c98f9bcb8e email_notifications: Use assert to verify only one recipient.
This ValueError had no test coverage, because the code path wasn't
actually possible with how the caller is constructed.

Rather than writing a highly artificial test for this as proposed in
2022-08-25 12:26:02 -07:00
Zixuan James Li c8d5959689 settings: Make AVATAR_SALT mandatory.
This also allows us to remove some assertions as we now know that
AVATAR_SALT will never be None.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-25 12:13:03 -07:00
Anders Kaseorg 4a61e36def CVE-2022-36048: Rewrite only specific local links to relative.
Due to mismatches between the URL parsers in Python and browsers, it
was possible to hoodwink rewrite_local_links_to_relative into
generating links that browsers would interpret as absolute.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-24 16:29:09 -07:00
Mateusz Mandera c1c9024af5 email_mirror: Replace disallowed characters in incoming email subject.
These characters are not allowed and trying to create a Zulip message
with those characters throws a JsonableError in check_stream_topic.

We don't want to reject emails with those chars in the subject, so
it's best to just modify it appropriately.
2022-08-22 17:16:20 -07:00
Zixuan James Li 32a77fc0bf test_helpers: Move patch_queue_publish to test_service_bot_system.
This prepares us to tighten the type annotations for this helper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 15:46:16 -07:00
Zixuan James Li 21fd62427d typing: Remove ViewFuncT.
This removes ViewFuncT and all the associated type casts with ParamSpec
and Concatenate. This provides more accurate type annotation for
decorators at the cost of making the concatenated parameters
positional-only. This change does not intend to introduce any other
behavioral difference. Note that we retype args in process_view as
List[object] because the view functions can not only be called with
arguments of type str.

Note that the first argument of rest_dispatch needs to be made
positional-only because of the presence of **kwargs.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 15:46:16 -07:00
Israel Ebonko 1f286ab283 integration: Add wekan integration.
Fixes #13086

Co-authored-by: Ganesh Pawar (ganpa3) <pawarg256@gmail.com>
2022-08-22 15:37:22 -07:00
Zixuan James Li 965ed728e1 type_debug: Remove type_debug.
This module was originally introduced in 2016 to assist adding mypy
annotations to the project. Back then static type checking was not that
established throughout the codebase, so it was helpful to be able to
print out the types for type checking purposes.

This workflow is no longer helpful for improving type annotations right
now, and it has been unused for a while.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 13:19:09 -07:00
Zixuan James Li de22b72670 rate_limit: Remove rate_limit_ip.
Because rate_limit_request_by_ip is the only caller of it, it is safe
for us to inline RateLimitedIpAddr and remove this helper. This ensures
that we have consistent internals for rate limiting functions, which all
have a should_rate_limit check.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 730d3ec929 typing: Fix type annotation of assert_(not_)in_success_response.
This allows us to use them with HttpResponse objects returned by
calling a view function directly.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 26a518267a rate_limit: Replace rate_limit with inlined rate limit checks.
This change incorporate should_rate_limit into rate_limit_user and
rate_limit_request_by_ip. Note a slight behavior change to other callers
to rate_limit_request_by_ip is made as we now check if the client is
eligible to be exempted from rate limiting now, which was previously
only done as a part of zerver.lib.rate_limiter.rate_limit.

Now we mock zerver.lib.rate_limiter.RateLimitedUser instead of
zerver.decorator.rate_limit_user in
zerver.tests.test_decorators.RateLimitTestCase, because rate_limit_user
will always be called but rate limit only happens the should_rate_limit
check passes;

we can continue to mock zerver.lib.rate_limiter.rate_limit_ip, because the
decorated view functions call rate_limit_request_by_ip that calls
rate_limit_ip when the should_rate_limit check passes.

We need to mock zerver.decorator.rate_limit_user for SkipRateLimitingTest
now because rate_limit has been removed. We don't need to mock
RateLimitedUser in this case because we are only verifying that
the skip_rate_limiting flag works.

To ensure coverage in add_logging_data, a new test case is added to use
a web_public_view (which decorates the view function with
add_logging_data) with a new flag to check_rate_limit_public_or_user_views.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 2aac1dc40a rate_limit: Move rate_limit_remote_server to zilencer.auth.
This allows us to avoid importing from zilencer conditionally in
zerver.lib.rate_limiter, as we make rate limiting self-contained now.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 29bad25f83 zilencer: Only apply rate limit to remote server.
This refactors the test case alongside, since normal views accessed by
remote server do not get rate limited by remote server anymore.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 79e86471e7 rate_limit: Extract should_rate_limit from rate_limit.
This allows us to reapply this check for other rate limiting functions.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li d4b6d36740 typing: Tighten type annotation of zerver.lib.rest.
By replacing Any with object we enforce type narrowing before using the
kwargs when a more specific type is required.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-15 19:51:08 -07:00
Zixuan James Li eb88fee745 rest: Remove kwargs from rest_path.
The only caller that passes the kwargs argument is the avatar rest_path.
The application of kwargs can be rewritten with a wrapper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-15 19:51:08 -07:00
Zixuan James Li fcba35036b rest: Add assertions for entries from supported methods.
Mypy considers that "Tuple[Any, ...]" is incompatible with
"Union[Tuple[Callable[..., HttpResponse], Set[str]], HttpResponse]".

handler, view_flags = entry is sufficient to suppress the error, but we
also add assertions for full measure.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-15 17:20:04 -07:00
Zixuan James Li dd2fd8edda rest: Implement get_target_view_function_or_response.
As noted in the docstring, this is a temporary helper function that
separates routing for paths that support multiple HTTP methods from
`rest_dispatch` itself. We will need to replace this helper with
class-based views in the future. The helper will also be handy to
reduce duplication when splitting up `rest_dispatch` by authentication
methods.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-13 14:53:51 -07:00