Commit Graph

15635 Commits

Author SHA1 Message Date
Zixuan James Li 56855e0187 typing: Add assertions for Optional values.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 7c62af0d73 typing: Add assertions before accessing settings.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li a09c9a4063 test_auth_backends: Remove unused return.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 0ef5eee3f0 error_notify: Fix type narrowing of settings.ERROR_BOT.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li c5075be9aa test_retention: Add type annotation for expected_result
Otherwise mypy infers the type of `expected_result` to be incompatible
with the first argument of `fix_ordering_of_result`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li e665ec8ae2 middleware: Add isinstance check before retrieving content.
StreamingHttpResponse is inferred without the isinstance check in the
else branch. We refactor this is shorten the code and also type narrow
it appropriately.
2022-07-15 14:00:56 -07:00
Zixuan James Li 2095258aa5 middleware: Assert request.method is not None.
`request.method` is not `None` in normal use cases, unless an
`HttpRequest` is directly instantiated without the method being set.
This situation does not apply to `WSGIRequest` at all.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li ece3fbdf95 migrations: Use the correct type annotation for the Model type.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 7d86d291d4 middleware: Remove inappropriate StreamingHttpResponse annotation.
Asserting response.stream is False is just suggesting the response being
an `HttpResponse`. This removes `StreamingHttpResponse` with the more
generic `HttpResponseBase` with an isinstance-check.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li da4654d838 test_send_email: Fix the to_email argument to pass str.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 45db04a39e test_auth_backends: Fix invalid assertions.
`m.output` is a `list` of `str`s. It does not make sense comparing it to
a `str`. Guessed the intention here is to use `self.assert_length`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 0952b024c1 scim: Add SCIMConfigDict.
This adds a `TypedDict` to provide better type annotation for the
configuration object for SCIM.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 2b1820cfcf message_edit: Avoid retyping variables.
Mypy disallows definition of the same variable with incompatible types.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li b775639f42 test: Use list comprehension for dictionary values.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 7950baafe2 tests: Add type narrowing for EmailMultiAlternatives.
Items in `django.core.mail.outbox` are by default typed as the less
general `EmailMessage` type. Before accessing the attribute
`alternatives`, we need to narrow the type to `EmailMultiAlternatives`.
Then narrow the tuple value we want to access to `str` before using
it in `assertIn` or `self.normalize_string`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 039c46a4d2 registration: Maintain immutability of the POST QueryDict.
Instead of mutating the original `QueryDict`, we shall create a new
one when updating the `QueryDict`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 199d89d761 registration: Fix invalid json response.
The body of `json_success` should be a `Mapping[str, object]`, optional
value is not allowed here.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 9d1ef72f08 test_helpers: Make use_db_models generic.
We use this decorator on subclasses of `MigrationsTestCase`, which does
not have `self`s being `MigrationsTestCase`, but the corresponding
subclass.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 13:49:45 -07:00
Zixuan James Li 1c4741cec3 models: Use more accurate type annotations for as_sql.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 13:48:42 -07:00
Tim Abbott 05b70ba74a decorator: Explicitly require req_secret in internal_notify_view.
It's hard to come up with a realistic story where this would matter:
SHARED_SECRET is generated automatically during server setup at the
same time as SECRET_KEY, which is a required setting, but it seems
preferable to be explicit that this is a required parameter for the
internal_notify authentication model.
2022-07-15 09:20:37 -07:00
Lauryn Menard 855e14272a backend: Migrate `secret` parameter to REQ framework.
Instead of using request.POST to get any potential `secret`
parameter used in `authenticate_notify` for `internal_notify_view`
decorator, moves it to the REQ framework parameters as `req_secret`.

Updates existing tests to explicitly test for a request without
`secret` parameter, which defaults to `None`; this is also tested
in `test_event_system.py`.
2022-07-15 09:20:37 -07:00
Zixuan James Li 75925fe059 middleware: Reorder middleware to clean up LogRequests hasattr checks.
Similar to the previous commit, we should access request.user only
after it has been initialized, rather than having awkward hasattr
checks.

With updates to the settings comments about LogRequests by tabbott.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-14 17:28:50 -07:00
Zixuan James Li 37a7d1fe7b middleware: Reorder middleware to avoid hasattr checks.
`request.user` gets set in Django's `AuthenticationMiddleware`, which
runs after our `HostDomainMiddleware`.

This makes `hasattr` checks necessary in any code path that uses the
`request.user` attribute. In this case, there are functions in
`context_processors` that get called in the middleware.

Since neither `CsrfMiddleware` nor `HostDomainMiddleware` are required
to run before `AuthenticationMiddleware`, moving it two slots up in
`computed_settings` is sufficient to avoid the `hasattr` checks.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-14 17:24:24 -07:00
Zixuan James Li 0056becd04 test_helpers: Default user_profile to AnonymousUser.
A request that has went through the auth middleware shouldn't have
`.user` being `None`. We should use `AnonymousUser` by default to
represent unauthenticated users.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-14 17:24:24 -07:00
Ganesh Pawar 135854db97 emoji: Use path converter for emoji name in URL.
If the emoji name contains forward slashes, the `str` converter
would treat it as a URL delimiter. Instead use the path converter, so
that forward slashes are included in the emoji name variable.

Fixes #22377
2022-07-14 14:29:40 -07:00
Aman Agrawal 4176e909fa events_register: Pass spectator set language to client in user_settings.
Fixes #22461
This fixes spectator language selection modal just using realm
default language as current selected language.
2022-07-14 14:27:32 -07:00
Sahil Batra 093a74bd54 models: Remove "role" field from Subscription.
This commit removes "role" field from subscription
objects since we are not moving forward with stream
administrator concept and instead working on new
permssions model as per #19525.
2022-07-13 17:07:44 -07:00
Sahil Batra 83383090f9 realm: Removed WILDCARD_MENTION_POLICY_STREAM_ADMINS option.
This commit removes WILDCARD_MENTION_POLICY_STREAM_ADMINS
option of wildcard_mention_policy since we are not moving
forward with stream administrator concept and instead working
on new permssions model as per #19525.

We also add a migration to change wildcard_mention_policy of
existing realms to WILDCARD_MENTION_POLICY_ADMINS. This change
is fine since we were already treating both the setting values
as same as stream admin concept was not implemented completely.
2022-07-13 17:07:44 -07:00
Sahil Batra d31fc27912 streams: Remove is_stream_admin property and its uses.
This commit removes the is_stream_admin property of Subscription
model and also updates check_stream_access_for_delete_or_update
to not return true when is_stream_admin is True.

We also removes the relevant tests.

This change is done as we would not be moving forward with the
stream administrator concept as we have decided to modify the
permissions model as per #19525.
2022-07-13 17:07:44 -07:00
Adam Sah f7f5f7a0b1 merge_streams: Improve order in which operations are executed.
As discussed in the new comments results in a better failure mode if
an error occurs while adding subscriptions; running the merge tool
again after fixing whatever caused the error will work just fine.
2022-07-13 16:19:59 -07:00
Adam Sah cb6382369e merge_streams: Move logic to zerver/actions/streams.py. 2022-07-13 16:12:36 -07:00
Anders Kaseorg 81892df176 requirements: Upgrade to Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-13 16:07:17 -07:00
Alya Abbott 7e3388dd9e images: Move /team images into landing-page directory. 2022-07-13 15:14:19 -07:00
Zixuan James Li 74f59bd8d0 2fa: Rename zulip_otp_required for clarity.
The name does not really comply with the actual behavior of
the decorator since it returns True for an unauthenticated user.
This makes it clear that the 2fa check only applies to users that
are already logged in.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-13 14:49:41 -07:00
Zixuan James Li 00bd7513f2 2fa: Verify 2FA authentication status with is_2fa_verified.
This replaces user.is_verified with is_2fa_verified.

The helper does extra checks such that the user being checked for 2fa
authentication status is valid.

`request.user.is_verified` is functionally the same as `is_verified`
from `django_otp.middleware`, except that the former is monkey-patched
onto the user object by the 2FA middleware. We use the latter wrapped
in `is_2fa_verified` instead to avoid accessing the patched attribute.

See also: 6b24d56e59/docs/source/overview.rst (authentication-and-verification)

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-13 14:49:41 -07:00
Zixuan James Li 3367839839 decorator: Add test case for unauthenticated 2fa.
This simulates the situation in which the user is not
authenticated (as an AnonymousUser) and have 2FA enabled.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-13 14:49:41 -07:00
Christopher Chong 5362a6b555 integrations: Add RhodeCode webhook integration.
Fixes #19627.
2022-07-13 14:10:00 -07:00
Lauryn Menard fed5ab0e1d api-docs: Add note to `/update-message` parameters.
Adds a note to the `content` and `stream_id` parameters for the
`/update-message` endpoint that indicates these parameters throw
an error when sent in the same request.

Also, updates the main description of the endpoint to include
changing a message's stream. And updates some of the parameter
descriptions to be more consistent with each other and clear.
2022-07-13 13:59:25 -07:00
Mateusz Mandera 2299aa3382 docs: Remove some outdated references to thumbnailing.md doc.
The doc was removed in 405bc8dabf
2022-07-12 17:44:24 -07:00
evykassirer 084b054827 emoji: Update emoji validation error messages with more information.
This change tells the user what a valid emoji name looks like, so that
they know what to fix if the name they chose doesn't pass the validator.
2022-07-12 16:43:29 -07:00
Mateusz Mandera e6264cd85b test_import_export: Test exclusion of private uploads from realm export. 2022-07-12 06:08:07 +00:00
Anders Kaseorg b35268e6bb CVE-2022-31134: Exclude private attachments from realm exports.
Zulip Server 2.1.0 and above have a UI tool, accessible only to server
owners and server administrators, which provides a way to download a
“public data” export. While this export tool is only accessible to
administrators, in many configurations server administrators are not
expected to have access to private messages and private
streams. However, the “public data” export which administrators could
generate contained the attachment contents for all attachments, even
those from private messages and streams.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-12 06:08:05 +00:00
Lauryn Menard 22c4763a61 models: Refactor UserBaseSettings `property_types` structure.
To prevent new contributors who are adding new user settings from
being confused about the legacy dicts for display settings and
notifications settings, we create `modern_settings` and
`modern_notifications_settings` dicts with comments documenting
that new user settings should be added to one of them.

Note that the `modern_notification_settings` is a placeholder for
new notifications settings as there have been no new notification
settings added since 430c5cb8 (when the legacy dicts were created)
and is currently annotated as `Dict[str, Any]` (which can be
removed if/when a new setting is added).
2022-07-07 17:16:14 -07:00
Lauryn Menard 0543f4596c integrations-docs: Update `git-webhook-url-with-branches.md`.
Updates `git-webhook-url-with-branches.md` (and two files that use
that file as an include link) for some of the follow-ups from #22315
to the Markdown parser. With this fix, all integrations docs that
reference this file as an include link should render the url as a
div element with `.codehilite` class.
2022-07-07 17:12:00 -07:00
Zixuan James Li 5d524aa43d custom_profile_fields: Use cast to assert non-optional items.
The mypy django-stubs plugin incorrectly infers optional value for order,
which makes the `value_list` to be typed as `Iterable[Optional[int]]`.
We use a type cast here to ensure that
`try_reorder_relam_custom_profile_fields` won't causes mypy to complain
about it.

TODO: Remove the cast when https://github.com/typeddjango/django-stubs/issues/444 gets resovled.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 5a934ef39f test_import_export: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 94732b5891 users: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 77b5c79aa2 realm_settings: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 552b58afb7 display_recipient: Tighten function signatures with ValuesQuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 07cc859120 topic: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 924739099b streams: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 27af5865b0 soft_deactivation: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li d98fda7286 message: Tighten function signatures with ValuesQuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li da01bdd41c cache: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li ea43edc9d8 user_groups: Tighten function signatures with generic QuerySet.
`member_ids` needs to be defined as an `Iterable` as it will otherwise
inferred to have incompatible types in the else branch.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li f5f94b9cad stream_subscription: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li d7d6aa31bb subscription_info: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 9d18845be5 models: Tighten function signatures with generic QuerySet.
TODO: For now, we import `_QuerySet` as `ValuesQuerySet`. But there
is a convenient reexport of `ValuesQuerySet` in `django_stubs_ext`
that does the same thing. Once we get django-stubs integrated,
we should import `ValuesQuerySet` from `django_stubs_ext` instead.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:43 -07:00
Zixuan James Li 6c7b2d621e typing: Avoid redefinition of incompatible QuerySets.
The pattern of using the same variable to apply filters
or alter the `QuerySet` in other ways might produce `QuerySet`s
with incompatible types. This behavior is not allowed by mypy.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:43 -07:00
Zixuan James Li 88f6f3fc0f user_topics: Properly type mute_cond with TypedDict.
The returned dictionary is not at all used outside the function, so it's
sufficient to make it available only within the helper function itself.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:43 -07:00
Zixuan James Li 8fa94138bc realm_domains: Type QuerySet.values as RealmDomainDict.
This ensure that the return type is compatible with the
actual type of `realm.realmdomain_set.values`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:42 -07:00
Zixuan James Li ab1bbdda65 typing: Broaden type annotations for QuerySet compatibility.
To explain the rationale of this change, for example, there is
`get_user_activity_summary` which accepts either a `Collection[UserActivity]`,
where `QuerySet[T]` is not strictly `Sequence[T]` because its slicing behavior
is different from the `Protocol`, making `Collection` necessary.

Similarily, we should have `Iterable[T]` instead of `List[T]` so that
`QuerySet[T]` will also be an acceptable subtype, or `Sequence[T]` when we
also expect it to be indexed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:42 -07:00
Aman Agrawal 40fcf5a633 message_flags: Filter msgs having (or not) the flag before updating.
We were blindly adding / removing flag from UserMessages without
check if they even need to be updated.

This caused server to repeatedly update flags for messages which
already had been updated, creating a confusion for other clients
like mobile.

Fixes #22164
2022-07-07 10:52:36 -07:00
Aman Agrawal 0ad282c11e models: Introduce functions to check if a flag is present in a UM.
Added 2 new functions which check if a flag is present in a
user message or not using bit comparison.

Refactor `where_unread/starred/active_push_notification` to use
these functions.

Existing tests are enough to check if these functions work correctly.
2022-07-07 10:52:36 -07:00
Alya Abbott c5508bb8fd help center: Move typing notifications documentation to a separate page. 2022-07-07 10:42:04 -07:00
Anders Kaseorg 2439914a50 settings: Add two_factor.plugins.phonenumber to INSTALLED_APPS.
I missed this in commit feff1d0411
(#22383) for upgrading to django-two-factor-auth 1.14.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-06 17:23:53 -07:00
Zixuan James Li e11013fc00 decorator: Remove unused Union.
The other variant of possible return type was removed in
7c9e8a5071, but the return type is not
accurately reflecting that.
2022-07-06 17:20:57 -07:00
Alex Vandiver 0830d5e7ea emoji: Write "original" file before attempting resize.
Resizing emoji can fail, especially for animated GIFs; in such cases,
it is useful to have the original data on hand, to be able to dissect
the failure.
2022-07-06 17:20:40 -07:00
Anders Kaseorg de31114d70 test_runner: Avoid undocumented writeln method.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:55:18 -07:00
Anders Kaseorg 8246ee7c57 mypy: Add links to specific mypy bugs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:58 -07:00
Anders Kaseorg feff1d0411 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Anders Kaseorg 6c79b8f2f1 test_tornado: Avoid deprecated AsyncHTTPTestCase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Anders Kaseorg b4cf9ad777 db: Use cursor_factory psycopg2 option.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Anders Kaseorg 55915c26a7 migrations: Replace NullBooleanField.
This was removed in Django 4.0 except in historical migrations.  We
might as well replace it everywhere.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:52:08 -07:00
Anders Kaseorg 4391234eb3 test_runner: Remove unused addInfo method.
It’s unused since commit 92d0290dc5
(#15629).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 13:46:05 -07:00
Anders Kaseorg b991c8019e slack_incoming: Simplify empty POST test.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 13:08:35 -07:00
Anders Kaseorg 7c992422f0 email_notifications: Give absolute path to html2text.
Our uWSGI configuration doesn’t correctly activate our virtualenv.  We
should investigate that, but until we do, we need to invoke html2text
by an absolute path.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-30 17:02:47 -07:00
Adam Sah 0073131983 realms: Extract do_set_realm_stream shared helper.
This deduplicates what were previously two nearly identical functions.
2022-06-29 16:43:55 -07:00
Zixuan James Li 0017f7a025 confirmation: Support more models as ConfirmationObjT.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-29 11:31:07 -07:00
Alex Vandiver bc25f016fc slack_incoming: Support more syntax.
This significantly extends the accepted support for Slack "blocks"[1], as
well as their legacy "attachments"[2] syntax.

Fixes: #21944.

See also #22228, which this does not solve, but assuming we can decide
which compromises to make, becomes a reasonably uncomplicated follow-up.

[1]: https://api.slack.com/reference/block-kit/blocks
[2]: https://api.slack.com/reference/messaging/attachments
2022-06-28 17:04:53 -07:00
Alex Vandiver 239fdc2222 slack_incoming: Stop passing around the previous body.
The only operation is to append -- let that happen in the caller.
2022-06-28 17:04:53 -07:00
Alex Vandiver 0511400d73 slack_incoming: Handle null blocks and attachments.
This is not accepted according to Slack's block-builder, but is
attested in the wild.
2022-06-28 17:04:53 -07:00
Alex Vandiver 74cf9a1e9f slack_incoming: Strengthen types using WildValue. 2022-06-28 17:04:53 -07:00
Anders Kaseorg 4fc3845dc2 tornado: Ignore StreamClosedError.
This was also hidden until 81f7192ca3
(#22301).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-28 16:35:49 -07:00
Anders Kaseorg 0d93ec6214 tornado: Remove dead check for message format.
This was for the old /messages/latest API that was removed in commit
e06722657a.

If we wanted a new check like this, it shouldn’t go in zulip_finish,
because that only runs when the client gets an asynchronous response
from polling an initially-empty queue, and not when the client gets a
synchronous response from polling a nonempty queue.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-28 16:35:49 -07:00
Zixuan James Li b65401ed47 tests: Tighten signature of the wrapped test client helpers.
We wrap methods of the django test client for the test suite, and
type keyword variadic arguments as `ClientArg` as it might called
with a mix of `bool` and `str`.

This is problematic when we call the original methods on the test
client as we attempt to unpack the dictionary of keyword arguments,
which has no type guarantee that certain keys that the test client
requires to be bool will certainly be bool.

For example, you can call
`self.client_post(url, info, follow="invalid")` without getting a
mypy error while the django test client requires `follow: bool`.

The unsafely typed keyword variadic arguments leads to error within
the body the wrapped test client functions as we call
`django_client.post` with `**kwargs` when django-stubs gets added,
making it necessary to refactor these wrappers for type safety.

The approach here minimizes the need to refactor callers, as we
keep `kwargs` being variadic while change its type from `ClientArg`
to `str` after defining all the possible `bool` arguments that might
previously appear in `kwargs`. We also copy the defaults from the
django test client as they are unlikely to change.

The tornado test cases are also refactored due to the change of
the signature of `set_http_headers` with the `skip_user_agent` being
added as a keyword argument. We want to unconditionally set this flag to
`True` because the `HTTP_USER_AGENT` is not supported. It also removes a
unnecessary duplication of an argument.

This is a part of the django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:12:00 -07:00
Zixuan James Li 4e518a3852 tests: Call client methods with explicit keyword arguments.
This is a prep commit for tightening the types for our wrapped test
client.

The callers of the test client methods are refactored to either call
them without unpacking at all or create a TypedDict for the keyword
arguments to be unpacked. This allows the type checker to know exactly what
keys are present and their corresponding type.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:12:00 -07:00
Zixuan James Li a3a0545aac typing: Populate POST and _files with `cast` and `setattr`.
`POST` is an immutable attribute and `_files` is an internal attribute
of `HttpRequest`. With type annotations provided by `django-stubs`, mypy
stops us from modifying these attributes. This uses `cast` and `setattr`
to avoid typing issues.

This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:07:18 -07:00
Zixuan James Li 27be27560b typing: Access LANGUAGE_CODE via `django.utils.translation.get_language`.
We no longer need to access the internal `LANGUAGE_CODE` attribute by
using `django.utils.translation.get_language`.

A test case overriding the translation is added to ensure the password
reset form sending to users requested from a wrong domain is properly
translated.

This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:07:18 -07:00
Zixuan James Li 417e1b5e81 logging_util: Save an assertion with __getitem__.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:05:24 -07:00
Zixuan James Li fc3de0ba41 typing: Tighten type annotation for serialized user groups.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:05:24 -07:00
Zixuan James Li 5285fbb4d0 typing: Fix type annotation for missing messages in soft reactivation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:05:24 -07:00
Zixuan James Li 9bfeebf064 user_profile: Fallback to "" for timezone upon creation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:05:24 -07:00
Zixuan James Li a9029c68ea tornado: Make _request an attribute on AsyncDjangoHandler.
For the same reason as `handler_id` has, we define `_request`
as an attribute. Note that the name `request` is already taken.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:03:09 -07:00
Zixuan James Li 56e1f3b725 tornado: Make handler_id an attribute on AsyncDjangoHandler.
This prevents us from relying on a side-effect of `allocate_handler_id`
that monkey-patches `handler_id` on the `AsyncDjangoHandler` object,
allowing mypy to acknowledge the existence of `handler_id` as an `int`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:03:09 -07:00
Zixuan James Li f0e505d557 retention: Guarantee type-safeness when calling move_rows.
This ensures that all the keyword arguments in `move_rows`
have the correct types. Note that `returning_id` is supposed to be a
flag instead of a `Composable` `Literal`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:01:35 -07:00
Anders Kaseorg bae4182e47 test_classes: Check responses in ZulipTestCase.register.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-28 13:15:21 -07:00
Anders Kaseorg 869fe60689 markdown: Parse included blocks in a new parser state.
This fixes inclusion of a multi-paragraph file into a list item.

Followup to commit dc33a0ae67 (#22315).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 17:37:00 -07:00
Anders Kaseorg a2e1d61172 integrations: Satisfy Python-Markdown’s archaic 4-space requirement.
Followup to commit dc33a0ae67 (#22315).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 17:36:17 -07:00
Anders Kaseorg 53231aa9d9 decorator: Type cache_info, cache_clear for ignore_unhashable_lru_cache.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 10:20:05 -07:00
Lauryn Menard 9db8a59a56 integrations-docs: Add line break in GitLab integration doc. 2022-06-27 10:14:51 -07:00
Anders Kaseorg a7e10ee47e tornado: Send request_started signal in Django thread.
Django’s ASGIHandler does this too and it seems like a good idea.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 10:14:02 -07:00
Anders Kaseorg 0dfde9db94 python: Simplify unnecessary getattr calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:38:22 -07:00
Anders Kaseorg 3bf8ee2156 python: Unquote some unnecessarily quoted type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:37:41 -07:00
Anders Kaseorg 6fe5cb65a1 templates: Fix Python-Markdown extension type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:37:14 -07:00
Anders Kaseorg dc33a0ae67 markdown: Rewrite include plugin without markdown-include.
markdown-include is GPL licensed.

Also, rewrite it as a block processor, so that it works correctly
inside indented blocks.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:36:31 -07:00
Anders Kaseorg 7f0e11bd06 markdown: Rename preprocessor_priorities module to priorities.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:36:31 -07:00
Anders Kaseorg f3254bb558 mattermost: Run html2text as a subprocess.
html2text is GPL licensed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:32:59 -07:00
Anders Kaseorg e10b7f2f3c email_notifications: Run html2text as a subprocess.
html2text is GPL licensed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:32:59 -07:00
Alex Vandiver 5529639b84 change_realm_subdomain: Support skipping the redirect.
Sometimes (e.g. when moving an old realm out of the way of an import
into that name) we do *not* wish to add a redirect realm.  Add a flag
to support that.
2022-06-25 08:44:24 -07:00
Alex Vandiver 3e4d07952c test_signup: Test redirects more explicitly. 2022-06-25 08:44:24 -07:00
Anders Kaseorg 236ef8a077 test_push_notifications: Simplify with Python 3.8 AsyncMock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:43:20 -07:00
Anders Kaseorg 5033beb99c request: Replace tornado_handler weak reference with tornado_handler_id.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:42:23 -07:00
Anders Kaseorg d03892391a event_queue: Take (typed) keyword parameters for fetch_events.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:42:23 -07:00
Anders Kaseorg 4d4d1e50cc tornado: Simplify in_tornado_thread usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:42:23 -07:00
Anders Kaseorg 952422d373 test_helpers: Remove default tornado_handler from HostRequestMock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:42:23 -07:00
Anders Kaseorg c1a54f4567 decorator: Only URL-decode application/x-www-form-urlencoded requests.
We previously parsed any request with method other than {GET, POST} and
Content-Type other than multipart/form-data as if it were
application/x-www-form-urlencoded.

Check that Content-Type is application/x-www-form-urlencoded before
parsing the body that way.  Restrict this logic to {DELETE, PATCH,
PUT} (having a body at all doesn’t make sense for {CONNECT, HEAD,
OPTIONS, TRACE}).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:36:16 -07:00
Zixuan James Li 6400cd93ea test_middleware: Narrow the type of open_graph_description.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li cf455131ef message_edit: Make propagate_mode optional.
When there is no topic/stream being change, `propagate_mode` becomes
unnecessary. We add an assertion to ensure that the previous assumption
that `propagate_mode` is not `None` still holds when either `topic_name`
or `new_stream` is not `None`.

We can possibly improve this by overloading `do_update_message` and
`check_update_message`, but that's beyond the scope of the PR and
feasibility of doing that should also be further discussed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li fa51d8718c home: Prevent mypy from inferring the type of page_params.
Without an explicit type annotation, mypy infers the type of
values in `page_params` upon its initialization as a `Union`, while
other computed values haven't been assigned yet. We break this
over-conservative inferred type by annotating `page_params` as a
`Dict[str, object]`.

We could have created a `TypedDict` to have accurate type annotation
for all of the fields, but it does not seem worth it at this point since
the data structure is not widely used in the backend.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 67fda5516f import_utils: Fix wrong usage of model_to_dict.
The argument `exclude` expects a `list` or `set` of field names,
not a `str`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 0ec561ab57 upload: Add assertions before accessing uploaded files.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 428dfd2693 typing: Add assertions for report returned from notify_server_error.
We don't have a specific type for the reports returned from the error
logging handlers. The check is necessary as they are currently typed
as `Dict[str, object]` in `run_handler`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 8ba8177dc5 typing: Correct the signautre of prepare_activation_url.
`request.session` is `django.contrib.sessions.backends.base.SessionBase`
`instead of django.contrib.sessions.models.Session`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 2ddea50745 typing: Correct type annotation of supported_backends.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 1860b0b7ac typing: Fix missing attribtute of RedisUtilsTest.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:05:12 -07:00
Zixuan James Li fd9a0f4274 typing: Apply trivial none-checks with assertions as necessary.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 58e95cbfae typing: Add none-check for test_users.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 146b0c482c typing: Add none-check for WebhookIntegration.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 058dab5818 typing: Add none-checks for stream.recipient_id.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 944e602788 streams: Refactor bulk_add_subscription handling recipient_ids set.
While it is possible to have `stream.recipient_id` being `None`,
the code works under the assumption that it is not. Potentially
we will get a runtime error, but it is not quite explicit without
the assertion.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li c567e44408 typing: Avoid redefinition of variables with different types.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 5fd278bf09 user_settings: Add assertions before attribute access.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 75df60bd46 send_email: Define variable typed as a Union.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 40b4da8f58 emoji: Add none checks for uploaded file name.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 10dc6ea39e typing: Narrow connection type before accessing pg_version.
This change is solely for removing the attribute error that mypy raises when we
 access `pg_version` on `connection`. django-stubs annotate `connection` as
`BaseDatabaseWrapper` while it is actually a proxy object, so we cannot
use an regular assertion with isinstance to narrow the type.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li d46f7566dd typing: Fix wrong type annotations for mute_stream.
This is a part of djang-stubs refactorings.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 4cf3ba5744 typing: Fix typical typing typos.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li 3dcb4ca20a commands: Use getattr to access patched attribute during runtime.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li a86ba33087 webhooks: Use setattr when assigning _all_event_types.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Anders Kaseorg 345ed1d09d middleware: Pass unhandled API exceptions through to the test suite.
This results in more useful stack traces in failing tests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:23:08 -07:00
Anders Kaseorg 81f7192ca3 tornado: Add missing await for finish calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Anders Kaseorg df69e1d979 mypy: Enable truthy-bool errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Anders Kaseorg 74e94ae78c integrations: Fix config_options type.
The “validator” component of the tuple does not follow the Validator
contract as of 7e9db327b3 (#15498).
Define a separate type for it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Anders Kaseorg c944adfcc6 mypy: Enable redundant-expr errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Alex Vandiver 160a8b9945 compilemessages: Use a consistent ordering for the languages list.
This makes the generated data more consistent.
2022-06-23 16:25:33 -07:00
Anders Kaseorg 3321a81c4a wordpress: Fix nonsense tests.
‘stream_name’ is not a cromulent keyword argument for client_post(),
‘unknown_action’ is malformed application/x-www-form-urlencoded, and
these two tests were duplicates of each other with different comments.
I’m not sure what they were intended to test, but here’s a guess.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 16:12:53 -07:00
Anders Kaseorg 20f9293f1f CVE-2022-31017: Fix edit event exposure in protected-history streams.
When editing an old message in a private stream with protected
history, the server would incorrectly send an API event including the
edited message to all of the stream’s current subscribers, including
those who should not have access to the old message. This API event is
ignored by official clients, so it could only be observed by a user
using a modified client or their browser’s developer tools.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-21 13:41:23 -07:00
Zixuan James Li aebed0e57f management: Remove rename_stream.
Now that it is trivial to rename a stream in the UI, And due
to the fact that the command has been broken for 3 years unnoticed,
it is unnecessary to maintain it anymore.

Fixes #22244.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-21 12:56:54 -07:00
Anders Kaseorg 0430705d13 test_tornado: Call process_event on first fetch_events return.
The 0.1 second delay was sometimes not long enough to guarantee we hit
the async response path, resulting in a nondeterministic coverage
failure.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-08 17:10:38 -07:00
Anders Kaseorg e112b619cc tornado: Fix race condition on handler._request.
Commit 6fd1a558b7 (#21469) introduced an
await point where get_events_backend calls fetch_events in order to
switch threads.  This opened the possibility that, in the window
between the connect_handler call in fetch_events and the old location
of this assignment in get_events_backend, an event could arrive,
causing ClientDescriptor.add_event to crash on missing
handler._request.  Fix this by assigning handler._request earlier.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-08 17:10:38 -07:00
Mateusz Mandera acfa55138e retention: Add docstring info on how archive cleaning works.
In particular, it's important to record the special treatment around
ArchivedAttachment rows not being deleted in this step.
2022-06-08 15:12:36 -07:00
David Rosa 52ef574d3e help-docs: Fix minor errors in "Link to a message or conversation".
Removes the ":" which have accidentally ended up in the "Get a link
to a specific topic" and "Get a link to a specific stream" headings.

Renames the "Via browser's address bar" tab to "Web" so that it
stays consistent with other help center articles.

Fixes part of #22147.
2022-06-08 12:20:32 -07:00
Zixuan James Li 426f8ce385 tests: Replace `HttpResponse` with `TestHttpResponse.`
Since `HttpResponse` is an inaccurate representation of the
monkey-patched response object returned by the Django test client, we
replace it with `_MonkeyPatchedWSGIResponse` as `TestHttpResponse`.

This replaces `HttpResponse` in zerver/tests, analytics/tests, coporate/tests,
zerver/lib/test_classes.py, and zerver/lib/test_helpers.py with
`TestHttpResponse`. Several files in zerver/tests are excluded
from this substitution.

This commit is auto-generated by a script, with manual adjustments on certain
files squashed into it.

This is a part of the django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-08 11:25:03 -07:00
Anders Kaseorg d5fea08b8a cache: Remove needless monkey patching.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-07 09:37:43 -07:00
Sahil Batra dfacd62d93 streams: Update error message returned for administrative access.
We have now decided to not continue with the stream administrator
concept as we are changing the permissions model to be based on
user groups as per #19525. So, this commit updates the error message
to "Must be an organization administrator".
2022-06-07 09:36:24 -07:00
Zixuan James Li a142fbff85 tests: Refactor away result.json() calls with helpers.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-06 23:06:00 -07:00
Anders Kaseorg a7f9c4f958 logging: Pass more format arguments to logging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-03 12:27:23 -07:00
Alex Vandiver d0c5389d60 gitlab: Fix event_name -> object_kind defaulting.
94457732c1 changed this from:

```py
event_name = payload.get("event_name", payload.get("object_kind")).tame(check_string)
```

...to:
```py
event_name = payload.get("event_name", payload["object_kind"]).tame(check_string)
```

Which causes a failure when `event_name` exists but `object_kind` does
not, since the default is evaluated first.

Switch to an `if` statement to clarify the fallbacks better.
2022-06-02 23:23:19 -07:00
Mateusz Mandera a671ae9749 get_old_unclaimed_attachments: Add docstring explaining the logic. 2022-06-02 17:32:23 -07:00
Mateusz Mandera 09dc166b45 do_delete_old_unclaimed_attachments: Consider ArchivedAttachment rows.
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
2022-06-02 17:32:23 -07:00
Mateusz Mandera 5ff4754090 test_upload: Fix some URLs to uploaded files.
Using http://localhost:9991 is incorrect - e.g. messages sent with file
urls constructed trigger do_claim_attachments to be called with empty
list in potential_path_ids.

realm.host should be used in all these places, like in the other tests
in the file.
2022-06-02 17:32:23 -07:00
Aman Agrawal 085443c722 home: Get default_language info from cookie for spectator.
This allows us to set default_language as cookie in frontend which
can be used on server load to send the relevant
translation data to the client.
2022-06-01 17:08:00 -07:00
Adam Sah ba5cf331a2 testing: 100% coverage for zerver/tests/test_digest.py. 2022-06-01 16:09:13 -07:00
Anders Kaseorg 9ee636e920 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 16:01:54 -07:00
Lauryn Menard 2047ca8f16 api-docs: Add changelog entry for user/stream ID narrow options.
Adds a 2.1 release changelog entry for adding support for user
and stream IDs in search/narrow options. Also, adds a Changes
note in the narrow parameter in the OpenAPI `get-messages`
endpoint definition.

Both link to the api documentation for constructing a narrow,
where the 2.1 release update is already mentioned.

Fixes #9474.
2022-06-01 15:11:26 -07:00
Lauryn Menard 82c9c6f599 api-docs: Expand use of `SimpleSuccess` response schema.
Use `SimpleSuccess` response schema for all endpoints that were
already returning a success (200) response without any data beyond
the `response` and `msg` fields, which are standard for all
endpoint responses.

Prep commit for adding `ignored_parameters_unsupported` to
`json_success` responses.
2022-06-01 15:09:39 -07:00
Zixuan James Li 4c6f2ae7be typing: Add assertions for authentication.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li c572d9be5a typing: Add none-checks for db queries.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li 4a5043dd6e typing: Add none-checks for miscellaneous cases.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li e338ada66c typing: Add none-checks for Recipient objects.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li bb6a934c8d typing: Add appropriate none-checks for LOCAL_UPLOADS_DIR.
This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li d5517932cd typing: Use BaseDatabaseSchemaEditor in place of DatabaseSchemaEditor.
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 14:18:53 -07:00
Zixuan James Li 63e9ae8389 typing: Apply trivial fixes to adjust edge cases in typing.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.

This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 12:03:51 -07:00
Zixuan James Li c34ac1fcd4 typing: Access url via key "Location" instead of attribute "url".
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 11:59:47 -07:00
Sahil Batra 9273c314cf invites: Capitalize "ID" in the error raised for invalid stream ids.
This commit changes "id" to "ID" in the error message returned when
invalid stream IDs are passed to multiuse invite endpoint.
2022-05-27 17:06:03 -07:00
Sahil Batra 708204290e streams: Capitalize "ID" in invalid stream errors in API.
This commit changes the error message from "Invalid stream id"
to "Invalid stream ID" for cases where invalid stream IDs are
passed to API endpoints to make it consistent with other similar
error messages.
2022-05-27 17:06:03 -07:00
Zixuan James Li 44ecd66eae types: Better types for API fields.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:43:00 -07:00
Zixuan James Li f1ef27cb85 password_reset: Soft reactivate upon password reset.
Fixes #22066

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:30:34 -07:00
Zixuan James Li 3e95b59f2e soft_deactivation: Add a helper for queuing soft_reactivation.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:28:52 -07:00
Anders Kaseorg bb6bd900cd response: Replace response.asynchronous attribute with new class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-27 14:27:34 -07:00
Alex Vandiver 134977b590 email: Attempt to suppress email auto-replies.
This applies a commonly-used, though non-RFC, header which suppresses
auto-replies to the message.  There is a small chance that this will
result in bad filters thinking the messages *from Zulip* are
themselves auto-replies, but this seems a small risk.

Fixes: #13193.
2022-05-26 18:04:12 -07:00
Aman Agrawal 3991e89f19 models: Add missing database index for open communities directory.
Enable database indexing of `want_advertise_in_communities_directory`
field of `Realm` model.
2022-05-26 11:47:24 -07:00
Chris Bobbe abfbacbdf6
openapi: Clarify custom_profile_field.value can be null in update-user event.
This has always been true, but wasn't documented correctly.
2022-05-26 10:52:13 -07:00
Lauryn Menard aa5536fe30 api-docs: Update `/set-typing-status` endpoint documentation.
Adds Changes notes for feature level 58 where support was added
for stream messages for the `/set-typing-status` endpoint
parameters.

Updates formatting for references to the `type`
parameter in the descriptions of other endpoint parameters.

Improves readability of and updates links in the endpoint's main
description.
2022-05-24 16:02:30 -07:00
Mateusz Mandera 4e1befa209 sessions: Rename get_session_user to get_session_user_id.
The function returns a user id, not a user - we generally try to reflect
that in function names accurately.
2022-05-23 15:57:10 -07:00
Mateusz Mandera 66d8c2ae15 sessions: Don't filter by expire_date in delete_realm_user_sessions.
The other delete_... functions just do .all() - we should be consistent
one way or another.
2022-05-23 15:57:10 -07:00
Mateusz Mandera 003d086f2e sessions: Only fetch user ids in delete_realm_user_sessions.
No need to fetch entire UserProfiles, especially since we're dealing
with all UserProfiles in a Realm here, which would be quite large.
2022-05-23 15:57:09 -07:00
Tim Abbott 0275358da8 openapi: Improve documentation of new_email field in events. 2022-05-23 15:53:05 -07:00
Anders Kaseorg 8b1e7a8b3e github: Accept null for issue and pull request body.
https://github.com/octokit/webhooks/blob/v5.6.0/payload-schemas/api.github.com/common/issue.schema.json#L84
https://github.com/octokit/webhooks/blob/v5.6.0/payload-schemas/api.github.com/common/pull-request.schema.json#L78

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-23 15:52:24 -07:00
Lauryn Menard 7ceb5f6311 api-docs: Add changelog for `stream_id` param to `mute-topic`.
Adds a changelog 2.0 entry for adding support for `stream_id`
parameter to the `mute-topic` endpoint. Also, adds Changes note
to the endpoint parameter description, and reorders/clarifies
that at least one (and only one) stream parameter must be provided
by the client and that the `string_id` parameter is preferred.

Fixes #11136.
2022-05-23 15:48:06 -07:00
Sahil Batra ce34b585a5 streams: Add endpoint "GET /streams/{stream_id}" to get stream by id.
Fixes #22082.
2022-05-23 15:14:04 -07:00
Alex Vandiver 3d66dd9eeb fill_memcached_caches: Document possible arguments to --cache. 2022-05-22 14:52:38 -07:00
Lauryn Menard 2d85612ff0 api-docs: Add `create_web_public_stream_policy` to realm update.
Adds `create_web_public_stream_policy` to the `get-events` API
documentation for the `realm op:update` event.

Also, fixes changelog entries for feature levels 103 and 104,
which are related to the API documentation changes or fix an
error in references to the undocumented endpoint `PATCH /realm`.
2022-05-19 13:37:40 -07:00
Sahil Batra 95a2b580a0 streams: Save one DB query in bulk_remove_susbcriptions.
We remove one call to get_occupied_streams to get occupied
streams before unsubscribing because we already know which
streams can become vacant, i.e. the one from which users are
being unsubscribed, and we can directly use the list of streams
from which users are being unsubscribed and get vacant streams
by checking which of these streams are not in get_occupied_streams
called after unsubscribing users.
2022-05-17 15:09:15 -07:00
Alex Vandiver 21ec1f07d4 name_restrictions: Add your-org.zulipchat.com as a reserved name.
This is used as a placeholder in the mobile app.
2022-05-17 14:58:31 -07:00
Sahil Batra 6b82cbe0a6 user_groups: Rename existing_subgroups variable to existing_direct_subgroup_ids.
This commit renames existing_subgroups variable to existing_direct_subgroup_ids
in add_subgroups_to_group_backend and remove_subgroups_from_group_backend functions
for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra 8edf30caf2 user_groups: Rename subgroups parameter to direct_subgroup_ids.
This commit renames subgroups parameter of do_send_create_user_group_event
to direct_subgroup_ids for better readability.
2022-05-17 14:51:55 -07:00
Sahil Batra dfd7902c77 user_groups: Rename subgroups fields to direct_subgroup_ids.
This commit renames subgroups and subgroup_ids field sent in user
group objects to direct_subgroup_ids for better readability.
2022-05-17 14:51:45 -07:00
Adam Sah 04ccd8c6d8 testing: 100% code coverage for zerver/lib/outgoing_webhook.py. 2022-05-17 13:51:00 -07:00
Alex Vandiver 814841c9ec puppet: Remove typo'd cron job.
54b6a83412 fixed the typo introduced in 49ad188449, but that does
not clean up existing installs which had the file with the wrong name
already.

Remove the file with the typo'd name, so two jobs do not race, and fix
the typo in the comment.
2022-05-16 14:57:21 -07:00
Alex Vandiver 513fcb7bd5 Revert "settings: Add web-public streams beta subdomain list."
This reverts commit 20368a936c.  It is
no longer in beta, and this configuration is no longer needed.
2022-05-16 14:49:14 -07:00
Alex Vandiver a6d1c41827 email_mirror: Handle invalid `charset="..."` values. 2022-05-16 12:04:38 -07:00
Anders Kaseorg 0043c0b6b2 django: Use HttpRequest.headers.
Fixes #14769.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d98e3ecb0d tests: Don’t reuse HttpRequest objects.
Django caches some information on HttpRequest objects, including the
headers dict, under the assumption that requests won’t be reused.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d3c6ca8def development: Redirect rather than mutate in register_development_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg 110f7a379a beanstalk: Move %40 kludge into authenticated_rest_api_view.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Alex Vandiver 803982e872 message_flags: Short-circuit if no messages changed.
Omit sending an event, and updating the database, if there are no
matching messages.
2022-05-12 21:57:55 -07:00
Sahil Batra 0bfe973b65 streams: Mark messages as read only for unsubscribed streams.
Previously, we were marking messages of all the streams passed
to bulk_remove_subscriptions even if user was not subscribed
to some of them and those streams would ideally not have
any unread messages. This code was added in 766511e519.

This commit changes the code to only mark messages of actually
unsubscribed streams as read.
2022-05-12 18:53:45 -07:00
Anders Kaseorg 1e453633b0 gitlab: Remove function signature inspection nonsense.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg f05fa9937b gci: Fix get_body_based_on_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg 4c772989a1 front: Fix get_body_based_on_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 18:04:44 -07:00
Anders Kaseorg 6d1b68c61b freshdesk: Remove unsafe TicketDict class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:56:46 -07:00
Anders Kaseorg cce142c61a middleware: Fix URL encoding of next parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:51:51 -07:00
Tim Abbott 86137dc02f users: clarify confusing check for last owner.
Conceptually, we're clearly intending to check whether the user we're
mutating is the last realm owner. The preexisting code was safe
because we've already checked that the target user is an owner, and
thus if we're the last owner, we're the target user.
2022-05-10 14:09:57 -07:00
Somesh Ranjan 0b1f8c05e3 org_settings: Add backend support to change bot role.
This commit attempts to add the backend support by extending the
/json/bots/{bot_id}/ url support to accept the role field as a
parameter. This was previously already possible via
`/json/users/{user_id}`, so this change just simplifies client
implementation.
2022-05-10 14:09:57 -07:00
Christopher Chong 94457732c1 github, gitlab: Replace the usage of dict.get(key) with dict[key]. 2022-05-10 13:50:20 -07:00
Christopher Chong 27a498e1e7 gitlab: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 76b1a682b7 github: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 2f097c524e gitea, gogs: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 83178a474f gci: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 087d4dff9d front: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 3cd41be3d0 freshstatus: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong f349da218c freshping: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong ec1e958e90 freshdesk: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 6e0e175183 flock: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 5833c05fa6 errbit: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 66fd6b1e55 dialogflow: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Christopher Chong 990a046fc7 delighted: Strengthen types using WildValue. 2022-05-10 13:50:20 -07:00
Lauryn Menard 0acf955265
openapi: Fix inconsistencies in OpenAPI error documentation.
Fixes a few small inconsistences / mistakes in the OpenAPI
documentation related to error documentation. Does not change
the rendered API documentation, which is likely why these were
not noticed sooner.
2022-05-06 15:07:08 -07:00
Alya Abbott 86b61d46eb portico: Add Recurse Center case study. 2022-05-05 16:43:00 -07:00
Tim Abbott 24f527cb59 home: Don't send /register response for spectators.
In very large communities, computing page_params can be quite
expensive. Because we've moved the homepage for communities with web
public streams enabled to be the Zulip app, and it's common for
automation to frequently poll the homepage of a Zulip organization,
we'd like to keep those homepages cheap (as the login pages are).

We address this by prototyping something we may end up wanting to do
anyway -- having the web application do a `POST /register` API call in
order to fetch most page_params, and merging those with the mostly
webapp configuration page_params that we leave in the / response for
convenience.

This exact implementation is messy in a few ways:

* We rely on the assumption that ui_init.initialize_everything happens
  before all code that needs to inspect the page_params properties we
  are fetching via /register. This is likely mostly true, but nothing
  in the implementation enforces it.
* The bundle of ~25 keys that are in page_params ideally would be
  considered individually, with some moved to the /register API
  response and perhaps others eliminated or namespaced inside a
  webapp_settings object.
* It's weird to have the spectators network sequence different that
  from logged-in users, and potentially a maintainability risk.
* We might be able to arrange that the initial `/` response be
  cacheable, now that we're no longer embedding our metadata inside
  it. We've made no effort to do that as of yet.

Despite those issues, this commit solves an immediate problem and will
give us helpful experience with a model closer to the one we'll want
in order to happily support a web client that can be run locally
against a production Zulip server's data.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2022-05-05 15:20:46 -07:00
Tim Abbott 2e86ea6540 events: Add support for spectator access to /register.
This is necessary for the mobile/terminal clients to build spectator
support down the line. We'll also be using it for the web application,
in an upcoming commit.
2022-05-05 15:20:44 -07:00
Tim Abbott 53518e8a24 events: Pass an explicit realm to do_events_register. 2022-05-05 15:17:07 -07:00
Tim Abbott d180f7ccfc right sidebar: Clean up rendered description logic.
Previously, we were masking the realm_description raw Markdown with
rendered Markdown, which was a type error.

When we switch to calling /register explicitly in a few commits, this
results in a bug, since the raw Markdown ends up taking priority.

Fix this by just using a different name for this different concept.
2022-05-05 15:17:07 -07:00
yogesh sirsat 08e8a21da0 custom_profile_fields: Clarify an external account error message.
This error message is for a very precise situation -- the pattern not
having the desired format. We should say that, rather than a generic
"Malformed".
2022-05-04 17:57:44 -07:00
NerdyLucifer 6a5d646739 settings (admin/org): Show error for same choices in "list of options".
Currently an user can create multiple options with same text/label in
the select/"list of options" custom profile field type.

Fix this issue by extending the validator to throw an error if there
are duplicate choices in the "list of options" in custom profile
field.

Tweaked by tabbott to use a simpler check.

Fixes: #21880
2022-05-04 17:55:28 -07:00
Adam Sah a9ce02348b testing: Improve backend code coverage for semaphore webhook. 2022-05-04 16:13:24 -07:00
Alex Vandiver 21e80e47bd message_edit: Filter UserMessage better, now that mentions are unneeded.
The previous commit removed the need for `render_incoming_message` to
take a list of mentioned users; simplify the callsites accordingly.
2022-05-04 14:45:18 -07:00
Alex Vandiver cd9c69cd12 message_send: Remove unnecessary user_ids argument.
cfcbf58cd1 rightly removed the use of `user_ids` in
`render_markdown`, which in turn makes it unnecessary in
`render_incoming_message`.

Remove the unnecessary parameter from `render_incoming_message`.
2022-05-04 14:45:18 -07:00
Alex Vandiver 74e9b086f9 embed_links: Check that the message still exists before proceeding. 2022-05-04 14:45:18 -07:00
Alex Vandiver de63000db6 embed_links: Take a lock on the message object while editing.
We leave the fetching of links outside of the lock, as they could take
seconds, which is an unreasonable amount of time to hold a lock on the
message row.  This may result in unnecessary work, in the case that
the message was since edited, but the unnecessary work is preferable
to blocking other work on the message row for the duration.
2022-05-04 14:45:18 -07:00
Sahil Batra 9a6886f630 users: Always pass delivery_email in user's own object.
This commit changes the code to always pass delivery_email
field in the user's own object in 'realm_users'.

This commit also fixes the events sent by notify_created_user.
In the "realm_user/add" event sent when creating the user,
the delivery_email field was set according to the access
for the created user itself as the created user was passed as
acting_user to format_user_row. But now since we have changed
the code to always allow the user themselves to have access
to the email, this bug was caught in tests and we fix the person
object in the event to have delivery_email field based on whether
the user receiving the event has access to email or not.
2022-05-04 12:52:43 -07:00
Sahil Batra aa98b39429 bulk_create: Copy realm-level defaults in bulk_create_users.
This commit adds code to copy the realm-level default of
settings while creating users through bulk_create_users.

We do not directly call 'copy_default_settings' as it
calls ".save()" but here we want to bulk_create the objects
for efficiency.

We also add the code to set realm-default of enter_sends as
True for the Zulip dev server as done in 754b547e8 and thus
we remove enter_sends argument from create_user_profile as
it is of no use now.
2022-05-04 12:52:43 -07:00
Sahil Batra 8ff0cec2ed users: Pass email_address_visibility as parameter to can_access_delivery_email.
This is a prep commit for adding user-level email visibility setting.
2022-05-04 12:52:43 -07:00
Lauryn Menard 44c9b788f9 settings: Add realm setting for Zulip communities directory.
Adds `want_advertise_in_communities_directory` to the realm model
to track organizations that give permission to be listed on such
a site / directory on zulip.com.

Adds a checkbox to the organization profile admin for
organizations to give permission to be advertised in the
Zulip communities directory.

Adds a help center article about the Zulip communities directory
and uses a shared intro documentation file to create sections in
the articles on creating an organization profile and moderating
open organizations.

Co-authored-by: Alya Abbott <alya@zulip.com>
2022-05-04 11:13:28 -07:00
Mateusz Mandera 163910b765 auth: Don't leak info about accounts in other realms in the login form.
The "wrong subdomain" error leaks information that we shouldn't reveal.
We want to return a generic failure message instead.
2022-05-03 17:06:51 -07:00
Mateusz Mandera 1faebecb7a auth: Change the logger for authing to the wrong subdomain situations.
It's better to use zulip.auth logger here instead of the root logger.
Also the level is adjusted to INFO, since WARNING is excessive.
2022-05-03 17:06:51 -07:00
Mateusz Mandera 71f6f2310c auth: Don't log email when user tried authing to the wrong subdomain.
We want to avoid logging this kind of potentially sensitive information.
Instead, it's more useful to log ids of the matching accounts on
different subdomains.
2022-05-03 17:06:51 -07:00
Anders Kaseorg f29553d809 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 10:10:06 -07:00