Commit Graph

15996 Commits

Author SHA1 Message Date
Mateusz Mandera b5b8cc06dc auth: Clean up redundant request.session.modified change.
This line was added in 94e099eaab,
presumably because of the
del request.session["multiuse_object_key"]
line that was just above it.
Looks like it should have been removed in
868a763cec, which eliminated that `del`
operation.
2022-11-10 17:46:53 -08:00
Alex Vandiver f3135eb414 export_search: Add ability to limit by date sent. 2022-11-10 16:49:19 -08:00
Alex Vandiver 2d5366f455 export_search: Add a compliance tool to export a subset of messages.
This is primarily for administrators needing to provide message
history for compliance or auditing purposes.  Search terms can be
pulled from a file, one per line, or from arguments provided on the
command line.
2022-11-10 16:49:19 -08:00
Alex Vandiver 1f18fa6580
register: Not all clients send a User-Agent header.
In 1fce1c3c73, we added logic to parse
the User-Agent in /register requests; this logic crashed if an HTTP request
was missing that header.

Includes a test for `/register` with no user agent passed; this should catch
similar regressions in the future.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2022-11-10 16:21:04 -08:00
Anders Kaseorg 70dbeb197f middleware: Set the correct options on the django_language cookie.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-09 14:24:22 -08:00
Lauryn Menard 62c9bb728e tests: Remove ignored `field_type` param in custom profile data test.
For PATCH requests to `/realm/profile_fields`, the field_type is
determined via the field.id, which is in the URL. Therefore,
`field_type` does not need to be passed as part of the request
data in tests.
2022-11-08 11:08:14 -08:00
Sahil Batra 7c4da60b53 message: Don't create historical UserMessage rows when marking as read.
We do not create historical UserMessage rows, for messages that didn't
have one, while marking messages as read and simply ignore those messages.
We do so because there is no user of creating UserMessage rows and it just
wastes storage.

Note that we still allow to mark messages from unsubscribed streams as
read but only those which have UserMessage rows for them to handle the
case when the unread messages were not marked as read while unsubscribing
from the stream due to some race condition. In such cases, messages
will not be included in the unread count shown in "All messages" menu
(and stream is anyways not present in the left sidebar), but the message
border on the left is green if viewing the stream after unsusbcribing it.
So, to avoid the confusion for users, the messages will be marked as read
when user scrolls down.
2022-11-08 08:50:17 -08:00
Zixuan James Li 390e2a12e5 webhooks: Use get_short_sha to get shortened sha.
This unifies the length of the shortened SHA our integrations generate,
and ensures that they are long enough for projects of various sizes with
a chosen value defined in get_short_sha.

Fixes #23475

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-08 08:20:51 -08:00
Zixuan James Li 4e46899494 webhooks: Pick a more reasonable length for short sha.
7 characters are not enough for large projects, so we change
it to reasonably longer. As an example, The Linux kernel needs
at least 11 characters of sha in its shortened form to identify
a revision. We pick 11 so it should work for most of the projects.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-08 08:20:51 -08:00
Lauryn Menard 2a1ceb58d9 api-docs: Return 404 for direct `/api-doc-template` call.
We use `templates/zerver/api/api-doc-template.md` as a base template
for the documented API endpoints in `zerver/openapi/zulip.yaml`.

Previously, if this template was called as an endpoint, then it
would fail an assertion check and send server error. Now we check
for specifically for that potential path and return a 404 error
response for no existing article.

Fixes #21876.
2022-11-08 08:10:15 -08:00
Matt Keller 8aa7ff4bbb slack: Parse emoji skin tone variants.
Fixes part of #23276.
2022-11-07 14:25:49 -08:00
Lauryn Menard fbe2ef9431 api-docs: Remove unused description in get-subscription-status. 2022-11-07 12:31:55 -08:00
Lauryn Menard 566a6823af api-docs: Use default example response description for get-messages. 2022-11-07 12:31:55 -08:00
Lauryn Menard 6f44681354 api-docs: Use default response example description for create-drafts. 2022-11-07 12:31:55 -08:00
Alya Abbott 9ce484e267 portico: Add a page about our values. 2022-11-07 12:27:09 -08:00
Alya Abbott 1ef94cc2b4 portico: Update /history page. 2022-11-07 12:27:09 -08:00
Lauryn Menard 19a4c3907f decorator: Put back check for dev env in `ignore_unhashable_lru_cache`.
Prior to 53231aa, the `ignore_unhashable_lru_cache` decorator had
a check for the development environment so that changes could be
seen on refresh.

Puts that check back in IgnoreUnhashableLruCacheWrapper class.
2022-11-07 12:17:59 -08:00
Lauryn Menard 66e607cf4c tests: Remove ignored `message_id` parameter in message edit tests.
For PATCH requests to edit messages, the message ID is in the URL
and does not need to be passed as part of the data in tests.
2022-11-07 12:16:28 -08:00
Anders Kaseorg 55342efd33 scim: Upgrade django-scim2; remove request.user monkey patching.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-05 15:13:50 -07:00
Anders Kaseorg 7e8ebd18b6 test_scim: Add test for missing or wrong authentication.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-05 15:13:46 -07:00
Alex Vandiver d8ebbedbbb outgoing_webhook: Ignore the exception on failure if the stream is gone.
In the outgoing webhook handler, there is potentially several seconds
of trying between when a message triggering an outgoing webhook
arrives, and when it fails.  In the meantime, the stream the
triggering message was on may have been deleted, causing the
"Failure!" message to have no valid stream to be sent to.

Rather than raise an exception in the outgoing webhook worker, ignore
the exception and move on.
2022-11-04 14:46:17 -07:00
Alex Vandiver eb7a2f2c38 queue: Do test retries in tests.
The lambda passed to `queue_json_publish` is used if
`settings.USING_RABBITMQ` is unset -- which is only true in tests.  As
such, this pattern causes failures to never actually retry within
tests.

This behaviour has existed ever since the outgoing webhook code was
introduced in 53a8b2ac87, with no explanation.  Not passing that
argument allows tests to verify the retry behaviour when webhooks
fail.
2022-11-04 14:46:17 -07:00
Zixuan James Li 7dbd98d25c rest: Restrict access to json views via basic auth.
Previously, test cases or clients accessing /json/ views using HTTP
Basic Auth would be accepted, while we intended to only allow clients
authenticated with a session cookie to access these views.

This adds a check on the accessed path to avoid this possibility.

It seems unlikely that any API clients clients were taking advantage
of this unintended quirk; so we're not going to bother documenting
this bug fix as an API change. In any case, it should be trivial for
anyone affected to consult the documentation and then switch their
/json/foo URL to a correct /api/v1/foo URL.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-04 14:44:07 -07:00
Sahil Batra 1fce1c3c73 register: Handle "Pronouns" type fields for older mobile clients.
Mobile clients older than v27.192 do not support PRONOUNS type
custom profile fields, so we instead change the type of it to
SHORT_TEXT in the data sent with register response and also in
the events sent to those clients.
2022-11-04 14:31:18 -07:00
Sahil Batra b2737b0878 migrations: Add migration to migrate pronouns custom profile fields.
This commit adds a migration to migrate SHORT_TEXT type profile
fields for pronouns to recently added PRONOUNS type.
2022-11-04 14:31:18 -07:00
Anders Kaseorg b9dfa23666 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 14:08:58 -07:00
Anders Kaseorg c6f6df986c openapi: Fix “contatining” typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 14:08:58 -07:00
M@ 47ecd33d3a
import: Narrow the id-window to just the current realm.
On multi-realm systems this results in traversal of all messages in
all realms and returns a massive payload of 1 row per stream on 
the server, not the intended one row per realm.
2022-11-03 12:13:43 -07:00
Anders Kaseorg 3d853caf16 ruff: Fix C417 Unnecessary `map` usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg ab864be904 ruff: Fix C416 Unnecessary `set` comprehension (rewrite using `set()`).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 77c15547e6 ruff: Fix C414 Unnecessary `list` call within `sorted()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 69e94b5991 ruff: Fix C413 Unnecessary `list` call around `sorted()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 9e53f0c756 ruff: Fix C405 Unnecessary `list` literal (rewrite as a `set` literal).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 033d2615f6 ruff: Fix B017 `assertRaises(Exception):` should be considered evil.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg e1a2894af9 ruff: Fix B011 Do not `assert False` (`python -O` removes these calls).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg adffad384c lint: Replace pycodestyle and pyflakes with ruff.
https://github.com/charliermarsh/ruff

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Tim Abbott 873ab01c7a message_flags: Fix accidentally quadratic behavior. 2022-11-02 21:05:24 -07:00
Sahil Batra aeb9a27d0e message: Ignore messages in unsubcribed streams when marking them unread.
Zulip's unread messages design has an invariant that all unread stream
messages must be in streams the user is subscribed to. For example, We
do not include the unread messages from unsubscribed streams in the
"unread_msgs" data structure in "/register" response and we mark all
unread messages as read when unsubscribing a user from a stream.

Previously, the mark as unread endpoint allowed violating that
invariant, allowing you to mark messages in any stream as unread.

Doing so caused the "message_details" data structures sent with
"update_message_flags" events to not contain messages from
unsubscribed streams, even though those messages were present in the
set of message IDs. These malformed events, in turn, caused exceptions
in the frontend's processing of such an event.

This change is paired with a separate UI change to not offer the "Mark
as unread" feature in such streams; with just this commit, that will
silently fail.

With some additions to the tests by tabbott.
2022-11-02 14:49:23 -07:00
Mateusz Mandera 6e336ef6f6 delete_topic: Use the same timeout pattern as /mark_all_as_read.
We don't want to use the nginx 502 timeout as an API pattern. In
/mark_all_as_read we've already figured out how we want to handle this
instead.
2022-11-02 16:50:06 -04:00
Zixuan James Li da9ad7c3a7 markdown: Pass realm down into render_stream_description.
This guarantees that the Realm is always non-None when we hit the
codepath is_static_or_current_realm_url via
do_change_stream_description, so that we can properly skip rewritting
some images.

Fixes #19405

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-01 08:20:47 -07:00
Mateusz Mandera 582d5b0aa3 realm_creation: Rework error pages.
The previous error page was inadequate for serving the two different
scenarios where we show errors in realm_creations, in particular
containing a misleading sentence about realm creation being disabled
(even in the case where it was actually enabled and the user simply had
an expired link).
2022-10-31 17:35:06 -07:00
Tim Abbott b057d8ea17 invites: Delete ScheduledEmails in revoke invites transactions.
This prevents likely rare race conditions where a reminder about a
revoked invitation might get sent.
2022-10-31 15:06:42 -07:00
Mateusz Mandera 3dd0545118 invites: Add atomic() around some invite-revoking blocks.
These should only ever happen together, so should be wrapped in
atomic().
2022-10-31 14:37:46 -07:00
Lauryn Menard 0c947985bd tests: Update tests to use example profile picture.
So that we can stop using Tim's photo for tests, adds an open
license profile picture to use instead.

Updates tests that used `tim.png` to use the new example profile
picture, which is located in `static/images/test-images/avatars/`.
2022-10-31 14:36:54 -07:00
Lauryn Menard 1eda42f509 images: Create `/authentication_backends` directory for icons.
Creates `static/images/authentication_backends` directory for icons
of backend authentication methods, which are used on the log-in page.

And updates the example documentation in the API `/server_settings`
endpoint.
2022-10-31 14:36:54 -07:00
Lauryn Menard 89713bab76 tests: Use Zulip icon for markdown test cases instead of octopus.
Updates the markdown test case that used `zulip-octopus.png` to
instead use an zulip logo that's also referenced in a frontend
puppeteer test, `static/images/logo/zulip-icon-128x128.png`.
2022-10-31 14:36:54 -07:00
Sahil Batra 888e4f9bf4 custom_profile_fields: Fix query for display_in_profile_summary limit.
The query in display_in_profile_summary_limit_reached should check
realm also since there is per-realm limit of 2 fields, otherwise
this will cause issues where multiple realms are hosted on the
same server.

Fixes #23368.
2022-10-31 14:10:47 -07:00
Mateusz Mandera 760587450e retention: Remove two redundant comments.
These two identical comments don't contribute anything useful and seem
just out of place at this point.
2022-10-31 10:23:57 -07:00
Mateusz Mandera 7b13204e8f retention: Use Message.realm to simplify private message query.
We no longer need to do the inner joins to figure out the message's
realm and split up the cross-realm and regular case - now we just look
at zerver_message.realm directly.
2022-10-31 10:23:57 -07:00
Mateusz Mandera 78a325ac58 internal_send_private_message: Fix edge case of message to system bot.
I don't think this is used anywhere outside of tests, but we should have
this logic correct. If this function is used to send a message from a
user to a cross-realm bot, the message.realm should be the realm of the
user.

In the normal case, where a user send a message to a cross-realm bot
through the API is already handled correctly, this bug is unrelated.
2022-10-31 10:23:57 -07:00