In the outgoing webhook handler, there is potentially several seconds
of trying between when a message triggering an outgoing webhook
arrives, and when it fails. In the meantime, the stream the
triggering message was on may have been deleted, causing the
"Failure!" message to have no valid stream to be sent to.
Rather than raise an exception in the outgoing webhook worker, ignore
the exception and move on.
The lambda passed to `queue_json_publish` is used if
`settings.USING_RABBITMQ` is unset -- which is only true in tests. As
such, this pattern causes failures to never actually retry within
tests.
This behaviour has existed ever since the outgoing webhook code was
introduced in 53a8b2ac87, with no explanation. Not passing that
argument allows tests to verify the retry behaviour when webhooks
fail.
Previously, test cases or clients accessing /json/ views using HTTP
Basic Auth would be accepted, while we intended to only allow clients
authenticated with a session cookie to access these views.
This adds a check on the accessed path to avoid this possibility.
It seems unlikely that any API clients clients were taking advantage
of this unintended quirk; so we're not going to bother documenting
this bug fix as an API change. In any case, it should be trivial for
anyone affected to consult the documentation and then switch their
/json/foo URL to a correct /api/v1/foo URL.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Mobile clients older than v27.192 do not support PRONOUNS type
custom profile fields, so we instead change the type of it to
SHORT_TEXT in the data sent with register response and also in
the events sent to those clients.
On multi-realm systems this results in traversal of all messages in
all realms and returns a massive payload of 1 row per stream on
the server, not the intended one row per realm.
Zulip's unread messages design has an invariant that all unread stream
messages must be in streams the user is subscribed to. For example, We
do not include the unread messages from unsubscribed streams in the
"unread_msgs" data structure in "/register" response and we mark all
unread messages as read when unsubscribing a user from a stream.
Previously, the mark as unread endpoint allowed violating that
invariant, allowing you to mark messages in any stream as unread.
Doing so caused the "message_details" data structures sent with
"update_message_flags" events to not contain messages from
unsubscribed streams, even though those messages were present in the
set of message IDs. These malformed events, in turn, caused exceptions
in the frontend's processing of such an event.
This change is paired with a separate UI change to not offer the "Mark
as unread" feature in such streams; with just this commit, that will
silently fail.
With some additions to the tests by tabbott.
This guarantees that the Realm is always non-None when we hit the
codepath is_static_or_current_realm_url via
do_change_stream_description, so that we can properly skip rewritting
some images.
Fixes#19405
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The previous error page was inadequate for serving the two different
scenarios where we show errors in realm_creations, in particular
containing a misleading sentence about realm creation being disabled
(even in the case where it was actually enabled and the user simply had
an expired link).
So that we can stop using Tim's photo for tests, adds an open
license profile picture to use instead.
Updates tests that used `tim.png` to use the new example profile
picture, which is located in `static/images/test-images/avatars/`.
Creates `static/images/authentication_backends` directory for icons
of backend authentication methods, which are used on the log-in page.
And updates the example documentation in the API `/server_settings`
endpoint.
Updates the markdown test case that used `zulip-octopus.png` to
instead use an zulip logo that's also referenced in a frontend
puppeteer test, `static/images/logo/zulip-icon-128x128.png`.
The query in display_in_profile_summary_limit_reached should check
realm also since there is per-realm limit of 2 fields, otherwise
this will cause issues where multiple realms are hosted on the
same server.
Fixes#23368.
We no longer need to do the inner joins to figure out the message's
realm and split up the cross-realm and regular case - now we just look
at zerver_message.realm directly.
I don't think this is used anywhere outside of tests, but we should have
this logic correct. If this function is used to send a message from a
user to a cross-realm bot, the message.realm should be the realm of the
user.
In the normal case, where a user send a message to a cross-realm bot
through the API is already handled correctly, this bug is unrelated.
Clarifies most of the narrow parameter descriptions by adding
information about what a user's message history includes, about
new bot users not generally being subscribed to streams, and
about the specific `streams:public` narrow.
Updates the main descriptions for the `/get-messages` and
`/check-messages-match-narrow` endpoints.
Fixes#19477.
This commit updates the urls for personal narrow sent in email
notifications to be of form "{user_id}-{encoded_full_name}" to
make it consistent with the urls that we use for such narrows
in webapp which were recently updated in b4eddad for improving
performance. We encode the full name in the same way that we do in
webapp by replacing the url characters encoded by browser with "-".
It doesn't seem to make sense to append _{number} to the status code in
that arg, because the resulting string stops looking like a status code
and actually makes this test fail in the follow-up commit with the
confusing error message of
Unknown response http status: 2000
So this just seems like a bug.
Updates the hash used for the recent conversations view to be
"#recent" instead of "#recent_topics".
We will need to keep the logic for handling "#recent_topics"
permanently because users potentially have messages from
Welcome Bot with links to that hash.
Including "recent_topics" as a web_public_allowed_hash in
hash_util.js can be changed once self-hosted servers cannot
upgrade directly to Zulip 5.x from the current version.
Fixes#23132.
Output message should talk about both the cases:
actual_count > expected_count and actual_count < expected_count.
The message now includes information for the case where
actual_query_count < expected_query_count.
Fixes: #23325
Replaces instances of "recent topics" in the web-app and documentation
to be "recent conversations".
Renames both `recent-topics.md` files in the help center to be
`recent-conversations.md` and updates/redirects links to new URL.
Does not update instances of "recent topics" in frontend code comments
and does not update the main overview changelog, for now.
Does not change case study text where "recent topics" was referenced
in a quote, but does change generic text references to be "recent
conversations".
This was broken by commit b945aa3443
(#22604), because email_to_domain implicitly lowercased the result.
No adjustment is needed for is_disposable_domain, which already
lowercases its argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Renames the help article on custom profile fields to reflect that
its content is not just about adding fields.
Adds a redirect from the old URL to the new URL and updates internal
links, linking to #add-a-custom-profile-field where appropriate.
Fixes#23170.
Adds new tab to `zerver/lib/markdown/tabbed_sections.py` to document
managing bots from both personal settings and organization settings.
Documents adding bots from the organization settings Bots panel.
Separates instructions for deactivating and reactivating a bot from
both personal settings and organization settings.
Fixes a few formatting issues such as missing bold formatting and
heading level.
Fixes: #23066.
On my data (about 10 million messages in 1600 streams) this used to take
about 40 hours, while the improved statement completes in roughly 30
seconds.
The old solution had postgres go through the entire table until the
first match for each stream. Thus, the time spent scanning the table
got longer and longer for each stream because postgres always started at
the beginning (and somehow it did not use any indices) and had to skip
over all rows until it found the first message from the stream that is
was looking for each time.
This new statement just performans a bulk operation, scanning the table
only once and then inserts the results directly into the destination
table.
Slightly more verbose inforation about this change can be found in:
https://chat.zulip.org/#narrow/stream/31-production-help/topic/Import.20Rocketchat.20data/near/1408867
Signed-off-by: Florian Pritz <bluewind@xinu.at>
This adds a helper based on testing patterns of using the "queries_captured"
context manager with "assert_length" to check the number of queries
executed for preventing performance regression.
It explains the rationale of checking the query count through an
"AssertionError" and prints the queries captured as assert_length does,
but with a format optimized for displaying the queries in a more
readable manner.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This adds CapturedQueryDict to provide a more accurate type annotation
for the return value of queries_captured. We also replace "Generator"
with "Iterator" because the latter two type parameters were unused.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We followed a same approach as in #22611 to mark migrations as noops. We
might eventually squash them.
Migration operations that only change the validators should be removed
as they are essentially noops that do not affect the database schema.
However, ./manage.py makemigration still generates a new migration for
validators change regardless. So we still have to keep one migration
that updates the validators to the latest state. We prefer to keep the
earliest one for this purpose.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
These used to only be shown conditional on the
{% if password_auth_enabled %} in the template. Meaning that if you had
an org with email auth disabled and a deactivated user tried to log in,
they wouldn't see the error shown and get confused.
This switches the position of where these error will be shown (above the
login+password form instead of below it), but it looks fine.
Send an empty list of `custom_profile_fields` in `page_params` for
spectators, rather than not sending the field at all.
Also, updates the user info popover to not show the manage user
three-dot menu when in a spectator view.
As noted in the previous commit, this causes bloat in memcached, for
no purpose. Log a warning when `cache_with_key` sees a QuerySet
returned from the function it is decorating.
Storing a QuerySet rather than the list version of the result in it
has a large overhead -- and, as noted by the type annotations, the
result is only ever used as a list. This difference is particularly
important because the cached `get_realm_user_dicts` can get extremely
large for realms with large numbers of users, potentially overflowing
the 1MB default object limit in memcached.
Switch all cases of `cache_with_key` which return QuerySets to
returning the list values of them.
Storing this key is superfluous, as it will be the same for all users,
and definitionally already known to fetch the cache for the realm. It
is also not currently used by the callsites that read rows from the
cache.
9381a3bd45 added support for linkifier pattern URLs containing
`%20`-style escapes, but only did so for the codepath which is used in
the message body -- topic links did not understand them.
Expand the support to include when they are substituted into topics.
Previously we did not send notification for topic-only edits.
Now, we add backend support for sending notification to topic-only
edits as well.
We would add support for this in webapp in further commits since
message edit UI will be updated as well. We just make sure that no
notifications are sent when editing topic using pencil icon in
message header.
We also change the API default for moving a topic to only notify the
new location, not the old one; this matches the current defaults in
the web UI.
Includes many tests.
We also update the puppeteer tests to test only content edit as
we are going to change the UI to not allow topic editing from
message edit UI. Also fixing the existing tests to pass while
doing topic edits is somewhat complex as notification message
is also sent to new topic by default.
Fixes#21712.
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
The previous commit did this for revoking sessions. send_events should
be handled similarly too, to correctly handle calling do_deactivate_user
inside a transaction.
These were useful as a transitional workaround to ignore type errors
that only show up with django-stubs, while avoiding errors about
unused type: ignore comments without django-stubs. Now that the
django-stubs transition is complete, switch to type: ignore comments
so that mypy will tell us if they become unnecessary. Many already
have.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.
Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
zerver/migrations/0240_usermessage_migrate_bigint_id_into_id.py needs
to be updated to account for Django 4.1 creating AutoField as an
identity column rather than a serial column.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The sequence value should reflect the last id, not the next id, to
avoid leaving a gap of 1. Also, it should take ArchivedUserMessage.id
into account to avoid collisions during future archiving.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
With django-stubs, these explicit copies of Django’s implicit id
fields are no longer needed for type checking. An exception is the
BigAutoField AbstractUserMessage.id, which is left alone.
This reverts commit c08ee904d8 (#15641).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Note that django_stubs_ext is required to be placed within common.in
because we need the monkeypatched types in runtime; django-stubs
itself is for type checking only.
In the future, we would like to pin to a release instead of a git
revision, but several patches we've contributed upstream have not
appeared in a release yet.
We also remove the type annotation for RealmAuditLog.event_last_message_id
here instead of earlier because type checking fails otherwise.
Fixes#11560.
Previously, we type the model fields with explicit type annotations
manually with the approximate types. This was because the lack of types
for Django.
django-stubs provides more specific types for all these fields that
incompatible with our previous approximate annotations. So now we can
remove the inline type annotations and rely on the types defined in the
stubs. This allows mypy to infer the types of the model fields for us.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We no longer need to annotate the type of objects returned
from queries since django-stubs plugin infers that already.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This script pulls from our previously custom-written emoji strings
and fills in the rest from CLDR. It also removes 4 custom emoji which
collide with some of the new CLDR names (they will now just be called
by their CLDR name).
This works around some regression in moto 1.3.15 that I bisected to
b8820009e8
where ‘tools/test-backend test_transfer’ fails when run by itself.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This removes everything from SCIMClient except the "is_authenticated`
method. Previously, "realm" and "name" were only needed for logging
purposes. It is the best to keep SCIMClient as minimal as possible, as
it is only intended to be used for authenticating requests to SCIM
views.
This change also gurantees that the "LogRequests" middleware will not
rely on the type unsafe access of the format_requestor_for_logs method
on SCIMClient.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The scim_client attribute on SCIMTestCase is currently unused since
9198fe4fac.
The creation of the SCIMClient instance was previously needed because
zerver.middleware.validate_scim_bearer_token returns SCIMClient from the db.
The attribute itself on the was never really used in the test case.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Alex Vandiver
Zixuan James Li
Sahil Batra
Anders Kaseorg
M@
Tim Abbott
Mateusz Mandera
Lauryn Menard
Alya Abbott
Prakhar Pratyush
David Rosa
Florian Pritz
PIG208
Julia Bichler
PIG208
evykassirer