Commit Graph

55837 Commits

Author SHA1 Message Date
Anders Kaseorg 1fa8862e48 ruff: Fix RUF019 Unnecessary key check before dictionary access.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 712917b2c9)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 3bf1934598 openapi: Validate real requests and responses, not fictional mocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 029e765e20)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 9f2172c0f9 openapi: Get parameters from requestBody too.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 5cac872e4b)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 899819fb2f openapi: Use Parameter class for generating curl examples.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a67d1b57b9)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 33e77b6d15 openapi: Represent OpenAPI parameters with a Parameter class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 131b230e2b)
2024-03-07 15:07:12 -08:00
Anders Kaseorg fc8e023da2 openapi: Remove confusing check for 200 responses.
This error message didn’t make sense for the check as written, and our
OpenAPI document already provides the expected format for our 200
responses.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0cee3bee00)
2024-03-07 15:07:12 -08:00
Anders Kaseorg b5a1a9af25 test_openapi: Provide missing defs_mapping argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit bb699f7ee3)
2024-03-07 15:07:12 -08:00
Anders Kaseorg cf034cf19f openapi: Loosen anchor parameter specification to string.
Real requests would not validate against the previous version.  There
seems to be no consistent way to determine whether a string parameter
should be coerced to an integer for validation against an allOf
schema (which works at the level of JSON objects, not strings).

See also https://github.com/python-openapi/openapi-core/issues/698.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0514f92bdb)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 9fd5e9af4a test_classes: Add Content-Type header to empty DELETE/POST bodies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0dd92d2116)
2024-03-07 15:07:12 -08:00
Anders Kaseorg f03cc51aca test_classes: Default client_post to application/x-www-form-urlencoded.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a356ec7011)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 09706f820d requirements: Upgrade to openapi-core pre-release.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit d9f2f23c6a)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 0c67227f88 Revert "zproject: Prevent having exactly 17/18 middlewares, for Python 3.11 bug."
This reverts commit cf0b803d50.

(cherry picked from commit 41f253774d)
2024-03-07 15:07:12 -08:00
Anders Kaseorg 81d78cb88c ci: Disable coverage on Debian 12.
This is probably a more effective workaround for
https://github.com/python/cpython/issues/106092.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 4f27381ebc)
2024-03-07 15:07:12 -08:00
Aman Agrawal b222c7237b css: Apply fa-chevron-down CSS to zulip-icon-chevron-down.
This was missed as part of #27577 migration.

(cherry picked from commit ff49fec7af)
2024-03-07 15:01:15 -08:00
Alya Abbott 8a90495226 docs: Add plan sign-up to mobile notifications sign-up instructions.
(cherry picked from commit 3b22637d77)
2024-03-07 15:01:15 -08:00
Tim Abbott 27507e5944 docs: Better document backup/export tool tradeoffs.
- More consistent export/import vs backup bullets at the top.

- Remove misleading documentation regarding the `zulip_org_id` reuse
  problems. This documentation was written for Zulip 2.1.0 in
  c6fe6cf0a4 and largely made obsolete
  in d800ac33a0 (Zulip 5.0).

- Light editing for readability/crispness.

Fixes #28925.

(cherry picked from commit 270deb0334)
2024-03-07 15:01:15 -08:00
Alex Vandiver dc29251edd log-search: Fix attempts to limit to events requests in Django logs.
(cherry picked from commit 4735be85a7)
2024-03-07 15:01:15 -08:00
Alex Vandiver 18ac602e29 log-search: Support user-agent / client limit.
(cherry picked from commit 4372138f13)
2024-03-07 15:01:15 -08:00
Alex Vandiver 442b4fb71e log-search: Remove old /report/ endpoints.
(cherry picked from commit ed0058dff6)
2024-03-07 15:01:15 -08:00
Mateusz Mandera 726812c80b context_managers: Open file in write mode in lockfile_nonblocking.
Otherwise this fails if the file doesn't yet exist.

(cherry picked from commit dd6516d157)
2024-03-07 15:01:15 -08:00
Mateusz Mandera 96001b19fc update_analytics_count: Use a correct lock mechanism.
Adds a re-usable lockfile_nonblocking helper to context_managers.

Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.

We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.

(cherry picked from commit f61ed58c8f)
2024-03-07 15:01:15 -08:00
Mateusz Mandera aecab44538 remote_billing: Fix phrase on the "server already on a plan" error page.
"All older versions" was incorrect.

(cherry picked from commit ff02b4dda1)
2024-03-07 15:01:15 -08:00
Mateusz Mandera 5cecbcdfb3 config_error: Support passing arguments specifying the "go back" link.
Depending on the kind of config error being shown, different "go back"
links may be more appropriate.
We probably hard-coded /login/ for it, because these config errors are
most commonly used for authentication backend config error, where it
makes sense to have /login/ as "go back", because the user most likely
indeed got there from the login page.

However, for remote_billing_bouncer_not_configured, it doesn't make
sense, because the user almost surely is already logged in and got there
by clicking "Plan management" inside the gear menu in the logged in app.

(cherry picked from commit fcc3d88daf)
2024-03-07 15:01:15 -08:00
Mateusz Mandera 7d0917c3c7 remote_billing: Tweak /self-hosted-billing/ endpoints access model.
It's best for these to just be consistent. Therefore:
1. The .../not-configured/ error page endpoint should be restricted to
   .has_billing_access users only.
2. For consistency, self_hosting_auth_view_common is tweaked to also do
   the .has_billing_access check as the first thing, to avoid revealing
   configuration information via its redirect/error-handling behavior.

The revealed configuration information seems super harmless, but it's
simpler to not have to worry about it and just be consistent.

(cherry picked from commit 7e8c645280)
2024-03-07 15:01:15 -08:00
Mateusz Mandera 9856588b07 config_error: Change the look of the page.
Replaces the green theme with the old Zulip octopus with a plain, white
page.

(cherry picked from commit 2dff547e99)
2024-03-07 15:01:15 -08:00
Mateusz Mandera 836dc9f1c9 test_helpers: Remove /self-hosted-billing/ from exempt_patterns.
This was added early on and hasn't been removed since, even though we
have tests for the endpoint for a while now.

(cherry picked from commit a7f187bec3)
2024-03-07 15:01:15 -08:00
Mateusz Mandera b67e3d5ffa remote_billing: Make "plan management" always available.
Just shows a config error page if the bouncer is not enabled. Uses a new
endpoint for this so that it can work nicely for both browser and
desktop app clients.
It's necessary, because the desktop app expects to get a json response
with either an error or billing_access_url to redirect to. Showing a
nice config error page can't be done via the json error mechanism, so
instead we just serve a redirect to the new error page, which the app
will open in the browser in a new window or tab.

(cherry picked from commit f6d526f798)
2024-03-07 15:01:15 -08:00
Alex Vandiver 6a6e59f4d6 message: Merge unnecessary cache_transformer step.
Having a non-identity `cache_transformer` is no different from running
it on every row of the query_function.  Simplify understanding of the
codepath used in caching by merging the pieces of code.

(cherry picked from commit 547c8f895d)
2024-03-07 09:48:20 -08:00
Alex Vandiver 8b23b81f56 message_cache: Rename "to_dict" functions which deal with bytes.
(cherry picked from commit 548bb5362e)
2024-03-07 09:48:20 -08:00
Alex Vandiver 73116023a6 message_cache: Rename update_to_dict_cache to update_message_cache.
This better describes what it does.

(cherry picked from commit 96119e45b9)
2024-03-07 09:48:20 -08:00
Alex Vandiver e15a175a42 message: Split MessageDict and friends into its own file.
(cherry picked from commit 93a071a1f8)
2024-03-07 09:48:20 -08:00
Alex Vandiver 4c1b1ae2f6 message: Move render_markdown into zerver.lib.markdown.
(cherry picked from commit 11bde84580)
2024-03-07 09:48:20 -08:00
Alex Vandiver 3e2b295140 message_edit: Carry the QuerySet through as much as possible.
Rather than pass around a list of message objects in-memory, we
instead keep the same constructed QuerySet which includes the later
propagated messages (if any), and use that same query to pick out
affected Attachment objects, rather than limiting to the set of ids.
This is not necessarily a win -- the list of message-ids *may* be very
long, and thus the query may be more concise, easier to send to
PostgreSQL, and faster for PostgreSQL to parse.  However, the list of
ids is almost certainly better-indexed.

After processing the move, the QuerySet must be re-defined as a search
of ids (and possibly a very long list of such), since there is no
other way which is guaranteed to correctly single out the moved
messages.  At this point, it is mostly equivalent to the list of
Message objects, and certainly takes no less memory.

(cherry picked from commit eaf58438ec)
2024-03-07 09:48:20 -08:00
Alex Vandiver 210c9aaf1c topic: Use a single SQL statement to propagate message moves.
Rather than use `bulk_update()` to batch-move chunks of messages, use
a single SQL query to move the messages.  This is much more efficient
for large topic moves.  Since the `edit_history` field is not yet
JSON (see #26496) this requires that PostgreSQL cast the current data
into `jsonb`, append the new data (also cast to `jsonb`), and then
re-cast that as text.

For single-message moves, this _increases_ the SQL query count by one,
since we have to re-query for the updated data from the database after
the bulk update.  However, this is overall still a performance
improvement, which improves to 2x or 3x for larger topic moves.  Below
is a table of duration in seconds to run `do_update_message` to move a
topic to a new stream, based on messages in the topic, for before and
after this change:

| Topic size |  Before  |  After  |
| ---------- | -------- | ------- |
| 1          |   0.1036 |  0.0868 |
| 2          |   0.1108 |  0.0925 |
| 5          |   0.1139 |  0.0959 |
| 10         |   0.1218 |  0.0972 |
| 20         |   0.1310 |  0.1098 |
| 50         |   0.1759 |  0.1366 |
| 100        |   0.2307 |  0.1662 |
| 200        |   0.3880 |  0.2229 |
| 500        |   0.7676 |  0.4052 |
| 1000       |   1.3990 |  0.6848 |
| 2000       |   2.9706 |  1.3370 |
| 5000       |   7.5218 |  3.2882 |
| 10000      |  14.0272 |  5.4434 |

(cherry picked from commit a2657b843c)
2024-03-07 09:48:20 -08:00
Alex Vandiver 9a2a5b5910 message: Add a bulk_access_stream_messages_query method.
This applies access restrictions in SQL, so that individual messages
do not need to be walked one-by-one.  It only functions for stream
messages.

Use of this method significantly speeds up checks if we moved "all
visible messages" in a topic, since we no longer need to walk every
remaining message in the old topic to determine that at least one was
visible to the user.  Similarly, it significantly speeds up merging
into existing topics, since it no longer must walk every message in
the new topic to determine if the user could see at least one.

Finally, it unlocks the ability to bulk-update only messages the user
has access to, in a single query (see subsequent commit).

(cherry picked from commit 7dcc7540f9)
2024-03-07 09:48:20 -08:00
Alex Vandiver 9ac6ca1545 streams: Remove a lie from the docstring.
(cherry picked from commit c118f1874e)
2024-03-07 09:48:20 -08:00
Alex Vandiver a25ee8aad0 message_edit: len(changed_messages) is always > 0.
It is only ever added to, and starts as a one-element list.

(cherry picked from commit db3bcb8889)
2024-03-07 09:48:20 -08:00
Tim Abbott eeedc34e58 message_edit: Ensure gaining_usermessage_user_ids is set.
This makes this code path a bit easier to follow.

(cherry picked from commit 68be070306)
2024-03-07 09:48:20 -08:00
Tim Abbott e928fce232 message_edit: Clarify unmodified_user_messages logic.
(cherry picked from commit fc679b09a3)
2024-03-07 09:48:20 -08:00
Alex Vandiver fd1db72276 tests: Reserve "Internal" client, used by email gateway and topic moves.
(cherry picked from commit 13b9c87f93)
2024-03-07 09:48:20 -08:00
Alex Vandiver aee36b2b49 tests: Clear in-memory Client caches before testing query counts.
This makes counts more apples-to-apples comparable when run
back-to-back.

(cherry picked from commit a84de411a9)
2024-03-07 09:48:20 -08:00
Aman Agrawal c440958ecf personal_menu: Fix clicking on `Set status` not working.
Use `tabindex` instead of `href` to set focus on `a` tag.

Ideally, we should use `button` for these elements but since
we want to keep the pattern for these dropdowns same where some
`a` elements do have a valid `href`.

(cherry picked from commit 95e74558c5)
2024-03-01 09:26:25 -08:00
Aman Agrawal 0a97e41581 Revert "desktop_app: Close popover when clicking on "Plan management"."
This reverts commit 1f79e6294f.
No longer required after
70ba7cc042

(cherry picked from commit 569b2c8a60)
2024-03-01 09:26:25 -08:00
Aman Agrawal 2530566774 popover_menu: Fix popover not closed on clicking external links.
For example, gear menu was not closed after `Integrations` button
was clicked since we don't have an event handler which opens
`/integrations` in a new tab but we let the browser navigate user
to `/integrations` after clicking on `a href='/integrations'`.

There was no handler for hiding the popover after clicking on such
links, so this commit adds one.

(cherry picked from commit 70ba7cc042)
2024-03-01 09:26:25 -08:00
Lalit edeea66b0f ts: Migrate `popover_menus` to typescript.
This commit migrates `popover_menus` module to TypeScript. Also adds
a placeholder types declaration file for `css_variables` module.

(cherry picked from commit 02257b8cbf)
2024-03-01 09:26:25 -08:00
Tim Abbott 3e012889aa docs: Document restarts after changing zulip.conf.
(cherry picked from commit 7d9c82cbea)
2024-03-01 09:26:25 -08:00
David Rosa 32990b79e2 docs: Split /production/deployment.html into multiple pages.
- Makes "Deployment options" easier to navigate by splitting the
  "Reverse proxies" and "System configuration" sections out into
  dedicated pages.

Fixes #28928.

(cherry picked from commit 2b95068406)
2024-03-01 09:26:25 -08:00
Alya Abbott d65fa3bb47 docs: Move installation troubleshooting section to troubleshooting.md.
(cherry picked from commit 78aad5a4df)
2024-03-01 09:26:25 -08:00
Alya Abbott 9fa6f34770 docs: Move installer details to Deployment options page.
(cherry picked from commit 112f3d123a)
2024-03-01 09:26:25 -08:00
Alya Abbott de0688ff84 docs: Edit installation instructions for clarity.
(cherry picked from commit 7f51161761)
2024-03-01 09:26:25 -08:00