Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/zerver/lib
History
Alex Vandiver 07c4291749 message: Rewrite personals query to be more performant and accurate. 
The previous query suffered from bad corner cases when the user had
received a large number of direct messages but sent very few,
comparatively.  This mean that the first half of the UNION would
retrieve a very large number of UserMessage rows, requiring fetching a
large number of Message rows, merely to throw them away upon
determining that the recipient was the current user.

Instead of merging two queries of "last 1k received" + "last 1k sent",
we instead make better use of the UserMessage rows to find "last 1k
sent or received."  This may change the list of recipients, as large
disparities in sent/received messages may result in pushing the
most-recently-sent users off of the list.  These are likely uncommon
edge cases, however -- and the disparity is the whole reason for the
performance problem.

This also provides more correct answers.  In the case where a user's
1001'th message sent was to person A today, but my most recent message
received was from them yesterday, the previous plan would show the
message I received yesterday message-id as the max, and not the more
recent message I sent today.

While we could theoretically raise the `RECENT_CONVERSATIONS_LIMIT` to
more frequently match the same recipient list as previously, this
increases the cost of the most common cases unreasonably.  With a
1000-message limit, the common cases are slightly faster, and the tail
latencies are very much improved; raising `RECENT_CONVERSATIONS_LIMIT`
would increase the result similarity to the old algorithm, at the cost
of the p50 and p75.

|        |   Old   |   New   |
| ------ | ------- | ------- |
| Mean   | 0.05287 | 0.02520 |
| p50    | 0.00695 | 0.00556 |
| p75    | 0.05592 | 0.03351 |
| p90    | 0.14645 | 0.08026 |
| p95    | 0.20181 | 0.10906 |
| p99    | 0.30691 | 0.16014 |
| p99.9  | 0.57894 | 0.19521 |
| max    | 22.0610 | 0.22184 |

On the whole, however, the much more bounded worst case are worth the
small changes to the resultset.
 		2024-01-18 09:30:20 -08:00
..
markdown markdown: Prevent OverflowError with large time integers. 2024-01-05 12:01:06 -08:00
upload s3: Add a setting for S3 addressing style. 2024-01-05 11:12:18 -08:00
url_preview python: Use urlsplit instead of urlparse. 2023-12-05 13:03:07 -08:00
webhooks mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
__init__.py
…
addressee.py actions: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
alert_words.py models: Extract zerver.models.alert_words. 2023-12-16 22:08:44 -08:00
async_utils.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
attachments.py models: Move some functions to zerver.lib.attachments. 2023-12-16 22:08:44 -08:00
avatar.py python: Consistently use from…import for urllib.parse. 2023-12-05 13:03:07 -08:00
avatar_hash.py utils: Remove make_safe_digest wrapper. 2023-07-19 10:54:05 -07:00
bot_config.py ruff: Fix UP032 Use f-string instead of `format` call. 2023-08-02 15:58:55 -07:00
bot_lib.py models: Extract zerver.models.users. 2023-12-16 22:08:44 -08:00
bot_storage.py
…
bulk_create.py models: Extract zerver.models.groups. 2023-12-16 22:08:44 -08:00
cache.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
cache_helpers.py models: Extract zerver.models.clients. 2023-12-16 22:08:44 -08:00
camo.py
…
ccache.py python: Delete superfluous parens. 2023-09-13 13:40:19 -07:00
compatibility.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
context_managers.py
…
create_user.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
data_types.py python: Elide unnecessary list wrappers. 2023-09-13 12:41:23 -07:00
db.py test_helpers: Fix logging in cursor_executemany mock. 2023-11-15 15:27:54 -08:00
debug.py
…
default_streams.py streams: Pass stream_weekly_traffic field in stream objects. 2023-08-06 18:06:42 -07:00
dev_ldap_directory.py dev_ldap_directory: Use f-strings for better readability. 2024-01-09 12:09:09 -08:00
digest.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
display_recipient.py models: Extract zerver.models.recipients. 2023-12-16 22:08:44 -08:00
domains.py
…
drafts.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
email_mirror.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
email_mirror_helpers.py
…
email_notifications.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
email_validation.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
emoji.py models: Extract zerver.models.realm_emoji. 2023-12-16 22:08:44 -08:00
emoji_utils.py emoji: Match emoji sequences in markdown. 2023-08-23 16:18:15 -07:00
event_schema.py events: Add 'onboarding_steps' event deprecating 'hotspots'. 2023-12-06 18:19:20 -08:00
events.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
exceptions.py remote_billing_page: Show error page for registration mismatch. 2023-12-10 19:33:48 -08:00
export.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
external_accounts.py ruff: Fix SIM118 Use `k not in d` instead of `k not in d.keys()`. 2023-07-24 10:39:28 -07:00
fix_unreads.py ruff: Fix PERF401 Use a list comprehension to create a transformed list. 2023-08-07 17:23:55 -07:00
generate_test_data.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
github.py
…
home.py views: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
hotspots.py banner: Make banner about automatic follow/unmute topics one-time only. 2023-12-06 18:19:20 -08:00
html_diff.py
…
html_to_text.py widgets: Rename confusing attribute name in `tabbed_sections.py`. 2023-08-31 11:55:28 -07:00
i18n.py i18n: Fix default language for users created via API/LDAP. 2023-10-01 21:10:13 +02:00
import_realm.py models: Extract zerver.models.recipients. 2023-12-16 22:08:44 -08:00
initial_password.py
…
integrations.py integrations: Extract integration event types returning function. 2023-08-30 15:54:13 -07:00
logging_util.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
management.py models: Extract zerver.models.clients. 2023-12-16 22:08:44 -08:00
mdiff.py
…
mention.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
message.py message: Rewrite personals query to be more performant and accurate. 2024-01-18 09:30:20 -08:00
migrate.py migration: Make 'rename_indexes_constraints' a lib function. 2023-12-06 18:19:20 -08:00
mobile_auth_otp.py
…
muted_users.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
name_restrictions.py requests: Add SELF_HOSTING_MANAGEMENT_SUBDOMAIN. 2023-11-22 14:22:26 -08:00
narrow.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
narrow_helpers.py narrow: Split out narrow_helpers. 2023-06-30 11:26:23 -07:00
notes.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
notification_data.py models: Extract zerver.models.scheduled_jobs. 2023-12-16 22:08:44 -08:00
onboarding.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
outgoing_http.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
outgoing_webhook.py models: Extract zerver.models.bots. 2023-12-16 22:08:44 -08:00
per_request_cache.py per-request caches: Add per_request_cache library. 2023-08-11 11:09:34 -07:00
presence.py models: Move query_for_ids to zerver.lib.query_helpers. 2023-12-16 22:08:44 -08:00
profile.py
…
push_notifications.py zilencer: Have server send realm_uuid to remaining bouncer endpoints. 2024-01-05 13:09:09 -08:00
pysa.py
…
query_helpers.py models: Move query_for_ids to zerver.lib.query_helpers. 2023-12-16 22:08:44 -08:00
queue.py queue: Only NAK the events if the channel is still open. 2023-12-12 09:20:29 -08:00
rate_limiter.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
realm_description.py
…
realm_icon.py
…
realm_logo.py
…
recipient_parsing.py request: Extract out methods from 'scheduled_messages' to reuse. 2023-10-10 17:15:28 -07:00
recipient_users.py models: Extract zerver.models.recipients. 2023-12-16 22:08:44 -08:00
redis_utils.py
…
remote_server.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
request.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
response.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
rest.py
…
retention.py models: Always search Messages with a realm_id or id limit. 2023-09-11 15:00:37 -07:00
safe_session_cached_db.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
scheduled_messages.py models: Extract zerver.models.scheduled_jobs. 2023-12-16 22:08:44 -08:00
scim.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
scim_filter.py
…
send_email.py send_email: Distinct emails means distinct, case-insensitively. 2024-01-04 10:46:53 -08:00
server_initialization.py models: Extract zerver.models.clients. 2023-12-16 22:08:44 -08:00
sessions.py models: Extract zerver.models.users. 2023-12-16 22:08:44 -08:00
singleton_bmemcached.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
soft_deactivation.py models: Extract zerver.models.scheduled_jobs. 2023-12-16 22:08:44 -08:00
sounds.py
…
sqlalchemy_utils.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
storage.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
stream_color.py
…
stream_subscription.py streams: Send user creation events on adding subscribers. 2023-11-21 23:58:45 -08:00
stream_topic.py
…
stream_traffic.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
streams.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
string_validation.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
subdomains.py python: Consistently use from…import for urllib.parse. 2023-12-05 13:03:07 -08:00
subscription_info.py models: Extract zerver.models.streams. 2023-12-16 22:08:44 -08:00
templates.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
test_classes.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
test_console_output.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
test_data.source.txt
…
test_fixtures.py
…
test_helpers.py auth: Add a configurable wrapper around authenticate calls. 2024-01-15 12:18:48 -08:00
test_runner.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
tex.py push_notification: Fix bad rendering of math formulas. 2023-11-26 23:30:24 -08:00
thumbnail.py
…
timeout.py mypy: Enable new error explicit-override. 2023-10-12 12:28:41 -07:00
timestamp.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
timezone.py
…
topic.py topic: Add comments calling out case-sensitive index usage. 2023-09-27 10:22:42 -07:00
transfer.py uploads: Allow uploads to set storage class. 2023-07-19 16:19:34 -07:00
typed_endpoint.py remote_realm: Add syncing of org_type. 2023-11-28 14:41:16 -08:00
types.py python: Consistently use from…import for datetime. 2023-12-05 12:01:18 -08:00
url_encoding.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
url_redirects.py help: Rename view-and-browse-images.md -> view-images-and-videos.md 2023-11-23 10:37:31 -08:00
user_agent.py
…
user_counts.py python: Elide unnecessary list wrappers. 2023-09-13 12:41:23 -07:00
user_groups.py models: Extract zerver.models.groups. 2023-12-16 22:08:44 -08:00
user_message.py
…
user_status.py users: Update presence and user status code to support restricted users. 2023-11-21 23:58:45 -08:00
user_topics.py lib: Rename *topic local variables to *topic_name. 2024-01-15 09:40:43 -08:00
users.py models: Extract zerver.models.realms. 2023-12-16 22:08:44 -08:00
utils.py utils: Remove make_safe_digest wrapper. 2023-07-19 10:54:05 -07:00
validator.py realm: Enfore length restriction on jitsi_server_url at API level. 2023-12-14 12:11:59 -08:00
widget.py widgets: Fix bug where a new line right after /todo broke rendering. 2023-09-08 15:39:07 -07:00
zcommand.py ruff: Fix UP032 Use f-string instead of `format` call. 2023-08-02 15:58:55 -07:00
zephyr.py
…