Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/zerver/lib
History
Alex Vandiver faeffa2466 queue_processors: Set a bounded prefetch size on rabbitmq queues. 
RabbitMQ clients have a setting called prefetch[1], which controls how
many un-acknowledged events the server forwards to the local queue in
the client.  The default is 0; this means that when clients first
connect, the server must send them every message in the queue.

This itself may cause unbounded memory usage in the client, but also
has other detrimental effects.  While the client is attempting to
process the head of the queue, it may be unable to read from the TCP
socket at the rate that the server is sending to it -- filling the TCP
buffers, and causing the server's writes to block.  If the server
blocks for more than 30 seconds, it times out the send, and closes the
connection with:

```
closing AMQP connection <0.30902.126> (127.0.0.1:53870 -> 127.0.0.1:5672):
{writer,send_failed,{error,timeout}}
```

This is https://github.com/pika/pika/issues/753#issuecomment-318119222.

Set a prefetch limit of 100 messages, or the batch size, to better
handle queues which start with large numbers of outstanding events.

Setting prefetch=1 causes significant performance degradation in the
no-op queue worker, to 30% of the prefetch=0 performance.  Setting
prefetch=100 achieves 90% of the prefetch=0 performance, and higher
values offer only minor gains above that.  For batch workers, their
performance is not notably degraded by prefetch equal to their batch
size, and they cannot function on smaller prefetches than their batch
size.

We also set a 100-count prefetch on Tornado workers, as they are
potentially susceptible to the same effect.

[1] https://www.rabbitmq.com/confirms.html#channel-qos-prefetch
 		2021-11-16 11:48:50 -08:00
..
markdown markdown: CSS-escape preview links. 2021-10-26 18:17:23 -07:00
url_preview url_preview: Interpret og:image relative to full page URL. 2021-10-21 12:20:37 -07:00
webhooks links: Replace all references to Django docs to link to /3.2/ version. 2021-11-08 18:06:16 -08:00
__init__.py
…
actions.py invites: Update error message when max limit for the day is reached. 2021-11-01 16:36:26 -07:00
addressee.py python: Convert deprecated Django ugettext alias to gettext. 2021-04-15 18:01:34 -07:00
alert_words.py python: Reformat with Black, except quotes. 2021-02-12 13:11:19 -08:00
attachments.py python: Fix mypy no_implicit_reexport errors. 2021-07-16 14:02:31 -07:00
avatar.py avatar: Remove ?x=x kludge. 2021-10-14 12:47:43 -07:00
avatar_hash.py typing: Improve typing with assertions. 2021-08-20 05:54:19 -07:00
bot_config.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
bot_lib.py python: Remove locally dead code. 2021-08-19 01:51:37 -07:00
bot_storage.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
bulk_create.py typing: Fix argument type for models in function signatures. 2021-08-20 05:54:19 -07:00
cache.py python: Remove default "utf8" argument for encode(), decode(). 2021-08-02 15:53:52 -07:00
cache_helpers.py typing: Use accurate type hints for dictionaries. 2021-08-20 06:02:28 -07:00
camo.py python: Convert "".format to Python 3.6 f-strings. 2021-08-02 15:53:52 -07:00
ccache.py python: Remove default "utf8" argument for encode(), decode(). 2021-08-02 15:53:52 -07:00
compatibility.py compatibility: Shift functions to new module. 2021-06-05 07:33:25 -07:00
context_managers.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
create_user.py settings: Remove realm-level default of enable_login_emails setting. 2021-09-28 12:41:09 -07:00
data_types.py python: Reformat with Black, except quotes. 2021-02-12 13:11:19 -08:00
db.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
debug.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
dev_ldap_directory.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
digest.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
display_recipient.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
domains.py python: Convert deprecated Django ugettext alias to gettext. 2021-04-15 18:01:34 -07:00
drafts.py drafts: Send events to clients when drafts change. 2021-07-30 10:41:52 -07:00
email_mirror.py rate_limit: Pass seconds to freedom to all RateLimited exceptions. 2021-11-04 20:34:39 -07:00
email_mirror_helpers.py email_mirror: Move ZulipEmailForwardUserError into email_mirror_helpers. 2021-08-31 16:37:54 -07:00
email_notifications.py user_groups: Rename get_user_group_members to get_user_group_direct_members. 2021-10-12 13:57:26 -07:00
email_validation.py python: Convert deprecated Django ugettext alias to gettext. 2021-04-15 18:01:34 -07:00
emoji.py emoji: Add backend changes to support still image for animated emojis. 2021-09-12 07:13:04 +00:00
error_notify.py bots: Assume STAFF_SUBDOMAIN to be the realm where ERROR_BOT lives. 2021-07-27 15:51:59 -07:00
event_schema.py events: Fix updating user email on changing email_address_visibility. 2021-10-04 17:24:31 -07:00
events.py lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
exceptions.py invites: Update error message when max limit for the day is reached. 2021-11-01 16:36:26 -07:00
export.py export: Remove `--delete-after-upload` flag in realm export. 2021-11-03 11:14:02 -07:00
external_accounts.py docs: Fix capitalization mistakes. 2021-05-10 09:57:26 -07:00
fix_unreads.py models: Use default db table name for UserTopic. 2021-09-17 12:14:28 -07:00
generate_test_data.py populate_db: Generate resolved topics for testing. 2021-08-03 16:42:09 -07:00
github.py fetch-contributor-data: Use builtin backoff. 2021-09-01 05:34:13 -07:00
home.py realm: Rename plan type constants to be more descriptive. 2021-10-19 12:20:39 -07:00
hotspots.py docs: Fix spelling errors caught by codespell. 2021-04-26 09:31:08 -07:00
html_diff.py html_diff: Handle empty differences between empty strings. 2021-10-18 18:27:40 -07:00
html_to_text.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
i18n.py request: Refactor ZulipRequestNotes to RequestNotes. 2021-09-03 08:48:45 -07:00
import_realm.py realm: Rename plan type constants to be more descriptive. 2021-10-19 12:20:39 -07:00
initial_password.py python: Remove default "utf8" argument for encode(), decode(). 2021-08-02 15:53:52 -07:00
integrations.py integrations: Add SonarQube webhook integration. 2021-08-05 09:53:42 -07:00
logging_util.py request: Refactor ZulipRequestNotes to RequestNotes. 2021-09-03 08:48:45 -07:00
management.py management: Fix typing for management scripts. 2021-08-20 05:54:18 -07:00
mdiff.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
mention.py models: Add related_name to UserGroup and UserGroupMembership fields. 2021-10-13 10:08:06 -07:00
message.py message: Make zero invalid value for message_content_delete_limit_seconds. 2021-09-30 14:45:39 -07:00
migrate.py typing: Replace CursorObj by CursorWrapper. 2021-08-20 05:54:19 -07:00
mobile_auth_otp.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
name_restrictions.py auth: Ensure a realm can't be created on SOCIAL_AUTH_SUBDOMAIN. 2021-09-10 10:47:15 -07:00
narrow.py spectators: Rename is_web_public_compatible functions. 2021-09-06 09:00:16 -07:00
notes.py request: Extract generic BaseNotes from RequestNotes. 2021-09-03 08:47:51 -07:00
notification_data.py notifications: Fix unnecessary wildcard mention notifications. 2021-08-13 09:48:18 -07:00
onboarding.py onboarding: Update Welcome Bot. 2021-10-15 12:59:20 -07:00
outgoing_http.py outgoing_http: Give an easy way to configure retries. 2021-09-01 05:34:13 -07:00
outgoing_webhook.py outgoing webhooks: Fix inconsistencies with Slack's API. 2021-09-23 11:19:20 -07:00
presence.py typing: Fix function signatures. 2021-08-20 05:54:19 -07:00
profile.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
push_notifications.py push_notifications: Truncate overly large remove events. 2021-11-02 21:24:14 -07:00
pysa.py
…
queue.py queue_processors: Set a bounded prefetch size on rabbitmq queues. 2021-11-16 11:48:50 -08:00
rate_limiter.py request: Refactor ZulipRequestNotes to RequestNotes. 2021-09-03 08:48:45 -07:00
realm_description.py markdown: Refactor out additional properties added to Message. 2021-06-24 18:14:53 -07:00
realm_icon.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
realm_logo.py realm: Rename plan type constants to be more descriptive. 2021-10-19 12:20:39 -07:00
redis_utils.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
remote_server.py push_notifs: Log the number of devices notification was sent to. 2021-09-29 15:50:06 -07:00
request.py request: Refactor ZulipRequestNotes to RequestNotes. 2021-09-03 08:48:45 -07:00
response.py json_error: Completely remove json_error. 2021-07-06 15:34:33 -07:00
rest.py users: Allow spectators to view user avatars. 2021-11-02 11:26:19 -07:00
retention.py typing: Fix argument type for models in function signatures. 2021-08-20 05:54:19 -07:00
scim.py lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
scim_filter.py auth: Add support for using SCIM for account management. 2021-10-14 12:29:10 -07:00
send_email.py send_email: Log IP address sources (and more realms) of send_email. 2021-08-31 16:36:42 -07:00
server_initialization.py realm: Create RealmUserDefault object for internal realm. 2021-10-29 16:19:55 -07:00
sessions.py sessions: Use SessionBase instead of Session. 2021-07-24 09:54:21 -07:00
soft_deactivation.py soft_deactivate: Handle multiple SUBSCRIPTION_DEACTIVATEDs. 2021-11-10 12:19:25 -08:00
sqlalchemy_utils.py python: Upgrade SQLAlchemy from 1.3.24 to 1.4.23. 2021-08-31 06:47:39 -07:00
storage.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
stream_subscription.py bulk_access_messages: Bulk fetch Subscription details. 2021-05-12 16:23:22 -07:00
stream_topic.py user_topic: Filter on visibility_policy in muted_topic code. 2021-09-17 12:14:28 -07:00
streams.py streams: Restrict creating web public streams based on new setting. 2021-10-05 09:56:00 -07:00
subdomains.py subdomains: Extend "static" to include resources hosted on S3. 2021-06-08 15:28:10 -07:00
templates.py api docs: Replace most markdown files with a common template. 2021-06-24 10:42:08 -07:00
test_classes.py realm: Rename plan type constants to be more descriptive. 2021-10-19 12:20:39 -07:00
test_console_output.py python: Remove locally dead code. 2021-08-19 01:51:37 -07:00
test_data.source.txt Rename default branch to ‘main’. 2021-09-06 12:56:35 -07:00
test_fixtures.py puppeteer_tests: Reset test environment after each run. 2021-03-25 12:58:36 -07:00
test_helpers.py tests: Move simulated_queue_client to the only test it is used in. 2021-11-16 11:48:50 -08:00
test_runner.py typing: Fix function signatures. 2021-08-20 05:54:19 -07:00
tex.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
thumbnail.py thumbnail: Remove unused is_camo_url parameter. 2021-08-19 01:51:37 -07:00
timeout.py timeout: Remove unnecessary varargs support. 2021-02-15 17:05:28 -08:00
timestamp.py python: Reformat with Black, except quotes. 2021-02-12 13:11:19 -08:00
timezone.py python: Reformat with Black, except quotes. 2021-02-12 13:11:19 -08:00
topic.py python: Upgrade SQLAlchemy from 1.3.24 to 1.4.23. 2021-08-31 06:47:39 -07:00
topic_mutes.py user_topic: Filter on visibility_policy in muted_topic code. 2021-09-17 12:14:28 -07:00
transfer.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
type_debug.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00
types.py saml: Implement IdP-initated logout for Keycloak. 2021-10-27 13:13:55 -07:00
unminify.py unminify: Fix lookup if source map does not exist in disk. 2021-03-16 14:46:18 -07:00
upload.py python: Remove re.UNICODE flag (redundant in Python 3). 2021-10-22 13:42:29 -07:00
url_encoding.py url_encoding: Use proper parsing for query string appending. 2021-10-14 12:47:43 -07:00
user_agent.py python: Reformat with Black, except quotes. 2021-02-12 13:11:19 -08:00
user_groups.py user_groups: Add a recursive group membership model. 2021-10-13 10:08:06 -07:00
user_mutes.py typing: Use assertions for function arguments. 2021-07-26 14:48:45 -07:00
user_status.py status: Extract format_user_status helper. 2021-09-02 16:50:25 -07:00
users.py types: Fix declared type of custom profile field values. 2021-09-22 16:22:46 -07:00
utils.py utils: Remove unused split_by function. 2021-08-19 01:51:37 -07:00
validator.py Revert "validator: Add generic check_or." 2021-09-28 09:28:56 -07:00
widget.py poll widget: Add server validation. 2021-06-14 17:46:16 -07:00
zcommand.py composebox_typeahead: Remove redundant options from the `/` typeahead. 2021-10-21 10:50:27 -07:00
zephyr.py python: Normalize quotes with Black. 2021-02-12 13:11:19 -08:00