This commit adds backend code to set email_address_visibility when
registering a new user. The realm-level default and the value of
source profile gets overridden by the value user selected during
signup.
This lets us simplify the long-ish ‘../../static/js’ paths, and will
remove the need for the ‘zrequire’ wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).
Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules. This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack. It also shrinks the release tarball by 3%.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is quite a bit faster:
```
%timeit calendar.timegm(now.timetuple())
2.91 µs ± 361 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
%timeit int(now.timestamp())
539 ns ± 27 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```
This is particularly important for the presence endpoint, which is a
tight loop of serializing datetimes.
As written, the QOS parameters are (re)set every time ensure_queue is
called, which is every time a message is enqueued. This is wasteful --
particularly QOS parameters only apply for consumers, and setting them
takes a RTT to the server.
Switch to only setting the QOS once, when a connection
is (re)established. In profiling, this reduces the time to call
`queue_json_publish("noop", {})` from 878µs to 150µs.
We add stream_permission_group_settings object which is
similar to property_types framework used for realm settings.
This commit also adds GroupPermissionSetting dataclass for
defining settings inside stream_permission_group_settings.
We add "do_change_stream_group_based_setting" function which
is called in loop to update all the group-based stream settings
and it is now used to update 'can_remove_subscribers_group'
setting instead of "do_change_can_remove_subscribers_group".
We also change the variable name for event_type field of
RealmAuditLog objects to STREAM_GROUP_BASED_SETTING_CHANGED
since this will be used for all group-based stream settings.
'property' field is also added to extra_data field to identify
the setting for which RealmAuditLog object was created.
We will add a migration in further commits which will add the
property field to existing RealmAuditLog objects created for
changing can_remove_subscribers_group setting.
This reverts commit 851d68e0fc.
That commit widened how long the transaction is open, which made it
much more likely that after the user was created in the transaction,
and the memcached caches were flushed, some other request will fill
the `get_realm_user_dicts` cache with data which did not include the
new user (because it had not been committed yet).
If a user creation request lost this race, the user would, upon first
request to `/`, get a blank page and a Javascript error:
Unknown user_id in get_by_user_id: 12345
...where 12345 was their own user-id. This error would persist until
the cache expired (in 7 days) or something else expunged it.
Reverting this does not prevent the race, as the post_save hook's call
to flush_user_profile is still in a transaction (and has been since
168f241ff0), and thus leaves the potential race window open.
However, it much shortens the potential window of opportunity, and is
a reasonable short-term stopgap.
This will allow us to re-use this logic later, when we add support for
re-checking notification settings just before sending email/push
notifications to the user.
Also, since this is essentially part of the notifiability logic,
this better belongs to `notification_data.py` and this change will
hopefully reduce the reading complexity of the message-send codepath.
This commits update the code to use user-level email_address_visibility
setting instead of realm-level to set or update the value of UserProfile.email
field and to send the emails to clients.
Major changes are -
- UserProfile.email field is set while creating the user according to
RealmUserDefault.email_address_visbility.
- UserProfile.email field is updated according to change in the setting.
- 'email_address_visibility' is added to person objects in user add event
and in avatar change event.
- client_gravatar can be different for different users when computing
avatar_url for messages and user objects since email available to clients
is dependent on user-level setting.
- For bots, email_address_visibility is set to EVERYONE while creating
them irrespective of realm-default value.
- Test changes are basically setting user-level setting instead of realm
setting and modifying the checks accordingly.
Previously, user objects contained delivery_email field
only when user had access to real email. Also, delivery_email
was not present if visibility setting is set to "everyone"
as email field was itself set to real email.
This commit changes the code to pass "delivery_email" field
always in the user objects with its value being "None" if
user does not have access to real email and real email otherwise.
The "delivery_email" field value is None for logged-out users.
For bots, the "delivery_email" is always set to real email
irrespective of email_address_visibility setting.
Also, since user has access to real email if visibility is set
to "everyone", "delivery_email" field is passed in that case
too.
There is no change in email field and it is same as before.
This commit also adds code to send event to update delivery_email
field when email_address_visibility setting changes to all the
users whose access to emails changes and also changes the code to
send event on changing delivery_email to users who have access
to email.
This commit renames parse_message_content_edit_or_delete_limit
to parse_message_time_limit_setting and also renames
MESSAGE_CONTENT_EDIT_OR_DELETE_LIMIT_SPECIAL_VALUES_MAP to
MESSAGE_TIME_LIMIT_SETTING_SPECIAL_VALUES_MAP.
We do this change since this function and object will also be
used for message move limit and it makes sense to have a more
generic name.
In Zulip, message topics are case-insensitive but case-preserving.
The `get_context_for_message` function erroneously did a
case-sensitive search, and thus only messages whose topic matched
exactly were pulled in as context.
Make the missed-message pipeline aware that message topics are not
case-sensitive. This means that, when collapsing adjacent messages,
we merge messages with topic headers which are "different"; create a
separate explicit "grouping" to know which to collapse.
The Content-Type of user-provided uploads was provided by the browser
at initial upload time, and stored in S3; however, 04cf68b45e
switched to determining the Content-Disposition merely from the
filename. This makes uploads vulnerable to a stored XSS, wherein a
file uploaded with a content-type of `text/html` and an extension of
`.png` would be served to browsers as `Content-Disposition: inline`,
which is unsafe.
The `Content-Security-Policy` headers in the previous commit mitigate
this, but only for browsers which support them.
Revert parts of 04cf68b45e, specifically by allowing S3 to provide
the Content-Disposition header, and using the
`ResponseContentDisposition` argument when necessary to override it to
`attachment`. Because we expect S3 responses to vary based on this
argument, we include it in the cache key; since the query parameter
has dashes in it, we can't use use the helper `$arg_` variables, and
must parse it from the query parameters manually.
Adding the disposition may decrease the cache hit rate somewhat, but
downloads are infrequent enough that it is unlikely to have a
noticeable effect. We take care to not adjust the cache key for
requests which do not specify the disposition.
Fixes the documentation generated from the Markdown macros
{settings_tab|your-bots} and {settings_tab|bot-list-admin} to
match the text labels in the Zulip UI and improves the text of
relative links to explicitly say if we are referring to the Bots
tab of the Personal or Organization settings menu.
Follow-up to #23256.
This code needs to be more flexible to improve the documentation
of items in the Personal and Organization settings menu when
using the `{settings_tab|[setting-name]}` Markdownm macro that
provides relative links or step-by-step instructions.
This commit moves the Markdown formatting code to a new function that
receives tuples from `link_mapping` as input. This is a preliminary
step to offer more flexibility than the current approach.
- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.
- Clean up the language.
- Add a prominent "Go to organization" button.
- Link to guides for new users and admins.
- Fix duplication bug in text email version.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Removes `base_path` argument when making the markdown extension for
parameters in documentation for API endpoints.
This seems to have been originally included for API parameters that
were documented in JSON files, which is no longer in use. Now all
API endpoints in the documentation are documented in
`zerver/openapi/zulip.yaml`.
Removes `base_path` argument when making the markdown extension for
return values in documentation for API endpoints.
This seems to have been a copy and paste error in commit d2ee99a2fd
when `zerver/lib/markdown/api_return_values_generator.py` was created.
Until now, custom emojis with "periods" in their name were allowed, even though
they don't really fit the pattern of how we name them, and in fact the Markdown
processor would not render such custom emoji. Fix this by just disallowing the
character.
Also update the error strings accordingly.
Note that this does not include a migration to eliminate any existing custom emoji with this
character in their name.
Fixes#24066.
We change the do_create_user function to use transaction.atomic
decorator instead of using with block. Due to this change, all
send_event calls are made inside transaction.on_commit.
Some other changes -
- Remove transaction.atomic decorator from send_inital_realm_messages
since it is now called inside a transaction.
- Made changes in tests which tests message events and notifications
to make sure on_commit callbacks are executed.
These files are not Jinja2 templates, so there's no reason that they needed
to be inside `templates/zerver`. Moving them to the top level reflects their
importance and also makes it feel nicer to work on editing the help center content,
without it being unnecessary buried deep in the codebase.
The content of a message is truncated to `MAX_MESSAGE_LENGTH`, which
is 1000 characters. Since the email gateway places attachments at the
very end of the extracted body, that means that they are the first
thing to get truncated off.
That is, if an incoming email message contains 1000 `a`s and an image
attachment, the link that attaches the attachment to the message will
get truncated off, leaving it dangling in the database.
Truncate the message body content separately from the attachment links
which are included at the end of the body.
Since we want to use `accounts/new/send_confirm` to know how many
users actually register after visiting the register page, we
added it to Google Tag Manager, but GTM tracks every user
registration separately due <email> in the URL
making it harder to track.
To solve this, we want to pass <email> as a GET parameter which
can be easily filtered inside GTM using a RegEx and all the
registrations can be tracked as one.
A missed message email notification, where the message is the welcome
message sent by the welcome bot on account creation, get sent when
the user somehow not focuses the browser tab during account creation.
No missed message email or push notifications should be sent for the
messages generated by the welcome bot.
'internal_send_private_message' accepts a parameter
'disable_external_notifications' and is set to 'True' when the sender
is 'welcome bot'.
A check is introduced in `trivially_should_not_notify`, not to notify
if `disable_external_notifications` is true.
TestCases are updated to include the `disable_external_notifications`
check in the early (False) return patterns of `is_push_notifiable` and
`is_email_notifiable`.
One query reduced for both `test_create_user_with_multiple_streams`
and `test_register`.
Reason: When welcome bot sends message after user creation
`do_send_messages` calls `get_active_presence_idle_user_ids`,
`user_ids` in `get_active_presence_idle_user_ids` remains empty if
`disable_external_notifications` is true because `is_notifiable` returns
false.
`get_active_presence_idle_user_ids` calls `filter_presence_idle_user_ids`
and since the `user_ids` is empty, the query inside the function doesn't
get executed.
MissedMessageHookTest updated.
Fixes: #22884
A separate function named `trivially_should_not_notify` is added which
extracts the common checks from `get_push_notification_trigger` and
`get_email_notification_trigger` which are users' notification settings
independent and thus don't depend on what type of notification (email/push)
it is.
Documents link to the bot's user card from the bot's name in
Organization settings > Bots, and information in the bot's user card.
Fixes part of #23970.
When the email mirror gateway is sending messages "as" a user (as
triggered by having access to the missed-message email address),
attachments were still created as the Email Gateway bot. Since the
sender (the end-user) was not the owner of those attachments (the
gateway bot), nor were they referenced yet anywhere, this resulted in
the attachments being "orphaned" and not allowed to be accessed by
anyone -- despite the attachment links being embedded in the message.
This was accompanied by the error:
```
WARN [] User 12345 tried to share upload 123/3LkSA4OcoG6OpAknS2I0SFAQ/example.jpf in message 123456, but lacks permission
INFO [zerver.lib.email_mirror] Successfully processed email from user 12345 to example-stream
```
We solve this by creating attachment objects as the users the message
will be sent from.
The max inline preview limit was previously increased to 10 by #20789.
However, as issue #23624 shows, it's still causing confusion for users
when they include more than 10 links.
Bump this limit up to 24, which is a multiple of the 4 image preview
per line logic.
Accessing .realm will cause a fetch query from the database if the
attribute hasn't been fetched already earlier in the codepath. That's
completely redundant if we're just comparing realms, and we should only
access .realm_id attribute. This seems to eliminate a query in some
codepaths, which is nice in this performance-sensitive function.
When file uploads are stored in S3, this means that Zulip serves as a
302 to S3. Because browsers do not cache redirects, this means that
no image contents can be cached -- and upon every page load or reload,
every recently-posted image must be re-fetched. This incurs extra
load on the Zulip server, as well as potentially excessive bandwidth
usage from S3, and on the client's connection.
Switch to fetching the content from S3 in nginx, and serving the
content from nginx. These have `Cache-control: private, immutable`
headers set on the response, allowing browsers to cache them locally.
Because nginx fetching from S3 can be slow, and requests for uploads
will generally be bunched around when a message containing them are
first posted, we instruct nginx to cache the contents locally. This
is safe because uploaded file contents are immutable; access control
is still mediated by Django. The nginx cache key is the URL without
query parameters, as those parameters include a time-limited signed
authentication parameter which lets nginx fetch the non-public file.
This adds a number of nginx-level configuration parameters to control
the caching which nginx performs, including the amount of in-memory
index for he cache, the maximum storage of the cache on disk, and how
long data is retained in the cache. The currently-chosen figures are
reasonable for small to medium deployments.
The most notable effect of this change is in allowing browsers to
cache uploaded image content; however, while there will be many fewer
requests, it also has an improvement on request latency. The
following tests were done with a non-AWS client in SFO, a server and
S3 storage in us-east-1, and with 100 requests after 10 requests of
warm-up (to fill the nginx cache). The mean and standard deviation
are shown.
| | Redirect to S3 | Caching proxy, hot | Caching proxy, cold |
| ----------------- | ------------------- | ------------------- | ------------------- |
| Time in Django | 263.0 ms ± 28.3 ms | 258.0 ms ± 12.3 ms | 258.0 ms ± 12.3 ms |
| Small file (842b) | 586.1 ms ± 21.1 ms | 266.1 ms ± 67.4 ms | 288.6 ms ± 17.7 ms |
| Large file (660k) | 959.6 ms ± 137.9 ms | 609.5 ms ± 13.0 ms | 648.1 ms ± 43.2 ms |
The hot-cache performance is faster for both large and small files,
since it saves the client the time having to make a second request to
a separate host. This performance improvement remains at least 100ms
even if the client is on the same coast as the server.
Cold nginx caches are only slightly slower than hot caches, because
VPC access to S3 endpoints is extremely fast (assuming it is in the
same region as the host), and nginx can pool connections to S3 and
reuse them.
However, all of the 648ms taken to serve a cold-cache large file is
occupied in nginx, as opposed to the only 263ms which was spent in
nginx when using redirects to S3. This means that to overall spend
less time responding to uploaded-file requests in nginx, clients will
need to find files in their local cache, and skip making an
uploaded-file request, at least 60% of the time. Modeling shows a
reduction in the number of client requests by about 70% - 80%.
The `Content-Disposition` header logic can now also be entirely shared
with the local-file codepath, as can the `url_only` path used by
mobile clients. While we could provide the direct-to-S3 temporary
signed URL to mobile clients, we choose to provide the
served-from-Zulip signed URL, to better control caching headers on it,
and greater consistency. In doing so, we adjust the salt used for the
URL; since these URLs are only valid for 60s, the effect of this salt
change is minimal.
Moving `/user_avatars/` to being served partially through Django
removes the need for the `no_serve_uploads` nginx reconfiguring when
switching between S3 and local backends. This is important because a
subsequent commit will move S3 attachments to being served through
nginx, which would make `no_serve_uploads` entirely nonsensical of a
name.
Serve the files through Django, with an offload for the actual image
response to an internal nginx route. In development, serve the files
directly in Django.
We do _not_ mark the contents as immutable for caching purposes, since
the path for avatar images is hashed only by their user-id and a salt,
and as such are reused when a user's avatar is updated.
Importing `upload_backend` directly means that in testing it must also
be mocked where it is imported, in order to correctly test the right
backend. Since `get_avatar_url` is part of the public
`ZulipUploadBackend` API, add another helper method to call that.
The `django-sendfile2` module unfortunately only supports a single
`SENDFILE` root path -- an invariant which subsequent commits need to
break. Especially as Zulip only runs with a single webserver, and
thus sendfile backend, the functionality is simple to inline.
It is worth noting that the following headers from the initial Django
response are _preserved_, if present, and sent unmodified to the
client; all other headers are overridden by those supplied by the
internal redirect[^1]:
- Content-Type
- Content-Disposition
- Accept-Ranges
- Set-Cookie
- Cache-Control
- Expires
As such, we explicitly unset the Content-type header to allow nginx to
set it from the static file, but set Content-Disposition and
Cache-Control as we want them to be.
[^1]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
Enforcing a consistent `type` helps us double-check that we're not
playing fast-and-loose with any file paths for local files. As noted
in the comment, this is purely for defense-in-depth.
Passing `write_local_file` a consistent `type` requires removing the
"avatars" out of `realm_avatar_and_logo_path` -- which makes it
consistent across upload backends.
This, in turn, requires a compensatory change to zerver.lib.export, to
be explicit that the realm icons are exported from the avatars
directory. This clarity is likely an improvement.
Updates the help center article to match the style and formatting
of "Import from Slack" and replaces existing content with its
corresponding Markdown macro.
‘logging.warning("Naive datetime:", item)’ is an invalid call that
crashes with “TypeError: not all arguments converted during string
formatting”. I take that to mean this check has not been tripped in
the six years it’s been there, and can safely be replaced with an
error.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Some email clients (notably, Gmail Web) support automatically threading
emails together if recipients and subjects match[1]. Manual testing
indicated that prefixing a subject with "[bracketed content]" does not
break this threading behavior, but the added checkmark in a resolved
topic's title does. Before sending an email notification, determine
whether the topic is resolved, and pass this information to the Jinja
template to properly format a threadable email subject.
Fixes: #22538
[1]: https://support.google.com/mail/answer/5900
Previously, stream names and topics (without consideration for their
resolution status) were concatenated in Python-land and passed through
to the template. To more cleanly separate concerns, and to prepare for
accounting for topic resolution status being a third, independent,
component of a subject line, instead pass stream and topic strings
independently to the Jinja template, which can format them as it sees
fit.
Additionally, migrate existing EditMessageTest to use this helper
method, with the side effect of migrating the tested flow from a
/json/messages URL to a /api/v1/messages URL.
This uses the linkifier index among the list of linkifiers in the
replacement as the priority to order the replacement order for
patterns in the topic. This avoids having multiple overlapping matches
that each produce a link.
The linkifier with the lowest id will be prioritized when its pattern
overlaps with another. Linkifiers are prioritized over raw URLs.
Note that the same algorithm is used for local echoing and the
backend markdown processor.
Fixes#23715.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The same pattern being matched multiple times in a topic cannot be
properly ordered using topic_name.find(match_text) and etc. when there
are multiple matches of the same pattern in the topic.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Moves files in `templates/zerver/help/include` that are used
specifically for API documentation pages to be in a new directory:
`templates/zerver/api/include`.
Adds a boolean parameter to `render_markdown_path` to be used
for help center documentation articles.
Also moves the test file `empty.md` to the new directory since
this is the default directory for these special include macros
that are used in documentation pages.
Moves files in `templates/zerver/help/include` that are used
specifically for integrations documentation to be in a new
directory: `templates/zerver/integrations/include`.
Adds a boolean parameter to `render_markdown_path` to be used
for integrations documentation pages.
As we have seen no further cases of this in production since #23215,
increase the severity to an error, and switch from returning a
list (which is not type-safe if the function declares a QuerySet
return) to returning the QuerySet without caching.
Failing to store the result in the cache, with an error, seems
superior to raising an exception; in both cases the next request will
redo the work, but we are guaranteed a worse user experience if we 500
the request.
Ref https://github.com/zulip/zulip/pull/23215#discussion_r994186493
remove_user_from_user_group's only caller has been removed in 271333301d.
Its usage has been superseded by remove_members_from_user_group.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
In 1fce1c3c73, we added logic to parse
the User-Agent in /register requests; this logic crashed if an HTTP request
was missing that header.
Includes a test for `/register` with no user agent passed; this should catch
similar regressions in the future.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
7 characters are not enough for large projects, so we change
it to reasonably longer. As an example, The Linux kernel needs
at least 11 characters of sha in its shortened form to identify
a revision. We pick 11 so it should work for most of the projects.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Prior to 53231aa, the `ignore_unhashable_lru_cache` decorator had
a check for the development environment so that changes could be
seen on refresh.
Puts that check back in IgnoreUnhashableLruCacheWrapper class.
In the outgoing webhook handler, there is potentially several seconds
of trying between when a message triggering an outgoing webhook
arrives, and when it fails. In the meantime, the stream the
triggering message was on may have been deleted, causing the
"Failure!" message to have no valid stream to be sent to.
Rather than raise an exception in the outgoing webhook worker, ignore
the exception and move on.
The lambda passed to `queue_json_publish` is used if
`settings.USING_RABBITMQ` is unset -- which is only true in tests. As
such, this pattern causes failures to never actually retry within
tests.
This behaviour has existed ever since the outgoing webhook code was
introduced in 53a8b2ac87, with no explanation. Not passing that
argument allows tests to verify the retry behaviour when webhooks
fail.
Previously, test cases or clients accessing /json/ views using HTTP
Basic Auth would be accepted, while we intended to only allow clients
authenticated with a session cookie to access these views.
This adds a check on the accessed path to avoid this possibility.
It seems unlikely that any API clients clients were taking advantage
of this unintended quirk; so we're not going to bother documenting
this bug fix as an API change. In any case, it should be trivial for
anyone affected to consult the documentation and then switch their
/json/foo URL to a correct /api/v1/foo URL.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Mobile clients older than v27.192 do not support PRONOUNS type
custom profile fields, so we instead change the type of it to
SHORT_TEXT in the data sent with register response and also in
the events sent to those clients.
On multi-realm systems this results in traversal of all messages in
all realms and returns a massive payload of 1 row per stream on
the server, not the intended one row per realm.
This guarantees that the Realm is always non-None when we hit the
codepath is_static_or_current_realm_url via
do_change_stream_description, so that we can properly skip rewritting
some images.
Fixes#19405
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
So that we can stop using Tim's photo for tests, adds an open
license profile picture to use instead.
Updates tests that used `tim.png` to use the new example profile
picture, which is located in `static/images/test-images/avatars/`.
We no longer need to do the inner joins to figure out the message's
realm and split up the cross-realm and regular case - now we just look
at zerver_message.realm directly.
This commit updates the urls for personal narrow sent in email
notifications to be of form "{user_id}-{encoded_full_name}" to
make it consistent with the urls that we use for such narrows
in webapp which were recently updated in b4eddad for improving
performance. We encode the full name in the same way that we do in
webapp by replacing the url characters encoded by browser with "-".
Updates the hash used for the recent conversations view to be
"#recent" instead of "#recent_topics".
We will need to keep the logic for handling "#recent_topics"
permanently because users potentially have messages from
Welcome Bot with links to that hash.
Including "recent_topics" as a web_public_allowed_hash in
hash_util.js can be changed once self-hosted servers cannot
upgrade directly to Zulip 5.x from the current version.
Fixes#23132.
Output message should talk about both the cases:
actual_count > expected_count and actual_count < expected_count.
The message now includes information for the case where
actual_query_count < expected_query_count.
Fixes: #23325
Replaces instances of "recent topics" in the web-app and documentation
to be "recent conversations".
Renames both `recent-topics.md` files in the help center to be
`recent-conversations.md` and updates/redirects links to new URL.
Does not update instances of "recent topics" in frontend code comments
and does not update the main overview changelog, for now.
Does not change case study text where "recent topics" was referenced
in a quote, but does change generic text references to be "recent
conversations".
This was broken by commit b945aa3443
(#22604), because email_to_domain implicitly lowercased the result.
No adjustment is needed for is_disposable_domain, which already
lowercases its argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Renames the help article on custom profile fields to reflect that
its content is not just about adding fields.
Adds a redirect from the old URL to the new URL and updates internal
links, linking to #add-a-custom-profile-field where appropriate.
Fixes#23170.
Adds new tab to `zerver/lib/markdown/tabbed_sections.py` to document
managing bots from both personal settings and organization settings.
Documents adding bots from the organization settings Bots panel.
Separates instructions for deactivating and reactivating a bot from
both personal settings and organization settings.
Fixes a few formatting issues such as missing bold formatting and
heading level.
Fixes: #23066.
On my data (about 10 million messages in 1600 streams) this used to take
about 40 hours, while the improved statement completes in roughly 30
seconds.
The old solution had postgres go through the entire table until the
first match for each stream. Thus, the time spent scanning the table
got longer and longer for each stream because postgres always started at
the beginning (and somehow it did not use any indices) and had to skip
over all rows until it found the first message from the stream that is
was looking for each time.
This new statement just performans a bulk operation, scanning the table
only once and then inserts the results directly into the destination
table.
Slightly more verbose inforation about this change can be found in:
https://chat.zulip.org/#narrow/stream/31-production-help/topic/Import.20Rocketchat.20data/near/1408867
Signed-off-by: Florian Pritz <bluewind@xinu.at>
This adds a helper based on testing patterns of using the "queries_captured"
context manager with "assert_length" to check the number of queries
executed for preventing performance regression.
It explains the rationale of checking the query count through an
"AssertionError" and prints the queries captured as assert_length does,
but with a format optimized for displaying the queries in a more
readable manner.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This adds CapturedQueryDict to provide a more accurate type annotation
for the return value of queries_captured. We also replace "Generator"
with "Iterator" because the latter two type parameters were unused.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Send an empty list of `custom_profile_fields` in `page_params` for
spectators, rather than not sending the field at all.
Also, updates the user info popover to not show the manage user
three-dot menu when in a spectator view.
As noted in the previous commit, this causes bloat in memcached, for
no purpose. Log a warning when `cache_with_key` sees a QuerySet
returned from the function it is decorating.
Storing this key is superfluous, as it will be the same for all users,
and definitionally already known to fetch the cache for the realm. It
is also not currently used by the callsites that read rows from the
cache.
9381a3bd45 added support for linkifier pattern URLs containing
`%20`-style escapes, but only did so for the codepath which is used in
the message body -- topic links did not understand them.
Expand the support to include when they are substituted into topics.
These were useful as a transitional workaround to ignore type errors
that only show up with django-stubs, while avoiding errors about
unused type: ignore comments without django-stubs. Now that the
django-stubs transition is complete, switch to type: ignore comments
so that mypy will tell us if they become unnecessary. Many already
have.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.
Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
zerver/migrations/0240_usermessage_migrate_bigint_id_into_id.py needs
to be updated to account for Django 4.1 creating AutoField as an
identity column rather than a serial column.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Note that django_stubs_ext is required to be placed within common.in
because we need the monkeypatched types in runtime; django-stubs
itself is for type checking only.
In the future, we would like to pin to a release instead of a git
revision, but several patches we've contributed upstream have not
appeared in a release yet.
We also remove the type annotation for RealmAuditLog.event_last_message_id
here instead of earlier because type checking fails otherwise.
Fixes#11560.
We no longer need to annotate the type of objects returned
from queries since django-stubs plugin infers that already.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This works around some regression in moto 1.3.15 that I bisected to
b8820009e8
where ‘tools/test-backend test_transfer’ fails when run by itself.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, we included all three message edit related settings
("allow_message_editing", "message_content_edit_limit_seconds" and
"edit_topic_policy") in the event data and api response irrespective
of which of these settings were changed. Now, we only include changed
settings and separate events are sent for each setting if more than
one of them is changed.
Note that the previous typed in event_schema.py for
`message_content_edit_limit_seconds` incorrectly did not allow `None`
as a value, which is used to encode no limit.
This refactors and renames user_ids_muting_topic to accept a parameter
'visibility_policy' and fetch user IDs that have a specific
visibility_policy(provided as the parameter) set for a topic.
Unfortunately, doing so requires forking common API documentation
text, since we're not making any changes to other endpoints that don't
allow unauthenticated requests at all.
Follow-up on #21995.
As mentioned in the TODO this commit deletes, the export with member
consent system was failing to account for the fact that if consenting
users only have access to a subset of messages of a stream with
protected history, only that subset should be exported - rather than all
the stream's messages.
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
SCIMClient is a type-unsafe workaround for django-scim2’s conflation
of SCIM users with Django users. Given that a SCIMClient is not a
UserProfile, it might as well not be a model at all, since it’s only
used to satisfy django-scim2’s request.user.is_authenticated queries.
This doesn’t solve the type safety issue with assigning a SCIMClient
to request.user, nor the performance issue with running the SCIM
middleware on non-SCIM requests. But it reduces the risk of potential
consequences worse than crashing, since there’s no longer a
request.user.id for Django to confuse with the ID of an actual
UserProfile.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fourth step in making user status `away` a deprecated way to access
`presence_enabled` for clients supporting older servers, and
checkpoint commit prior to deleting the `status` field from the
UserStatus model.
Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
We stop sending the `away=True` based on the user's `UserStatus`
object having `status=AWAY`, and instead send that value if
`!presence_enabled` for the user.
Third step in making user status `away` a deprecated way to access
`presence_enabled` for clients supporting older servers.
Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
Rename functions that refer to "user_info" without a reference to
"status" to help clarify in the backend between UserPresence
and UserStatus models.
Prep commit for migrating "unavailable" user status feature to
"invisible" user presence feature.
Rename functions that refer to "status" without a reference to
"presence" to help clarify in the backend between UserPresence
and UserStatus models.
Prep commit for migrating "unavailable" user status feature to
"invisible" user presence feature.
This help center article should include more features rather than just
focusing on the "go to conversation" button. We should broaden and
restructure this page to cover other advanced features.
Refactors the "Go to conversation" section as step-by-step instructions,
and adds a `keyboard_tip`.
Adds new section "Toggle between Ctrl+Enter and Enter".
Deletes the "Enable Enter to send" help center article, and adds its
content as a new subheading in this section.
Updates existing links accordingly and adds a URL redirect.
Documents "Enable Control + Enter to send".
Tweaks intro paragraph of "Mastering the compose box".
Fixes: #22817.
To allow `custom_profile_field` to display in user profile popover,
added new boolean field "display_in_profile_summary" in its model class.
In `custom_profile_fields.py`, functions are edited as per conditions,
like currently we can display max 2 `custom_profile_fields` except
`LONG_TEXT` and `USER` type fields.
Default external account custom profile fields made updatable for only
this new field, as previous they were not updatable.
Fixes part of: #21215
Sahil Batra
Anders Kaseorg
Alex Vandiver
Mateusz Mandera
Abhijeet Prasad Bodas
Alya Abbott
David Rosa
Lauryn Menard
Alya Abbott
Lalit
Tim Abbott
Lauryn Menard
Aman Agrawal
Prakhar Pratyush
Trident Pancake
Josh Klar
Josh Klar
Zixuan James Li
M@
Florian Pritz
PIG208
khanhdq
Kartik Srivastava
yogesh sirsat