Commit Graph

16909 Commits

Author SHA1 Message Date
Alex Vandiver d87895a3ef missed-message: Merge before calling handle_missedmessage_emails.
The MissedMessage queue worker is the single callsite of
`handle_missedmessage_emails`, which immediately transforms the list
of events into a dict keyed by message-id.

Skip the intermediate list step, and use defaultdict and a dataclass
to simplify and make explicit the pieces.  This removes the unused
user_profile_id and message_id pieces of the data structure.
2023-07-13 11:50:42 -07:00
Alex Vandiver c7d9a4784e missed-message: Remove unnecessary select_related().
This was added in ebb4eab0f99d; neither the `user_profile` nor the
`message` attribute are read off of the object.
2023-07-13 11:50:42 -07:00
Steve Howell 4533ff3671 onboarding: Rename variable to cutoff_date.
This is just as clear in terms of intent, and it's robust to
us tweaking the number of weeks.
2023-07-13 11:46:34 -07:00
Prakhar Pratyush 0891f9f65a mention: Determine @topic mention during message rendering.
This commit adds a boolean field `mentions_topic_wildcard`
to the `MessageRenderingResult` dataclass.

The field is set to true only if message rendering determines
the message has an actual topic wildcard mention in it (and not,
e.g., topic wildcard mention syntax inside a code block).

The rendered content for topic wildcard mention is
'<span class="topic-mention">{wildcard}</span>'.

The 'topic-mention' class is the identifier for the wildcard
mention being a topic wildcard mention.

We don't use 'data-user-id="*"' and "user-mention" class for
topic wildcard mentions and eventually plan to remove them for
stream wildcard mentions too in a separate mini-project.
2023-07-13 11:34:48 -07:00
Prakhar Pratyush 806d8f2dc7 test_markdown: Merge similar tests into a single test case.
This prep commit merges separate tests for '**@all**',
'**@stream**' and '**@everyone**' stream wildcard mentions
into a single test named 'test_mention_stream_wildcard'.

Similarly, it merges separate tests for '@all', '@stream',
and '@everyone' stream wildcard mentions into a single test
named 'test_mention_at_stream_wildcard'.

The aim is to finally have two separate tests for stream and
topic wildcard mentions (when we introduce topic wildcards)
instead of having separate tests for each mention text
(i.e. all, everyone, stream, topic).
2023-07-13 11:34:48 -07:00
Prakhar Pratyush c0c30bc5f7 topic_mentions: Fetch users to be notified of @topic mentions.
This commit adds the 'topic_wildcard_mention_user_ids' and
'topic_wildcard_mention_in_followed_topic_user_ids'
attributes to the 'RecipientInfoResult' dataclass.

Only topic participants are notified of @topic mentions.

Topic participants are anyone who sent a message to a topic
or reacted to a message on the topic.

'topic_wildcard_mention_in_followed_topic_user_ids' stores the
ids of the topic participants who follow the topic and have
enabled the wildcard mention notifications for followed topics.

'topic_wildcard_mention_user_ids' stores the ids of the topic
participants for whom 'user_allows_notifications_in_StreamTopic'
with setting 'wildcard_mentions_notify' returns True.
2023-07-13 11:34:48 -07:00
Prakhar Pratyush 1df63ed448 mention: Add 'has_topic_wildcards' to 'MentionData'.
This commit adds a 'has_topic_wildcards' instance variable
to the 'MentionData' class for the detection of
- possible topic wildcards mentions.

Fixes part of #22829.

Co-authored-by: Prakhar Pratyush <prakhar841301@gmail.com>
Co-authored-by: orientor <aditya.verma@students.iiit.ac.in>
2023-07-13 11:34:48 -07:00
Prakhar Pratyush 3f6b41e4be test_notifications: Update tests to cover the corner case properly.
This commit updates the existing tests in 'test_email_notifications'
and 'test_push_notifications' to properly configure user settings
and visibility policies before running the actual tests.

Earlier, the tests were passing, but the corner case expected
to be covered wasn't covered.

This should have been included in
d80779435a.
2023-07-13 11:34:48 -07:00
Prakhar Pratyush 2869de8026 test_notifications: Remove unnecessary comments.
These comments should not have been included in
a8fd9eb701.

We covered the case "Private message should soft reactivate
the user" earlier in the test. So the comment was rightly added
there.

During stream wildcard or group mention, no such personal mention
is involved; hence, the comments are not needed.
2023-07-13 11:34:48 -07:00
Prakhar Pratyush 2b42df4ef1 mention: Replace 'wildcard' with 'stream_wildcard'.
This is a prep commit to replace 'wildcard' with 'stream_wildcard'.

This wasn't included in 179d5cb because we didn't decide to
use a different rendered_content for topic wildcard mention,
i.e., ''<span class="user-mention topic-mention">{wildcard}</span>'.

Our intention was not to create separate tests for both stream
and topic wildcard mentions, as they were expected to have the
same rendered content format.
2023-07-13 11:34:48 -07:00
Steve Howell 418057048a onboarding: Backfill unread messages up to 12 weeks old.
Previously this limit was 1 week, which was fine for busy
organizations, but for organizations that send a few messages a week,
or have occasional bursts of activity but the last one was a few weeks
ago, this should give a significantly better new user experience.

There are still caps like 1000 messages total and 20
unread, but we're a bit more flexible about time.
2023-07-13 10:40:12 -07:00
Steve Howell 890732a88f soft activation: Avoid QuerySet and use List instead. 2023-07-13 08:09:14 -07:00
Anders Kaseorg 7e707270f0 models: Convert deprecated index_together option to indexes.
index_together is slated for removal in Django 5.1:
https://docs.djangoproject.com/en/4.2/internals/deprecation/#deprecation-removed-in-5-1

We set the optional index names to match the previously generated
index names to avoid adding new migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-12 07:12:43 -07:00
nimish c238327899 settings: Change "Display settings" to "Preferences".
This includes changing the URL to #settings/preferences, with a
transparent redirect so that existing links, like the one from Welcome
Bot, continue to work.
2023-07-12 07:09:03 -07:00
Anders Kaseorg 63be67af80 logging_util: Remove dependence on get_current_request.
Pass the HttpRequest explicitly through the two webhooks that log to
the webhook loggers.

get_current_request is now unused, so remove it (in the same commit
for test coverage reasons).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-11 22:23:47 -07:00
Anders Kaseorg f66e2c3112 sentry: Remove dependence on get_current_request.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-11 22:23:47 -07:00
abdullahm1 5a90f9c404 tests: Use time_machine for testing scheduled message delivery. 2023-07-11 17:34:58 -07:00
Lauryn Menard 3dfdbbc775 welcome-emails: Separate followup_day1 email from other welcome emails.
The initial followup_day1 email confirms that the new user account
has been successfully created and should be sent to the user
independently of an organization's setting for send_welcome_emails.

Here we separate out the followup_day1 email into a separate function
from enqueue_welcome_emails and create a helper function for setting
the shared welcome email sender information.

The followup_day1 email is still a scheduled email so that the initial
account creation and log-in process for the user remains unchanged.

Fixes #25268.
2023-07-11 14:15:52 -07:00
Lauryn Menard 0e1acd595b welcome-emails: Use followup_day2 for scheduled email tests.
The followup_day2 email is scheduled with a delay as a welcome email
and is therefore more likely to exist as a scheduled email in these
deactivation cases.
2023-07-11 14:15:52 -07:00
Lauryn Menard dd59e83d54 welcome-emails: Make some code comments and docstrings more evergreen. 2023-07-11 14:15:52 -07:00
Lauryn Menard c323afd9d7 test-example: Revise comment with number of emails generated.
Updates comment to not include the number of emails generated so
that it doesn't need to be updated every time a new email is added.
The current count in the comment is already out-of-date.
2023-07-11 14:15:52 -07:00
Zixuan James Li 84723654c8 webhooks: Use 200 status code for unknown events.
Because the third party might not be expecting a 400 from our
webhooks, we now instead use 200 status code for unknown events,
while sending back the error to Sentry. Because it is no longer an error
response, the response type should now be "success".

Fixes #24721.
2023-07-11 13:51:37 -07:00
Sahil Batra 2e4f7f6336 user_groups: Remove "@" from name of role-based system groups.
This commit removes "@" from name of role-based system groups
since we have added a restricion on having user group names
starting with "@" in the previous commit as they look odd in
mention syntax.

We also add a migration in this commit to update the name of
role-based system groups in existing realms to remove "@"
from the name. This migration also updates the names of
non-system user groups by removing the invalid prefixes
from their names and if there is a group already with that
name, we insted name the group as "group:{group_id}".

Fixes #26148.
2023-07-11 13:46:02 -07:00
Sahil Batra 929bf1243e user_groups: Disallow certain prefixes in group name.
We do not allow user group names to start with "@", "role:",
"user:", "stream:" and "channel:".

Group names starting with "@" look odd in mentions and
"role:", "user:" and "stream:" prefixes are reserved for
system groups which will be used in the new groups-based
permission model. We do not allow "channel:" prefix for
now just to be safe in a case where we use it instead of
"stream:" prefix for stream based groups in future.

Fixes part of #26148.
2023-07-11 13:46:02 -07:00
Sahil Batra ea3a7a9e6f user_groups: Add API restrictions for long user group names.
Previously we had database level restriction on length of
user group names. Now we add the same restriction to API
level as well, so we can return a better error response.
2023-07-11 13:46:02 -07:00
Steve Howell 89381a8072 cache: Eliminate get-stream-by-name cache.
We remove the cache functionality for the
get_realm_stream function, and we also change it to
return a thin Stream object (instead of calling
select_related with no arguments).

The main goal here is to remove code complexity, as we
have been prone to at least one caching validation bug
related to how Realm and UserGroup interact. That
particular bug was more theoretical than practical in
terms of its impact, to be clear.

Even if we were to be perfectly disciplined about only
caching thin stream objects and always making sure to
delete cache entries when stream data changed, we would
still be prone to ugly situations like having
transactions get rolled back before we delete the cache
entry. The do_deactivate_stream is a perfect example of
where we have to consider the best time to unset the
cache. If you unset it too early, then you are prone to
races where somebody else churns the cache right before
you update the database. If you set it too late, then
you can have an invalid entry after a rollback or
deadlock situation. If you just eliminate the cache as
a moving part, that whole debate is moot.

As the lack of test changes here indicates, we rarely
fetch streams by name any more in critical sections of
our code.

The one place where we fetch by name is in loading the
home page, but that is **only** when you specify a
stream name. And, of course, that only causes about an
extra millisecond of time.
2023-07-11 13:45:40 -07:00
Steve Howell 046e4c715b cache: Use DB for all bulk get-stream-by-name queries.
This changes bulk_get_streams so that it just uses the
database all the time.  Also, we avoid calling
select_related(), so that we just get back thin and
tidy Stream objects with simple queries.

About not caching any more:

It's actually pretty rare that we fetch streams by name
in the main application. It's usually API requests that
send in stream names to find more info about streams.

It also turns out that for large queries (>= ~30 rows
for my measurements) it's more efficent to hit the
database than memcached. The database is super fast at
scale; it's just the startup cost of having Django
construct the query, and then having the database do
query planning or whatever, that slows us down. I don't
know the exact bottleneck, but you can clearly measure
that one-row queries are slow (on the order of a full
millisecond or so) but the marginal cost of additional
rows is minimal assuming you have a decent index (20
microseconds per row on my droplet).

All the query-count changes in the tests revolve around
unsubscribing somebody from a stream, and that's a
particularly odd use case for bulk_get_streams, since
you generally unsubscribe from a single stream at a
time. If there are some use cases where you do want to
unsubscribe from multiple streams, we should move
toward passing in stream ids, at least from the
application. And even if we don't do that, our cost for
most queries is a couple milliseconds.
2023-07-11 13:45:40 -07:00
Steve Howell adb548c7a2 stream creation: Avoid stream.realm references.
We want to avoid Django going back to the database to
get a realm object that the caller already has.

It's actually currently the case that we often
pre-fetch realm objects when we get stream objects
using get_stream (using a call to select_related() with
no arguments), but that is an expensive operation that
we want to avoid going forward.

This commit prepares us to just fetch slim objects.
2023-07-11 13:45:40 -07:00
Satyam Bansal 328c104424 integrations: Separate issue milestoned events in GitHub Integration.
This commit creates separate events for issue milestoned and
demilestoned notifications. This allows the end-users to choose
whether they want these notifications or not.

Fixes #25793.
2023-07-11 08:58:31 -07:00
Satyam Bansal 34f31ab9d2 integrations: Improve GitHub issue milestoned notifications.
Earlier, the notifications had no information about the milestone
that was added or removed.
2023-07-11 08:58:31 -07:00
Satyam Bansal 1c567ae616 integrations: Add issue demilestoned fixture to GitHub Integration. 2023-07-11 08:58:31 -07:00
Satyam Bansal f8ac308ec2 integrations: Add issue milestoned fixture to GitHub Integration. 2023-07-11 08:58:31 -07:00
Zixuan James Li 3349ac9f86 user_groups: Audit UserGroup group based setting changes.
This add audit log entries when any group based setting of a user group
is updated. We store both the old and new values in extra_data, along
with the name of that setting. Entries populated during user group creation
are hardcoded to track "can_mention_group".

Potentially we can adjust "set_defaults_for_group_settings" so that it
populates realm audit logs with it, but that is out of scope for this change.

We use an atomic transaction so that the audit logs are committed
together with the updates.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 4d0b7fe682 user_groups: Audit UserGroup properties changes.
This add audit log entries when the name or description of a user group
is updated. We store both the old and new values in extra_data. We wrap
the functions inside an atomic transaction so that the audit logs and
the updates are committed together.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 3035854dca user_groups: Audit UserGroup supergroup memberships changes.
This is mostly the same as tracking subgroup changes, except that now
modified_user_group is the subgroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li ad698d597a user_groups: Audit UserGroup subgroup memberships changes.
It's worth noting that instead of adding another field to the
RealmAuditLog model, we store the modified subgroup ids in extra_data as
a JSON encoded dict with the key "subgroup_ids". We don't create audit
log entries for supergroup changes at this point.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 44781ddfa9 user_groups: Audit UserGroup memberships changes.
This also add audit log entries during user creation and role change,
because we modify system group memberships there.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 63f5936207 user_groups: Audit UserGroup creation.
We also create RealmAuditLog entries for the initial memberships that
get added along with the creation of a UserGroup. System user groups are
not created with members so no audit logs are populated for that.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 71de14ab43 models: Add modified_user_group.
This also adds the supported event types for changes to UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Alex Vandiver a076d49be7 sentry: Reduce http timeout.
This helps reduce the impact on busy uwsgi processes in case there are
slow timeout failures of Sentry servers.  The p99 is less than 300ms,
and p99.9 per day peaks at around 1s, so this will not affect more
than .1% of requests in normal operation.

This is not a complete solution (see #26229); it is merely stop-gap
mitigation.
2023-07-10 13:46:16 -07:00
Lauryn Menard 3d8090a116 sentry-webhook: Revise documentation page to be clearer. 2023-07-10 13:43:28 -07:00
Steve Howell b31bbc6148 signup: Clean up add_new_user_history.
Various cleanups:

    * clean up comments
    * improve names for constants and variables
    * express first ORM query as a single statement
    * use set differences to simplify logic
    * avoid all the reversing churn
    * avoid early-exit idiom since this function is so small

Note that it's plausible that we should just combine the two
queries and let the database exclude the already-used ids,
but that felt a little risky for now.  As I mentioned on
Zulip, I think the one-week window has dubious value, but
I am biased by having wasted time chasing down a test
flake related to the time window.
2023-07-10 13:41:28 -07:00
Steve Howell 225e826fb2 deactivate streams: Remove unused "log" parameter. 2023-07-10 13:41:28 -07:00
Steve Howell bc3afe9127 default stream groups: Make deleting streams efficient.
This pulls one query out the loop, and then it makes
another query a bulk query, and then it finally eliminates
an unnecessary query at the end.
2023-07-10 13:41:28 -07:00
Steve Howell 87d1208d53 tests: Improve test for default stream groups. 2023-07-10 13:41:28 -07:00
Steve Howell 1156a50109 signup: Avoid bloated Stream objects for default streams.
Basically, I eliminate the use of select_all() in a query
that still makes a single round trip.  We have good test
enforcement that Django never needs to lazily fetch
objects off the Stream object. (It used to be common
to fetch stream.realm a while back, but we upgraded
bulk_add_subscription, in particular, a while back.)
2023-07-10 13:41:28 -07:00
Steve Howell 8894ff89ac signup: Extract set_up_streams_for_new_human_user.
We extract code from process_new_human_user with
no modifications.

This has all the best outcomes of extracting a function:

    * better profile info
    * easier to test for query counts (signup gets real noisy)
    * simplifies a long, messy function

It has no real drawbacks, since the helper function doesn't need
to pass back any intermediate state to the parent for the rest
of what the parent does.

When you profile test_signup and test_invite, with a decent
sample size, the set_up_streams_for_new_human_user function
does about 20% of the work for process_new_human_user, which
is a lot considering that most tests don't create a ton of
pre-registered or default streams.
2023-07-10 13:41:28 -07:00
Steve Howell d6ef94f63f page load: Improve default_streams performance.
At least as measured by test_events.py, which has over 1000
calls to fetch initial data for page loads, this should
be about a 10% improvement in how much time the server
spends fetching data.

We mostly avoid a select_related() query that did this nastiness:

    INNER JOIN "zerver_realm" ON ("zerver_stream"."realm_id" = "zerver_realm"."id")
    INNER JOIN "zerver_usergroup" ON ("zerver_stream"."can_remove_subscribers_group_id" = "zerver_usergroup"."id")
    INNER JOIN "zerver_realm" T4 ON ("zerver_usergroup"."realm_id" = T4."id")
    INNER JOIN "zerver_usergroup" T5 ON ("zerver_usergroup"."can_mention_group_id" = T5."id")
    INNER JOIN "zerver_realm" T6 ON (T5."realm_id" = T6."id")
    INNER JOIN "zerver_usergroup" T7 ON (T5."can_mention_group_id" = T7."id")
    INNER JOIN "zerver_realm" T8 ON (T7."realm_id" = T8."id")
    INNER JOIN "zerver_usergroup" T9 ON (T7."can_mention_group_id" = T9."id")
    INNER JOIN "zerver_realm" T10 ON (T9."realm_id" = T10."id")
    INNER JOIN "zerver_usergroup" T11 ON (T9."can_mention_group_id" = T11."id")
    WHERE "zerver_stream"."id" IN (SELECT U0."stream_id" FROM "zerver_defaultstream" U0 WHERE U0."realm_id" = 2

Future commits will address the codepath for creating users.
2023-07-10 13:41:28 -07:00
Steve Howell 763b5e0741 default streams: Extract library functions.
I created zerver/lib/default_streams.py, so that various
views and events.py don't have to awkwardly reach into
an "actions" file.

I copied over two functions verbatim from actions/default_streams.py:

    get_default_streams_for_realm
    streams_to_dicts_sorted

The latter only remains as an internal detail in the new library.

I also created two new helpers:

    get_default_stream_ids_for_realm:

        This is both faster and easier to use in all the places
        where we only need to get a set of default stream ids.

    get_default_streams_for_realm_as_dicts:

        This just wraps the prior calls to
        streams_to_dicts_sorted(get_default_streams_for_realm(...)),
        and it doesn't yet address the slowness of the underlying
        code.

        All the "real" code should be functionally the same.

        In a few tests I now use this wrapper instead of
        calling get_default_streams_for_realm, just to get
        slightly deeper coverage.
2023-07-10 13:41:28 -07:00
Lauryn Menard d84fd73db4 markdown-processor: Update insertion_index check for multiple classes.
Updates find_proper_insertion_index to check for the inline image
classes as matching at least one of the classes in the element's
attrib["class"] so that cases where an inline preview image has
multiple classes, like YouTube video previews, will have the
correct insertion index.

Fixes #26186.
2023-07-07 11:07:45 -04:00