Commit Graph

45626 Commits

Author SHA1 Message Date
Steve Howell 49a1d5a9a3 refactor: Reduce scope of message var. 2021-12-30 11:36:52 -08:00
Steve Howell d7f90acbf6 refactor: Extract remove_subscriber. 2021-12-30 11:36:52 -08:00
Steve Howell e742ebb8e8 refactor: Limit scope of e.target.
Now our click handlers get stream_id directly from
e.target, and then downstream code is no longer
coupled to the event semantics.

Note that we'll probably just know the stream_id
more directly after future commits.

We also remove a little bit of redundant error
handling.
2021-12-30 11:36:52 -08:00
Steve Howell cb21506ea7 node tests: Sort list of exempt files. 2021-12-30 11:36:52 -08:00
Steve Howell 0777cd73bb stream edit: Extract stream_subscribers_ui.js.
This is a fairly straightforward extraction.

It's good to test this with Iago, and then go into
Manage Streams and add/remove subscribers for a stream
like devel.

I copy/pasted two small functions that will soon
diverge from stream_edit. The get_stream_id function
will either use a module variable (since we're
generally only editing subscribers for one stream, and
we already have the singleton assumption with
`input_pill`) or a more strict CSS selector.  And then
get_sub_for_target depends on get_stream_id. We may not
always need full subs, anyway, and when we adapt some
of this code for creating streams, things are likely to
change.

I stopped exporting a couple functions that have no
callers outside of this module.

The main entry point for the module is
enable_subscriber_management.

We continue to export invite_user_to_stream and
remove_user_from_stream, which should possibly be just
pulled into their own module to lessen some
dependencies, but they don't have too much baggage,
since they just wrap channel calls.
2021-12-30 11:36:52 -08:00
Steve Howell d938f73d05 refactor: Move code for can't-subscribe popover. 2021-12-30 11:36:52 -08:00
Steve Howell a9271e7a99 performance: Cache stream lookups in MentionBackend.
This is useful when you subscribe a bunch of folks
to a stream and need to send them all PMs telling
them about the new subscription.
2021-12-30 11:28:15 -08:00
Steve Howell 4adcaf92f7 refactor: Attach get_stream_name_map to MentionData.
This diff looks slightly noisy, but the main chunk of
code that we moved here has the same logic as before,
and it just gets realm_id from MentionBackend now, instead
of having our markdown processor have to supply it.

We basically want MentionData to be the gatekeeper of
mention data, and then we delegate backend tasks to
MentionBackend.

Soon we will add a cache to MentionBacked, which will
justify this change a bit more.
2021-12-30 11:28:15 -08:00
Steve Howell 05eb4cfa5f mypy: Fix argument type for get_active_streams.
We now make it mandatory to pass in the Realm object.

If this function was ever called with None, I am scared
to know what the expected results were at the time of
writing.
2021-12-30 11:28:15 -08:00
Steve Howell 0359c083d1 refactor: Extract get_linkable_streams.
This is a one-liner with two purposes:

    * We want the comment to explain the business rule.

    * We want to just work in id space.
2021-12-30 11:28:15 -08:00
Steve Howell c4bd4496dd peformance: Cache user mentions for multiple PMs.
It's slightly annoying to plumb Optional[MentionBackend]
down the stack, but it's a one-time change.

I tried to make the cache code relatively unobtrusive
for the single-message use case.

We should be able to eliminate redundant stream queries
using similar techniques.

I considered caching at the level of rendering the message
itself, but this involves nearly as much plumbing, and
you have to account for the fact that several users on
your realm may have distinct default languages (French,
Spanish, Russian, etc.), so you would not eliminate as
many query hops. Also, if multiple streams were involved,
users would get slightly different messages based on
their prior subscriptions.
2021-12-30 11:28:15 -08:00
Steve Howell c6448263c3 refactor: Add MentionBackend.
We will eventually use this to avoid redundant
queries.

The diff is slightly noisy here, but there are no
logic changes.
2021-12-30 11:28:15 -08:00
Steve Howell a22f49bf83 refactor: Extract UserFilter.
This is setting us up for future commits.
2021-12-30 11:28:15 -08:00
Steve Howell ea252ab53e refactor: Convert FullNameInfo to a dataclass.
As part of this we no longer query for email, which
is a vestige of when we used emails to identify users
on the frontend.
2021-12-30 11:28:15 -08:00
Steve Howell f5fc348786 mypy: Add explicit types for dbdata references.
When our handlers specifically reference self.md.zulip_db_data,
we now use an explicit type.

We probably want a more robust solution here, such as a semgrep
rule.
2021-12-30 11:28:15 -08:00
Steve Howell df84892aad markdown: Convert DbData to a dataclass. 2021-12-30 11:28:15 -08:00
Steve Howell 4e551f8279 refactor: Introduce get_stream_name_map.
We only need a name -> id map, and the FullNameInfo
type was a lie.
2021-12-30 11:28:15 -08:00
Steve Howell c04a8097f3 mypy: Add EmojiInfo type.
We now serialize still_url as None for non-animated emojis,
instead of omitting the field. The webapp does proper checks
for falsiness here.  The mobile app does not yet use the field
(to my knowledge).

We bump the API version here. More discussion here:

https://chat.zulip.org/#narrow/stream/378-api-design/topic/still_url/near/1302573
2021-12-30 11:28:14 -08:00
Steve Howell a6201b430f tests: Improve checks for subscribing users.
We now check both the notification messages for
all three of Hamlet's peers.

And we count queries.
2021-12-30 11:23:25 -08:00
Steve Howell fd925e6045 streams: Add id to user mentions for stream notifications. 2021-12-30 11:23:25 -08:00
Lauryn Menard a16fcd3172 tests: Improve testing helper event schema for `update_message`.
Further clarifies the fields returned by `update_message` event
for the type of change (content, topic and/or stream).

Follow-up task from #20587.
2021-12-30 08:35:35 -08:00
Biki-das b3c2f88b7b typeahead: Remove unnecessary topic_names variable.
Also fix a comment typo in the same file.
2021-12-30 06:56:29 -08:00
Tim Abbott 877c275380 typeahead: Use early return pattern for enter logic.
This makes this code more readable.
2021-12-30 06:55:57 -08:00
BIKI DAS ad61d06cea
python: Remove unnecessary list comprehension.
`all` can take a generator, not just a list.  

Using a generator expression here is simpler and faster.
2021-12-30 06:51:50 -08:00
Julia Bichler b673e966ca spectators: Prevent toggling TODO checkboxes.
Checkboxes now don't change if a user who is not logged in
clicks on them.

Fixes #20300.
2021-12-29 17:10:21 -08:00
Anders Kaseorg b0b8f84949 test_console_output: Avoid appending to bytes in a loop.
Appending to bytes in a loop leads to a quadratic slowdown since
Python doesn’t optimize this for bytes like it does for str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-29 16:50:08 -08:00
Anders Kaseorg f9271098bf upload-release: Avoid appending to bytes in a loop.
Appending to bytes in a loop leads to a quadratic slowdown since
Python doesn’t optimize this for bytes like it does for str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-29 16:50:08 -08:00
Anders Kaseorg 80e4d5436d requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-29 16:48:47 -08:00
Alex Vandiver c094867a74 puppet: Add aarch64 build hashes to external dependencies.
wal-g does not ship aarch64 binaries, currently; the compilation
process([1]) is somewhat complicated, so we defer the decision about
how to support wal-g for aarch64 until a later date.

[1]: https://github.com/wal-g/wal-g/blob/master/docs/PostgreSQL.md#installing
2021-12-29 16:35:15 -08:00
Alex Vandiver f166f9f7d6 puppet: Centralize versions and sha256 hashes of external dependencies.
This will make it easier to update versions of these dependencies.
2021-12-29 16:35:15 -08:00
Alex Vandiver 74b804880e bootstrap-aws: Support aarch64. 2021-12-29 16:35:15 -08:00
Alex Vandiver 57662689a9 puppet: Provide a constant homedir for grafana user.
The homedir of a user cannot be changed if any processes are running
as them, so having it change over time as upgrades happen will break
puppet application, as the old grafana process under supervisor will
effectively lock changes to the user's homedir.

Unfortunately, that means that this change will thus fail to
puppet-apply unless `supervisorctl stop grafana` is run first, but
there's no way around that.
2021-12-29 16:35:15 -08:00
Alex Vandiver 6e55e52694 puppet: Pull out grafana $data_dir. 2021-12-29 16:35:15 -08:00
Alex Vandiver 51d3862c7e puppet: Move wal-g to external_dep, in /srv/zulip-wal-g-*. 2021-12-29 16:35:15 -08:00
Alex Vandiver 1e4e6a09af puppet: Stop making resources for external binaries and directories.
In the event that extracting doesn't produce the binary we expected it
to, all this will do is create an _empty_ file where we expect the
binary to be.  This will likely muddle debugging.

Since the only reason the resourfce was made in the first place was to
make dependencies clear, switch to depending on the External_Dep
itself, when such a dependency is needed.
2021-12-29 16:35:15 -08:00
Alex Vandiver 3c163a7d5e puppet: Move slash out of $dir by convention. 2021-12-29 16:35:15 -08:00
Alex Vandiver bb5a2c8138 puppet: Move prometheus to external_dep. 2021-12-29 16:35:15 -08:00
Alex Vandiver 2d6c096904 puppet: Move node_exporter to external_dep. 2021-12-29 16:35:15 -08:00
Alex Vandiver d2a78bac7e puppet: Adjust wal-g release version and SHA256.
wal-g apparently removed the 1.1.1 release; replace it with the
equivalent rc.
2021-12-29 16:35:15 -08:00
Sahil Batra a59d429045 settings: Fix code for special case of theme settings subsection.
We handle "Theme settings" subsection separately in
get_subsection_property_elements as it contains unique
radio-button structure for emojiset setting.

This should have been fixed while reorganizing the section
to have color scheme and emoji related settings under same
subsection in adb612a0b4.

Fixes #20644.
2021-12-29 16:32:12 -08:00
parth 4edf029ad5 invitations: Don't notify now-deactivated users.
While accepting an invitation from a user, there was no condition in
place to check if the user sending the invitation was now
now-deactivated.

Skip sending notifications about newly-joined users to users who are
now disabled.

Fixes #18569.
2021-12-29 16:21:19 -08:00
Steve Howell 1e4593b2ae performance: Avoid Recipient lookup.
We don't have to go to the database to get the Recipient
fields for `user_profile.recipient`.

See also 85ed6f332a from a little
over a year ago--it's very similar.
2021-12-28 12:15:02 -08:00
Steve Howell 01ebb2c85f refactor: Pass realm to bulk_remove_subscriptions.
We made a very similar change to bulk_add_subscriptions
earlier in the year.
2021-12-28 12:15:02 -08:00
Steve Howell ebbd5f168b refactor: Pass realm to notify_subscriptions_removed. 2021-12-28 12:15:02 -08:00
Steve Howell 966d88a78a stream colors: Fix stream color assignment.
The bug here probably didn't come up too much in
practice, but if we were adding a user to multiple
streams when they already had used all N available
colors, all the new streams would be assigned the same
color, since the size of used_colors would stay at N,
thwarting our little modulo-len hackery.

It's not a terrible bug, since users can obviously
customize their stream colors as they see fit.

Usually when we are adding a user to multiple streams,
the users are fairly new, and thus don't have many
existing streams, so I have never heard this bug
reported in the field.

Anyway, assigning the colors in bulk seems to make more
sense, and I added some tests.

For the situations where all the colors have already
been used, I didn't put a ton of thought into exactly
which repeated colors we want to choose; instead, I
just ensure they're different modulo 24. It's possible
that we should just have more than 24 canned colors, or
we should just assign the same default color every time
and let users change it themselves (once they've gone
beyond the 24, to be clear). Or maybe we can just do
something smarter here. I don't have enough time for a
deep dive on this issue.
2021-12-28 12:15:02 -08:00
Steve Howell fe3295d395 performance: Avoid monster query for existing subs.
Part of our codepath for subscribing users involves
fetching the users' existing subscriptions to make sure
we can do things like properly report to the clients
that the users were already subscribed.  This codepath
used to be coupled to code that helped users maintain
unique stream colors.

Suppose you are creating a new stream, and you are
importing users from an older stream with 15k
subscribers, and each of your users is subscribed to
about 20 streams.

The prior code, instead of filtering on recipient_id,
would literally look at every subscription for every
user, which was kind of crazy if you didn't understand
the pick-stream-color complications.

Before this commit, we would fetch 300k rows with 15
columns each (granted, all but one of the columns are
bool/int). That's a total of 4.5 million tiny objects
that we had to glom into Django ORM objects and slice
and dice.

After this commit, we would fetch exactly zero rows
for the are-they-already-subscribed logic.

Yes, ZERO.

If we were to mistakenly try to re-add the same 15k
subscribers to the new stream (under the new code), we
will now fetch 15k Sub rows instead of 300k.

It is worth looking at the prior commit. We go through
great pains to ensure that users get new stream colors
when we invite them to a stream, and we still fetch a
bunch of data for that. Instead of 4.5 million cells,
it's more like 600k cells (2 columns per row), and it's
less than that insofar as some users may only
have 24 distinct colors among their many streams.
It's a lot of work.
2021-12-28 12:15:02 -08:00
Steve Howell f638fd6f72 performance: Get used stream colors in separate trip.
This commit sets us up for the next commit, which will
save us a very expensive query.

If you are adding 15k users to a stream, and each user
has about 20 existing streams, then we need to retrieve
300k rows from the database to figure out which stream
colors they already have.  We don't need all the extra
fields from Subscription, so now we get just the two
values we need for making a color map.

In the next commit we'll eliminate the other use case
for the big query, and I will explain in greater
depth how splitting out the color-picking code can
be a huge win. It is possible that some product decisions
could make this codepath easier. We could also do some
engineering specific to stream colors, such as caching
which colors users have already used.

This does cost us an extra round trip to the database.
2021-12-28 12:15:02 -08:00
Steve Howell 56da570422 code cleanup: Remove unused parameter in pick_color. 2021-12-28 12:15:02 -08:00
YashRE42 5e194afda3 reactions: Destructure during make..reaction call from set..reaction.
This is a pure code refactor for readability.
2021-12-28 11:12:02 -08:00
YashRE42 6fa92cce7e reactions: Use parameter destructuring in make_clean_reactions.
This is a pure code refactor for readability.
2021-12-28 11:11:57 -08:00