Commit Graph

45693 Commits

Author SHA1 Message Date
Alex Vandiver 51d3862c7e puppet: Move wal-g to external_dep, in /srv/zulip-wal-g-*. 2021-12-29 16:35:15 -08:00
Alex Vandiver 1e4e6a09af puppet: Stop making resources for external binaries and directories.
In the event that extracting doesn't produce the binary we expected it
to, all this will do is create an _empty_ file where we expect the
binary to be.  This will likely muddle debugging.

Since the only reason the resourfce was made in the first place was to
make dependencies clear, switch to depending on the External_Dep
itself, when such a dependency is needed.
2021-12-29 16:35:15 -08:00
Alex Vandiver 3c163a7d5e puppet: Move slash out of $dir by convention. 2021-12-29 16:35:15 -08:00
Alex Vandiver bb5a2c8138 puppet: Move prometheus to external_dep. 2021-12-29 16:35:15 -08:00
Alex Vandiver 2d6c096904 puppet: Move node_exporter to external_dep. 2021-12-29 16:35:15 -08:00
Alex Vandiver d2a78bac7e puppet: Adjust wal-g release version and SHA256.
wal-g apparently removed the 1.1.1 release; replace it with the
equivalent rc.
2021-12-29 16:35:15 -08:00
Sahil Batra a59d429045 settings: Fix code for special case of theme settings subsection.
We handle "Theme settings" subsection separately in
get_subsection_property_elements as it contains unique
radio-button structure for emojiset setting.

This should have been fixed while reorganizing the section
to have color scheme and emoji related settings under same
subsection in adb612a0b4.

Fixes #20644.
2021-12-29 16:32:12 -08:00
parth 4edf029ad5 invitations: Don't notify now-deactivated users.
While accepting an invitation from a user, there was no condition in
place to check if the user sending the invitation was now
now-deactivated.

Skip sending notifications about newly-joined users to users who are
now disabled.

Fixes #18569.
2021-12-29 16:21:19 -08:00
Steve Howell 1e4593b2ae performance: Avoid Recipient lookup.
We don't have to go to the database to get the Recipient
fields for `user_profile.recipient`.

See also 85ed6f332a from a little
over a year ago--it's very similar.
2021-12-28 12:15:02 -08:00
Steve Howell 01ebb2c85f refactor: Pass realm to bulk_remove_subscriptions.
We made a very similar change to bulk_add_subscriptions
earlier in the year.
2021-12-28 12:15:02 -08:00
Steve Howell ebbd5f168b refactor: Pass realm to notify_subscriptions_removed. 2021-12-28 12:15:02 -08:00
Steve Howell 966d88a78a stream colors: Fix stream color assignment.
The bug here probably didn't come up too much in
practice, but if we were adding a user to multiple
streams when they already had used all N available
colors, all the new streams would be assigned the same
color, since the size of used_colors would stay at N,
thwarting our little modulo-len hackery.

It's not a terrible bug, since users can obviously
customize their stream colors as they see fit.

Usually when we are adding a user to multiple streams,
the users are fairly new, and thus don't have many
existing streams, so I have never heard this bug
reported in the field.

Anyway, assigning the colors in bulk seems to make more
sense, and I added some tests.

For the situations where all the colors have already
been used, I didn't put a ton of thought into exactly
which repeated colors we want to choose; instead, I
just ensure they're different modulo 24. It's possible
that we should just have more than 24 canned colors, or
we should just assign the same default color every time
and let users change it themselves (once they've gone
beyond the 24, to be clear). Or maybe we can just do
something smarter here. I don't have enough time for a
deep dive on this issue.
2021-12-28 12:15:02 -08:00
Steve Howell fe3295d395 performance: Avoid monster query for existing subs.
Part of our codepath for subscribing users involves
fetching the users' existing subscriptions to make sure
we can do things like properly report to the clients
that the users were already subscribed.  This codepath
used to be coupled to code that helped users maintain
unique stream colors.

Suppose you are creating a new stream, and you are
importing users from an older stream with 15k
subscribers, and each of your users is subscribed to
about 20 streams.

The prior code, instead of filtering on recipient_id,
would literally look at every subscription for every
user, which was kind of crazy if you didn't understand
the pick-stream-color complications.

Before this commit, we would fetch 300k rows with 15
columns each (granted, all but one of the columns are
bool/int). That's a total of 4.5 million tiny objects
that we had to glom into Django ORM objects and slice
and dice.

After this commit, we would fetch exactly zero rows
for the are-they-already-subscribed logic.

Yes, ZERO.

If we were to mistakenly try to re-add the same 15k
subscribers to the new stream (under the new code), we
will now fetch 15k Sub rows instead of 300k.

It is worth looking at the prior commit. We go through
great pains to ensure that users get new stream colors
when we invite them to a stream, and we still fetch a
bunch of data for that. Instead of 4.5 million cells,
it's more like 600k cells (2 columns per row), and it's
less than that insofar as some users may only
have 24 distinct colors among their many streams.
It's a lot of work.
2021-12-28 12:15:02 -08:00
Steve Howell f638fd6f72 performance: Get used stream colors in separate trip.
This commit sets us up for the next commit, which will
save us a very expensive query.

If you are adding 15k users to a stream, and each user
has about 20 existing streams, then we need to retrieve
300k rows from the database to figure out which stream
colors they already have.  We don't need all the extra
fields from Subscription, so now we get just the two
values we need for making a color map.

In the next commit we'll eliminate the other use case
for the big query, and I will explain in greater
depth how splitting out the color-picking code can
be a huge win. It is possible that some product decisions
could make this codepath easier. We could also do some
engineering specific to stream colors, such as caching
which colors users have already used.

This does cost us an extra round trip to the database.
2021-12-28 12:15:02 -08:00
Steve Howell 56da570422 code cleanup: Remove unused parameter in pick_color. 2021-12-28 12:15:02 -08:00
YashRE42 5e194afda3 reactions: Destructure during make..reaction call from set..reaction.
This is a pure code refactor for readability.
2021-12-28 11:12:02 -08:00
YashRE42 6fa92cce7e reactions: Use parameter destructuring in make_clean_reactions.
This is a pure code refactor for readability.
2021-12-28 11:11:57 -08:00
YashRE42 06d8e6dfd5 reactions: Refactor add_clean_reactions to make_clear_reaction.
This is a pure code refactor for readability.

Previously, we were relying on there being a side effect to
add_clean_reaction which was necessitated by the presence of an output
parameter, `message` (or more specifically `message.clean_reaction`).
Output parameters are confusing.
Hence, this commit changes to have a make_clean_reaction function that
returns a reaction.
2021-12-28 11:11:51 -08:00
YashRE42 0b0637de78 reactions: Remove `export` from add_clean_reaction() function.
This function is not called by any external files or tests.
2021-12-28 11:11:43 -08:00
YashRE42 1dffad98fa reactions: Use parameter destructuring in insert_new_reaction.
This is a pure code refactor for readability.
2021-12-28 11:11:39 -08:00
YashRE42 0438ed5777 reactions: Use parameter destructuring in update_existing_reaction.
This is a pure code refactor for readability.
2021-12-28 11:11:37 -08:00
YashRE42 0991ed4526 reactions: Use parameter destructuring in view.remove_reaction.
This is a pure code refactor for readability.
2021-12-28 11:11:31 -08:00
Abhijeet Prasad Bodas aa18e797a8 test_event_queue: Generalize some helpers.
This will later allow us to also use these when
writing new tests for bots.
2021-12-28 10:59:04 -08:00
Abhijeet Prasad Bodas acdce4df47 actions: Fix misleading comment about wildcard mentions.
Having the `wildcard_mentions_notify` setting turned on does
not necessarily mean that the user will receive notification
for that message. There is more nuance to this, as explained
in the updated comment.
2021-12-28 10:58:54 -08:00
Eeshan Garg 2393342e03 webhooks/jira: Handle anomalous payloads properly.
We recently ran into a payload in production that didn't contain
an event type at all. A payload where we can't figure out the event
type is quite rare. Instead of letting these payloads run amok, we
should raise a more informative exception for such unusual payloads.
If we encounter too many of these, then we can choose to conduct a
deeper investigation on a case-by-case basis.

With some changes by Tim Abbott.
2021-12-28 10:56:25 -08:00
Mateusz Mandera c5c3ab66d6 remote_server: Migrate RemoteZulipServer.uuid to be UUIDField.
Given that these values are uuids, it's better to use UUIDField which is
meant for exactly that, rather than an arbitrary CharField.

This requires modifying some tests to use valid uuids.
2021-12-28 10:11:34 -08:00
Mateusz Mandera e48120fd12 remote_server: Validate zulip_org_id submitted by registering server.
zulip_org_id is supposed to be a UUID, so we want to actually validate
the format, not only check the length.
2021-12-28 10:11:34 -08:00
BIKI DAS 42dd58cffe
docs: Fix a few typos in documentation. 2021-12-28 09:36:59 -08:00
Steve Howell d62b39450e performance: Optimize send_subscription_add_events.
We avoid repeating the same calculations over and
over again for the same stream.

This helps, but the real bottleneck in this function
is that send_event usually takes at least a millisecond,
and that adds up quickly if you're doing something
like subscribing 5k users to a new stream.
2021-12-28 09:33:16 -08:00
Anders Kaseorg bc69f213a0 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 60eed65832 scim: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg c8dd90f32b bot_config: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 575932f4e0 actions: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 95cddff39b test_scim: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg b73a6b7b06 test_stripe: What even is how to code I don’t know help.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg f45b245f74 test_urls: Fix get_callback_string logic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 48190cf744 test_timezone: Fix ambiguous_abbrevs type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg c4c28e06d9 test_openapi: Replace convert_regex_to_url_pattern.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg e3a8f992d5 test_openapi: Fix __wrapped__ accesses.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg d40f3d54f1 test_console_output: Implement the entire TextIO contract.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 702ce071f4 python: Accept Optional[FrameType] in signal handlers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Alex Vandiver 7a9074ecfd puppet: Use shorter local variable for supervisor conf.d dir. 2021-12-28 09:24:01 -08:00
Alex Vandiver 670fad0cc4 puppet: Drop now-unnecessary supervisor file removals. 2021-12-28 09:24:01 -08:00
BIKI DAS d7f5d041b9
typo: Rename default_hander to default_handler. 2021-12-28 09:19:57 -08:00
BIKI DAS c1134a8bda
docs: Fix "should should" typo. 2021-12-28 09:19:04 -08:00
Anders Kaseorg 2ff3b51e03 test-documentation: Suppress Unicode Private Use Area validator warning.
sphinx_rtd_theme references the Font Awesome fa-link icon directly
with U+F0C1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 16:55:31 -08:00
Anders Kaseorg 591bd3f4a1 webhooks: Rename Yo App to Yo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 1d3520db12 webhooks: Remove space from UptimeRobot.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 68c99511a2 webhooks: Fix TeamCity capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 65868b09eb webhooks: Add missing space in Review Board.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00