Commit Graph

44374 Commits

Author SHA1 Message Date
Alex Vandiver d4f6b93859 scheduled_email: Create ScheduledEmail objects in a transaction.
This fixes two bugs: the most obvious is that there is a race where a
ScheduledEmail object could be observed in the window between creation
and when users are added; this is a momentary instance when the object
has no users, but one that will resolve itself.

The more subtle is that .save() will, if no records were found to be
updated, _re-create_ the object as it exists in memory, using an
INSERT[1].  Thus, there is a race with `deliver_scheduled_emails`
between when the users are added, and when `email.save()` runs:

 1. Web request creates ScheduledEmail object
 2. Web request creates ScheduledEmailUsers object
 3. deliver_scheduled_emails locks the former, preventing updates.
 4. deliver_scheduled_emails deletes both objects, commits, releasing lock
 5. Web request calls `email.save()`; UPDATE finds no rows, so it
    re-creates the ScheduledEmail object.
 6. Future deliver_scheduled_emails runs find a ScheduledEmail with no
    attending ScheduledEmailUsers objects

Wrapping the logical creation of both of these in a single transaction
avoids both of these races.

[1] https://docs.djangoproject.com/en/3.2/ref/models/instances/#how-django-knows-to-update-vs-insert
2021-08-19 00:44:33 -07:00
Alex Vandiver 4c518c2bba scheduled_email: Consistently lock users table.
Only clear_scheduled_emails previously took a lock on the users before
removing them; make deliver_scheduled_emails do so as well, by using
prefetch_related to ensure that the table appears in the SELECT.  This
is not necessary for correctness, since all accesses of
ScheduledEmailUser first access the ScheduledEmail and lock it; it is
merely for consistency.

Since SELECT ... FOR UPDATE takes an UPDATE lock on all tables
mentioned in the SELECT, merely doing the prefetch is sufficient to
lock both tables; no `on=(...)` is needed to `select_for_update`.

This also does not address the pre-existing potential deadlock from
these two use cases, where both try to lock the same ScheduledEmail
rows in opposite orders.
2021-08-19 00:44:33 -07:00
Alex Vandiver ebaafb32f3 send_email: Change clear_scheduled_emails to only take one user.
No codepath except tests passes in more than one user_profile -- and
doing so is what makes the deduplication necessary.

Simplify the API by making it only take one user_profile id.
2021-08-19 00:44:33 -07:00
Suyash Vardhan Mathur 6fbe7ad61e apidocs: Remove automatically added : at end of response. 2021-08-17 06:25:02 -07:00
Eeshan Garg 190ce61eed landing-page: Fix mobile scrolling bug when sidebar is toggled.
On mobile, when the sidebar is toggled, the following three issues
are encountered:
- When none of the sidebar menus are expanded, the sidebar has no
  scrollbar, which is expected. But if you scroll, the background
  content scrolls, which is a bug.
- When some of the sidebar menus are expanded such that the content
  overflows and is "scrollable", once you get to the end of the
  sidebar content, the background content keeps scrolling in a weird
  way.
- If the mobile screen is wide enough, if you scroll the sidebar
  content, it scrolls as expected. But if you move the pointer to
  the side of the background content that is still visible, you
  can scroll the background content even though it should be fixed.

This commit fixes all of the above issues.
2021-08-17 06:23:46 -07:00
Abhijeet Prasad Bodas ac70a2d2e1 notifications: Fix unnecessary wildcard mention notifications.
This fixes a bug where email notifications were sent for wildcard
mentions even if the `enable_offline_email_notifications` setting was
turned off.
This was because the `notification_data` class incorrectly considered
`wildcard_mentions_notify` as an indeoendent setting, instead of a wrapper
around `enable_offline_email_notifications` and `enable_offline_push_notifications`.

Also add a test for this case.
2021-08-13 09:48:18 -07:00
Alex Vandiver ec354ee013 changelog: Add 2.1.8 release notes. 2021-08-12 01:06:15 +00:00
Alex Vandiver fc9e5f5187 github: Use logging.exception so we know which exception happened. 2021-08-10 22:38:18 -07:00
Anders Kaseorg 2bfe8d5c90 node_tests: Fix incorrect usage of assert.throws.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-10 13:18:53 -07:00
PIG208 c99c423c81 support: Remove `is True` in comparisons.
This is a followup for #19432 that fixes a remaining issue of the
original PR.
2021-08-10 12:08:13 -07:00
Anders Kaseorg 1bdb7b1141 mypy: Add boto3-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg bfdb2f4628 export: Fix error message generation in _check_key_metadata.
There is no key.name.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 05124737f7 mypy: Add backoff-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 09564e95ac mypy: Add types-psycopg2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 5c90522e69 mypy: Add types-Pillow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg b798b34b23 generate_integration_bots_avatars: Fix transparency initialization.
None results in an uninitialized image (that happens to be transparent
most of the time); we want to explicitly initialize the image to
transparent.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 14f0594795 upload: Replace exif_rotate with Pillow exif_transpose.
Fixes #18599.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 6289803368 mypy: Add types-oauthlib.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 4c73926ad9 mypy: Add types-jsonschema.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 58cfc152a0 mypy: Add types-commonmark.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Priyansh Garg 54452fef6c data_import: Fix channel mentions in Rocket.Chat import.
While the STREAM_LINK_REGEX and STREAM_TOPIC_LINK_REGEX
identifies the stream and topic mentions in the content
correctly (tested by printing out the matches), the
stream/topic mentions are still not linked to the
corresponding streams/topics for imported messages, as
a `zulip_message` instance is required for linking these
mentions to actual streams/topics (see `StreamPattern`
class in `markdown/__init__.py`) which is not provided
while processing the markdown for imported messages.
2021-08-09 06:38:26 -07:00
Priyansh Garg aed4e48da7 data_import: Import attachments from Rocket.Chat. 2021-08-09 06:38:26 -07:00
Priyansh Garg 65e28907cb data_import: Import custom emoji from Rocket.Chat. 2021-08-09 06:38:26 -07:00
Tim Abbott a7890f046b api: Fix tests for avatar_url. 2021-08-09 02:15:55 -07:00
PIG208 a583e9ffad webhooks: Refactor alertmanager to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 4905d73fe4 auth: Refactor oauth_redirect_to_root to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 bf86649f4d billing: Refactor initial_upgrade to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 6a61dc2de2 billing: Refactor billing_home to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 0dac524ea4 registration: Refactor view functions in registration.py to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 15eeb2cb25 message: Refactor send_message_backend to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 94685e1afb analytics: Refactor the support view to use REQ. 2021-08-08 17:11:18 -07:00
Tim Abbott 6e335b8c7b api docs: Fix unexpected editor substitution in changelog. 2021-08-08 17:10:54 -07:00
Gaurav Pandey 067a32925d api: Default `client_gravatar` to `true`.
Default `client_gravatar` to `true` for all endpoints passing the
parameter to avoid extra payload.

Fixes #11214.
2021-08-08 16:53:05 -07:00
Sahil Batra 5459a92e4a setting: Use "unlimited" instead of "forever" for retention setting.
This commit updates both the stream-level and realm-level message
retention setting to use 'unlimited' instead of 'forever' to set
message retention setting to "retain messages forever".
2021-08-08 15:56:57 -07:00
Sahil Batra 524b177719 api: Remove incorrect realm setting in 'realm/update_dict' schema.
We incorrectly include many realm settings in the data section of
'realm/update_dict' schema. It should only contain the settings
related to message edit, realm icon, realm logo and authentication
methods and not other settings, becausea all the other settings send
'realm/update' event and not 'realm/update_dict' event.

This commit only removes 'message_retention_days' and others will
be removed separately.
2021-08-08 15:09:47 -07:00
Sahil Batra 0e5b2326b2 stream settings: Rearrange personal stream settings.
We rearrange the personal stream settings to show all the
notification settings together under a separate heading.

Fixes #19508.
2021-08-08 15:05:56 -07:00
Sahil Batra 4deb9dc834 stream settings: Remove border around personal stream settings. 2021-08-08 15:05:56 -07:00
akshatdalton 1a63c2d187 notification: Fix "narrow to..." link present above composebox.
Earlier copy to the link present above the composebox (which
appears when a message is sent outside the current narrow), it
used to provide the link to the home page which is fixed now.
Now, copy to the link gives the link of the narrow where message
is sent.
2021-08-07 09:51:04 -07:00
Ganesh Pawar 378395bcc4 user_status: Show change status modal when clicked on emoji in buddy list.
Fixes #19506
2021-08-07 07:19:03 -07:00
Ganesh Pawar c6368f5638 buddy_list: Add a self class to the current user entry. 2021-08-07 07:19:03 -07:00
Anders Kaseorg dd2f979541 templates: Add missing tabnabbing protection for target="_blank" links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-07 06:25:30 -07:00
Alya Abbott 78184c5bc5 portico: Add comparison checkboxes to /for/education.
On narrow width screens, user can now scroll the table,
instead of table collapsing all the text together making
it look weird.
2021-08-07 06:23:31 -07:00
Eeshan Garg f2d58f1d5d landing-page: Fix flickering sidebar transition upon resizing.
When one resizes the window and tries to switch to the vertical
sidebar menu, the CSS transition flickers in and out. This is
less than ideal.

This commit implements a solution to this problem. The solution is
inspired by the following helpful article:

https://ishadeed.com/article/layout-flickering/
2021-08-07 06:22:29 -07:00
Eeshan Garg 132498894f header: Stop trying to detect if touch is enabled for click events.
We found a bug in our top-level nav on certain Samsung phones where
tapping on the dropdowns didn't do anything. As it turns out, trying
to figure out whether touch is enabled on a device using CSS media
queries such as `(hover: none)` is very tricky business! While it
may work on some devices, it may break spectacularly on others.

This commit presents a potential solution to this. Media queries
about widths are a lot more device-independent than hover media
queries. So, this is what we do now:

- We fire click events regardless of whether the top-level nav is in
  the vertical sidebar orientation or the horizontal orientation.
  Since we are no longer dependent on hover media queries for click
  events, this fixes the mobile bug mentioned above because taps or
  clicks will always work regardless of device.
- The mouseover/mouseout events are only fired when the header is
  in the horizontal orientation and the primary input mechanism
  supports hovering over elements. This allows us to support hovering
  over menus on desktop. However, since mouseout/mouseover events are
  irrelevant for mobile, we are fine on that front since clicks/taps
  will still work.

NOTE that the above approach also takes care of weird edge cases where
you have a horizontal orientation on a device such as the iPad Pro that
is flipped in the horizontal position. Since clicks work anyway, these
edge cases are largely taken care of.
2021-08-07 06:22:29 -07:00
Eeshan Garg 13b0114bcc landing_page: Switch to sidebar menu earlier at width <= 1024px.
Right now, some of our horizontally spaced out top-level dropdowns
(such as "Solutions") get truncated on screen widths less than
1024 px. We switch to the vertical sidebar menu at widths less than
686px.

Looking at a lot of mainstream websites, a few of them switch to a
vertical sidebar menu on much wider widths than we do. Plus,
switching to a vertical orientation is a much cleaner way to fix
this issue than playing around with legacy Bootstrap code. Therefore,
we should toggle on the sidebar menu at width <= 1024px.
2021-08-07 06:22:29 -07:00
Anders Kaseorg b44e6123fe gear_menu: Fix template variables passed from page_params.
Commit 9049fb3bd4 (#19176) broke these
by changing {page_params} to {...page_params}.  We could change it
back, but it’s better to be explicit about which items we use from
page_params.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-07 06:19:43 -07:00
Priyam Seth 5a4318d046 compose: Fix compose fade not updating on deleting user pill.
This commit makes the compose.update_fade as an exported
function and adds the listener to update the fade manually
as deleting the pill was not calling this functions.

Fixes #18865
2021-08-06 14:22:55 -07:00
Mateusz Mandera da815db372 test_external: Extract rate_limit_rule() decorator. 2021-08-06 12:17:44 +02:00
Mateusz Mandera ddcfd9e2ee rate_limit: Rate limit the /accounts/find/ endpoint.
Closes #19287

This endpoint allows submitting multiple addresses so we need to "weigh"
the rate limit more heavily the more emails are submitted. Clearly e.g.
a request triggering emails to 2 addresses should weigh twice as much as
a request doing that for just 1 address.
2021-08-06 12:17:44 +02:00
Mateusz Mandera 29b3e81dd4 test_external: Assert that requests don't get rate limited too soon.
This assertion was missing in do_test_hit_ratelimits, allowing a test to
pass even if the requests started being blocked sooner than expected.
2021-08-06 12:17:44 +02:00