Commit Graph

6949 Commits

Author SHA1 Message Date
Alex Vandiver 915c2b2fd9 muting: Fix a race in topic unmuting.
Rather than check if the topic exists and then try to delete it, just
try to delete it, and catch the lack of matching rows.
2022-01-18 14:15:06 -08:00
Mateusz Mandera f8b06ed952 events: Send invites_changed event if user deactivation revokes invites.
revoke_invites_generated_by_user should send invites_changed event if it
actually revokes some invitations. This is called in the user
deactivatoin codepath.
2022-01-18 14:12:55 -08:00
Mateusz Mandera 74d2aea76a apply_events: Update state["subscribers"] upon "remove user" event.
Event of type "realm_user", op "remove", emitted by do_deactivate_user
should remove the user id from subscriptions in the state. We weren't
catching this bug, because test_do_deactivate_bot uses a newly created
bot, so no stream subscriptions are affected. The bug shows up if
deactivating e.g. cordelia - thus we want to have two tests instead,
one for testing bot deactivation and one for user deactivation.
2022-01-18 14:12:55 -08:00
Steve Howell dd1c9c45c7 stream colors: Try harder to avoid collisions.
We now use recipient_id % 24 for new stream colors
when users have already used all 24 of our canned
colors.

This fix doesn't address the scenario that somebody
dislikes one of our current canned colors, so if a
user continually changes canned color N to some other
color for new streams, their new streams will continue
to include color N (and the user will still need to
change them).

This fix doesn't address the fact that it can be expensive
during bulk-add situations to query for all the colors
that users have already used up.

See https://chat.zulip.org/#narrow/stream/3-backend/topic/assigning.20stream.20colors
for more discussion.
2022-01-18 13:56:54 -08:00
Sahil Batra 06cba4ae1f actions: Use transaction.atomic in do_change_bot_owner. 2022-01-18 12:43:04 -08:00
Sahil Batra 7c44151135 actions: Use transaction.atomic in do_change_tos_version. 2022-01-18 12:43:04 -08:00
Sahil Batra 06d715a41d actions: Use transaction.atomic in do_change_icon_source. 2022-01-18 12:43:04 -08:00
Sahil Batra 64d1dc6525 actions: Use transaction.atomic in do_change_logo_source. 2022-01-18 12:43:04 -08:00
Sahil Batra 8945a64024 actions: Use transaction.atomic in do_change_realm_org_type. 2022-01-18 12:43:04 -08:00
Sahil Batra c8f81ded4e actions: Use transaction.atomic in do_change_default_sending_stream. 2022-01-18 12:43:04 -08:00
Sahil Batra cb43bdab93 actions: Use transaction.atomic for do_change_default_all_public_streams. 2022-01-18 12:43:04 -08:00
Sahil Batra 4a7461361e actions: Use transaction.atomic for do_change_default_events_register_stream. 2022-01-18 12:43:04 -08:00
Sahil Batra 5c758af3b4 actions: Use transaction.atomic for do_change_user_setting. 2022-01-18 12:43:04 -08:00
Alex Vandiver 19f891968d markdown: Increase the maximum number of image previews per message.
The limit here is purely to prevent breakage in case of a pathological
number of images in a single message; 5 images is entirely possible in
a reasonable message, and causes user confusion when they are not
expended.

Increase the limit to 10 per message.
2022-01-14 11:30:07 -08:00
Anders Kaseorg bea64f8608 python: Convert deprecated Django ugettext alias to gettext again.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Commit e7ed907cf6 (#18174) fixed this
before, but new instances have been added.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:28:12 -08:00
Anders Kaseorg 78e54a0d7a python: Replace deprecated jinja2.utils.Markup with markupsafe.Markup.
Fixes “DeprecationWarning: 'jinja2.Markup' is deprecated and will be
removed in Jinja 3.1. Import 'markupsafe.Markup' instead.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:22:48 -08:00
Sahil Batra 9b8713fc1e users: Send peer_add subscription events on reactivating users.
The subscriber list was not updating without a refresh on
reactivating user, because the subscriptions data with the
client was not updated on reactivation.

This commit adds code to send peer_add subscription events
on reactivating the user.

We do not send peer_remove events on deactivating the user,
but the subscriber list is still live-updated because we
have the data of the streams which the deactivated user is
susbcribed to and the clients itself updates the data and UI
on receiving event of deactivation of user, which it is not
possible when reactivating the user.

Fixes #20383.
2022-01-12 14:30:21 -08:00
Mateusz Mandera 93e8740218 do_deactivate_user: Revoke invitations generated by the user.
Leaving old invitations valid, potentially for a very long time, is
clearly unexpected and undesired behavior under normal circumstances. A
user shouldn't be able to e.g. generate a multiuse invite link, get
banned from the organization by being deactivated and then just re-join
using the link they've created for themselves.
2022-01-12 13:53:34 -08:00
Mateusz Mandera 76f1e902a6 notify_invites_changed: Fix passing of deleted objects to the function.
do_revoke_user_invite and do_revoke_multi_use_invite were using objects
after their deletion to pass the argument to notify_invites_changed. We
should avoid that. The function was only using the .realm attribute of
the received objects, so it's simpler to make it just take realm as its
argument.
2022-01-12 13:53:34 -08:00
Mateusz Mandera ff688c3a8d actions: Give do_get_user_invites a more specific name.
The added docstrings elaborates on why the new name is more appropriate.
2022-01-12 13:53:34 -08:00
BIKI DAS fb1ff71b2e
python: Remove duplicates of global imports inside functions.
Neither of these appears to have been intentional.
2022-01-12 13:50:12 -08:00
Anders Kaseorg 96159688c2 test_helpers: Fix exception safety of stdout_suppressed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-11 19:38:36 -08:00
Alex Vandiver 1f9373585a string_validation: Use enumerate rather than `range(len(...))`. 2022-01-11 15:59:50 -08:00
Alex Vandiver ce09c8b65f string_validation: Make `unicode_non_chars` a set, for efficiency. 2022-01-11 15:59:38 -08:00
Alex Vandiver df50280c54 string_validation: Loosen to allow some `Cn` unicode characters.
Under the unicodedata distributed with Python 3.6, some Emoji are
classified as `Cn`, and not `So`:

```
$ unicode 1f929 --long
U+1F929 GRINNING FACE WITH STAR EYES
UTF-8: f0 9f a4 a9 UTF-16BE: d83edd29 Decimal: &#129321; Octal: \0374451
🤩
Category: So (Symbol, Other); East Asian width: W (wide)
Unicode block: 1F900..1F9FF; Supplemental Symbols and Pictographs
Bidi: ON (Other Neutrals)

$ python3.6 -c 'import unicodedata; print(unicodedata.category("\U0001f929"))'
Cn

$ python3.7 -c 'import unicodedata; print(unicodedata.category("\U0001f929"))'
So
```

Drop `Cn` from the list of excluded Unicode character classes, and
replace it with an explicit list of the 66 non-characters, which are
invariant.

Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Alex Vandiver eb872b5bcd actions: Use check_stream_topic when editing message topics. 2022-01-11 15:17:53 -08:00
Alex Vandiver 4f482c234c string_validation: Standardize missing topic with missing stream name.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Alex Vandiver 58c8eebda2 string_validation: Make check_stream_topic merely check, not alter.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Alex Vandiver 1cdb93f6aa string_validation: Factor out topic validation.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Alex Vandiver 94dbb540b1 string_validation: Give a more specific message for empty stream names.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Alex Vandiver 3574637fbf string_validation: Factor out stream name validation.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Eeshan Garg d6b92092dd streams: Add RealmAuditLog entries for post policy changes. 2022-01-10 18:29:04 -08:00
Eeshan Garg c30458e174 streams: Add notifications for posting policy changes.
An explanatory note on the changes in zulip.yaml and
curl_param_value_generators is warranted here. In our automated
tests for our curl examples, the test for the API endpoint that
changes the posting permissions of a stream comes before our
existing curl test for adding message reactions.

Since there is an extra notification message due to the change in
posting permissions, the message IDs used in tests that come after
need to be incremented by 1.

This is a part of #20289.
2022-01-10 18:29:04 -08:00
Eeshan Garg 625af3cea9 streams: Add extra line break to description change notification.
The extra line break above "Old description:" aids readability.
2022-01-10 11:36:19 -08:00
Eeshan Garg f97093ba32 streams: Add RealmAuditLog entries for description changes. 2022-01-07 16:13:11 -08:00
Eeshan Garg 80f30f187e streams: Add notifications for description changes.
This is a part of #20289.
2022-01-07 16:13:11 -08:00
Eeshan Garg 1b303e7b2f signup: Stop prompting self-hosted customers for marketing emails.
Fixes #20595.
2022-01-06 13:22:50 -08:00
Mateusz Mandera 30ccb76e19 do_delete_user: Preserve date_joined value of the user. 2022-01-04 15:42:03 -08:00
Mateusz Mandera 444bb6d0e9 do_delete_user: Create RealmAuditLog entries. 2022-01-04 15:42:03 -08:00
Mateusz Mandera 208c0c3034 do_delete_user: Use get_fake_email_domain for dummy user email domain.
Otherwise the dummy user can be created with an invalid email domain -
e.g. in development environment with the domain
"@http://localhost:9991". get_fake_email_domain exists exactly for
handling these kinds of scenarios.
2022-01-04 15:42:03 -08:00
Mateusz Mandera dffdeb48e7 do_delete_user: Make the replacement dummy user inactive.
Otherwise, the dummy user will show up in the user list in the right
sidebar.
2022-01-04 15:42:03 -08:00
Alex Vandiver fc13dd6f3d user_groups: Don't use access_user_group_by_id for notifications.
Stop using `access_user_group_by_id` in notifications codepaths, as it
is meant to be used to check for _write_ access, not read
access (which is not limited).  In the notification codepaths, there
are no ACLs to apply, and the ID is known-good; just load it
directly. The `for_mention` flag is removed, as it was not used in the
mention codepaths at all, only the notification ones.
2022-01-04 14:45:04 -08:00
Abhijeet Prasad Bodas 15e8717847 notifications: Don't enqueue notifications for bots.
This replaces the temporary (and testless) fix in
24b1439e93 with a more permanent
fix.

Instead of checking if the user is a bot just before
sending the notifications, we now just don't enqueue
notifications for bots. This is done by sending a list
of bot IDs to the event_queue code, just like other
lists which are used for creating NotificationData objects.

Credit @andersk for the test code in `test_notification_data.py`.
2022-01-03 09:55:06 -08:00
Steve Howell a9271e7a99 performance: Cache stream lookups in MentionBackend.
This is useful when you subscribe a bunch of folks
to a stream and need to send them all PMs telling
them about the new subscription.
2021-12-30 11:28:15 -08:00
Steve Howell 4adcaf92f7 refactor: Attach get_stream_name_map to MentionData.
This diff looks slightly noisy, but the main chunk of
code that we moved here has the same logic as before,
and it just gets realm_id from MentionBackend now, instead
of having our markdown processor have to supply it.

We basically want MentionData to be the gatekeeper of
mention data, and then we delegate backend tasks to
MentionBackend.

Soon we will add a cache to MentionBacked, which will
justify this change a bit more.
2021-12-30 11:28:15 -08:00
Steve Howell 0359c083d1 refactor: Extract get_linkable_streams.
This is a one-liner with two purposes:

    * We want the comment to explain the business rule.

    * We want to just work in id space.
2021-12-30 11:28:15 -08:00
Steve Howell c4bd4496dd peformance: Cache user mentions for multiple PMs.
It's slightly annoying to plumb Optional[MentionBackend]
down the stack, but it's a one-time change.

I tried to make the cache code relatively unobtrusive
for the single-message use case.

We should be able to eliminate redundant stream queries
using similar techniques.

I considered caching at the level of rendering the message
itself, but this involves nearly as much plumbing, and
you have to account for the fact that several users on
your realm may have distinct default languages (French,
Spanish, Russian, etc.), so you would not eliminate as
many query hops. Also, if multiple streams were involved,
users would get slightly different messages based on
their prior subscriptions.
2021-12-30 11:28:15 -08:00
Steve Howell c6448263c3 refactor: Add MentionBackend.
We will eventually use this to avoid redundant
queries.

The diff is slightly noisy here, but there are no
logic changes.
2021-12-30 11:28:15 -08:00
Steve Howell a22f49bf83 refactor: Extract UserFilter.
This is setting us up for future commits.
2021-12-30 11:28:15 -08:00
Steve Howell ea252ab53e refactor: Convert FullNameInfo to a dataclass.
As part of this we no longer query for email, which
is a vestige of when we used emails to identify users
on the frontend.
2021-12-30 11:28:15 -08:00
Steve Howell f5fc348786 mypy: Add explicit types for dbdata references.
When our handlers specifically reference self.md.zulip_db_data,
we now use an explicit type.

We probably want a more robust solution here, such as a semgrep
rule.
2021-12-30 11:28:15 -08:00
Steve Howell df84892aad markdown: Convert DbData to a dataclass. 2021-12-30 11:28:15 -08:00
Steve Howell 4e551f8279 refactor: Introduce get_stream_name_map.
We only need a name -> id map, and the FullNameInfo
type was a lie.
2021-12-30 11:28:15 -08:00
Steve Howell c04a8097f3 mypy: Add EmojiInfo type.
We now serialize still_url as None for non-animated emojis,
instead of omitting the field. The webapp does proper checks
for falsiness here.  The mobile app does not yet use the field
(to my knowledge).

We bump the API version here. More discussion here:

https://chat.zulip.org/#narrow/stream/378-api-design/topic/still_url/near/1302573
2021-12-30 11:28:14 -08:00
Lauryn Menard a16fcd3172 tests: Improve testing helper event schema for `update_message`.
Further clarifies the fields returned by `update_message` event
for the type of change (content, topic and/or stream).

Follow-up task from #20587.
2021-12-30 08:35:35 -08:00
BIKI DAS ad61d06cea
python: Remove unnecessary list comprehension.
`all` can take a generator, not just a list.  

Using a generator expression here is simpler and faster.
2021-12-30 06:51:50 -08:00
Anders Kaseorg b0b8f84949 test_console_output: Avoid appending to bytes in a loop.
Appending to bytes in a loop leads to a quadratic slowdown since
Python doesn’t optimize this for bytes like it does for str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-29 16:50:08 -08:00
parth 4edf029ad5 invitations: Don't notify now-deactivated users.
While accepting an invitation from a user, there was no condition in
place to check if the user sending the invitation was now
now-deactivated.

Skip sending notifications about newly-joined users to users who are
now disabled.

Fixes #18569.
2021-12-29 16:21:19 -08:00
Steve Howell 1e4593b2ae performance: Avoid Recipient lookup.
We don't have to go to the database to get the Recipient
fields for `user_profile.recipient`.

See also 85ed6f332a from a little
over a year ago--it's very similar.
2021-12-28 12:15:02 -08:00
Steve Howell 01ebb2c85f refactor: Pass realm to bulk_remove_subscriptions.
We made a very similar change to bulk_add_subscriptions
earlier in the year.
2021-12-28 12:15:02 -08:00
Steve Howell ebbd5f168b refactor: Pass realm to notify_subscriptions_removed. 2021-12-28 12:15:02 -08:00
Steve Howell 966d88a78a stream colors: Fix stream color assignment.
The bug here probably didn't come up too much in
practice, but if we were adding a user to multiple
streams when they already had used all N available
colors, all the new streams would be assigned the same
color, since the size of used_colors would stay at N,
thwarting our little modulo-len hackery.

It's not a terrible bug, since users can obviously
customize their stream colors as they see fit.

Usually when we are adding a user to multiple streams,
the users are fairly new, and thus don't have many
existing streams, so I have never heard this bug
reported in the field.

Anyway, assigning the colors in bulk seems to make more
sense, and I added some tests.

For the situations where all the colors have already
been used, I didn't put a ton of thought into exactly
which repeated colors we want to choose; instead, I
just ensure they're different modulo 24. It's possible
that we should just have more than 24 canned colors, or
we should just assign the same default color every time
and let users change it themselves (once they've gone
beyond the 24, to be clear). Or maybe we can just do
something smarter here. I don't have enough time for a
deep dive on this issue.
2021-12-28 12:15:02 -08:00
Steve Howell fe3295d395 performance: Avoid monster query for existing subs.
Part of our codepath for subscribing users involves
fetching the users' existing subscriptions to make sure
we can do things like properly report to the clients
that the users were already subscribed.  This codepath
used to be coupled to code that helped users maintain
unique stream colors.

Suppose you are creating a new stream, and you are
importing users from an older stream with 15k
subscribers, and each of your users is subscribed to
about 20 streams.

The prior code, instead of filtering on recipient_id,
would literally look at every subscription for every
user, which was kind of crazy if you didn't understand
the pick-stream-color complications.

Before this commit, we would fetch 300k rows with 15
columns each (granted, all but one of the columns are
bool/int). That's a total of 4.5 million tiny objects
that we had to glom into Django ORM objects and slice
and dice.

After this commit, we would fetch exactly zero rows
for the are-they-already-subscribed logic.

Yes, ZERO.

If we were to mistakenly try to re-add the same 15k
subscribers to the new stream (under the new code), we
will now fetch 15k Sub rows instead of 300k.

It is worth looking at the prior commit. We go through
great pains to ensure that users get new stream colors
when we invite them to a stream, and we still fetch a
bunch of data for that. Instead of 4.5 million cells,
it's more like 600k cells (2 columns per row), and it's
less than that insofar as some users may only
have 24 distinct colors among their many streams.
It's a lot of work.
2021-12-28 12:15:02 -08:00
Steve Howell f638fd6f72 performance: Get used stream colors in separate trip.
This commit sets us up for the next commit, which will
save us a very expensive query.

If you are adding 15k users to a stream, and each user
has about 20 existing streams, then we need to retrieve
300k rows from the database to figure out which stream
colors they already have.  We don't need all the extra
fields from Subscription, so now we get just the two
values we need for making a color map.

In the next commit we'll eliminate the other use case
for the big query, and I will explain in greater
depth how splitting out the color-picking code can
be a huge win. It is possible that some product decisions
could make this codepath easier. We could also do some
engineering specific to stream colors, such as caching
which colors users have already used.

This does cost us an extra round trip to the database.
2021-12-28 12:15:02 -08:00
Steve Howell 56da570422 code cleanup: Remove unused parameter in pick_color. 2021-12-28 12:15:02 -08:00
Abhijeet Prasad Bodas acdce4df47 actions: Fix misleading comment about wildcard mentions.
Having the `wildcard_mentions_notify` setting turned on does
not necessarily mean that the user will receive notification
for that message. There is more nuance to this, as explained
in the updated comment.
2021-12-28 10:58:54 -08:00
Eeshan Garg 2393342e03 webhooks/jira: Handle anomalous payloads properly.
We recently ran into a payload in production that didn't contain
an event type at all. A payload where we can't figure out the event
type is quite rare. Instead of letting these payloads run amok, we
should raise a more informative exception for such unusual payloads.
If we encounter too many of these, then we can choose to conduct a
deeper investigation on a case-by-case basis.

With some changes by Tim Abbott.
2021-12-28 10:56:25 -08:00
Steve Howell d62b39450e performance: Optimize send_subscription_add_events.
We avoid repeating the same calculations over and
over again for the same stream.

This helps, but the real bottleneck in this function
is that send_event usually takes at least a millisecond,
and that adds up quickly if you're doing something
like subscribing 5k users to a new stream.
2021-12-28 09:33:16 -08:00
Anders Kaseorg bc69f213a0 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 60eed65832 scim: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg c8dd90f32b bot_config: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 575932f4e0 actions: Placate mypy 0.930.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg d40f3d54f1 test_console_output: Implement the entire TextIO contract.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 702ce071f4 python: Accept Optional[FrameType] in signal handlers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 591bd3f4a1 webhooks: Rename Yo App to Yo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 1d3520db12 webhooks: Remove space from UptimeRobot.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 65868b09eb webhooks: Add missing space in Review Board.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg c02c053ec3 webhooks: Fix Mailchimp capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg dc72f79a83 webhooks: Fix Canarytokens pluralization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Shlok Patel b1436aed9c production: Create stream in an atomic transaction.
To avoid the window between stream creation and creation of the
Recipient object, we create the stream in an atomic transaction.

Fixes #20127
2021-12-21 15:45:45 -08:00
Anders Kaseorg dc18aadeb2 test_classes: Type kwargs for client_get and friends.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-17 08:03:52 -08:00
Anders Kaseorg 27977eddeb export: Use tar -C to switch directories.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-17 08:01:53 -08:00
Tim Abbott 22b5e105e6 upload: Remove incorrect animated GIF asserts.
GIF files can be `.GIF`, and also we determine the file format by
inspecting the image data, so there's no reason to have this
assertion.

(The code for serving still images does not rely on the file being a
GIF.)
2021-12-16 16:13:00 -08:00
Sahil Batra 1b23cbdf3e do_change_user_role: Use transaction.atomic. 2021-12-16 14:24:30 -08:00
Sahil Batra 168f241ff0 do_create_user: Use transaction.atomic.
Have kept process_new_human_user out of
the atomic block because it involves many
different operations and also sends events.
Tried enclosing event in on_commit but that
would need many changes in the tests, so have
skipped it for now.
2021-12-16 14:24:30 -08:00
Lauryn Menard 9321095406 tests: Update event tests for `do_update_message` action.
Updates testing helpers in `event_schema.py` for `do_update_message` so
that all stream message fields are present in any edits / updates to
stream messages. Adds verfication tests of events returned from private
message edits and from stream message content-only and topic-only edits.
2021-12-16 11:01:31 -08:00
Lauryn Menard 3b72da8a7c api: Include `stream_id` field for all edits to stream messages.
Updates the `update_message` event type to always include a `stream_id`
field when the message being edited is a stream message. This change
aligns with the current definition of the `\get-events` endpoint
in the OpenAPI documentation.
2021-12-16 11:01:31 -08:00
Tim Abbott af27675857 send_custom_email: Add support for emailing remote server contacts.
This isn't a fully reusable tool, since it has copy about terms of
service, but it's at least readily modified and has tests.
2021-12-14 18:11:23 -08:00
Tim Abbott f287606198 send_custom_email: Make options a mandatory kwarg. 2021-12-14 18:11:23 -08:00
Alex Vandiver 4b1fd209be send_email: Don't abort on an EmailNotDeliveredException.
It is better to press on, than stop halfway through due to a user
whose email no longer works.  The exception is already logged, which
is sufficient here, as this is generally run interactively.
2021-12-14 17:07:34 -08:00
Alex Vandiver 45736aea3c email: Don't send overly-long "To" addresses.
This parallels b7fa41601d, but with "To"
addresses, not "From" addresses.
2021-12-14 15:37:12 -08:00
Steve Howell 53ffb8152f tests: Use read_test_image_file helper. 2021-12-13 12:25:19 -08:00
Mateusz Mandera 74f4e3e914 do_change_realm_subdomain: Use transaction.atomic. 2021-12-11 10:39:07 -08:00
Mateusz Mandera 1692b2e81b do_reactivate_realm: Use transaction.atomic. 2021-12-11 10:39:07 -08:00
Mateusz Mandera 466e0bcdb3 do_set_realm_signup_notifications_stream: Use transaction.atomic. 2021-12-11 10:39:07 -08:00
Mateusz Mandera cddfd2cc92 do_set_realm_notifications_stream: Use transaction.atomic. 2021-12-11 10:39:07 -08:00
Mateusz Mandera 4999f68ba9 do_set_realm_message_editing: Use transaction.atomic. 2021-12-11 10:39:07 -08:00
Mateusz Mandera dc9aac9253 do_set_realm_authentication_methods: Use transaction.atomic. 2021-12-11 10:38:14 -08:00
Tim Abbott 31842e1377 export: Fix empty realm_icons directory in single-user exports. 2021-12-10 12:05:34 -08:00
Steve Howell 2902f8b931 tests: Ensure stream senders get a UserMessage row.
We now complain if a test author sends a stream message
that does not result in the sender getting a
UserMessage row for the message.

This is basically 100% equivalent to complaining that
the author failed to subscribe the sender to the stream
as part of the test setup, as far as I can tell, so the
AssertionError instructs the author to subscribe the
sender to the stream.

We exempt bots from this check, although it is
plausible we should only exempt the system bots like
the notification bot.

I considered auto-subscribing the sender to the stream,
but that can be a little more expensive than the
current check, and we generally want test setup to be
explicit.

If there is some legitimate way than a subscribed human
sender can't get a UserMessage, then we probably want
an explicit test for that, or we may want to change the
backend to just write a UserMessage row in that
hypothetical situation.

For most tests, including almost all the ones fixed
here, the author just wants their test setup to
realistically reflect normal operation, and often devs
may not realize that Cordelia is not subscribed to
Denmark or not realize that Hamlet is not subscribed to
Scotland.

Some of us don't remember our Shakespeare from high
school, and our stream subscriptions don't even
necessarily reflect which countries the Bard placed his
characters in.

There may also be some legitimate use case where an
author wants to simulate sending a message to an
unsubscribed stream, but for those edge cases, they can
always set allow_unsubscribed_sender to True.
2021-12-10 09:40:04 -08:00
Tim Abbott 4cb189fc63 settings: Rename TOS_VERSION to TERMS_OF_SERVICE_VERSION.
The previous version was appropriate in a setting where it was only
used for Zulip Cloud, but it's definitely clearer to spell it out.
2021-12-09 17:51:16 -08:00
odunybrad 90aa45a316 emoji: Add database-level uniqueness constraint for RealmEmoji.
While races here are unlikely, it is most correct to enforce this
invariant at the database layer, and having a database-level
constraint makes the models file a bit more readable.
2021-12-09 17:48:53 -08:00
Steve Howell 9a39ca217f user export: Show less info for recipients.
For PM and huddles, show full names but no
emails or other crufty fields.
2021-12-09 17:20:01 -08:00
Steve Howell 6a5c407b05 user export: Be more selective about exported messages. 2021-12-09 17:20:01 -08:00
Steve Howell fa654fd7a0 user export: Ignore realm icon and logo.
These are not considered to be "personal"
info, even if you upload them, so we
don't export them.

Generally the only folks who upload
these are admins, who can easily get
them in other ways. In fact, anybody
can get these via the app.
2021-12-09 17:20:01 -08:00
Steve Howell 8f991f8eb1 export: Make sure messages are sorted **across** files.
We now ensure that all message ids are sorted BEFORE
we split them into batches.

We now do a few extra "slim" queries to get message
ids up front.

But, now, when we divide them into batches, we no
longer run 2 or 3 different complicated queries in
a loop. We just basically hydrate our message ids,
so `write_message_partials` should be easy to reason
about.

This change also means that for tiny realms with
< 1000 messages you will always have just one
json file, since we aggregate the ids from the
queries before batching.
2021-12-09 12:22:34 -08:00
Steve Howell cef0e11816 export: Add get_id_list_gently_from_database.
This is slightly overkill for the single-user use
case, but for small queries it's barely any overhead,
and it's a nice abstraction.
2021-12-09 12:22:34 -08:00
Steve Howell 8ea320812f user exports: Chunkify messages in sorted order.
This accomplishes a few things:

    * It extracts `chunkify` rather than having us
      clumsily track chunking-related stuff in a
      big loop that is doing other stuff.

    * It makes it so that all message ids
      in message-000001.json < message-000002.json.

    * It makes it easier for us to customize
      the messages we send to a single user
      (coming soon).

BTW we probably have a slicker version of chunkify
somewhere in our codebase, but I couldn't remember
where.
2021-12-09 12:22:34 -08:00
Steve Howell 2a73964e16 user export: Add reactions.
We may eventually try to attach these to the messages
in the message-NNNNNN.json files, but for now they're
fine in user.json.
2021-12-09 12:22:34 -08:00
Steve Howell f810833df5 export: Improve export_usermessages_batch.
We no longer jankily read our input file into
an "output" variable.  Instead, we do things
in a type-safe way.
2021-12-09 08:36:40 -08:00
Steve Howell 5c1e8cb8dc mypy: Add MessagePartial TypedDict. 2021-12-09 08:36:40 -08:00
Steve Howell 09c57a3f9f export: Log more consistently and sort ids.
Now all file writes go through our three
helper functions, and we consistently
write a single log message after the file
gets written.

I killed off write_message_exports, since
all but one of its callers can call
write_table_data, which automatically
sorts data.  In particular, our Message
and UserMessage data will now be sorted
by ids.
2021-12-09 08:36:40 -08:00
Steve Howell 6ec49951c6 minor: Avoid creating intermediate list for message_ids.
This probably just postpones the list creation until
Django builds the "IN" query, but semantically it's
good to work in sets where we don't have any
meaningful ordering of the list that gets used.
2021-12-08 16:12:54 -08:00
Steve Howell f8ed099d3c export: Sort table data for most tables.
This affects most of our tables, but it excludes
table(s) like Message that go through kind of
unique codepaths.
2021-12-08 16:12:54 -08:00
Steve Howell a1d3f12e53 refactor: Extract write_table_data().
The immediate benefit of this is stronger mypy
checks (avoiding the ugly union caused by message
files).

The subsequent commit will add sorting.

We have test coverage on all these lines insofar
as if you comment out the lines, tests will
explode (i.e. more than superficial line
coverage).
2021-12-08 16:12:54 -08:00
Steve Howell c76ca2d0df export: Sort records.json files by path. 2021-12-08 16:12:54 -08:00
Steve Howell 2ef38e3d48 refactor: Extract write_records_json_file. 2021-12-08 16:12:54 -08:00
Steve Howell b79cfc19ab user export: Broaden query for RealmAuditLog.
We now check acting_user as well as modified_user
to see if a row pertains to our exported user.
2021-12-08 16:01:38 -08:00
Steve Howell 927b04368e minor: Use virtual_parent for custom fetchers.
The distinction here wasn't super meaningful
due to the way we order our "elif" statements,
but we want to reserver "normal_parent" for the
majority of use cases, where you simply tell
the Config what the "foreign_key" is.
2021-12-08 15:58:07 -08:00
Steve Howell 50120a9387 export: Remove config parameter for custom fetchers. 2021-12-08 15:58:07 -08:00
Steve Howell 54a3a423e5 mypy: Fix CustomFetch=Any hack. 2021-12-08 15:58:07 -08:00
Steve Howell 4128b52ac5 export: Rename custom fetchers. 2021-12-08 15:58:07 -08:00
Steve Howell a2c4931316 exports: Use realm for RealmAuditLog in realm exports.
For realm-wide exports, there is no reason to query
inefficiently against a list of modified users.

We move the Config out of the common child configs.
2021-12-08 15:58:07 -08:00
Steve Howell 8dd3c1038f exports: Rename parent_key to include_rows.
Even though Django usually treats foo__in
and foo_id__in identically for filters where
foo is a ForeignKey type, we want to insist
on somewhat more consistent syntax, because
we have the odd combo of type and type_id
in Recipient, where type_id is kinda like a
foreign key, but not a ForeignKey.

So we assert for now that all our include_rows
values end in "_id__in".
2021-12-08 15:58:07 -08:00
Steve Howell 02207f47d5 minor: Move code blocks to be alphabetical. 2021-12-08 15:58:07 -08:00
Steve Howell aae9f1b6f5 export: Make Config errors more clear. 2021-12-08 15:58:07 -08:00
Nikhil Maske 091772b534 hotspots: Remove intro_reply hotspot.
Zulip shows two guides on How to reply, first one by
the welcome bot and second one is intro_reply hotspot.
To simply and avoid redundancy, intro_reply hotspot is
removed.

Fixes #20482.
2021-12-07 21:55:59 -08:00
Eeshan Garg 3714a30e63 stream notifications: Add helper for silent user mention syntax.
In many of our stream notification messages, we make use of the
same silent user mention syntax, the template for which was always
hardcoded. This commit adds a helper function that all relevant
callers can call to get the right syntax when mentioning users.

Thanks to Tim Abbott for this suggestion!
2021-12-07 14:53:50 -08:00
Eeshan Garg 8ebe05f644 streams: Add RealmAuditLog entry for message retention updates. 2021-12-07 14:53:50 -08:00
Eeshan Garg d2901892e2 streams: Add notifications for message retention policy updates.
This is a part of #20289.
2021-12-07 14:53:50 -08:00
Steve Howell 6d09eab285 export: Export file images for single users.
We don't have automated test coverage on this yet,
but below are the results from manual testing.

Note that we include the realm icon and logo even
though they were not created by Cordelia.

    ./manage.py export_single_user cordelia@zulip.com

    $ (cd /tmp/zulip-export-4v3mo802/ && find .)
    .
    ./emoji
    ./emoji/2
    ./emoji/2/emoji
    ./emoji/2/emoji/images
    ./emoji/2/emoji/images/3.jpg
    ./emoji/records.json
    ./messages-000001.json
    ./realm_icons
    ./realm_icons/2
    ./realm_icons/2/night_logo.original
    ./realm_icons/2/night_logo.png
    ./realm_icons/2/icon.png
    ./realm_icons/2/icon.original
    ./realm_icons/records.json
    ./avatars
    ./avatars/2
    ./avatars/2/c5125af0447f4d66ce34c1b32eac75ac27ebe0e7.original
    ./avatars/2/c5125af0447f4d66ce34c1b32eac75ac27ebe0e7.png
    ./avatars/records.json
    ./uploads
    ./uploads/2
    ./uploads/2/68
    ./uploads/2/68/xyEkC5dTIp8m42_6HJ3kBfdt
    ./uploads/2/68/xyEkC5dTIp8m42_6HJ3kBfdt/denver.jpg
    ./uploads/2/96
    ./uploads/2/96/ol5WE6RTUntvuPDSpJUrYTim
    ./uploads/2/96/ol5WE6RTUntvuPDSpJUrYTim/denver.jpg
    ./uploads/records.json
    ./user.json
2021-12-07 11:16:52 -08:00
Steve Howell b8d9143318 export: Validate emoji paths.
(We lift the RealmEmoji query to be used by
both local and S3 storage helpers.)
2021-12-07 11:16:52 -08:00
Steve Howell ef6d9b10d2 refactor: Extract get_emoji_path. 2021-12-07 11:16:52 -08:00
Steve Howell 5a41904201 export: Add handle_system_bots flag.
We will set this to False for single-user exports.
2021-12-07 11:16:52 -08:00
Steve Howell 0e19deb558 exports: Limit s3 upload exports with path_id checks. 2021-12-07 11:16:52 -08:00
Steve Howell f6cbf931ae refactor: Pass attachments to export_uploads_from_local.
The next commit will use attachments in the s3 path.
2021-12-07 11:16:52 -08:00
Steve Howell 03f40a64d4 refactor: Pass valid_hashes to export_files_from_s3. 2021-12-07 11:16:52 -08:00
Steve Howell 15bc677f35 export: Pass users to export_avatars_from_local. 2021-12-07 11:16:52 -08:00
Eeshan Garg 2cdaae681d actions: Rename do_change_plan_type -> do change_realm_plan_type.
We will soon be adding an equivalent function for RemoteZulipServer,
so it makes sense to rename this function to be more descriptive.
2021-12-06 16:18:53 -08:00
Steve Howell 42ecabe967 export: Add check_metadata flag. 2021-12-06 15:09:37 -08:00
Steve Howell 0166f13d83 s3 exports: Validate user metadata for all assets.
This preps us to download assets for just a single
user.
2021-12-06 15:09:37 -08:00
Steve Howell 946ab22bba refactor: Lift users query to caller.
This preps us to reuse this code for single users
(after a few more subsequent changes).
2021-12-06 15:09:37 -08:00
Steve Howell b0e5c1d3b9 export: Remove paranoid assertion.
There are tactical reasons to remove this assertion.

Basically, the reason it's safe to remove is that it's
been around a long time and we would have seen this
operationally. Also, the check to make sure that the
S3 filename thingy matches the avatar hash is a much
stronger check.

We will soon restore a stronger version of this check
that applies to all of our asset types (emojis/avatars/etc.).
2021-12-06 15:09:37 -08:00
Steve Howell 59951ae52b refactor: Move metadata checks for s3 export.
This technically broadens the check for user_profile_id,
but we write that metadata on every record.
2021-12-06 15:09:37 -08:00
Steve Howell a27a6a4548 refactor: Inline _check_key_metadata.
This was only called in one place.
2021-12-06 15:09:37 -08:00
Steve Howell db39948be5 refactor: Pass in flavor to export_files_from_s3. 2021-12-06 15:09:37 -08:00
Steve Howell f4354c896b refactor: Pass object_key into export_files_from_s3.
This makes it easier to read the calling code and see
the big picture of how the four asset types are
organized.

I also handle uploads first, to be similar to the local
code.

This code is well tested--you can modify any of the callers
to pass in a wrong value of `object_key` and get a failing
test.
2021-12-06 15:09:37 -08:00
Tim Abbott fee05339a2 actions: Extract ONBOARDING_RECENT_TIMEDELTA. 2021-12-06 14:07:54 -08:00
Steve Howell 4088be6017 import/export: Add UserStatus table.
(We support both realm and single-user exports.)
2021-12-06 13:27:25 -08:00
Steve Howell 45addcd506 cosmetic: Sort DATE_FIELDS in export.py source. 2021-12-06 13:27:25 -08:00
Steve Howell f83907d3bb export: Add MutedUser table.
Note that the import was already implemented, but
its test was flawed.
2021-12-06 13:27:25 -08:00
Steve Howell 624260dd70 import: Extract re_map_realm_emoji_codes. 2021-12-06 13:27:24 -08:00
Steve Howell 6ec5a5193a tests: Verify emoji codes. 2021-12-06 13:22:47 -08:00
Palash e1796b0c8f onboarding: Manage occurrence of 'help' command.
"help" command occurs in the command list in
initial pms or when bot doesn't understand the message. It doesn't
occur when the bot is respoding to the "help" command itself.
2021-12-06 11:32:00 -08:00
Sahil Batra b68ebf5a22 message: Check wildcard mention restrictions while editing message.
This commit adds code to check whether a user is allowed to use
wildcard mention in a large stream or not while editing a message
based on the realm settings.

Previously this was only checked while sending message, thus user
was easily able to use wildcard mention by first sending a normal
message and then using a wildcard mention by editing it.
2021-12-06 10:22:29 -08:00
Aman Agrawal d2d6e1577f events: Explicitly send empty drafts to spectators.
This is purely for readability reasons, we still send empty
drafts without this change since UserProfile is None.

Fixes #20297
2021-12-03 14:34:21 -08:00
Palash a444560684 onboarding: Replace 'edit profile' with 'profile' in bot command list.
And change the tests accordingly.
2021-12-02 08:12:51 -08:00
Palash 6404b7bf61 onboarding: Make Welcome Bot more interactive.
1. The initial welcome message now contains less detail.
2. The bot now responds to these commands: "apps", "edit profile",
"dark mode", "light mode", "streams", "topics", "message formatting",
"keyboard shortcuts" and "help" - the bot still responds if there are
slight variations in these commands.
3. Tests have been made to check if bot responds to the advertised
commands (with variations) and gives a negative message if it doesn't
understand the message.

With substantial tweaks by tabbott.

Fixes #19900.
2021-12-01 16:30:48 -08:00
Mateusz Mandera 994ab8115e scim: Order Users by id when queried using filter syntax.
django-scim2 doesn't order the rows when fetching them in reponse to a
query using the filter syntax. We ensure that ORDER BY id is always
appended to the SQL queries.
2021-11-26 16:06:16 -08:00
Steve Howell 946e8064a3 user exports: Add several tables to the tarball.
We add the following tables to the user export:

    AlertWord
    CustomProfileFieldValue
    RealmAuditLog
    Service
    UserActivity
    UserActivityInterval
    UserCount
    UserGroup
    UserHotspot
    UserPresence
    UserTopic

Except for UserCount, we achieve this by sharing
code with the realm export via
add_user_profile_child_configs.

UserCount is handled slightly differently than realm
exports due to which key we trigger off.

It's possible that RealmAuditLog is incomplete for
single users, since we may also want rows where they
are the acting_user. This commit finds rows where
they are the modified_user. For non-admins I believe
it's rarely the case that they are the actor, and
they will tend to be the modified user if the two
fields are different at all.  For admins it's
arguable we want to see both changes they enacted
as well as changes that affected them.
2021-11-25 08:36:43 -08:00
Sahil Batra 2c1a87166c register: Pass settings.WEB_PUBLIC_STREAMS_ENABLED to clients.
This commit adds 'server_web_public_streams_enabled' field to
the register response to pass settings.WEB_PUBLIC_STREAMS_ENABLED
to clients.
2021-11-23 10:48:20 -08:00
Shlok Patel b3c58f454f api: Prevent special characters in topics.
Special characters, including `\r`, `\n`, and more esoteric codepoints
like non-characters, can negatively affect rendering and UI behaviour.

Check for, and prevent making new messages with, characters in the
Unicode categories of `Cc` (control characters), `Cs`, (surrogates),
and `Cn` (unassigned, non-characters).

Fixes #20128.
2021-11-22 22:09:06 -08:00
Sahil Batra eef9dd963b zcommand: Replace "mode" with "theme" in the returned message.
This commit replaces "dark mode" and "light mode" with "dark theme"
and "light theme" in the message returned and shown in a little
popup in the UI, when color scheme settings are changed through
slash commands.
2021-11-22 14:59:16 -08:00
Aman Agrawal 2c4af0e111 integration: Fix overflowing display name on open collective page.
Removed extra text to stop display name from flowing out of logo
container.
2021-11-21 21:05:06 -08:00
Aman Agrawal 334f046842 events: Don't send data related to custom profile field to spectator.
Since spectators can't access personal profile settings and
can't view profile for other users. Hence, we don't send realm
custom profile field data and user's profile data to spectators.

Fixes #20301.
2021-11-20 21:01:38 -08:00
Sahil Batra b72035abfe misc: Replace "Light mode" with "Light theme". 2021-11-19 11:49:59 -08:00
Sahil Batra 5024ba98cd misc: Rename "dark mode" to "dark theme".
Fixes part of #20228.
2021-11-18 16:57:11 -08:00
Sahil Batra 08404f2824 help: Rename night-mode.md to dark-theme.md.
Fixes part of #20228.
2021-11-18 16:55:35 -08:00
Aman Agrawal 00d3b88257 realm: Enable spectator access by default in development.
Enable spectator access for test `zulip` realm in developement
setup.

Add option in `do_create_realm` to configure
`enable_spectator_access` field of `Realm`.
2021-11-18 12:55:12 -08:00
Lauryn Menard 5300ad8ca0 api_documentation: Display `null` data type if potential value.
If null is a potential value of data type for a return value or
parameter in the API endpoint, then it is rendered as an option.

This currently relies on the 'nullable' setting in the OpenAPI spec
that was removed in the 3.1.0 release. If/when the OpenAPI version
is updated, then how the `data_type` for parameters and return values
is rendered will need to be reworked.

Fixes #20264.
2021-11-17 13:20:24 -08:00
Alex Vandiver faeffa2466 queue_processors: Set a bounded prefetch size on rabbitmq queues.
RabbitMQ clients have a setting called prefetch[1], which controls how
many un-acknowledged events the server forwards to the local queue in
the client.  The default is 0; this means that when clients first
connect, the server must send them every message in the queue.

This itself may cause unbounded memory usage in the client, but also
has other detrimental effects.  While the client is attempting to
process the head of the queue, it may be unable to read from the TCP
socket at the rate that the server is sending to it -- filling the TCP
buffers, and causing the server's writes to block.  If the server
blocks for more than 30 seconds, it times out the send, and closes the
connection with:

```
closing AMQP connection <0.30902.126> (127.0.0.1:53870 -> 127.0.0.1:5672):
{writer,send_failed,{error,timeout}}
```

This is https://github.com/pika/pika/issues/753#issuecomment-318119222.

Set a prefetch limit of 100 messages, or the batch size, to better
handle queues which start with large numbers of outstanding events.

Setting prefetch=1 causes significant performance degradation in the
no-op queue worker, to 30% of the prefetch=0 performance.  Setting
prefetch=100 achieves 90% of the prefetch=0 performance, and higher
values offer only minor gains above that.  For batch workers, their
performance is not notably degraded by prefetch equal to their batch
size, and they cannot function on smaller prefetches than their batch
size.

We also set a 100-count prefetch on Tornado workers, as they are
potentially susceptible to the same effect.

[1] https://www.rabbitmq.com/confirms.html#channel-qos-prefetch
2021-11-16 11:48:50 -08:00
Alex Vandiver 7c3507feef queue: Allow passing down a prefetch count to pika. 2021-11-16 11:48:50 -08:00
Alex Vandiver d1822b5630 tests: Move simulated_queue_client to the only test it is used in. 2021-11-16 11:48:50 -08:00
Alex Vandiver 6b6dcf6ce1 soft_deactivate: Handle multiple SUBSCRIPTION_DEACTIVATEDs.
Race conditions in stream unsubscription may lead to multiple
back-to-back SUBSCRIPTION_DEACTIVATED RealmAuditLog entries for the
same stream.  The current logic constructs duplicate UserMessage
entries for such, which then later fail to insert.

Keep a set of message-ids that have been prep'd to be inserted, so
that we don't duplicate them if there is a duplicated
SUBSCRIPTION_DEACTIVATED row.  This also renames the `message` local
variable, which otherwise overrode the `message` argument of a
different type.
2021-11-10 12:19:25 -08:00
YashRE42 778b7c0bdb links: Replace all references to Django docs to link to /3.2/ version.
Previously, our codebase contained links to various versions of the
Django docs, eg https://docs.djangoproject.com/en/1.8/ref/
request-response/#django.http.HttpRequest and https://
docs.djangoproject.com/en/2.2/ref/settings/#std:setting-SERVER_EMAIL
opening a link to a doc with an outdated Django version would show a
warning "This document is for an insecure version of Django that is no
longer supported. Please upgrade to a newer release!".

Most of these links are inside comments.

Following the replacement of these links in our docs, this commit uses
a search with the regex "docs.djangoproject.com/en/([0-9].[0-9]*)/"
and replaces all matches with "docs.djangoproject.com/en/3.2/".

All the new links in this commit have been generated by the above
replace and each link has then been manually checked to ensure that
(1) the page still exists and has not been moved to a new location
(and it has been found that no page has been moved like this), (2)
that the anchor that we're linking to has not been changed (and it has
been found that no anchor has been changed like this).

One comment where we mentioned a Django version in text before linking
to a page for that version has also been changed, the comment
mentioned the specific version when a change happened, and the history
is no longer relevant to us.
2021-11-08 18:06:16 -08:00
Alex Vandiver e3ec2e398c rate_limit: Pass seconds to freedom to all RateLimited exceptions. 2021-11-04 20:34:39 -07:00
Shlok Patel 893c9bc896 export: Remove `--delete-after-upload` flag in realm export.
For export realm following changes have been made:
- `./manage.py export --upload` would delete `.tar.gz` and unpacked dir
- `./manage.py export` would only delete `unpacked dir`

Besides, we have removed `--delete-after-upload` as we have set it as
the default.

Fixes #20081
2021-11-03 11:14:02 -07:00
Tim Abbott 4d055a6695 push_notifications: Truncate overly large remove events.
Fixes #19224.
2021-11-02 21:24:14 -07:00
Aman Agrawal 3e689ebae9 users: Allow spectators to view user avatars.
If realm is web_public, spectators can now view avatar of other
users.

There is a special exception we had to introduce in rest model to
allow `/avatar` type of urls for `anonymous` access, because they
don't have the /api/v1 prefix.

Fixes #19838.
2021-11-02 11:26:19 -07:00
Sahil Batra e6106cb334 invites: Update error message when max limit for the day is reached.
This commit updates the error message returned when the maximum
invite limit for the day. We update the error returned by API to
only mention that the limit is reached and add the suggestion
to use multi-use link or contact support in the message shown
in webapp.
2021-11-01 16:36:26 -07:00
Sahil Batra 84cd68a5aa realm: Create RealmUserDefault object for internal realm.
We create RealmUserDefault object for internal realm just
for consistency. The code in migration does so but it
was missed to add the code when creating new internal realm.
2021-10-29 16:19:55 -07:00
Mateusz Mandera 4105ccdb17 saml: Implement IdP-initated logout for Keycloak.
Fixes #13948.
2021-10-27 13:13:55 -07:00
Alex Vandiver 6a40c17ccf markdown: CSS-escape preview links.
This adds `soupsieve` as an explicit dependency, but intentionally
does not adjust the provision version, as it was already an indirect
dependency.
2021-10-26 18:17:23 -07:00
Alex Vandiver 52f74bbd9b markdown: Run URL preview links through camo.
Not proxying these requests through camo is a security concern.
Furthermore, on the desktop client, any embed image which is hosted on
a server with an expired or otherwise invalid certificate will trigger
a blocking modal window with no clear source and a confusing error
message; see zulip/zulip-desktop#1119.

Rewrite all `message_embed_image` URLs through camo, if it is enabled.
2021-10-26 18:17:23 -07:00
Anders Kaseorg 58920affd4 python: Remove re.UNICODE flag (redundant in Python 3).
https://docs.python.org/3/library/re.html#re.A

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-22 13:42:29 -07:00
Alex Vandiver 9381a3bd45 linkifiers: Support URL percent-encoded bytes.
Supporting URL percent-encoded bytes is possible using `%%20`, but this
is not necessarily very understandable to end-users, even those that
understand percent encoding.

Allow `%20` in linkifier URL format strings, and transform them into
`%%20` in the pattern just before they are applied in markdown
translation.  Care must be taken here, such that already-escaped `%`s
are not escaped an extra time.

We do this before rendering, and not before storage, as
a simplification; the JS-side linkifier at present only understands
`%(foo)s` and thus needs no changes, and to avoid an un-escaping pass
before showing in the admin UI.
2021-10-22 13:00:20 -07:00
Anders Kaseorg 4839b7ed27 url_preview: Interpret og:image relative to full page URL.
og:image is supposed to be an absolute URL, but some sites incorrectly
provide a relative URL.  In this case, it makes more sense to
interpret it relative to the full page URL after redirects, rather
than relative to just the domain part of the page URL before
redirects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-21 12:20:37 -07:00
Jonny Tran ddf4053a73 composebox_typeahead: Remove redundant options from the `/` typeahead.
Removes the `/day` and `/night` options from the typeahead menu while
still allowing the commands to be used. Typing `/day` and `/night`
will now suggest `/light` and `/dark`, respectively. Also changes the
`Dark mode` and `Light mode` popups that appear after using the
corresponding command.

Fixes #18318.
2021-10-21 10:50:27 -07:00
Anders Kaseorg d1c05467ec push_notifications: Fix RemoteZulipServer annotation to work at runtime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-20 00:48:55 -07:00
Alex Vandiver c0b557fb2e push_notifications: Move zilencer import inside ZILENCER_ENABLED check. 2021-10-20 06:16:10 +00:00
Alex Vandiver 111ee64e36 push_notifications: Pass down the remote server and user-id for logs.
This makes logging more consistent between FCM and APNs codepaths, and
makes clear which user-ids are for local users, and which are opaque
integers namespaced from some remote zulip server.
2021-10-19 22:04:24 -07:00
Alex Vandiver 5bcd3c01cb push_notifications: Add log line with user-id, UUID, and devices.
Being able to determine how many distinct users are getting push
notifications per remote host is useful, as is the distribution of
device counts.  This parallels the log line in
handle_push_notification for push notifications from local realms,
handled via the event queue.
2021-10-19 22:04:24 -07:00
rht bb8504d925 lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
Eeshan Garg b325a4f1be realm: Rename plan type constants to be more descriptive.
It is confusing to have the plan type constants not be namespaced
by the thing they represent. We already have a namespacing
convention in place for constants, so we should use it for
Realm.plan_type as well.
2021-10-19 12:20:39 -07:00
Alex Vandiver b02754adec html_diff: Handle empty differences between empty strings.
`rendered_content` in historical messages may be empty; examining the
history of them may thus require diff'ing two empty strings, which
itself produces an empty string.

Use `lxml.html.fragment_fromstring` to be able to successfully parse
these, rather than 500.

Part of #19559.
2021-10-18 18:27:40 -07:00
Mateusz Mandera f5a65846a8 scim: Override django-scim2 logic of exception handling in views.
As detailed in the comments, the default behavior is undesirable for us
because we can't really predict all possibilities of exceptions that may
be raised - and thus putting str(e) in the http response is potentially
insecure as it may leak some unexpected sensitive information that was
in the exception.

As a hypothetical example - KeyError resulting from some buggy
some_dict[secret_string] call would leak information. Though of course
we aim to never write code like that.
2021-10-17 21:33:03 -07:00
Vishnu KS aff52722a7 billing: Add PLUS as a plan_type to Realm. 2021-10-15 17:27:50 -07:00
Alya Abbott 60535adfc7 onboarding: Update Welcome Bot.
The most important thing is linking to the new user guide.
2021-10-15 12:59:20 -07:00
Anders Kaseorg 3bd3173b1f avatar: Remove ?x=x kludge.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Anders Kaseorg 58771d6e66 avatar: Use append_url_query_string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00