Commit Graph

13133 Commits

Author SHA1 Message Date
Aman Agrawal 5e83965e80 giphy: Use GIPHY web SDK to allow inserting GIFs in compose box.
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.

We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
2021-04-05 15:04:49 -07:00
LoopThrough-i-j ffd0d822fe actions: Send reaction events to subscribers with history access.
Previously, if a user subscribed to a stream with
history_public_to_subscribers, and then was looking at old messages in
the stream, they would not get live-updates for that stream, because
of the structure in how notify_reaction_update only looked at
UserMessage rows (we had a previous workaround involving the
`historical` field in `UserMessage` which had already made it work if
the user themselves added the reaction).

We fix this by including all subscribers with history access in the
set of recipients for update events.

Fixes a bug that was confused with #16942.
2021-04-05 13:43:39 -07:00
LoopThrough-i-j 277fbb3f02 stream_subscription: Add subscribe_ids_with_stream_history_access.
This new function returns the set of `user_ids` with access to the
stream's full history, for use in send_event calls.
2021-04-05 13:23:11 -07:00
PIG208 e86b2c8d58 embedded bot: Use server settings for storage_size_limit. 2021-04-04 18:05:30 -07:00
Cyril Pletinckx b7fa41601d emails: Truncate overly-long From fields for RFC compatibility.
Amazon SES has a limit on the size of address fields, and rejects
emails with too-long "From" combinations of name and address. This
limit is set to 320 bytes and comes from an RFC limitation on the
size of addresses. This RFC standard states that an email address
should not be composed of a local part (before the '@') longer than
64 bytes and a domain part (after the '@') longer than 255 bytes.
It is possible that Amazon SES misinterprets this limitation as it
checks the length of the combination of the name and the email
address of the sender.

To ensure that this problem is not encountered in the send_email
module of Zulip the length of this combination is now checked
against this limit and the from_name field is removed to only
keep the from_address field when it is necessary in order to
stay below 320 bytes.

If the from_address field alone is longer than 320 bytes the
sending process will raise an SMTPDataError exception.

Tests for this new check are added to the backend test suite in
order to test if build_email correctly outputs an email with filled
from_name and from_address fields when the total length is lower
than 320 bytes and that it correctly throws the from_name field
away when necessary.

Fixes: #17558.
2021-04-03 08:13:26 -07:00
Siddharth Asthana 47e478945b support: Create RealmAuditLog when updating realm_subdomain. 2021-04-03 08:01:46 -07:00
Siddharth Asthana 80c9243c6a support: Create RealmAuditLog when updating sponsorship status. 2021-04-03 08:01:46 -07:00
Siddharth Asthana ddbc6d7662 support: Create RealmAuditLog when realm reactivation email is sent. 2021-04-03 08:01:46 -07:00
Siddharth Asthana 233c4d520c support: Create RealmAuditLog when updating billing_method.
This commit also makes acting_user as a mandantory argument and fixes
the tests accordingly.
2021-04-03 08:01:46 -07:00
Siddharth Asthana 44c34cb39a support: Create RealmAuditLog when approving sponsorship. 2021-04-03 08:01:46 -07:00
Siddharth Asthana 6945ed3587 support: Pass acting_user to attach_discount_to_realm. 2021-04-03 08:01:46 -07:00
Siddharth Asthana c3f37c2a64 support: Pass acting_user to do_change_plan_type.
acting_user is now a mandatory field, so wherever this function is used,
we are passing acting_user as well.
2021-04-03 08:01:46 -07:00
Mateusz Mandera ba7f2a22e6 docs: Add redirects for moved pages about stream archiving.
We're renaming "stream deletion" language to "stream archiving"
and these pages were moved in the process, so we should keep redirects
for them for a while.
2021-04-02 22:08:15 -07:00
Mateusz Mandera 82b43a8cfe streams: Use "archive stream" phrasing instead of "delete stream".
The previous phrasing was misleading in relation to what the action
actually does.
2021-04-02 22:06:48 -07:00
Mateusz Mandera d5871f1005 streams: Allow admins to fetch private streams via the get streams API.
We send a list of all private streams to realm admins in
fetch_initial_state since 73c30774cb
anyway and this API code just didn't catch up.
2021-04-02 16:27:50 -07:00
Anders Kaseorg ceb7e2d2bd Revert "markdown: Add support to shorten GitHub links."
This reverts commit 9c6d8d9d81 (#16916).

This feature has known bugs, and also wants some design changes to
make it customizable like linkifiers, so we’re retargeting this to
post-4.x.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-02 15:52:34 -07:00
Alex Vandiver 4f6fc728cd tornado: Explicitly mark requests as varying by cookie.
The Session middleware only adds `Vary: cookie` if it sees an access
to the from inside of it.  Because we are effectively, from the Django
session middleware's point of view, returning the static content of
`request.saved_response` and never accessing the session, it does not
set `Vary: cookie` on longpoll requests.

Explicitly mark Tornado requests as varying by cookie.
2021-04-02 14:55:22 -07:00
shanukun bcc3bb03fe refactor: Make acting_user a mandatory kwarg for bulk_remove_subscriptions. 2021-04-02 14:44:41 -07:00
shanukun 0bf067b681 refactor: Make acting_user a mandatory kwarg for bulk_add_subscriptions. 2021-04-02 14:44:41 -07:00
shanukun 790085832c refactor: Make acting_user a mandatory kwarg for ensure_stream. 2021-04-02 14:44:41 -07:00
shanukun c39ffe8811 refactor: Make acting_user a mandatory kwarg for do_deactivate_stream. 2021-04-02 14:44:41 -07:00
shanukun f442e9fb03 refactor: Make acting_user a mandatory kwarg for do_scrub_realm. 2021-04-02 14:44:41 -07:00
shanukun 4b00e5da72 refactor: Make acting_user a mandatory kwarg for do_deactivate_realm. 2021-04-02 14:44:41 -07:00
shanukun 626cf52723 refactor: Make acting_user a mandatory kwarg for do_set_realm_signup_notifications_stream. 2021-04-02 14:44:41 -07:00
shanukun 00d998b955 refactor: Make acting_user a mandatory kwarg for do_set_realm_notifications_stream. 2021-04-02 14:44:41 -07:00
PIG208 8002c4ff10 python: Use the correct typing for update_message_backend.
This is likely a typo introduced in e2c14724d9.
2021-04-01 17:49:28 -07:00
WookieMonkeys 1b6f68bb59 stream: Add entropy to deactivated streams.
Adding an additional `!` to the stream name each time a stream is
deactivated, to a maximum of 21 times, effectively limits number of
times a stream with a given name can be deactivated.  This is unlikely
to come up in common usage, but may be confusing when testing.

Change what we prepend to deactivated stream names to something with
more entropy than just `!`, by instead prepending a substring of hash
of the stream's ID.  `!`s.  Using 128 bits of the hash means that it
will require more than 10^18th renames to have a 1% chance of collision.

Because too-long stream names are also truncated at 60 characters,
having this entropy in the beginning of the name also helps address
potential issues from stream names that differed only in, e.g. the
60th character.

Fixes #17016.
2021-04-01 17:16:35 -07:00
Tim Abbott 08116a17b0 typing: Move to parameter validation to view code. 2021-04-01 08:30:47 -07:00
Tim Abbott 2a8e9db8f1 typing: Remove obsolete block comment.
The legacy feature described here was removed in
d5cc29755e.
2021-04-01 08:13:23 -07:00
Dinesh ddca602123 typing_notifications: Do op validation in view function.
Instead of validating `op` value later, this commit does that
in `REQ`.

Also helps avoiding duplication of this validation when
stream typing notifications feature is added.
2021-04-01 07:50:02 -07:00
Dinesh 2d40224bb6 api docs: Fix incorrect comment in delete_stream documentation. 2021-04-01 07:49:42 -07:00
Riken Shah 08212ef74a puppeteer_tests: Remove login test.
This commit deletes the `01-login.ts` test because it was
redundant, We are already checking for log-in in all the
other tests.
2021-03-31 16:55:54 -07:00
Alex Vandiver 21bafe1e1e tornado: Drop unused command-line flags.
These flags were put in place in the first commit that introduced
Tornado (9afd63692f) with unclear
utility.

Remove them, since they have never been documented, and do not have a
clear need.
2021-03-31 14:19:38 -07:00
Alex Vandiver 07779ea879 middleware: Do not trust X-Forwarded-For; use X-Real-Ip, set from nginx.
The `X-Forwarded-For` header is a list of proxies' IP addresses; each
proxy appends the remote address of the host it received its request
from to the list, as it passes the request down.  A naïve parsing, as
SetRemoteAddrFromForwardedFor did, would thus interpret the first
address in the list as the client's IP.

However, clients can pass in arbitrary `X-Forwarded-For` headers,
which would allow them to spoof their IP address.  `nginx`'s behavior
is to treat the addresses as untrusted unless they match an allowlist
of known proxies.  By setting `real_ip_recursive on`, it also allows
this behavior to be applied repeatedly, moving from right to left down
the `X-Forwarded-For` list, stopping at the right-most that is
untrusted.

Rather than re-implement this logic in Django, pass the first
untrusted value that `nginx` computer down into Django via `X-Real-Ip`
header.  This allows consistent IP addresses in logs between `nginx`
and Django.

Proxied calls into Tornado (which don't use UWSGI) already passed this
header, as Tornado logging respects it.
2021-03-31 14:19:38 -07:00
Sundar Guntnur 5aefb5e656 webhook: Catch potential JsonableError when parsing widget_content.
The `widget_content` key is expected to contain a string which parses
as JSON; in the event that it does not, log the error and notify the
bot owner, instead of failing silently.

Fixes #16850.
2021-03-31 13:31:42 -07:00
m-e-l-u-h-a-n aea31eb31f api: Add REALM_DEACTIVATED error code.
In `validate_account_and_subdomain` we check
if user's realm is not deactivated. In case
of failure of this check, we raise our standard
JsonableError. While this works well in most
cases but it creates difficulties in handling
of users with deactivated realms for non-browser
clients.

So we register a new REALM_DEACTIVATED error
code so that clients can distinguish if error
is because of deactivated account. Following
these changes `validate_account_and_subdomain`
raises RealmDeactivatedError if user's realm
is deactivated.

This error is also documented in
`/api/rest-error-handling`.

Testing: I have mostly relied on automated
backend tests to test this.

Fixes #17763.
2021-03-31 08:46:13 -07:00
m-e-l-u-h-a-n 2eeb82edba api: Add USER_DEACTIVATED error code.
In validate_account_and_subdomain we check if
user's account is not deactivated. In case of
failure of this check we raise our standard
JsonableError. While this works well in most
cases but it creates difficulties in handling
of deactivated accounts for non-browser clients.

So we register a new USER_DEACTIVATED error
code so that clients can distinguish if error
is because of deactivated account. Following
these changes `validate_account_and_subdomain`
raises UserDeactivatedError if user's account
is deactivated.

This error is also documented in
`/api/rest-error-handling`.

Testing: I have mostly relied on automated
backend tests to test this.

Partially addresses issue #17763.
2021-03-31 08:46:13 -07:00
Tim Abbott 36e320cf18 api docs: Fix link to old Help Center linkifiers URL. 2021-03-30 16:51:42 -07:00
Sumanth V Rao e12f682e2e markdown: Include text & url in `topic_links` parameter of our API.
The linkifier code now includes both the shortened text and the expanded
URL, sorted by the order of the occurrence in a topic. This list is passed
back in the `topic_links` parameter of the /messages and the /events APIs.

topic_links earlier vs now:

earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32']

now: [{'url': 'https://www.google.com', 'text': 'https://www.google/com},
      {'url': 'https://github.com/zulip/zulip/32', 'text': '#32'}]

Similarly, the topic_links local echo logic in the frontend now returns
back an object.

Fixes: #17109.
2021-03-30 15:53:07 -07:00
Sumanth V Rao 7f6fe128f6 hotspots: Add TUTORIAL_ENABLED setting to toggle INTRO_HOTSPOTS.
We add a TUTORIAL_ENABLED setting for self-hosters who want to
disable the tutorial entirely on their system. For this, the
default value (True) is placed in default_settings.py, which
can be overwritten by adding an entry in /etc/zulip/settings.py.
2021-03-30 14:46:42 -07:00
lukem1 105a5a95ee topic_mutes: Filter deactivated streams from get_topic_mutes.
Updated database query to filter out deactivated streams from the
return of the get_topic_mutes method. Added optional
include_deactivated parameter to the method to make the behavior
default but overrideable. Added test case in test_muting for these
changes. Fixes blueslip warnings thrown by muting.js set_muted_topics
when passed deactivated streams via page_params.
2021-03-30 12:11:35 -07:00
Vishnu KS 92316ef4d1 statuspage: Properly detect the update is for component or incident.
The value of "status_indicator" can be "none" for both the component
and incident updates[1]. Also, it is not at all necessary that the value of
"status_indicator" is always "none" for incident updates[2][3]. So our previous
logic of using the value of "status_indicator" to determine whether
the update is that of a component or incident was incorrect. Instead, we
should use "incident" or "component" keys to determine the type of update.

This commit fixes issues [1] and [2] in sentry.

1. https://sentry.io/organizations/zulip/issues/2303217561
2. https://sentry.io/organizations/zulip/issues/2303197407
3. https://support.atlassian.com/statuspage/docs/enable-webhook-notifications/
2021-03-30 12:06:09 -07:00
Mateusz Mandera 353e1a2016 migrations: Subscription.is_user_active denormalization - final step.
With the previous two commits deployed, we're ready to use the
denormalization to optimize the query.

With dev environment db prepared using
./manage.py populate_db --extra-users=2000 --extra-streams=400
this takes the execution time of the query in
bulk_get_subscriber_user_ids from 1.5-1.6s to 0.4-0.5s on my machine.
2021-03-30 09:29:36 -07:00
Mateusz Mandera 1e37fde59f migrations: Subscription.is_user_active denormalization - step two.
This adds a migration to do the backfill and add the index that'll be
used to optimize the bulk_get_subscriber_user_ids query in the next
commit.
2021-03-30 09:29:36 -07:00
Tim Abbott 53ed759fc1 users: Fix ordering issue with deactivating bots.
The new comment explains the issue in some detail, but basically if we
deactivate the bots first, then an error partway through is corrected
by a retry; if we deactivate the user first, then we may leak
undeactivated bots if a failure occurs.
2021-03-30 09:21:41 -07:00
Mateusz Mandera f329878376 migrations: Subscription.is_user_active denormalization - step one.
This adds the is_user_active with the appropriate code for setting the
value correctly in the future. In the following commit a migration to
backfill the value for existing Subscriptions will be added.

To ensure correct user_profile.is_active handling also in tests, we
replace all direct .is_active mutation with calls to appropriate
functions.
2021-03-30 09:19:03 -07:00
Mateusz Mandera d236d3f738 users: Improve db transaction structure in user (de)activation process.
These procedures should be done atomically overall, with the exception
of the code that sends events to avoid block if there's a delay
communicating with Tornado.
We add the savepoint=False on underlying function that already
executes inside an atomic context - to avoid the overhead of creating
savepoints where they aren't needed.
2021-03-30 09:15:24 -07:00
Mateusz Mandera 0e6d230804 users: Fix do_deactivate_user to save is_mirror_user.
This was a bug - is_mirror_user was not listed in update_fields despite
being changed.
2021-03-30 09:15:24 -07:00
sahil839 54be0dd1a4 streams: Add moderators option in stream_post_policy.
This commit adds a new option of STREAM_POST_POLICY_MODERATORS
in stream_post_policy which will allow only realm admins and
moderators to post in that stream.
2021-03-30 09:06:20 -07:00
sahil839 a061240251 streams: Extract helper for checking access to stream based on post-policy.
We extract a helper which checks whether to allow the sender to send the
message to a stream according to the stream_post_policy. The purpose
of extracting it out is to avoid additional code for checking the access
for bot owners in case of bot sending the messages and instead calling
the handler two times - one time for sender and one time for bot owner if
sender is a bot.
2021-03-30 09:06:20 -07:00