Commit Graph

6040 Commits

Author SHA1 Message Date
Tim Abbott f52005b638 email: Set List-Unsubscribe headers where we have unsubscribe links.
Since we have a convention of using `unsubscribe_link` as the context
variable name, this will apply to all of our emails with confirmation
links.
2021-04-28 15:33:32 -07:00
PIG208 9c0ce19104 reactions: Extract check_add_reaction from add_reaction. 2021-04-28 09:11:08 -07:00
strifel 02c5676a67 ldap: Add advanced LDAP realm access control.
This allows access to be more configurable than just setting one
attribute.  This can be configured by setting the setting
AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL.
2021-04-28 09:08:13 -07:00
PIG208 427506b742 embedded bot: Return message id for send_message and send_reply.
This change is made to comply with the corresponding views for
the API. The incrementor implementation in zulip_bots won't work
otherwise if send_message and send_reply return None as it needs
the message id.
2021-04-28 08:32:21 -07:00
PIG208 d20deccee7 tests: Add a test for fixtures for non-webhook integrations.
This commit create a directory to store the mock message for nagios and
more will be added.

The json files in this directory will be used to config the screenshot
generating script for the documentations of non-webhook integrations.
2021-04-28 08:16:51 -07:00
PIG208 562cbcac1c tools: Refactor ScreenshotConfig for non-webhook integrations. 2021-04-28 08:16:51 -07:00
Adam Birds 188273d8f5 integrations: Add JSON Printer webhook integration.
I have added the JSON Printer integration which will show any webhook
payload inside a code block which is useful for webhook testing.

Fixes #17969.
2021-04-28 07:23:18 -07:00
Tim Abbott be2d93036d emails: Include marketing unsubscribe link in send_custom_email context. 2021-04-27 21:36:09 -07:00
Tim Abbott 50f00d3e97 emails: Add unsubscribe link infrastructure for marketing emails.
This is intended to be used in any marketing emails that we send with
the send_custom_email infrastructure.
2021-04-27 21:36:09 -07:00
Tim Abbott 1afc0496ca event_schema: Fix translation of typing events message_type.
This was incorrectly/lazily marked as a string, not as an enum of two
string values, in the previous commit.
2021-04-27 21:35:32 -07:00
Dinesh 27e4f5da92 typing: Support sending stream/topic typing status.
This extends the /json/typing endpoint to also accept
stream_id and topic. With this change, the requests
sent to /json/typing should have these:
* `to`: a list set to
    - recipients for a PM
    - stream_id for a stream message
* `topic`, in case of stream message
along with `op`(start or stop).

On receiving a request with stream_id and topic, we send
typing events to clients with stream_typing_notifications set
to True for all users subscribed to that stream.
2021-04-27 20:52:21 -07:00
Dinesh 734d935d4a client_capabilities: Add stream_typing_notifications. 2021-04-27 20:52:21 -07:00
Tim Abbott 05f2ebb2b8 settings: Include units in SERVER_UPGRADE_NAG_DEADLINE_DAYS name.
When naming things with units of time involved, it's always good to
include units in the name, so that the meaning is clear.
2021-04-27 17:15:16 -07:00
Tim Abbott a3640223b8 security: Check when server directory was unpacked for upgrade nag.
The comment explains in more detail, but this should help avoid cases
where a Zulip server accidentally avoids the nag by having upgraded to
a 2-year old Zulip version from a 3-year-old version 2 months ago.
2021-04-27 17:09:27 -07:00
Tim Abbott 5607365b88 emails: Fix incorrect onboarding email going to realm owners.
This was apparently missed when implementing realm owners last year.
2021-04-27 17:09:27 -07:00
Nyior 8e2042d378 security: Add tooling to nag users if a Zulip server is very old.
This will help ensure that users upgrade their Zulip server.

Essentially rewritten by tabbott.

Fixes part of #17826.
2021-04-27 12:10:48 -07:00
Siddharth Asthana a81c4b5e4c send_custom_email: Add dry-run flag to verify recipients and email.
Add a `--dry-run` flag to send_custom_email management command
in order to provide a mechanism to verify the emails of the recipients
and the text of the email being sent before actually sending them.

Add tests to:
- Check that no emails are actually sent when we are in the dry-run mode.
- Check if the emails are printed correctly when we are in the dry-run mode.

Fixes #17767
2021-04-26 17:31:34 -07:00
Cyril Pletinckx 9afde790c6 email: Open a single SMTP connection to send email batches.
Previously the outgoing emails were sent over several SMTP
connections through the EmailSendingWorker; establishing a new
connection each time adds notable overhead.

Redefine EmailSendingWorker worker to be a LoopQueueProcessingWorker,
which allows it to handle batches of events. At the same time, persist
the connection across email sending, if possible.

The connection is initialized in the constructor of the worker
in order to keep the same connection throughout the whole process.
The concrete implementation of the consume_batch function is simply
processing each email one at a time until they have all been sent.

In order to reuse the previously implemented decorator to retry
sending failures a new method that meets the decorator's required
arguments is declared inside the EmailSendingWorker class. This
allows to retry the sending process of a particular email inside
the batch if the caught exception leaves this process retriable.

A second retry mechanism is used inside the initialize_connection
function to redo the opening of the connection until it works or
until three attempts failed. For this purpose the backoff module
has been added to the dependencies and a test has been added to
ensure that this retry mechanism works well.

The connection is closed when the stop method is called.

Fixes: #17672.
2021-04-26 17:27:22 -07:00
Alex Vandiver 0ad17925eb send_email: Remove unnecessary send_email_from_dict.
This was introduced in 8321bd3f92 to serve as a sort of drop-in
replacement for zerver.lib.queue.queue_json_publish, but its use has
been subsequently cut out (e.g. `9fcdb6c83ac5`).

Remote its last callsite.
2021-04-26 17:27:22 -07:00
Mateusz Mandera b7b1ec0aeb outgoing_webhook: Improve invalid json handling when parsing response.
It's better to just raise JsonableError here, as that makes this error
processed in the central place for this kind of thing in do_rest_call:
---------
except JsonableError as e:
    response_message = e.msg
    logging.info("Outhook trigger failed:", stack_info=True)
    fail_with_message(event, response_message)
    response_message = f"The outgoing webhook server attempted to send a message in Zulip, but that request resulted in the following error:\n> {e}"
    notify_bot_owner(event, failure_message=response_message)
    return None
----------

which does all the things that are supposed to happen -
fail_with_message, appropriate logging and notifying the bot owner.
2021-04-26 09:32:35 -07:00
Mateusz Mandera b998138d3a outgoing_webhook: Handle valid, but unexpected json in response.
Responses such as "null" or "true" are valid json, but json.loads
returns different objects than dicts that the codepath expects.

Fixes #18223.
2021-04-26 09:32:35 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Anders Kaseorg e3f2ffa681 docs: Capitalize “Markdown” consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Anders Kaseorg 178736c8eb docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Aman Agrawal ebe822341d message_edit: Don't add content edit to all messages.
Remove content edit keys if present in edit_history_event
when passing to update_messages_for_topic_edit.

Since content edit is only applied to the edited_message,
this shouldn't be part of the rest of the messages for which
topic was edited. This was a bug identified by
editing topic and content of a message at the same time
when more than 1 message is affected.
2021-04-24 13:51:49 -07:00
Aman Agrawal 79d748ba7d message_edit: Use update_edit_history to update message history. 2021-04-23 15:12:09 -07:00
Aman Agrawal de50f4ae25 message_edit: Extract update_edit_history. 2021-04-23 15:12:09 -07:00
Aman Agrawal 736fdcda49 update_messages_for_topic_edit: Remame `message` variable. 2021-04-23 15:12:09 -07:00
Mateusz Mandera 1a8ad796f8 models: Replace __id syntax with _id where possible.
model__id syntax implies needing a JOIN on the model table to fetch the
id. That's usually redundant, because the first table in the query
simply has a 'model_id' column, so the id can be fetched directly.
Django is actually smart enough to not do those redundant joins, but we
should still avoid this misguided syntax.

The exceptions are ManytoMany fields and queries doing a backward
relationship lookup. If "streams" is a many-to-many relationship, then
streams_id is invalid - streams__id syntax is needed. If "y" is a
foreign fields from X to Y:
class X:
  y = models.ForeignKey(Y)

then object x of class X has the field x.y_id, but y of class Y doesn't
have y.x_id. Thus Y queries need to be done like
Y.objects.filter(x__id__in=some_list)
2021-04-22 14:53:00 -07:00
Alex Vandiver 11177a40da soft_deactivate: Log and continue on failure to catch up a user.
There exists a logic bug (see #18236) which causes duplicate
usermessage rows to be inserted.  Currently, this stops catch-up for
all users.

Catch and record the exception for each affected user, so we at least
make catch-up progress on other users.
2021-04-22 14:38:03 -07:00
Mateusz Mandera 8d4ab69a46 docs: Move the /configure-missed-message-emails help page.
configure-message-notification-emails is the correct name now.
2021-04-21 10:10:54 -07:00
Mateusz Mandera 977a2f7fa0 emails: Rename "missed message email" to "message notification email". 2021-04-21 10:10:54 -07:00
Mateusz Mandera 716449030d emails: Rename missed message email sender to "Zulip notifcations".
It was decided that this is more appropriate naming. "Missed message"
gives it a bit of a sound like something went wrong.
2021-04-21 10:10:54 -07:00
Tim Abbott 6346b9d3eb models: Replace user_profile__is_active queries with is_user_active.
This saves a couple database queries by using the recently added
denormalization for Subscription objects.
2021-04-19 18:30:31 -07:00
Tim Abbott a1cfe25f8d streams: Move can_access_stream_user_ids into streams.py.
This belongs either here or in stream_subscription.py, which arguably
should just be merged into streams.py anyway.
2021-04-19 18:30:31 -07:00
Tim Abbott e4932bd952 actions: Fixed deactivated user IDs being included for stream events.
This was a mostly harmless bug, since those users cannot have active
clients, but fixing it will improve performance in any Zulip
organization where the vast majority of users are deactivated.
2021-04-19 18:30:31 -07:00
akshatdalton 6509c4f8f4 linkifiers: Add an API to support the editing of linkifier.
This commit adds an API to `zproject/urls.py` to edit/update
the realm linkifier. Its helper function to update the
database is added in `zerver/lib/actions.py`.

`zulip.yaml` is documented accordingly as well, clearly
stating that this API updates one linkifier at a time.

The tests are added for the API and helper function which
updates the realm linkifier.

Fixes #10830.
2021-04-19 18:01:45 -07:00
akshatdalton b29bd71a9c Refactor: Use `id` instead of `pk` as key.
Use `id` instead of `pk` as key to get RealmFilter
object in `do_remove_linkifier` function in `actions.py`.
2021-04-19 18:01:45 -07:00
Mateusz Mandera ccfcc186ad subs: Fix subscriber_..._history_access to not exclude subbed guests.
Guests are supposed to have stream history access to public streams
they're subscribed to.
2021-04-19 10:10:51 -07:00
Mateusz Mandera 68d1f2d7ef streams: Add realm check in can_access_stream_history.
The caller is supposed validate the stream and user realm match, but
since this is a security-sensitive function, we should have this
defensive code to protect against some validation bugs in the caller
leading to this being called incorrectly and returning True.
2021-04-19 10:10:51 -07:00
Mateusz Mandera f5c4005f8a actions: Fix some lists incorrectly named "subscribers".
These contain subscriptions, not subscribers.
2021-04-19 10:10:51 -07:00
Mateusz Mandera 4e26a9e9d6 subs: Fix codepaths incorrectly fetching subs of deactivated users.
Fixes #17922.

These two places fetch subscriptions for the sake of getting user ids to
send events to. Clearly deactivated users should be excluded from that.
2021-04-19 10:10:51 -07:00
Mateusz Mandera 50bfbb588e subs: Allow filtering by is_user_active in get_active_subscriptions.
get_active_subscriptions_for_stream_id should allow specifying whether
subscriptions of deactivated users should be included in the result.
Active subs of deactivated users are  a subtlety that's easy to miss
when writing relevant code, so we make include_deactivated_users a
mandatory kwarg - this will force callers to definitely give thought to
whether such subs should be included or not.

This commit is just a refactoring, we keep original behavior everywhere
- there are places where subs of deactivates users should probably be
excluded but aren't - we don't fix that here, it'll be addressed in
follow-up commits.
2021-04-19 10:10:51 -07:00
Mateusz Mandera c3a8a15bae delete_messages: Pass a list of user ids in the event in all cases.
The bulk deletion codepath was using dicts instead of user ids in the
event, as opposed to the other codepath which was adjusted to pass just
user ids before. We make the bulk codepath consistent with the other
one. Due to the dict-type events happening in 3.*, we move the goal for
deleting the compat code in process_notification to 5.0.
2021-04-16 09:54:14 -07:00
Anders Kaseorg f59f2ca165 requirements: Re-drop direct dependency on mock.
This was dropped in commit 840cf4b885
(#15091), but commit 1432067959
(#17047) mistakenly reintroduced it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 21:47:33 -07:00
Anders Kaseorg bdb20a8002 integrations: Convert deprecated Django url to path.
django.conf.urls.url is actually a deprecated alias of
django.urls.re_path, but we want path instead of re_path.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg 2939d29b6d python: Convert deprecated Django smart_text alias to smart_str.
django.utils.encoding.smart_text is a deprecated alias of
django.utils.encoding.smart_str as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg dcdb00a5e6 python: Convert deprecated Django is_safe_url.
django.utils.http.is_safe_url is a deprecated alias of
django.utils.http.url_has_allowed_host_and_scheme as of Django 3.0,
and will be removed in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg e7ed907cf6 python: Convert deprecated Django ugettext alias to gettext.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Adam Birds 545cd961f4 integrations: Add docs for GitHub Actions integration.
I have added a documentation page for the GitHub Actions integration to
`/integrations/doc/github-actions` with a link to the Zulip GitHub
Actions repository.

Tweaked by tabbott to add cross-links with the main GitHub integration.
2021-04-15 16:42:31 -07:00