Commit Graph

1113 Commits

Author SHA1 Message Date
Shlok Patel 033e61a055 webhooks/github: Include discussion ID in messages.
We add discussion id and url in the comments and highlighted title to
the body of disscussion message to make it more meaningful and accessible.

Fixes #19938.
2021-10-26 17:52:57 -07:00
Shlok Patel 0678e2610f webhooks: Add support for GitHub discussions messages.
We aim to use Zulip topics thoughtfully in displaying messages from
discussions, as well as linking to the discussion in every message so
that it's easy to view them.

Fixes #19938.
2021-10-22 13:40:05 -07:00
Anders Kaseorg 5360bd76ac help: Add newline after admonition header for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:39:25 -07:00
PIG208 53888e5a26 request: Refactor ZulipRequestNotes to RequestNotes.
This utilizes the generic `BaseNotes` we added for multipurpose
patching. With this migration as an example, we can further support
more types of notes to replace the monkey-patching approach we have used
throughout the codebase for type safety.
2021-09-03 08:48:45 -07:00
Anders Kaseorg 1ce12191aa docs: Update links for other repository branch renames.
GitHub redirects these, but we should use the canonical URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 15:28:01 -07:00
Eeshan Garg 8697d98652 webhooks/clubhouse: Ignore label removals for story batch updates.
9ac55a8cf6 introduced support for
batch updates to stories. However, that commit didn't skip label
removals, as we already do in non-batch story payloads. This led
to an exception for batch story update payloads where labels were
removed but none were added.
2021-08-20 23:27:14 -07:00
PIG208 3b11c36ed9 typing: Fix function signatures.
This fixes mypy errors for function signatures discovered with
django-stubs.
2021-08-20 05:54:19 -07:00
Anders Kaseorg 4206e5f00b python: Remove locally dead code.
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
PIG208 a583e9ffad webhooks: Refactor alertmanager to use REQ. 2021-08-08 17:11:18 -07:00
tushar912 83f6557f43 integrations: Add SonarQube webhook integration.
Fixes #13395.
2021-08-05 09:53:42 -07:00
Lefteris Kyriazanos 2b70e88fda integrations: Add basic open collective integration.
Add basic open collective integration for the user donation
event.
Fixes #18319
2021-08-03 16:09:33 -07:00
Anders Kaseorg 5483ebae37 python: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg ad5f0c05b5 python: Remove default "utf8" argument for encode(), decode().
Partially generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Mateusz Mandera c34260426a bots: Pass realm to remaining get_system_bot calls in tests. 2021-07-26 15:33:13 -07:00
PIG208 5359d20a13 webhooks: Add support to event filtering system for webhooks.
This add support to event filtering system for most webhooks that
require trivial changes to adapt this feature.
2021-07-24 15:10:09 -07:00
PIG208 987f859681 webhooks: Detect view function with default naming convention. 2021-07-24 15:10:09 -07:00
Anders Kaseorg fb3ddf50d4 python: Fix mypy no_implicit_reexport errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-16 14:02:31 -07:00
Alex Vandiver 43f3f9221d slack_incoming: Handle explicit None text payload in webhook. 2021-07-15 09:37:39 -07:00
PIG208 c03b9c95ad request: Store client information using ZulipRequestNotes.
This concludes the HttpRequest migration to eliminate arbitrary
attributes (except private ones that are belong to django) attached
to the request object during runtime and migrated them to a
separate data structure dedicated for the purpose of adding
information (so called notes) to a HttpRequest.
2021-07-14 12:01:07 -07:00
PIG208 e373df88e0 webhooks: Make github webhook support event filtering system. 2021-07-13 16:47:18 -07:00
Riken Shah a96c614420 message: Fix the edit topic bug by removing leading \n from msg content.
The reason for this bug is because of different striping
processes in the backend and frontend, i.e The frontend
checks if the message's `raw_content` has changed to
decide if the `content` of the message should be sent in
the request to the backend, or not. So, it removes the
leading new line ('\n') from the message `raw_content`
when checking it, which is causing the "Error saving edit:
You don't have permission to edit this message" error.

This commit fixes it by removing the leading new line
when cleaning message content.

The bug was explained by @punchagan and its solution
by @timabbott.
2021-07-06 09:41:54 -07:00
PIG208 83ea6c37c5 webhook: Add expect_noop option for check_webhook.
This change allow check_webhook to raise an error when a message is
sent and vice versa. This is useful when one payload is not expecting
any output messages.
2021-07-05 19:17:11 -07:00
PIG208 2f9c586af5 webhooks: Allow event registration using webhook_view decorator.
In addition to event filtering, we add support for registering supported
events for a webhook integration using the webhook_view decorator.

The event types are stored in the view function directly as a function
attribute, and can be later accessed via the module path and the view
function name are given (which is already specified the integrations.py)

Note that the WebhookTestCase doesn't know the name of the view function
and the module of the webhook. WEBHOOK_DIR_NAME needs to be overridden
if we want exceptions to raised when one of our test functions triggered
a unspecified event, but this practice is not enforced.

all_event_type does not need to be given even if event filters are used
in the webhook. But if a list of event types is given, it will be possible
for us to include it in the documentation while ensuring that all the
tested events are included (but not vice versa at the current stage, as
we yet not required all the events included in the list to be tested)

This guarantees that we can always access the list of all the tested
events of a webhook. This feature will be later plumbed to marcos to
display all event types dynamically in doc.md.
2021-07-05 19:10:31 -07:00
PIG208 dcbb2a78ca python: Migrate most json_error => JsonableError.
JsonableError has two major benefits over json_error:
* It can be raised from anywhere in the codebase, rather than
  being a return value, which is much more convenient for refactoring,
  as one doesn't potentially need to change error handling style when
  extracting a bit of view code to a function.
* It is guaranteed to contain the `code` property, which is helpful
  for API consistency.

Various stragglers are not updated because JsonableError requires
subclassing in order to specify custom data or HTTP status codes.
2021-06-30 16:22:38 -07:00
PIG208 542516b07b integrations: Support params for events filtering.
We modify check_send_webhook_message to make it accept three new
parameters: only_events and exclude_events that are retrieved using REQ,
and complete_event_type, which is passed by the incoming webhook view
that is filtered according to the former two parameters.

Part of #18525.
2021-06-29 17:07:46 -07:00
PIG208 5ecbfecd77 webhook: Rename FIXTURE_DIR_NAME to WEBHOOK_DIR_NAME.
Since FIXTURE_DIR_NAME is the name of the folder that contains the view
and tests modules of the webhook and another folder called "fixtures" that
store the fixtures, it is more appropriate to call it WEBHOOK_DIR_NAME,
especially when we want to refer to the view module using this variable.
2021-06-29 17:01:54 -07:00
PIG208 b720e10d88 integrations: Reformat expected messages for readability. 2021-06-29 17:01:54 -07:00
akshatdalton 8d158852fd webhooks/slack_incoming: Update regex to correctly convert emphasis.
This was a bug where we were converting _text_ to **text**.
For emphasis we use the *text* format.
2021-06-24 09:43:32 -07:00
Riken Shah 4f54e15993 refactor: Convert `clean-unused-caches` to`clean_unused_caches.py`.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
2021-06-12 07:28:16 -07:00
Tim Abbott 17fe09bb6d webhooks: Update link to BuildBot documentation. 2021-06-10 17:15:55 -07:00
PIG208 e8b92b79e5 integrations: Add V3 support for PagerDuty. 2021-06-09 16:34:34 -07:00
PIG208 32ca263914 integrations: Change format of templates for PagerDuty V3.
Because the payload of V3 will no longer include the description,
We replace the ":" by "." in the message and create the new string
template for trigger messages.
2021-06-09 16:34:34 -07:00
PIG208 2ee00972a0 doc: Change supported extension type to reflect the change. 2021-06-02 09:31:38 -07:00
Adam Birds ee375e2999 integrations: Add Freshstatus webhook integration. 2021-05-27 23:38:11 -07:00
Adam Birds 303bb61f9c integrations: Add Sonarr Integration. 2021-05-26 17:12:28 -07:00
Adam Birds 17fe2e4a08 integrations: Add Radarr inegration. 2021-05-26 17:11:57 -07:00
Adam Birds 11e7f27031 integrations: Add Lidarr integration. 2021-05-26 17:11:24 -07:00
Adam Birds c7421cd69c integrations: Add `label_create_activity` to unsupported pivotal events.
Fixes #18580.
2021-05-25 20:56:52 -07:00
AdamVB a9e73358bf
integrations: Enhance Grafana integration with alert state.
Having the alert state in the message body is useful when alert topics 
are not defined by alert description but encoded in the url.

E.g. in large environments having a topic for each alert [alerting] and [ok] would 
make it harder to properly track if an alert has been resolved.

When each alert is in a single topic, so far, the alert state has been missing.

This change will add the current alert state and a fitting icon in front
of the alert name.(Similar to the prometheus alertmanager integration)

The test cases have been amended to cover all possible alert states, even
though realistically grafana only fires the ok and alerting states via
webhook.
2021-05-24 14:25:33 -07:00
PIG208 d8db797798 doc: Fix configurations for images in the doc of newrelic.
The screenshot generating mechanism doesn't work for newrelic and
causes error because its configuration file doesn't exist. This
commit fixes the configuration and re-generate the screenshots.
2021-05-10 17:50:32 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Adam Birds 589a7799dc integrations: Add Uptime Robot integration.
Note that the documentation cannot fully use our macros, because
Uptime Robot requires an & of the end of the URL, because of how it
passes its payload.

Fixes #13854. Fixes #13939.
2021-05-09 20:34:19 -07:00
Adam Birds 49b920d33a integrations: Fix bug with github multiple reviews requested on PR.
Fixes a bug in which multiple of the same messages are posted when
multiple reviewers are added to a PR.

Fixes #18393.
2021-05-08 07:56:48 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Adam Birds a3ba8b9efd integrations: Add Freshping webhook integration.
I have added a webhook integration for Freshping.
2021-05-06 11:18:35 -07:00
Adam Birds c72ef7be12 integrations: Move `get_setup_webhook_message` to `common.py`.
Move `get_setup_webhook_message` to
`zerver/lib/webhooks/common.py` so multiple integrations can use this
rather than just those which import `zerver/lib/webhooks/git.py`. Also
added the documentation for this.
2021-05-06 11:18:35 -07:00
Anders Kaseorg 871e73ab8f mypy: Don’t use Iterable for values iterated multiple times.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:06:17 -07:00
PIG208 9ac55a8cf6 integrations: Support handling batch updates for Clubhouse.
As the user can select multiple stories and edit multiple
properties at the same time, this can generate requests
without a "primary_id" containing multiple actions, while
each action contains multiple changes.

Fixes: #18022
2021-04-28 08:12:47 -07:00
PIG208 e1a37d2e0a integrations: Support adding multiple labels at a time for Clubhouse.
Fixes: #18022
2021-04-28 08:12:47 -07:00
PIG208 191041f80f integrations: Add additional tests to the Clubhouse integration.
Fixes: #18022
2021-04-28 08:12:47 -07:00
PIG208 f5528c38c2 integrations: Allow PR updates to multiple stories for Clubhouse.
The fixture "story_update_add_github_pull_request" is changed here as it
doesn't make sense to link a story to a PR without having "pull_request_ids"
changed. The previous example is likely a mistake which occurs when you try
to add a PR that has already been added to a story. This commit also allows
comments under the PR that link it to a story to be sent to the stream.

Fixes: #18022
2021-04-28 08:12:47 -07:00
PIG208 41543fabb8 integrations: Fix patch decorators for unittests of Clubhouse.
Incorrectly patching zerver.lib.webhooks.common.check_send_webhook
_message does not create a mock for the webhook as desired, causing
us to do tests with mock that has never been called.
2021-04-28 08:12:47 -07:00
PIG208 2b8fee7952 integrations: Support actions without primary_id for clubhouse.
Clubhouse has a feature for the user to select multiple stories and
update them at once. This will generate a request without primary_id.

Fixes: #18022
2021-04-28 08:12:47 -07:00
PIG208 c3d15eca7c integrations: Refactor clubhouse webhook's helper functions.
Instead of considering only the action with the primary id, this
refactors the helper functions for generating the topic and body
for the stream messages to accept an arbitrary action and generate
the corresponding message for each of the events.

Fixes: #18022
2021-04-28 08:12:47 -07:00
Adam Birds 188273d8f5 integrations: Add JSON Printer webhook integration.
I have added the JSON Printer integration which will show any webhook
payload inside a code block which is useful for webhook testing.

Fixes #17969.
2021-04-28 07:23:18 -07:00
Anders Kaseorg 178736c8eb docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Anders Kaseorg bdb20a8002 integrations: Convert deprecated Django url to path.
django.conf.urls.url is actually a deprecated alias of
django.urls.re_path, but we want path instead of re_path.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg e7ed907cf6 python: Convert deprecated Django ugettext alias to gettext.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Adam Birds 545cd961f4 integrations: Add docs for GitHub Actions integration.
I have added a documentation page for the GitHub Actions integration to
`/integrations/doc/github-actions` with a link to the Zulip GitHub
Actions repository.

Tweaked by tabbott to add cross-links with the main GitHub integration.
2021-04-15 16:42:31 -07:00
Arun Sankar 146b32d63a test users: Add an escape char to a test username.
Changed the name of the test-user cordelia from `Cordelia Lear` to
`Cordelia, Lear's daughter`.

This change will enable us to test users with escape characters in
their names.

I also updated the Node, Puppeteer, Backend tests and Fixtures to
support this change.
2021-04-13 11:42:06 -07:00
Anders Kaseorg f0e655f1d8 request: Rename validator parameter of REQ to json_validator.
This makes it much more clear that this feature does JSON encoding,
which previously was only indicated in the documentation.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 14:13:06 -07:00
Vishnu KS 92316ef4d1 statuspage: Properly detect the update is for component or incident.
The value of "status_indicator" can be "none" for both the component
and incident updates[1]. Also, it is not at all necessary that the value of
"status_indicator" is always "none" for incident updates[2][3]. So our previous
logic of using the value of "status_indicator" to determine whether
the update is that of a component or incident was incorrect. Instead, we
should use "incident" or "component" keys to determine the type of update.

This commit fixes issues [1] and [2] in sentry.

1. https://sentry.io/organizations/zulip/issues/2303217561
2. https://sentry.io/organizations/zulip/issues/2303197407
3. https://support.atlassian.com/statuspage/docs/enable-webhook-notifications/
2021-03-30 12:06:09 -07:00
Adam Birds 25658153ca integration: Update Zendesk Documentation.
I have updated the docs for the Zendesk to integration to include
numbers to make it easily readable.

Fixes part of #17633.
2021-03-25 01:40:38 -07:00
Tim Abbott c3998a3d3b integrations: Make Hello World example less confusing. 2021-03-25 01:40:38 -07:00
Adam Birds b9549250f3 integrations: Update Hello World Docs.
I have the updated the documentation page for the hello world
integration to include numbers to bring it up to standard and make it
more readable.

Fixes part of #17633.
2021-03-25 01:36:05 -07:00
Adam Birds 1dc4dfef92 integrations: Update deskdotcom integration docs.
I have updated the documentation for the deskdotcom integration to
include numbers for ease of readability.

Fixes part of #17633.
2021-03-25 01:35:57 -07:00
Alex Vandiver e637004695 sentry: Do not assume context lines exist in stacktrace.
`context_line` may not be set.
2021-03-22 12:16:10 -07:00
Sourabh f7ac4bbc5f doc: Correct the location of `Make a Zap` button.
In Zulip's documentation of Zapier, it's mentioned that location of 
the "Make a Zap" button is in the upper right, but it's in the upper left.
2021-03-21 17:39:53 -07:00
Adam Birds 7fd7a1917b integrations: Update Zabbix Documentation.
I have added a note about the fact these instructions are for Zabbix 5.2
and above and the workflow for other versions of Zabbix may be
different.
2021-03-18 15:00:10 -07:00
Adam Birds 3649da27bf integrations: Update Yo Documentation.
I have updated the documentation for the Yp integration to include
number for ease of readability.

Fixes part of #17633.
2021-03-16 15:29:47 -07:00
Adam Birds 795e41f317 integrations: Update Pingdom Documentation.
I have updated the Pingdom documentation to include numbers to increase
ease of readability.

Fixes part of #17633.
2021-03-16 15:29:47 -07:00
Adam Birds 8337dce913 integrations: Update HomeAssistant Documentation.
I have updated the docs for the homeassistant integration to include
numbers to increase visibility.

Fixies part of #17633.
2021-03-16 15:29:47 -07:00
Abhijeet Prasad Bodas 9223dced3b refactor: Rename filter to linkifier in frontend code and docs.
This only leaves `page_params.realm_filters`, which
will be changed in further commits along with the
API change.
2021-03-15 11:19:59 -07:00
Adam Birds fa8914085e integrations: Update Zabbix Documentation.
I have updated the documentation for the Zabbix integration to give the
correct instructions for the latest version of Zabbix (5.2). The old
instructions are now obsolete.

I have also updated the message that is PMd to a user if the webhook
doesn't receive a complete payload to also align with the new
instructions.
2021-03-14 19:33:23 -07:00
Mateusz Mandera 03e54232cf integrations: Use get_user function in dialogflow integration.
Using get_user_profile_by_email is invalid, as it omits the realm, and
also fetches via .delivery_email - our convention is that .email is
supposed to be used for user-facing purposes like this.
2021-03-11 20:30:57 -08:00
Tim Abbott 28d437672f webhooks: Fix spelling of milliseconds. 2021-03-05 12:22:50 -08:00
Gaurav Pandey 3d7462a0e7 integration: Add jotform integration.
Fixes #16554
2021-03-04 11:40:58 -08:00
Tushar Upadhyay 6250902edc
webhooks: Add support for more GitHub pull_request events.
Add support for the locked, unlocked, auto-merge-enabled, and auto-merge-disabled actions 
for the pull_request event.

Fixes #16258.
2021-03-01 14:16:08 -08:00
Anders Kaseorg 1212083218 webhooks: Strengthen format_pull_request_event type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-15 17:05:28 -08:00
Anders Kaseorg 3117969f03 webhooks: Remove get_event_handler ritual and Any casts.
It looks like this ritual was born when a type comment wasn’t working
because it was mistyped without the colon.

Signed-off-by: Anders Kaseorg <anders@zulip.com>'
2021-02-15 17:05:28 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 9773c0f1a8 python: Fix string literal concatenation mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 08:02:51 -05:00
ritik 50b0496a3a bitbucket webhook: Fully deprecate username field. 2021-02-05 09:46:27 -05:00
Alex Vandiver 01658e39a9 sentry: Verify version is supported, first.
Raven SDK does not send a `title` field.
2021-01-17 11:15:40 -08:00
cozyrohan 16d1ab3d5f webhooks/github: Fix repeating description for edits and updates.
This change updates the GitHub Integration webhook
get_opened_or_update_pull_request_body method so that
the description is only printed if it actually changes.
If the update event is a result of some other
attribute update, such as an asignee change, then the
description is not included in the message sent to
the zulip stream.

Fixes #16345
2021-01-04 14:34:17 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Max Zawisa 0e40cc72af newrelic: Added owner field and cleaned up code.
I reformatted the tests and view to include information about who
acknowledged and closed the alert. Only includes the information about
the owner if there was an owner.

Made a few small changes to the refactored bit as requested in review.
2020-12-15 12:04:46 -08:00
Max Zawisa 57e847ab89 newrelic: refactor of time input handling.
Moved time formatting check and conversion to
zerver/lib/webhooks/common.py. Updated tests slightly to match new
output. Removed duration from the calculation because the difference
is less than the precision of output and it complicated the error
handling.
2020-12-15 12:04:46 -08:00
Max Zawisa ec00557962 docs: Updated New Relic documentation.
The docs are updated to work with the new webhook and new process on
https://one.newrelic.com.
2020-12-15 12:04:46 -08:00
Puneeth Chaganti 5dc3489166 webhooks/sentry: Fix URL generated in transform_webhook_payload.
The URL incorrectly had `event` in the URL path, instead of `events`.

Closes #16783
2020-12-02 12:28:45 -08:00
Puneeth Chaganti b7a08323aa webhooks/sentry: Use received key when timestamp key is absent. 2020-12-02 12:28:45 -08:00
Vishnu KS dabbc3445a webhooks: Properly format the currency amount for refunds.
By default all Stripe API amounts are in the currency's smallest unit.
It's upto us to convert it to a bigger unit and show it to the end user.
And refund event used to show the currency in the smallest unit which makes
the output wrong when it comes to most currencies like USD, Europ, INR etc
which uses a bigger unit(eg Dollar instead of Cents) as the standard.
2020-11-29 18:11:24 -08:00
Max Zawisa f05a04e000
webhooks: Update NewRelic webhook for new format.
Update the New Relic webhook and tests to match the format specified
in the New Relic documentation. The new format sends a json body
instead of using url parameters. The old format is no longer supported
by New Relic according to their support staff; as a result, the fixtures for 
the old test cases were removed. Added fixtures for new test cases.

Fixes: #16393.
2020-11-18 16:19:08 -08:00
Vishnu KS 5eb63ddb7a webhooks: Handle dispute events with object IDs prefixed with du.
Sometimes the dispute object IDs are prefixed with `du` instead of `dp`.

https://freenode.logbot.info/stripe/20200605#c4059469

The correct long-term fix here would be to stop using object IDs to
detect the object type of these events and instead maybe make use of
"object" key instead.

https://stripe.com/docs/api/disputes/object#dispute_object-object
2020-11-16 17:05:54 -08:00
Puneeth Chaganti 358f1f9ba7 webhooks/sentry: Support integration configured as webhook.
Sentry allows adding simple webhooks without going through the process
of creating an Internal Integration in Sentry's Integration
Platform[1] (which our docs recommend).

The payload from sent from such a (simple) webhook integration is
slightly different from the payload sent by an Internal Integration
webhook. This commit tries to wrangle this payload into a form that is
usable by our webhook handler to send a notification message.

[1]: https://sentry.io/integration-platform/
2020-11-09 12:02:49 -08:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Steve Howell f29b2884ca bitbucket2: Format user info consistently.
We now use get_user_info() to format all of our
users in messages.
2020-09-23 15:31:38 -07:00
Steve Howell e0b6619dac bitbucket2: Simplify how we display user for fork events.
Even before GDPR changes, it was strange that we displayed
users differently for fork events vs. all other events.

After GDPR, we don't even get the `username` field any
more.

So now we simply use `display_name` if available, and then
we try `nickname`.

See https://developer.atlassian.com/cloud/bitbucket/bitbucket-api-changes-gdpr/
for more context.
2020-09-23 15:31:38 -07:00
Steve Howell 1ef8d79352 bitbucket: Decouple BITBUCKET_FORK_BODY between versions.
We were trying to share the same format string between
the two different versions of bitbucket, but this only
creates confusion, as the two versions are only close
enough to be confusing.

The format string might be the same, but the semantics
are different, as well as the eventual outputs.

For example, the {username} piece here is simple in version
2, but in version 3 we append a url to the user's name.
2020-09-23 15:31:38 -07:00
Hemanth V. Alluri f57196b1d0 webhooks/gitlab: Add fixture for MR opened with multiple assignees.
Fixes #16173.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-09-13 20:54:14 -07:00
Hemanth V. Alluri 54aba8d402 webhooks/gitlab: Use information about all assignees.
Previously, the GitLab webhook code, namely the `get_objects_assignee`
method first tried to get a single assignee and if that failed then it
looks for multiple assignees and then it would return the first
assignee that it found (there's actually a code smell here - a loop
which would always return on the first iteration).

Instead, this commit will change that behavior to first check for
multiple assignees first then for a single assignee if we can't find
multiple assignees. Ultimately it will return a list of all of the
assignees (however many that might be [0, n]). This method has then
aptly been renamed to `get_assignees`.

Finally, we tweked the code using this method to always use it's
output as an "assignees" parameter to templates (there's also an
assignee parameter which we want to avoid here for consistency).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-09-13 20:54:14 -07:00
Hemanth V. Alluri bbe7a54171 webhooks/gitlab: Make permissions consistent across all fixtures.
For some reasons, some of the fixtures had the +x bit set, while
some didn't. What this commit does is make sure that no fixture
is marked as "executable" (for anyone).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-09-13 20:54:14 -07:00
Anders Kaseorg f39d4cf1f0 sentry: Pass format arguments to logging.
semgrep 0.23 correctly flags this as a violation of our logging-format
rule.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 15:37:28 -07:00
Alex Vandiver 3f6e4ff303 webhooks: Move the extra logging information into a formatter.
This clears it out of the data sent to Sentry, where it is duplicative
with the indexed metadata -- and potentially exposes PHI if Sentry's
"make this issue public" feature is used.
2020-09-11 16:43:29 -07:00
Alex Vandiver 4917391133 webhooks: Derive payload from request itself. 2020-09-10 17:47:22 -07:00
Alex Vandiver a1f5f6502c webhooks: In logger, pull user from request, rather than parameter.
request.user is set by validate_api_key, which is called by
webhook_view and authenticated_rest_api_view.
2020-09-10 17:47:22 -07:00
Alex Vandiver d04db7c5fe webhooks: Remove repetitive argument to UnsupportedWebhookEventType.
The name of the webhook can be added by the webhook decorator.
2020-09-10 17:47:21 -07:00
Alex Vandiver cf6ebb9c8d webhooks: Rename api_key_only_webhook_view to webhook_view.
There are no other types of webhook views; this is more concise.
2020-09-10 17:47:21 -07:00
Alex Vandiver 9ea9752e0e webhooks: Rename UnexpectedWebhookEventType to UnsupportedWebhookEventType.
Any exception is an "unexpected event", which means talking about
having an "unexpected event logger" or "unexpected event exception" is
confusing.  As the error message in `exceptions.py` already explains,
this is about an _unsupported_ event type.

This also switches the path that these exceptions are written to,
accordingly.
2020-09-10 17:47:21 -07:00
Alex Vandiver 8016769613 webhooks: Move UnexpectedWebhookEventType into zerver.lib.exceptions.
8e10ab282a moved UnexpectedWebhookEventType into
`zerver.lib.exceptions`, but left the import into
`zserver.lib.webhooks.common` so that webhooks could continue to
import the exception from there.

This clutters things and adds complexity; there is no compelling
reason that the exception's source of truth should not move alongside
all other exceptions.
2020-09-10 17:47:21 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bef46dab3c python: Prefer kwargs form of dict.update.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Steve Howell dad0073331 trello webhook: Ignore copyCard actions.
See https://github.com/zulip/zulip/issues/16185,
which encourages folks to fill in for these
missing actions.
2020-09-03 10:44:39 -07:00
Steve Howell 5bff66b450 github webhook: Always send messages for team edits.
If there are unsupported keys, we still log an error,
but we now also send a message to the stream.  (This
is a good tradeoff for the github webhook, since users
can just turn off notifications if they find it spammy.
Also, we intend to support "repository" soon.)

This is a bit of an experiment to see how this plays
in the field:

    * will customers notice the change?
    * will Sentry reports look any different?
2020-09-03 10:44:39 -07:00
Steve Howell c6b9a23c17 github webhooks: Fix message for unsupported team payloads.
The main thing fixed here is that we weren't turning
on our keys into a list.  And then I refined the message
a bit more, including sorting the keys.

I also avoid the unnecessary "else".
2020-09-03 10:44:39 -07:00
Steve Howell 8785790a27 github tests: Add test_team_edited_error_handling.
This doesn't test much interesting yet, but it
will soon.
2020-09-03 10:44:39 -07:00
Steve Howell cc2dbefc60 mypy: Use better types for EVENT_FUNCTION_MAPPER.
The EVENT_FUNCTION_MAPPER maps a string event name
to a function handler.  Before this we circumvented
mypy checks with a call to get_body_function_based_on_type,
which specified Any as the type of our event function.

Now the types are rigorous.

This change was impossible without the recent commit
to introduce the Helper class.
2020-09-03 10:44:39 -07:00
Steve Howell 4de2b78c25 github refactor: Add Helper class.
The Helper class will soon grow, but the immediate
problem it solves is the need to jankily inspect
the parameters of our get_*_body function.

Most of the changes were handled by an ad hoc
munge.py script.

The substantive changes were adding the Helper
class and passing it in.

And then the linter discovered a place where
the optional include_title parameter wasn't used
(which is one of the reasons to avoid the janky
inspect-signature technique).

As a side note, none of the include_title parameters
needed a default value of False, as we always passed
in an explicit value.

We test cover both sides of include_title, which
you can verify by hard coding it to either True or
False (and seeing the relevant failures), although I
suspect most individual codepaths
only test one value, based on whether "topic" is in
the fixture or not.

Finally, I know Helper is not a great name, but I
intend to evolve the class a bit before deciding
whether a more descriptive name is helpful here.
(For example, an upcoming commit will add a
log_unexpected helper method.)
2020-09-03 10:44:39 -07:00
Steve Howell ead7cbea40 github refactor: Handle header_event explicitly.
We get the header_event one level up the call
stack now, too.

It's somewhat annoying that we have our own
concept of "event" here, instead of just returning
our event handlers directly, or just calling them
directly, but it's a bit non-trivial to fix that
right away.

In passing, I remove the strange OR for "ping",
which is already a key in EVENT_FUNCTION_MAPPER.
2020-09-03 10:44:39 -07:00
Steve Howell 0d9b1817f9 github refactor: Use early-exit idiom.
We also comment a bit more explicitly about the
None case.
2020-09-03 10:44:39 -07:00
Steve Howell 5c916135c9 github webhooks: Avoid string interpolation.
We know the event explicitly here.
2020-09-03 10:44:39 -07:00
Steve Howell 425db931a8 github webhook: Explicitly ignore team actions. 2020-09-03 10:44:39 -07:00
Steve Howell 294fd59983 github webhook: Ignore more pull_request actions.
See https://github.com/zulip/zulip/issues/16258 for
possible follow up here.

We now ignore the following two new pull_request
actions (as well as the three existing ones
from before):

    approved
    converted_to_draft

As the issue above indicates, we may want to actually
support "approved" if we can find somebody to work
on the webhook.  (And then the issue goes a little
broader than what changed here.)
2020-09-03 10:44:39 -07:00
Steve Howell 5dea85a186 github tests: Extract test_ignored_pull_request_actions.
We consolidate the tests and remove the fixtures, which
just have a lot of noisy fields that we ignore.  Also,
pull_request__request_review_removed was named improperly.
2020-09-03 10:44:39 -07:00
Steve Howell 4c2787c35f github tests: Add test_ignored_events. 2020-09-03 10:44:39 -07:00
Steve Howell 040bf82122 github webhook: Remove unused exception class. 2020-09-03 10:44:39 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 72d2e5df15 isort: Enable black profile.
Our isort configuration was almost Black-compatible, but we were
missing ensure_newline_before_comments.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:00:07 -07:00
Alex Vandiver cec594e525 trello: Remove unnecessary "is not supported" verbiage.
UnexpectedWebhookEventType already encapsulates that it is not
supported, and having extra wording here disrupts how the exception
stringifies.
2020-08-31 14:24:35 -07:00
Alex Vandiver a350a9b3d8 trello: Simplify logic for ignored card actions.
Rather than catching, checking action type, and possibly re-raising,
instead return None explicitly from `get_subject_and_body`, which
already signals for a blank success result.  This collocates the logic
of the action types in one place, and removes the complexity of the
re-raise.
2020-08-31 14:24:35 -07:00
Alex Vandiver 398102e215 sentry: When there are multiple stacktraces, take the most recent.
Sentry may get reported multiple exceptions stacks, in the case where
a `raise ...` was caught, and a new exception was `raise`d.  In this
case, the `filename` is the most recent exception -- but the
exceptions are stored in the `exception` key in the order in which
they occurred.  As such, taking the first value with a `stacktrace`
will result in showing the wrong line, or in no stack trace being
resolved at all.

Look from the last `exception` backwards, for matching stacks.
2020-08-31 14:14:20 -07:00
Steve Howell f429df3401 github tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell dfe6960584 minor: Format IGNORED_EVENTS for github. 2020-08-24 12:34:46 -07:00
Steve Howell 0863874050 jira: Ignore worklog_updated event. 2020-08-24 12:34:46 -07:00
Steve Howell 7b344044b4 jira: Ignore version_released event. 2020-08-24 12:34:46 -07:00
Steve Howell 52c17fc731 minor: Format IGNORED_EVENTS for jira. 2020-08-24 12:34:46 -07:00
Steve Howell 56bfae3fac jira tests: Test ignored events. 2020-08-24 12:34:46 -07:00
Steve Howell 1a011b2adb trello: Ignore dueComplete field.
It's possible that this is a new name for the "due"
field, but it's not totally clear.

In the exception we saw in the field:

    payload['action']['data']['old']['dueComplete'] = False
    payload['action']['data']['card']['dueComplete'] = True
2020-08-24 12:34:46 -07:00
Steve Howell 1bbf5491a7 trello: Ignore changes to card "cover".
These events happen when somebody changes the
background image for a Trello card, or something
similar.
2020-08-24 12:34:46 -07:00
Steve Howell 43cc4a2c50 trello tests: Test that we ignore "pos" data.
We will extend this test soon, once we begin
ignoring other fields.
2020-08-24 12:34:46 -07:00
Steve Howell bfe04ff6cf trello: Ignore updateCheckItem/updateList actions. 2020-08-24 12:34:46 -07:00
Steve Howell 37159fc235 trello tests: Generalize test for createCheckItem.
We remove the fixture for create_check_item, which
has been bit-rotting for as long as we have ignored
this type of card data.

Our new test is more powerful, in the sense that it
shows we successfully ignore all fixtures of this
type.

If we want to handle this, we'll just need to get
new, representative fixture data from trello.
2020-08-24 12:34:46 -07:00
Steve Howell b193c22e4d trello tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell 6d65af1790 webhook tests: Remove needless get_body functions.
These all did essentially what the parent implementation
does, except for checking that it's valid json.
2020-08-24 12:34:46 -07:00
Steve Howell 7fbe08f515 webhook tests: Introduce get_payload.
We introduce get_payload for the relatively
exceptional cases where webhooks return payloads
as dicts.

Having a simple "str" type for get_body will
allow us to extract test helpers that use
payloads from get_body() without the ugly
`Union[str, Dict[str, str]]` annotations.

I also tightened up annotations in a few places
where we now call get_payload (using Dict[str, str]
instead of Dict[str, Any]).

In the zendesk test I explicitly stringify
one of the parameters to satisfy mypy.
2020-08-24 12:34:46 -07:00
Steve Howell f7e4cc28eb mypy: Use str for get_body in bitbucket.
We don't need to claim a union type here.
2020-08-24 12:34:46 -07:00
Steve Howell fae3535e79 webhook tests: Avoid check_webhook for multi messages.
If we're not passing in expected_topic or expected_message
to check_webhook, it's better to just call send_webhook_payload,
since we'll want to explicitly check our messages
anyway.

This preps us to always require those fields for
check_webhook, which can prevent insidious testing no-ops.
2020-08-24 12:34:46 -07:00
Steve Howell f74aa29a1c webhook tests: Extract assert_stream_message.
This forces us to be a bit more explicit about testing
the three key values in any stream message, and it
also de-clutters the code a bit.  I eventually want
to phase out do_test_topic and friends, since they
have the pitfall that you can call them and have them
do nothing, because they don't actually require
values to be be passed in.

I also clean up the code a bit for the tests that
have two new messages arriving.
2020-08-24 12:34:46 -07:00
Steve Howell 3a710ab996 webhook tests: Remove stream_name parameter.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.

And then the other two callers are already
checking topics, so they might as well check
stream names, too.

We also have the two stream-oriented callers
make their own call to "subscribe".  And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.

Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper.  This is partly to satisfy mypy, but
it's also a good sanity check.

This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
2020-08-24 12:34:46 -07:00
Steve Howell 2fdf966d9f webhook tests: Rename helper to send_webhook_payload.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.

In passing I also remove "bytes" from the Union type for
"payload" parameter.
2020-08-24 12:34:46 -07:00
Steve Howell 388053db6b webhook tests: Rename main helper to check_webhook.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.

Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.

We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used.  So
for now I just made sure the docstrings of the two
sibling functions reference each other.
2020-08-24 12:34:46 -07:00
Steve Howell 00001a396b webhook tests: Shorten topic constants.
The "EXPECTED_" prefix and "_EVENTS" suffix
usually provided more noise than signal.

We also use module constants to avoid the "self."
noise.  It also makes it a bit more clear which
constants actually have to be in the class (e.g.
"FIXTURE_DIR_NAME") to do their job.
2020-08-24 12:34:46 -07:00
Steve Howell fbef6e3c03 zendesk tests: Fix anti-patterns.
We had optional parameters for expected_topic and
expected_message, which are trivial to eliminate,
since the integration is really simple.

And we were doing strange things trying to reset
class variables at the end of tests.  Now we just
set them explicitly in the tests.
2020-08-24 12:34:46 -07:00
Steve Howell 3de2e4f034 dialogflow test: Fix nonsensical tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored.  So the tests weren't
testing anything.

Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".

I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
2020-08-24 12:34:46 -07:00
Alex Vandiver aa5f98907c sentry: Deal more gracefully with missing stacktraces.
Not all languages produce exceptions with context or filenames.  See
https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Alex Vandiver 0db311ddad sentry: Stacks are returned most-recent last.
Per [1], the sentry API returns frames sorted from oldest to newest.
As such, matching against the first filename that matches is most
likely not the right frame.

Match against the last frame with the guilty filename.

[1] https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 123790a72d webhooks: Fix invalid JSON in Hello World test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 5043f6039b webhooks: Fix invalid JSON in Freshdesk test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg ec6856bc3b webhooks: Fix invalid JSON in Dialogflow test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver e3a093b73f sentry: Add two more languages seen. 2020-08-11 10:15:02 -07:00
Alex Vandiver 34250d76a2 sentry: Don't drop (and server error) on languages we don't know.
It is better to show the stacktrace without highlighting than to drop
it entirely.
2020-08-11 10:15:02 -07:00
Alex Vandiver 569de55b82 sentry: Rename mapping variable to be more explicitly-named. 2020-08-11 10:15:02 -07:00
Alex Vandiver 712affd4df sentry: Document default topic. 2020-08-11 10:15:02 -07:00
Anders Kaseorg f166f4c28c webhooks: Fix invalid JSON in Papertrail test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 15:07:27 -07:00
Anders Kaseorg ae3af9cd56 webhooks: Fix invalid JSON in Jira test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 15:06:05 -07:00
Hemanth V. Alluri 4e1024da5c webhooks/gitlab: Add an option to exclude MR title from topics.
Since the title of a merge request can often change, it shouldn't be a
part of the topic that we send the message to. Otherwise things would
get messy and confusing.

But at the same time we don't want to make this mandatory. So we add
a new boolean GET parameter that can toggle whether or not the topic
should include the MR title (`use_merge_request_title`).

Fixes #15951.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-31 10:33:52 -07:00
davidscherer 0e90442d8b
integrations: Don't call prettify_date on Trello card descriptions.
Card descriptions aren't dates, and calling prettify_date on them results in removing upper case T characters, replacing uppercase Z characters with " UTC", etc. in descriptions when they appear in Zulip.

This was pretty clearly just a copy/paste mistake (these functions are very closely parallel to the *_due_date_* functions above, which do work on dates and call prettify_date).
2020-07-28 17:58:19 -07:00
Hemanth V. Alluri eb2809effe webhooks/freshdesk: Remove key checking step.
The idea behind doing this is that we would rather let the code error
out rather than add to the logs. It's webhook code usually never uses
the logging module so this section of legacy code needed to be changed
or removed.

Assists PR #15942.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 10:55:11 -07:00
David Wood 366cb30db1 integrations: Re-add Canarytoken integration.
This commit re-adds the integration for canarytokens.org, now separate
from the primary Thinkst integration.

Signed-off-by: David Wood <david@davidtw.co>
2020-07-20 15:34:40 -07:00
David Wood f1c9d9c2fe integrations: Fix Thinkst Canary integration.
This commit fixes the Thinkst Canary integration which - based on the
schema in upstream documentation - incorrectly assumed that some fields
would always be sent, which meant that the integration would fail. In
addition, this commit adjusts support for canarytokens to only support
the canarytoken schema with Thinkst Canaries (not Thinkst's
canarytokens.org).

Signed-off-by: David Wood <david@davidtw.co>
2020-07-20 15:34:40 -07:00
Steve Howell 0a88202763 webhooks: Ignore short_name to guess user.
The short_name field isn't super reliable, and
we are about to eliminate it.
2020-07-17 11:04:54 -07:00
Anders Kaseorg 654a03c897 webhooks: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg 768e8ccc55 tests: Make all tests inherit ZulipTestCase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg f2e7076e2a decorator: Replace type: ignore with cast, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Till Schneidereit 6d6d43188d webhooks/github: Include full repository name in notification messages.
This changes the notification messages for events that currently just
include the string `"the repository"` to also include the full (`org/repo`)
name of the affected repository. Messages for the following events are
changed:
- `public`
- `star`
- `watch`
- `repository`
- `team_add`

Background: we're using the GitHub integration for org-wide notifications
for the [Bytecode Alliance Zulip](bytecodealliance.zulipchat.com/), and
having all messages just say "the repository" isn't ideal. Even now one
can hover over the link to see the repo's url, but it'd be much nicer if
the message just contained the full name.

I also changed the message for `star` to include a link to the repository,
same as the `watch` notification.
2020-06-29 13:12:16 -07:00
Anders Kaseorg 596bfb47c4 papertrail: Replace custom validator with check_dict.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 114f859d3a api_travis_webhook: Fix message type annotation.
Not all values of this dict are strings, just the ones we use.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 87f7874a79 CVE-2020-12759: Fix reflected XSS vulnerability in Dropbox webhook.
Also check the challenge argument’s presence before using it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-16 22:46:16 -07:00
Tim Abbott f4ac4be851 sentry webhook: Check for platform support early.
Otherwise, we don't know whether the event format might not have the
`filename` or other parameters.
2020-06-16 16:49:31 -07:00
Tim Abbott 54604257e0 sentry: Provide more clarity around unsupported platforms. 2020-06-16 14:04:16 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Tim Abbott 05455f432e sentry: Enable the javascript name for that platform.
Hopefully this will mean our Sentry integration properly handles
javascript events now.
2020-06-15 13:01:39 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Hemanth V. Alluri 545f40b2c2 webhooks/circleci: Add support for super minimal payloads.
After merging PR #15355 we found that CircleCI may send
super minimal payloads. This does not seem to depend on
.circleci/config.yml since we received two different
types of payloads off of the same configuration.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-14 22:35:41 -07:00
Hemanth V. Alluri 5bbfaa225b webhooks/circleci: Overhaul the CircleCI integration.
This change should add a lot more information to
our CircleCI integration's output.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-14 13:14:55 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
David Wood f0f42f7a94 integrations: Add Thinkst Canary integration.
This commit adds an integration for Thinkst Canaries - physical, VM and
cloud-based canaries for detecting attackers to a network. Thinkst
Canaries can send webhook alerts when canaries have been tripped, and
this integration will post Zulip messages when these webhooks are
received.

Signed-off-by: David Wood <david@davidtw.co>
2020-06-09 16:28:28 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 8c15081069 python: Further pyupgrade changes.
Generated by pyupgrade --py3-plus --keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:23:10 -07:00
Johannes Hochreiter d78d9099d1 gitlab: added pipeline number to event notification
- link to pipeline now also features the pipeline number
- updated tests accordingly
2020-06-07 14:01:04 -07:00
Johannes Hochreiter 60077f3097 gitlab: Added artifact link to pipeline event body
- if artifact was built file name, download link and browse link will be added to pipeline event body
- added test for pipeline with built artifacts
2020-06-07 14:01:04 -07:00
Mateusz Mandera 5b746b4128 docs: Add missing step to dropbox webhook instructions. 2020-06-05 10:21:25 -07:00
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Anders Kaseorg 95f29b5409 tests: Avoid deprecated TestCase method aliases.
Fixes these warnings with python -Wd:

/home/circleci/zulip/zerver/tests/test_middleware.py:43: DeprecationWarning: Please use assertRegex instead.
  self.assertRegexpMatches(
/home/circleci/zulip/zerver/webhooks/travis/tests.py:40: DeprecationWarning: Please use assertNotEqual instead.
  self.assertNotEquals(msg.topic_name(), self.TOPIC)

https://docs.python.org/3/library/unittest.html#deprecated-aliases

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-04 14:15:52 -07:00
Hemanth V. Alluri 9a9c1e0794 webhooks/stripe: Fix the invoice_created event.
The previous code for this event was using a key that's not actually
a part of the payload. So here we simple remove the usage of that key
and add a (previously missing) test for this event.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-03 17:18:43 -07:00
Puneeth Chaganti bb37e4ef00 integrations: Remove corrupted png image in bitbucket docs. 2020-06-01 18:03:06 -07:00
Imran Iqbal 465947d1bb
webhooks: Fix travis display emoji for `Pending` status.
The 🔄 emoji is spelled  :counterclockwise: in Zulip.
2020-05-28 15:02:30 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Hemanth V. Alluri ab03659095 webhooks/github: Provide more detailed info for unexpected events.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-21 12:31:43 -07:00
Imran Iqbal da084528db feat(travis): display emoji for `Pending` status
* Encountered `No emoji specified for status 'Pending'` when using `on_start: always`:

```yaml
notifications:
  webhooks:
    ...
    on_start: always    # default: never
```
2020-05-20 13:09:30 -07:00
Hemanth V. Alluri c8f4177872 webhooks/github: Add a support for team/edited events.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-16 14:59:08 -07:00
Hemanth V. Alluri 7e7847b55a webhooks/stripe: Add support for pyr_ events.
"pyr_" events are like refund events some source called it a pseudo
refund event (https://stackoverflow.com/questions/46296374/how-can-
i-get-the-original-charge-and-refund-ids-of-an-automatic-payout).
Though due to the lack of any documentation on this event I'm not
confident if this is the right fix.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-14 12:02:30 -07:00
Hemanth V. Alluri 20b39bd54e webhooks/stripe: Update refund updated event message.
Also add missing test coverage for this event.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-14 12:02:30 -07:00
cestrell 297185cc12 integrations: Use new function in Github releases.
Use get_release_event_message from webhooks/git.py to format release
events using the newly implemented release message template.

Tweaked by tabbott to handle name=None.

Builds on #14746. Proposed in #14934.
2020-05-12 17:42:49 -07:00
cestrell e3f0b2f20f integrations: Add Gogs webhook for release event.
Extends Gogs integrations in order to support a published release.
Tested on my local Ubuntu development server running on WSL2.

Fixes #14746.
2020-05-12 17:39:38 -07:00
Hemanth V. Alluri cd69da26dc webhooks/github: Ignore the label event. 2020-05-12 09:32:17 -07:00
Hemanth V. Alluri 4dd957faa2 webhooks/github: Add support for pull request ready for review.
GitHub supports opening a draft/WIP pull request and then marking it
as ready for review later on. This PR supports the ready_for_review
action for pull_request events.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-12 09:32:17 -07:00
Tim Abbott d3c7579730 gitlab: Fix missing coverage/bug for job hooks corner case.
Apparently, the change and test I added before didn't quite cover the
corner case that was broken.  This does, and exposes a second bug as
well, which we fix.
2020-05-11 16:13:23 -07:00
Tim Abbott d4f875f7c5 webhooks: Add missing gitlab test fixture. 2020-05-11 15:04:34 -07:00
Tim Abbott 11027161c3 webhooks: Fix handling of GitLab Job Hook events.
This fixes an exception for these events.

Fixture from https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#job-events.
2020-05-11 14:37:01 -07:00
arpit551 f3ab8e66dc ci: add configuration for posting to czo whenever CI fails in master.
Since notify is mostly depreciated  in the latest versions of CircleCI.
Although we can use use notify in CircleCI 2.0 but currently
there is no documentation regarding it.

We could have use notify here rather than this hacky solution but
if we use notify it was not possible to trigger CircleCI webhook only
for the main Zulip repository.

Also corrected the circle ci webhook for the case where we don't receive
previous in post request
2020-05-11 13:45:22 -07:00
Jenny Ghose 180c16c80e integrations: Add an incoming webhook for Grafana.
Tweaked by tabbott to use formatted suggested in one of the various
duplicate PRs for this issue, showing the rule name clearly.

Fixes #12951.
2020-05-11 00:27:38 -07:00
Tim Abbott a0c2121958 docs: Advertise Slack-compatible webhook a bit.
This should make it discoverable enough that users will try it out and
send us feedback.
2020-05-11 00:07:28 -07:00
Chris Heald c80e913c7a webhooks: Add a webhook capable of parsing Slack payloads.
This adds a webhook that can be used to interpret standard Slack
payloads. Since there are a ton of existing Slack integrations out
there, having a webhook which can accept standard Slack payloads can
significantly ease transition pains. Obviously this can't do everything
that Slack payloads can (particularly WRT their widgets/interactions),
but we can ingest text and parse out multi-block payloads into a message
relatively reasonably.
2020-05-11 00:07:28 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Hemanth V. Alluri fb757e91c1 webhooks/sentry: Add back support for the legacy integrations.
When the integration was originally rewritten, support for the
deprecated webhook payloads was removed. We later noticed that some
people using Zulip were still using versions of Sentry that required
the older integration code.

Thus this commit adds back the older integration code and whenever the
Sentry webhook payload does not have a "data" field (which must be
present in all modern payloads as per the documentation at
https://docs.sentry.io/workflow/integrations/integration-platform/webhooks)
we will use the older Sentry integration code.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-07 11:26:19 -07:00
Johannes Hochreiter c20fdb0d8b gitlab: Add pipeline and job URLs to pipeline event body. 2020-05-06 13:36:39 -07:00
pemontto 1688863be0 gitlab: Add support for unapproved merge request. 2020-05-06 13:34:47 -07:00
pemontto 740e01e456 gitlab: Add support for Gitlab system hooks.
Fixes #13960.
2020-05-06 13:34:47 -07:00
pemontto 46a6eb7cf8 gitlab: Add project name and url to custom topic messages. 2020-05-06 13:34:43 -07:00
Hemanth V. Alluri 04811e724d webhooks/sentry: Rewrite the sentry webhook for the latest SDKs.
Sentry has client SDKs for many programming languages and frameworks.
Sentry has deprecated their old "Raven" series of client SDKs in favor
of a new series of client SDKs following their unified API format.

As it stood, our Sentry integration was already outdated being written
for the version 5 payloads (the Raven SDKs stopped at version 6 which
is already vastly different from version 5) when the current and
prominently used version is version 7.

This commit completely rewrites the existing Sentry integration.

Tested and supported events:
- Issue created, resolved, assigned, and ignored events.
- "Sentry events" for "capture exception" and "capture message" with
the Golang, Node.js, and Python SDKs (other SDKs should also work but
only these were used for testing).

For reference:
- Old (Raven) SDK for python:
    https://github.com/getsentry/raven-python
- New (Unified API format) SDK for python:
    https://github.com/getsentry/sentry-python

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-02 13:39:57 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
pemontto 01a57ddd9d webhooks: Use Gitlab project web_url over repo homepage.
This is preferred because the `repository` field isn't available for
project-level events.
2020-04-24 17:50:47 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Puneeth Chaganti 6abb412538 webhooks/teamcity: Change teamcity build triggering user to iago.
To be able to get a screenshot of the personal message using the
`generate-integration-docs-screenshot` tool, this commit changes the
personal build to be triggered by iago, instead of cordelia.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 177a547587 webhooks/github: Delete a duplicate fixture.
The request_review_pull_request.json file is a duplicate of
pull_request__review_requested.json with data for a different pull-request.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 32afb4076a webhooks/semaphore: Remove accidentally created fixture file.
aa12002be7 seems to have accidentally placed a
copy of `sentry/fixtures/exception_message.json` here.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 36623cddfd webhooks/dialogflow: Remove default value for email parameter.
The webhook view used a default value for the email, which gave
non-informative errors when the webhook is incorrectly configured without
the email parameter.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 97792466ec webhooks/basecamp: Add support for todo_uncompleted event type.
We have a fixture for this event in the repo, but the event was not listed
in the supported event types list. It is also documented in the Basecamp API
docs here:
https://github.com/basecamp/bc3-api/blob/master/sections/webhooks.md
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 01efb351d5 webhooks/basecamp: Fix typo in todo_due_on_changed event name.
The event name seems to have been incorrectly called `todo_due_date_changed`
instead of `todo_due_on_changed`. The API docs for webhooks don't mention
the correct event name, but the TODO json payload[1] seems to contain the
`due_on` field, aside from the fixture actually referring to
`todo_due_on_changed` event type.

[1]: https://github.com/basecamp/bc3-api/blob/master/sections/todos.md
2020-04-17 09:41:55 -07:00
Puneeth Chaganti b04813d351 webhook/semaphore: Add only summary line to the notification message. 2020-04-14 12:42:36 -07:00
Puneeth Chaganti 504216c04e webhooks: Remove deprecated Stash webhooks' fixture. 2020-04-12 11:04:30 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Abhinav 41fc7b2ae1 webhooks/semaphore: Add support for Semaphore 2.0 notifications.
Semaphore has currently has two different versions of their product -
Classic and 2.0. This commit adds support for Semaphore 2.0, along side
Semaphore Classic, using the same webhook. This would let the integration
work seamlessly for users who have already configured a Zulip integration in
their Semaphore 2.0 projects.

Semaphore 2.0 currently only supports GitHub and their payloads do not
contain URLs for common entities like commits, pull requests and tags. We
construct URLs for them using templates, but also try to support other
services by providing notifications without URLs.

Closes #14171

Co-authored-by: Puneeth Chaganti <punchagan@muse-amuse.in>
2020-04-09 12:41:40 -07:00