Commit Graph

692 Commits

Author SHA1 Message Date
Tim Abbott d3c7579730 gitlab: Fix missing coverage/bug for job hooks corner case.
Apparently, the change and test I added before didn't quite cover the
corner case that was broken.  This does, and exposes a second bug as
well, which we fix.
2020-05-11 16:13:23 -07:00
Tim Abbott d4f875f7c5 webhooks: Add missing gitlab test fixture. 2020-05-11 15:04:34 -07:00
Tim Abbott 11027161c3 webhooks: Fix handling of GitLab Job Hook events.
This fixes an exception for these events.

Fixture from https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#job-events.
2020-05-11 14:37:01 -07:00
arpit551 f3ab8e66dc ci: add configuration for posting to czo whenever CI fails in master.
Since notify is mostly depreciated  in the latest versions of CircleCI.
Although we can use use notify in CircleCI 2.0 but currently
there is no documentation regarding it.

We could have use notify here rather than this hacky solution but
if we use notify it was not possible to trigger CircleCI webhook only
for the main Zulip repository.

Also corrected the circle ci webhook for the case where we don't receive
previous in post request
2020-05-11 13:45:22 -07:00
Jenny Ghose 180c16c80e integrations: Add an incoming webhook for Grafana.
Tweaked by tabbott to use formatted suggested in one of the various
duplicate PRs for this issue, showing the rule name clearly.

Fixes #12951.
2020-05-11 00:27:38 -07:00
Tim Abbott a0c2121958 docs: Advertise Slack-compatible webhook a bit.
This should make it discoverable enough that users will try it out and
send us feedback.
2020-05-11 00:07:28 -07:00
Chris Heald c80e913c7a webhooks: Add a webhook capable of parsing Slack payloads.
This adds a webhook that can be used to interpret standard Slack
payloads. Since there are a ton of existing Slack integrations out
there, having a webhook which can accept standard Slack payloads can
significantly ease transition pains. Obviously this can't do everything
that Slack payloads can (particularly WRT their widgets/interactions),
but we can ingest text and parse out multi-block payloads into a message
relatively reasonably.
2020-05-11 00:07:28 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Hemanth V. Alluri fb757e91c1 webhooks/sentry: Add back support for the legacy integrations.
When the integration was originally rewritten, support for the
deprecated webhook payloads was removed. We later noticed that some
people using Zulip were still using versions of Sentry that required
the older integration code.

Thus this commit adds back the older integration code and whenever the
Sentry webhook payload does not have a "data" field (which must be
present in all modern payloads as per the documentation at
https://docs.sentry.io/workflow/integrations/integration-platform/webhooks)
we will use the older Sentry integration code.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-07 11:26:19 -07:00
Johannes Hochreiter c20fdb0d8b gitlab: Add pipeline and job URLs to pipeline event body. 2020-05-06 13:36:39 -07:00
pemontto 1688863be0 gitlab: Add support for unapproved merge request. 2020-05-06 13:34:47 -07:00
pemontto 740e01e456 gitlab: Add support for Gitlab system hooks.
Fixes #13960.
2020-05-06 13:34:47 -07:00
pemontto 46a6eb7cf8 gitlab: Add project name and url to custom topic messages. 2020-05-06 13:34:43 -07:00
Hemanth V. Alluri 04811e724d webhooks/sentry: Rewrite the sentry webhook for the latest SDKs.
Sentry has client SDKs for many programming languages and frameworks.
Sentry has deprecated their old "Raven" series of client SDKs in favor
of a new series of client SDKs following their unified API format.

As it stood, our Sentry integration was already outdated being written
for the version 5 payloads (the Raven SDKs stopped at version 6 which
is already vastly different from version 5) when the current and
prominently used version is version 7.

This commit completely rewrites the existing Sentry integration.

Tested and supported events:
- Issue created, resolved, assigned, and ignored events.
- "Sentry events" for "capture exception" and "capture message" with
the Golang, Node.js, and Python SDKs (other SDKs should also work but
only these were used for testing).

For reference:
- Old (Raven) SDK for python:
    https://github.com/getsentry/raven-python
- New (Unified API format) SDK for python:
    https://github.com/getsentry/sentry-python

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-02 13:39:57 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
pemontto 01a57ddd9d webhooks: Use Gitlab project web_url over repo homepage.
This is preferred because the `repository` field isn't available for
project-level events.
2020-04-24 17:50:47 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Puneeth Chaganti 6abb412538 webhooks/teamcity: Change teamcity build triggering user to iago.
To be able to get a screenshot of the personal message using the
`generate-integration-docs-screenshot` tool, this commit changes the
personal build to be triggered by iago, instead of cordelia.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 177a547587 webhooks/github: Delete a duplicate fixture.
The request_review_pull_request.json file is a duplicate of
pull_request__review_requested.json with data for a different pull-request.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 32afb4076a webhooks/semaphore: Remove accidentally created fixture file.
aa12002be7 seems to have accidentally placed a
copy of `sentry/fixtures/exception_message.json` here.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 36623cddfd webhooks/dialogflow: Remove default value for email parameter.
The webhook view used a default value for the email, which gave
non-informative errors when the webhook is incorrectly configured without
the email parameter.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 97792466ec webhooks/basecamp: Add support for todo_uncompleted event type.
We have a fixture for this event in the repo, but the event was not listed
in the supported event types list. It is also documented in the Basecamp API
docs here:
https://github.com/basecamp/bc3-api/blob/master/sections/webhooks.md
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 01efb351d5 webhooks/basecamp: Fix typo in todo_due_on_changed event name.
The event name seems to have been incorrectly called `todo_due_date_changed`
instead of `todo_due_on_changed`. The API docs for webhooks don't mention
the correct event name, but the TODO json payload[1] seems to contain the
`due_on` field, aside from the fixture actually referring to
`todo_due_on_changed` event type.

[1]: https://github.com/basecamp/bc3-api/blob/master/sections/todos.md
2020-04-17 09:41:55 -07:00
Puneeth Chaganti b04813d351 webhook/semaphore: Add only summary line to the notification message. 2020-04-14 12:42:36 -07:00
Puneeth Chaganti 504216c04e webhooks: Remove deprecated Stash webhooks' fixture. 2020-04-12 11:04:30 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Abhinav 41fc7b2ae1 webhooks/semaphore: Add support for Semaphore 2.0 notifications.
Semaphore has currently has two different versions of their product -
Classic and 2.0. This commit adds support for Semaphore 2.0, along side
Semaphore Classic, using the same webhook. This would let the integration
work seamlessly for users who have already configured a Zulip integration in
their Semaphore 2.0 projects.

Semaphore 2.0 currently only supports GitHub and their payloads do not
contain URLs for common entities like commits, pull requests and tags. We
construct URLs for them using templates, but also try to support other
services by providing notifications without URLs.

Closes #14171

Co-authored-by: Puneeth Chaganti <punchagan@muse-amuse.in>
2020-04-09 12:41:40 -07:00
Tim Abbott 0599273cfa bitbucket: Fix support for pushes that update a tag.
Previously, we threw an exception if created/closed were both unset;
apparently that can happen when pushing an update to a tag.
2020-04-07 17:00:20 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Tim Abbott cb1fb94ac8 webhooks: Fix Jira webhook constructing subject still too early.
For event types that we don't yet support, like worklog_created (and
likely many more in the future), it doesn't make sense to call a
function that only parses issue events correctly.
2020-03-27 06:19:00 -07:00
Tim Abbott 9d346673f1 webhooks: Fix handling of additional old-format Jira events.
This should hopefully get the Jira webhook working correctly again for
all event types we handle.
2020-03-26 16:39:03 -07:00
Tim Abbott d52a891858 jira: Fix handling of sprint_started events.
Now that we inspect more attributes of events in determining subjects,
we need to filter events like sprint_started before fetching the subject.
2020-03-26 16:34:02 -07:00
Tim Abbott 6df86dab3e jira: Handle comment_created events without issue details.
I'm not sure what causes some Jira webhook events to not include the
metadata that other events do, but it's definitely a format sent by
real installations of Jira (likely a very old version, since this has
fields missing from what modern Jira does) and we've seen it in
production.

The best we can do is encourage users to upgrade Jira for better data.
2020-03-22 21:43:21 -07:00
Steve Howell 1306239c16 tests: Use email/delivery_email more explicitly.
We try to use the correct variation of `email`
or `delivery_email`, even though in some
databases they are the same.

(To find the differences, I temporarily hacked
populate_db to use different values for email
and delivery_email, and reduced email visibility
in the zulip realm to admins only.)

In places where we want the "normal" realm
behavior of showing emails (and having `email`
be the same as `delivery_email`), we use
the new `reset_emails_in_zulip_realm` helper.

A couple random things:

    - I fixed any error messages that were leaking
      the wrong email

    - a test that claimed to rely on the order
      of emails no longer does (we sort user_ids
      instead)

    - we now use user_ids in some place where we used
      to use emails

    - for IRC mirrors I just punted and used
      `reset_emails_in_zulip_realm` in most places

    - for MIT-related tests, I didn't fix email
      vs. delivery_email unless it was obvious

I also explicitly reset the realm to a "normal"
realm for a couple tests that I frankly just didn't
have the energy to debug.  (Also, we do want some
coverage on the normal case, even though it is
"easier" for tests to pass if you mix up `email`
and `delivery_email`.)

In particular, I just reset data for the analytics
and corporate tests.
2020-03-19 16:04:03 -07:00
Steve Howell c235333041 test performance: Pass in users to api_* helpers.
This reduces query counts in some cases, since
we no longer need to look up the user again. In
particular, it reduces some noise when we
count queries for O(N)-related tests.

The query count is usually reduced by 2 per
API call.  We no longer need to look up Realm
and UserProfile.  In most cases we are saving
these lookups for the whole tests, since we
usually already have the `user` objects for
other reasons.  In a few places we are simply
moving where that query happens within the
test.

In some places I shorten names like `test_user`
or `user_profile` to just be `user`.
2020-03-11 14:18:29 -07:00
Steve Howell 626ad0078d tests: Add uuid_get and uuid_post.
We want a clean codepath for the vast majority
of cases of using api_get/api_post, which now
uses email and which we'll soon convert to
accepting `user` as a parameter.

These apis that take two different types of
values for the same parameter make sweeps
like this kinda painful, and they're pretty
easy to avoid by extracting helpers to do
the actual common tasks.  So, for example,
here I still keep a common method to
actually encode the credentials (since
the whole encode/decode business is an
annoying detail that you don't want to fix
in two places):

    def encode_credentials(self, identifier: str, api_key: str) -> str:
        """
        identifier: Can be an email or a remote server uuid.
        """
        credentials = "%s:%s" % (identifier, api_key)
        return 'Basic ' + base64.b64encode(credentials.encode('utf-8')).decode('utf-8')

But then the rest of the code has two separate
codepaths.

And for the uuid functions, we no longer have
crufty references to realm.  (In fairness, realm
will also go away when we introduce users.)

For the `is_remote_server` helper, I just inlined
it, since it's now only needed in one place, and the
name didn't make total sense anyway, plus it wasn't
a super robust check.  In context, it's easier
just to use a comment now to say what we're doing:

    # If `role` doesn't look like an email, it might be a uuid.
    if settings.ZILENCER_ENABLED and role is not None and '@' not in role:
        # do stuff
2020-03-11 14:18:29 -07:00
Sourabh Singh 1b3cfecf2a
webhooks: Add team reviewers support in github webhook.
The github webhook implementation previously ignored the "team reviewers"
part of pull_request events, resulting in inaccurate output.

Fixes: #14096.
2020-03-10 16:29:59 -07:00
orientor 7d2bb707e7 integrations: Improved Taiga Integration.
This adds links to the users and other small improvements.

Progress towards #13698.
2020-03-04 16:52:52 -08:00
Chris Heald 18e3982acd integrations: Add AlertManager webhook. 2020-02-14 17:43:15 -08:00
Chris Heald a91358e186 webhooks: Fix hellosign webhook.
Hellosign now posts their callback as form/multipart, which Django only
permits to be read once. Attempts to access request.body after the
initial read throw "django.http.request.RawPostDataException: You
cannot access body after reading from request's data stream".

Fixes #13847.
2020-02-12 22:36:11 -08:00
Hemanth V. Alluri 7858e5ee40 webhooks/ansibletower: Update for AWX 9.1.1.
Add a simple compatibility function for AWX 9.x.x. Before AWX 9.x.x
a "friendly_name" key was sent by default. Afterwards it was removed
from being a default key but we can still more or less determine if
the triggering event was a job from the REST-style URL.

Note: It is also technically possible to add the key back by defining
a custom notification template in AWX/Tower.

Resolves #13295.
2020-02-02 20:21:03 -08:00
Mateusz Mandera 90bc2ad8ce webhooks: Fix typo in test_stream_error_pm_to_bot_owner comment. 2020-01-30 12:13:54 -08:00
Jonathan Cobb c7433c83ff integrations: Add errbit integration.
Fixes #13685.
2020-01-16 15:33:51 -08:00
Tim Abbott c10cc24ee8 python: Sort webhooks imports with isort. 2020-01-14 13:07:47 -08:00
shubhamgupta2956 92817e147d webhooks: Remove TrelloWebhookException classes.
This removes zerver/webhooks/trello/view/exceptions.py, which
contained legacy Trello webhook exception related classes.  We replace
them with UnexpectedWebhookEventType, which results in our standard
exception handling for unknown event types running (avoiding too-high
priority error logging).

Fixes #13467.
2020-01-07 18:02:40 -08:00
Tim Abbott 2e6e187c16 integrations: Deduplicate gogs and gitea integrations. 2019-11-18 12:08:09 -08:00
Jan Koscisz b88192d5bb integrations: Add Gitea integration.
Gitea integration adopted from Gogs integration with minor
adjusting. More events are now handled.

Fixes #13346
2019-11-18 11:55:24 -08:00
Jan Koscisz ace1503cb6 integrations: Add missing event handling to Gogs.
Issues event where not handled by Gogs integrations.
This added them to supported events.
2019-11-18 11:54:44 -08:00