Commit Graph

679 Commits

Author SHA1 Message Date
Hemanth V. Alluri 04811e724d webhooks/sentry: Rewrite the sentry webhook for the latest SDKs.
Sentry has client SDKs for many programming languages and frameworks.
Sentry has deprecated their old "Raven" series of client SDKs in favor
of a new series of client SDKs following their unified API format.

As it stood, our Sentry integration was already outdated being written
for the version 5 payloads (the Raven SDKs stopped at version 6 which
is already vastly different from version 5) when the current and
prominently used version is version 7.

This commit completely rewrites the existing Sentry integration.

Tested and supported events:
- Issue created, resolved, assigned, and ignored events.
- "Sentry events" for "capture exception" and "capture message" with
the Golang, Node.js, and Python SDKs (other SDKs should also work but
only these were used for testing).

For reference:
- Old (Raven) SDK for python:
    https://github.com/getsentry/raven-python
- New (Unified API format) SDK for python:
    https://github.com/getsentry/sentry-python

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-02 13:39:57 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
pemontto 01a57ddd9d webhooks: Use Gitlab project web_url over repo homepage.
This is preferred because the `repository` field isn't available for
project-level events.
2020-04-24 17:50:47 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Puneeth Chaganti 6abb412538 webhooks/teamcity: Change teamcity build triggering user to iago.
To be able to get a screenshot of the personal message using the
`generate-integration-docs-screenshot` tool, this commit changes the
personal build to be triggered by iago, instead of cordelia.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 177a547587 webhooks/github: Delete a duplicate fixture.
The request_review_pull_request.json file is a duplicate of
pull_request__review_requested.json with data for a different pull-request.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 32afb4076a webhooks/semaphore: Remove accidentally created fixture file.
aa12002be7 seems to have accidentally placed a
copy of `sentry/fixtures/exception_message.json` here.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 36623cddfd webhooks/dialogflow: Remove default value for email parameter.
The webhook view used a default value for the email, which gave
non-informative errors when the webhook is incorrectly configured without
the email parameter.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 97792466ec webhooks/basecamp: Add support for todo_uncompleted event type.
We have a fixture for this event in the repo, but the event was not listed
in the supported event types list. It is also documented in the Basecamp API
docs here:
https://github.com/basecamp/bc3-api/blob/master/sections/webhooks.md
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 01efb351d5 webhooks/basecamp: Fix typo in todo_due_on_changed event name.
The event name seems to have been incorrectly called `todo_due_date_changed`
instead of `todo_due_on_changed`. The API docs for webhooks don't mention
the correct event name, but the TODO json payload[1] seems to contain the
`due_on` field, aside from the fixture actually referring to
`todo_due_on_changed` event type.

[1]: https://github.com/basecamp/bc3-api/blob/master/sections/todos.md
2020-04-17 09:41:55 -07:00
Puneeth Chaganti b04813d351 webhook/semaphore: Add only summary line to the notification message. 2020-04-14 12:42:36 -07:00
Puneeth Chaganti 504216c04e webhooks: Remove deprecated Stash webhooks' fixture. 2020-04-12 11:04:30 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Abhinav 41fc7b2ae1 webhooks/semaphore: Add support for Semaphore 2.0 notifications.
Semaphore has currently has two different versions of their product -
Classic and 2.0. This commit adds support for Semaphore 2.0, along side
Semaphore Classic, using the same webhook. This would let the integration
work seamlessly for users who have already configured a Zulip integration in
their Semaphore 2.0 projects.

Semaphore 2.0 currently only supports GitHub and their payloads do not
contain URLs for common entities like commits, pull requests and tags. We
construct URLs for them using templates, but also try to support other
services by providing notifications without URLs.

Closes #14171

Co-authored-by: Puneeth Chaganti <punchagan@muse-amuse.in>
2020-04-09 12:41:40 -07:00
Tim Abbott 0599273cfa bitbucket: Fix support for pushes that update a tag.
Previously, we threw an exception if created/closed were both unset;
apparently that can happen when pushing an update to a tag.
2020-04-07 17:00:20 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Tim Abbott cb1fb94ac8 webhooks: Fix Jira webhook constructing subject still too early.
For event types that we don't yet support, like worklog_created (and
likely many more in the future), it doesn't make sense to call a
function that only parses issue events correctly.
2020-03-27 06:19:00 -07:00
Tim Abbott 9d346673f1 webhooks: Fix handling of additional old-format Jira events.
This should hopefully get the Jira webhook working correctly again for
all event types we handle.
2020-03-26 16:39:03 -07:00
Tim Abbott d52a891858 jira: Fix handling of sprint_started events.
Now that we inspect more attributes of events in determining subjects,
we need to filter events like sprint_started before fetching the subject.
2020-03-26 16:34:02 -07:00
Tim Abbott 6df86dab3e jira: Handle comment_created events without issue details.
I'm not sure what causes some Jira webhook events to not include the
metadata that other events do, but it's definitely a format sent by
real installations of Jira (likely a very old version, since this has
fields missing from what modern Jira does) and we've seen it in
production.

The best we can do is encourage users to upgrade Jira for better data.
2020-03-22 21:43:21 -07:00
Steve Howell 1306239c16 tests: Use email/delivery_email more explicitly.
We try to use the correct variation of `email`
or `delivery_email`, even though in some
databases they are the same.

(To find the differences, I temporarily hacked
populate_db to use different values for email
and delivery_email, and reduced email visibility
in the zulip realm to admins only.)

In places where we want the "normal" realm
behavior of showing emails (and having `email`
be the same as `delivery_email`), we use
the new `reset_emails_in_zulip_realm` helper.

A couple random things:

    - I fixed any error messages that were leaking
      the wrong email

    - a test that claimed to rely on the order
      of emails no longer does (we sort user_ids
      instead)

    - we now use user_ids in some place where we used
      to use emails

    - for IRC mirrors I just punted and used
      `reset_emails_in_zulip_realm` in most places

    - for MIT-related tests, I didn't fix email
      vs. delivery_email unless it was obvious

I also explicitly reset the realm to a "normal"
realm for a couple tests that I frankly just didn't
have the energy to debug.  (Also, we do want some
coverage on the normal case, even though it is
"easier" for tests to pass if you mix up `email`
and `delivery_email`.)

In particular, I just reset data for the analytics
and corporate tests.
2020-03-19 16:04:03 -07:00
Steve Howell c235333041 test performance: Pass in users to api_* helpers.
This reduces query counts in some cases, since
we no longer need to look up the user again. In
particular, it reduces some noise when we
count queries for O(N)-related tests.

The query count is usually reduced by 2 per
API call.  We no longer need to look up Realm
and UserProfile.  In most cases we are saving
these lookups for the whole tests, since we
usually already have the `user` objects for
other reasons.  In a few places we are simply
moving where that query happens within the
test.

In some places I shorten names like `test_user`
or `user_profile` to just be `user`.
2020-03-11 14:18:29 -07:00
Steve Howell 626ad0078d tests: Add uuid_get and uuid_post.
We want a clean codepath for the vast majority
of cases of using api_get/api_post, which now
uses email and which we'll soon convert to
accepting `user` as a parameter.

These apis that take two different types of
values for the same parameter make sweeps
like this kinda painful, and they're pretty
easy to avoid by extracting helpers to do
the actual common tasks.  So, for example,
here I still keep a common method to
actually encode the credentials (since
the whole encode/decode business is an
annoying detail that you don't want to fix
in two places):

    def encode_credentials(self, identifier: str, api_key: str) -> str:
        """
        identifier: Can be an email or a remote server uuid.
        """
        credentials = "%s:%s" % (identifier, api_key)
        return 'Basic ' + base64.b64encode(credentials.encode('utf-8')).decode('utf-8')

But then the rest of the code has two separate
codepaths.

And for the uuid functions, we no longer have
crufty references to realm.  (In fairness, realm
will also go away when we introduce users.)

For the `is_remote_server` helper, I just inlined
it, since it's now only needed in one place, and the
name didn't make total sense anyway, plus it wasn't
a super robust check.  In context, it's easier
just to use a comment now to say what we're doing:

    # If `role` doesn't look like an email, it might be a uuid.
    if settings.ZILENCER_ENABLED and role is not None and '@' not in role:
        # do stuff
2020-03-11 14:18:29 -07:00
Sourabh Singh 1b3cfecf2a
webhooks: Add team reviewers support in github webhook.
The github webhook implementation previously ignored the "team reviewers"
part of pull_request events, resulting in inaccurate output.

Fixes: #14096.
2020-03-10 16:29:59 -07:00
orientor 7d2bb707e7 integrations: Improved Taiga Integration.
This adds links to the users and other small improvements.

Progress towards #13698.
2020-03-04 16:52:52 -08:00
Chris Heald 18e3982acd integrations: Add AlertManager webhook. 2020-02-14 17:43:15 -08:00
Chris Heald a91358e186 webhooks: Fix hellosign webhook.
Hellosign now posts their callback as form/multipart, which Django only
permits to be read once. Attempts to access request.body after the
initial read throw "django.http.request.RawPostDataException: You
cannot access body after reading from request's data stream".

Fixes #13847.
2020-02-12 22:36:11 -08:00
Hemanth V. Alluri 7858e5ee40 webhooks/ansibletower: Update for AWX 9.1.1.
Add a simple compatibility function for AWX 9.x.x. Before AWX 9.x.x
a "friendly_name" key was sent by default. Afterwards it was removed
from being a default key but we can still more or less determine if
the triggering event was a job from the REST-style URL.

Note: It is also technically possible to add the key back by defining
a custom notification template in AWX/Tower.

Resolves #13295.
2020-02-02 20:21:03 -08:00
Mateusz Mandera 90bc2ad8ce webhooks: Fix typo in test_stream_error_pm_to_bot_owner comment. 2020-01-30 12:13:54 -08:00
Jonathan Cobb c7433c83ff integrations: Add errbit integration.
Fixes #13685.
2020-01-16 15:33:51 -08:00
Tim Abbott c10cc24ee8 python: Sort webhooks imports with isort. 2020-01-14 13:07:47 -08:00
shubhamgupta2956 92817e147d webhooks: Remove TrelloWebhookException classes.
This removes zerver/webhooks/trello/view/exceptions.py, which
contained legacy Trello webhook exception related classes.  We replace
them with UnexpectedWebhookEventType, which results in our standard
exception handling for unknown event types running (avoiding too-high
priority error logging).

Fixes #13467.
2020-01-07 18:02:40 -08:00
Tim Abbott 2e6e187c16 integrations: Deduplicate gogs and gitea integrations. 2019-11-18 12:08:09 -08:00
Jan Koscisz b88192d5bb integrations: Add Gitea integration.
Gitea integration adopted from Gogs integration with minor
adjusting. More events are now handled.

Fixes #13346
2019-11-18 11:55:24 -08:00
Jan Koscisz ace1503cb6 integrations: Add missing event handling to Gogs.
Issues event where not handled by Gogs integrations.
This added them to supported events.
2019-11-18 11:54:44 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Jan Koscisz 12bd3c04c0 integration: Fix missing taiga test action.
Previously testing webhook from taiga site would result in error.
This adds missing 'test' action to taiga webhooks.

Fixes #13317.
2019-10-28 14:58:24 -07:00
chgl bea9e41fbd webhooks: Add Harbor webhook integration. 2019-10-21 15:51:35 -07:00
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Rishi Gupta 3646936cfb integrations: Advertise our native Zapier integration. 2019-09-22 18:47:42 -07:00
Rishi Gupta 74d5d3db26 integration docs: Remove angle brackets from trello bash command.
If you accidentally leave them in, you'll get a confusing error message.
2019-09-19 17:48:16 -07:00
Puneeth Chaganti df134be235 webhooks/papertrail: Requests from Papertrail are not JSON requests.
Papertrail sends requests with the content type
`application/x-www-form-urlencoded`, with the payload parameter holding the
JSON body. This commit fixes the papertrail integration to use the payload
parameter in the request's POST data instead of trying to parse the
request's entire body as JSON.

Papertrail documentation here:
https://help.papertrailapp.com/kb/how-it-works/web-hooks#encoding
2019-09-04 13:16:17 -07:00
Rishi Gupta e1848e83bc integrations: Update instructions for AppFollow. 2019-08-22 13:17:06 -07:00
Hemanth V. Alluri 2e20dcc053 webhooks/jira: Update webhook to support improved comment events.
Atlassian announced that it will no longer provide information about
comments along with their "issue" type event payloads for Jira. So
we must now update the Jira integration to appropriately respond to
"comment" type events (the reason why we didn't do this before was
that initially, the "comment" type event payloads didn't contain
sufficient information about their issues, but this payload has
since been improved).

Note: This commit does *not* remove support for the older "issue"
type event payloads where information about comments was included.
This way we can maintain compatibility with old self-hosted versions
of self hosted Jira (2016 and before).

Source:
https://developer.atlassian.com/cloud/jira/platform/change-notice-
removal-of-comments-from-issue-webhooks/

Fixes #13012
2019-08-19 14:22:42 -07:00
Brutus5000 a386e3aebb github webhook: For release events show actions and tag name. 2019-08-17 12:35:16 -07:00
Anders Kaseorg e0cf3d0e76 zerver: Accept HEAD requests wherever GET requests are accepted.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00