Add HelloWorld webhook integration + Walkthough.

Improves webhook integration docs and Hello World webhook. Includes many suggested improvements from @timabbott and @tomaszkolek.
2016-05-20 10:38:46 -07:00 · 2016-05-20 10:38:46 -07:00 · 31efecf03d
parent 2a37dafcbb
commit 31efecf03d
10 changed files with 489 additions and 0 deletions
--- a/docs/images/helloworld-webhook.png
+++ b/docs/images/helloworld-webhook.png
--- a/docs/integration-guide.md
+++ b/docs/integration-guide.md
@ -62,6 +62,14 @@ with the product at all.
  don't have an API or webhook we can use -- sometimes the right API
  is just not properly documented.

+* A helpful tool for testing your integration is
+  [UltraHook](http://www.ultrahook.com/), which allows you to receive webhook
+  calls via your local Zulip dev environment. This enables you to do end-to-end
+  testing with live data from the service you're integrating and can help you
+  spot why something isn't working or if the service is using custom HTTP
+  headers.
+
+
 ## Writing Webhook integrations

 New Zulip webhook integrations can take just a few hours to write,
@ -106,6 +114,377 @@ Here's how we recommend doing it:

 * Finally, write documentation for the integration (see below)!

+### Files that need to be updated
+
+* `templates/zerver/integrations.html`: Edit to add end-user documentation and
+  integration icon. See [Documenting your
+  integration](#documenting-your-integration) for details.
+* `zerver/test_hooks.py`: Edit to include tests for your webbook. See [Testing
+  and writing tests](testing.html) for details.
+* `zproject/urls.py`: Edit to add externally available url of the webhook and
+  associate with the function added to `zerver/views/webhooks/mywebhook.py`
+
+### Files that need to be created
+
+Select a name for your webhook and use it consistently. The examples below are
+for a webhook named 'MyWebHook'.
+
+* `static/images/integrations/logos/mywebhook.png`: An image to represent
+  your integration in the user interface. Generally this Should be the logo of the
+  platform/server/product you are integrating. See [Documenting your
+  integration](#documenting-your-integration) for details.
+* `static/images/integrations/mywebbook/001.png`: A screen capture of your
+  integration for use in the user interface. You can add as many images as needed
+  to effectively document your webhook integration. See [Documenting your
+  integration](#documenting-your-integration) for details.
+* `zerver/fixtures/mywebhook/mywebhook_messagetype.json`: Sample json payload data
+  used by tests. Add one fixture file per type of message supported by your
+  integration. See [Testing and writing tests](testing.html) for details.
+* `zerver/views/webhooks/mywebhook.py`: Includes the main webhook integration
+  function including any needed helper functions.
+
+### Walkthrough of `Hello World` webhook
+
+Below explains each part of a simple webhook integration, called **Hello
+World**. This webhook sends a "hello" message to the `test` stream and includes
+a link to the Wikipedia article of the day, which it formats from json data it
+receives in the http request.
+
+Use this walkthrough to learn how to write your first webhook
+integration.
+
+#### Step 0: Create fixtures
+
+The first step in creating a webhook is to examine the data that the
+service you want to integrate will be sending to Zulip.
+
+You can use [requestb.in](http://requestb.in/) or a similar tool to capture
+webook payload(s) from the service you are integrating. Examining this
+data allows you to do two things:
+
+1. Determine how you will need to structure your webook code, including what
+   message types your integration should support and how; and,
+2. Create fixtures for your webook tests.
+
+Fixtures enable the testing of webhook integration code without the need to
+actually contact the service being integrated.
+
+Because `Hello World` is a very simple webhook that does one thing, it requires
+only one fixture, `zerver/fixtures/helloworld/helloworld_hello.json`:
+
+```
+{
+  "featured_title":"Marilyn Monroe",
+  "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe",
+}
+```
+
+When writing your own webhook integration, you'll want to write a test function
+for each distinct message condition your webhook supports. You'll also need a
+corresponding fixture for each of these tests. See [Step 3: Create
+tests](#step-3-create-tests) or [Testing](testing.html) for further details.
+
+#### Step 1: Create main webhook code
+
+The majority of the code for your webhook integration will be in a single
+python file in `zerver/views/webhooks/`. The name of this file should be the
+name of your webhook, all lower-case, with file extension `.py`:
+`mywebhook.py`.
+
+The Hello World integration is in `zerver/views/webhooks/helloworld.py`:
+
+```
+from __future__ import absolute_import
+from django.utils.translation import ugettext as _
+from zerver.lib.actions import check_send_message
+from zerver.lib.response import json_success, json_error
+from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view
+from zerver.lib.validator import check_dict, check_string
+
+from zerver.models import Client, UserProfile
+
+from django.http import HttpRequest, HttpResponse
+from six import text_type
+from typing import Dict, Any, Iterable, Optional
+
+@api_key_only_webhook_view('HelloWorld')
+@has_request_variables
+def api_helloworld_webhook(request, user_profile, client,
+                           payload=REQ(argument_type='body'),
+                           stream=REQ(default='test'),
+                           topic=REQ(default='Hello World')):
+    # type: (HttpRequest, UserProfile, Client, Dict[str, Iterable[Dict[str, Any]]], text_type, Optional[text_type]) -> HttpResponse
+
+  # construct the body of the message
+  body = 'Hello! I am happy to be here! :smile:'
+
+  # try to add the Wikipedia article of the day
+  # return appropriate error if not successful
+  try:
+      body_template = '\nThe Wikipedia featured article for today is **[{featured_title}]({featured_url})**'
+      body += body_template.format(**payload)
+  except KeyError as e:
+      return json_error(_("Missing key {} in JSON").format(str(e)))
+
+  # send the message
+  check_send_message(user_profile, client, 'stream', [stream], topic, body)
+
+  # return json result
+  return json_success()
+
+```
+
+The above code imports the required functions and defines the main webhook
+function `api_helloworld_webook`, decorating it with `api_key_only_webhook_view` and
+`has_request_variables`.
+
+You must pass the name of your webhook to the `api_key_only_webhook_view`
+decorator. Here we have used `HelloWorld`. To be consistent with Zulip code
+style, use the name of the product you are integrating in camel case, spelled
+as the product spells its own name (except always first letter upper-case).
+
+You should name your webhook function as such `api_webhookname_webhook` where
+`webhookname` is the name of your webhook and is always lower-case.
+
+At minimum, the webhook function must accept `request` (Django
+[HttpRequest](https://docs.djangoproject.com/en/1.8/ref/request-response/#django.http.HttpRequest)
+object), `user_profile` (Zulip's user object), and `client` (Zulip's analogue
+of UserAgent). You may also want to define additional parameters using the
+`REQ` object.
+
+In the example above, we have defined `payload` which is populated from the
+body of the http request, `stream` with a default of `test` (available by
+default in Zulip dev environment), and `topic` with a default of `Hello World`.
+
+The line that begins `# type` is a mypy type annotation. See [this
+page](mypy.html) for details about how to properly annotate your webhook
+functions.
+
+In the body of the function we define the body of the message as `Hello! I am
+happy to be here! :smile:`. The `:smile:` indicates an emoji. Then we append a
+link to the Wikipedia article of the day as provided by the json payload. If
+the json payload does not include data for `featured_title` and `featured_url`
+we catch a `KeyError` and use `json_error` to return the appropriate
+information: a 400 http status code with relevant details.
+
+Then we send a public (stream) message with `check_send_message` which will
+validate the message and then send it.  
+
+Finally, we return a 200 http status with a JSON format success message via
+`json_success()`.
+
+#### Step 2: Create an api endpoint for the webhook
+
+In order for a webhook to be externally available, it must be mapped to a url.
+This is done in `zproject/urls.py`. Look for the lines:
+
+```
+# Incoming webhook URLs
+urls += [
+    # Sorted integration-specific webhook callbacks.
+```
+
+And you'll find the entry for Hello World:
+
+```
+url(r'^api/v1/external/helloworld$',  'zerver.views.webhooks.helloworld.api_helloworld_webhook'),
+```
+
+This tells the Zulip api to call the `api_helloworld_webhook` function in
+`zerver/views/webhooks/helloworld.py` when it receives a request at
+`/api/v1/external/helloworld`.
+
+At this point, if you're following along and/or writing your own Hello World
+webhook, you have written enough code to test your integration.
+
+You can do so by using Zulip itself or curl on the command line.
+
+Using `manage.py` from within Zulip Dev environment:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./manage.py send_webhook_fixture_message \
+> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \
+> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'
+```
+After which you should see something similar to:
+
+```
+2016-07-07 15:06:59,187 INFO     127.0.0.1       POST    200 143ms (mem: 6ms/13) (md: 43ms/1) (db: 20ms/9q) (+start: 147ms) /api/v1/external/helloworld (helloworld-bot@zulip.com via ZulipHelloWorldWebhook)
+```
+
+Using curl:
+
+```
+curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>
+```
+
+After which you should see:
+```
+{"msg":"","result":"success"}
+```
+
+Using either method will create a message in Zulip:
+
+![Image of Hello World webhook message](images/helloworld-webhook.png)
+
+#### Step 3: Create tests
+
+Every webhook integraton should have a corresponding test class in
+`zerver/tests/test_hooks.py`.
+
+You should name the class `<WebhookName>HookTests` and this class should accept
+`WebhookTestCase`. For our HelloWorld webhook, we name the test class
+`HelloWorldHookTests`:
+
+```
+class HelloWorldHookTests(WebhookTestCase):
+    STREAM_NAME = 'test'
+    URL_TEMPLATE = "/api/v1/external/helloworld?&api_key={api_key}"
+    FIXTURE_DIR_NAME = 'helloworld'
+
+    # Note: Include a test function per each distinct message condition your integration supports
+    def test_hello_message(self):
+        # type: () -> None
+        expected_subject = u"Hello World";
+        expected_message = u"Hello! I am happy to be here! :smile: \nThe Wikipedia featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/Marilyn_Monroe)**";
+
+        # use fixture named helloworld_hello
+        self.send_and_test_stream_message('hello', expected_subject, expected_message,
+                                          content_type="application/x-www-form-urlencoded")
+
+    def get_body(self, fixture_name):
+        # type: (text_type) -> text_type
+        return self.fixture_data("helloworld", fixture_name, file_type="json")
+
+```
+
+When writing tests for your webook, you'll want to include one test function
+(and corresponding fixture) per each distinct message condition that your
+integration supports.
+
+If, for example, we added support for sending a goodbye message to our `Hello
+World` webook, we would add another test function to `HelloWorldHookTests`
+class called something like `test_goodbye_message`:
+
+```
+    def test_goodbye_message(self):
+        # type: () -> None
+        expected_subject = u"Hello World";
+        expected_message = u"Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";
+
+        # use fixture named helloworld_goodbye
+        self.send_and_test_stream_message('goodbye', expected_subject, expected_message,
+                                          content_type="application/x-www-form-urlencoded")
+```
+
+As well as a new fixture `helloworld_goodbye.json` in
+`zerver/fixtures/helloworld/`:
+
+```
+{
+  "featured_title":"Goodbye",
+  "featured_url":"https://en.wikipedia.org/wiki/Goodbye",
+}
+```
+
+Once you have written some tests, you can run just these new tests from within
+the Zulip dev environment with this command:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./tools/test-backend zerver.tests.test_hooks.HelloWorldHookTests
+```
+
+(Note: You must run the tests from `/srv/zulip` directory.)
+
+You will see some script output and if all the tests have passed, you will see:
+
+```
+Running zerver.tests.test_hooks.HelloWorldHookTests.test_hello_message
+DONE!
+```
+
+#### Step 4: Create documentation
+
+Next, we add end-user documentation for our webhook integration to
+`templates/zerver/integrations.html`.
+
+First, add a `div` that displays the logo of your integration and a link to its
+documentation:
+
+```
+ <div class="integration-lozenge integration-helloworld">
+   <a class="integration-link integration-helloworld" href="#helloworld">
+      <img class="integration-logo" src="/static/images/integrations/logos/helloworld.png" alt="Hello World logo" />
+      <span class="integration-label">Hello World</span>
+   </a>
+ </div>
+```
+
+And second, a div with the usage instructions:
+
+```
+<div id="helloworld" class="integration-instructions">
+
+    <p>Learn how Zulip integrations work with this simple Hello World example!</p>
+
+    <p>The Hello World webhook will use the <code>test<code> stream, which is
+    created by default in the Zulip dev environment. If you are running
+    Zulip in production, you should make sure this stream exists.</p>
+
+    <p>Next, on your <a href="/#settings" target="_blank">Zulip
+    settings page</a>, create a Hello World bot.  Construct the URL for
+    the Hello World bot using the API key and stream name:
+      <code>{{ external_api_uri }}/v1/external/helloworld?api_key=abcdefgh&amp;stream=test</code>
+    </p>
+
+    <p>To trigger a notication using this webhook, use `send_webhook_fixture_message` from the Zulip command line:</p>
+    <div class="codehilite">
+      <pre>(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./manage.py send_webhook_fixture_message \
+> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \
+> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'</pre>
+    </div>
+
+    <p>Or, use curl:</p>
+    <div class="codehilite">
+      <pre>curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key></pre>
+    </div>
+
+    <p><b>Congratulations! You're done!</b><br /> Your messages may look like:</p>
+
+    <img class="screenshot" src="/static/images/integrations/helloworld/001.png" />
+</div>
+```
+
+Both blocks should fall alphabetically so we add these two divs between the
+blocks for Github and Hubot, respectively.
+
+See [Documenting your integration](#documenting-your-integration) for further
+details, including how to easily create the message screenshot.
+
+#### Step 5: Preparing a pull request to zulip/zulip
+
+When you have finished your webhook integration and are ready for it to be
+available in the Zulip product, follow these steps to prepare your pull
+request:
+
+1. Run tests including linters and ensure you have addressed any issues they
+   report. See [Testing](testing.html) for details.
+2. Read through [Code styles and conventions](code-style.html) and take a look
+   through your code to double-check that you've followed Zulip's guidelines.
+3. Take a look at your git history to ensure your commits have been clear and
+   logical (see [Version Control](version-control.html) for tips). If not,
+   consider revising them with `git rebase --interactive`. For most webhooks,
+   you'll want to squash your changes into a single commit and include a good,
+   clear commit message.
+4. Push code to your fork.
+5. Submit a pull request to zulip/zulip.
+
+If you would like feedback on your integration as you go, feel free to submit
+pull requests as you go, prefixing them with `[WIP]`.
+
 ## Writing Python script and plugin integrations integrations

 For plugin integrations, usually you will need to consult the
--- a/static/images/integrations/helloworld/001.png
+++ b/static/images/integrations/helloworld/001.png
--- a/static/images/integrations/logos/helloworld.png
+++ b/static/images/integrations/logos/helloworld.png
--- a/templates/zerver/integrations.html
+++ b/templates/zerver/integrations.html
@ -131,6 +131,12 @@
            <span class="integration-label">Freshdesk</span>
         </a>
       </div>
+       <div class="integration-lozenge integration-helloworld">
+         <a class="integration-link integration-helloworld" href="#helloworld">
+            <img class="integration-logo" src="/static/images/integrations/logos/helloworld.png" alt="Hello World logo" />
+            <span class="integration-label">Hello World</span>
+         </a>
+       </div>
       <div class="integration-lozenge integration-git">
         <a class="integration-link integration-git" href="#git">
            <img class="integration-logo" src="/static/images/integrations/logos/git.png" alt="Git logo" />
@ -1153,6 +1159,38 @@

    </div>

+    <div id="helloworld" class="integration-instructions">
+
+        <p>Learn how Zulip integrations work with this simple Hello World example!</p>
+
+        <p>The Hello World webhook will use the <code>test</code> stream, which is
+        created by default in the Zulip dev environment. If you are running
+        Zulip in production, you should make sure this stream exists.</p>
+
+        <p>Next, on your <a href="/#settings" target="_blank">Zulip
+        settings page</a>, create a Hello World bot.  Construct the URL for
+        the Hello World bot using the API key and stream name:
+          <code>{{ external_api_uri }}/v1/external/helloworld?api_key=abcdefgh&amp;stream=test</code>
+        </p>
+
+        <p>To trigger a notication using this webhook, use `send_webhook_fixture_message` from the Zulip command line:</p>
+        <div class="codehilite">
+          <pre>(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./manage.py send_webhook_fixture_message \
+> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \
+> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=&lt;api_key&gt;'</pre>
+        </div>
+
+        <p>Or, use curl:</p>
+        <div class="codehilite">
+          <pre>curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://localhost:9991/api/v1/external/helloworld\?api_key\=&lt;api_key&gt;</pre>
+        </div>
+
+        <p><b>Congratulations! You're done!</b><br /> Your messages may look like:</p>
+
+        <img class="screenshot" src="/static/images/integrations/helloworld/001.png" />
+    </div>
+
    <div id="hubot" class="integration-instructions">


--- a/zerver/fixtures/helloworld/helloworld_goodbye.json
+++ b/zerver/fixtures/helloworld/helloworld_goodbye.json
@ -0,0 +1,4 @@
+{
+  "featured_title":"Goodbye",
+  "featured_url":"https://en.wikipedia.org/wiki/Goodbye",
+}
--- a/zerver/fixtures/helloworld/helloworld_hello.json
+++ b/zerver/fixtures/helloworld/helloworld_hello.json
@ -0,0 +1,4 @@
+{
+  "featured_title":"Marilyn Monroe",
+  "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe",
+}
--- a/zerver/tests/test_hooks.py
+++ b/zerver/tests/test_hooks.py
@ -1703,3 +1703,31 @@ class TrelloHookTests(WebhookTestCase):
        # type: () -> None
        expected_message = u"TomaszKolek renamed the board from Welcome Board to [New name](https://trello.com/b/iqXXzYEj)."
        self.send_and_test_stream_message('renaming_board', u"New name.", expected_message)
+
+class HelloWorldHookTests(WebhookTestCase):
+    STREAM_NAME = 'test'
+    URL_TEMPLATE = "/api/v1/external/helloworld?&api_key={api_key}"
+    FIXTURE_DIR_NAME = 'hello'
+
+    # Note: Include a test function per each distinct message condition your integration supports
+    def test_hello_message(self):
+        # type: () -> None
+        expected_subject = u"Hello World";
+        expected_message = u"Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/Marilyn_Monroe)**";
+
+        # use fixture named helloworld_hello
+        self.send_and_test_stream_message('hello', expected_subject, expected_message,
+                                          content_type="application/x-www-form-urlencoded")
+
+    def test_goodbye_message(self):
+        # type: () -> None
+        expected_subject = u"Hello World";
+        expected_message = u"Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";
+
+        # use fixture named helloworld_goodbye
+        self.send_and_test_stream_message('goodbye', expected_subject, expected_message,
+                                          content_type="application/x-www-form-urlencoded")
+
+    def get_body(self, fixture_name):
+        # type: (text_type) -> text_type
+        return self.fixture_data("helloworld", fixture_name, file_type="json")
--- a/zerver/views/webhooks/helloworld.py
+++ b/zerver/views/webhooks/helloworld.py
@ -0,0 +1,35 @@
+# Webhooks for external integrations.
+from __future__ import absolute_import
+from django.utils.translation import ugettext as _
+from zerver.lib.actions import check_send_message
+from zerver.lib.response import json_success, json_error
+from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view
+from zerver.lib.validator import check_dict, check_string
+from zerver.models import Client, UserProfile
+
+from django.http import HttpRequest, HttpResponse
+from six import text_type
+from typing import Dict, Any, Iterable, Optional
+
+@api_key_only_webhook_view('HelloWorld')
+@has_request_variables
+def api_helloworld_webhook(request, user_profile, client,
+                           payload=REQ(argument_type='body'), stream=REQ(default='test'),
+                           topic=REQ(default='Hello World')):
+    # type: (HttpRequest, UserProfile, Client, Dict[str, Iterable[Dict[str, Any]]], text_type, Optional[text_type]) -> HttpResponse
+
+    # construct the body of the message
+    body = 'Hello! I am happy to be here! :smile:'
+
+    # try to add the Wikipedia article of the day
+    # return appropriate error if not successful
+    try:
+        body_template = '\nThe Wikipedia featured article for today is **[{featured_title}]({featured_url})**'
+        body += body_template.format(**payload)
+    except KeyError as e:
+        return json_error(_("Missing key {} in JSON").format(str(e)))
+
+    # send the message
+    check_send_message(user_profile, client, 'stream', [stream], topic, body)
+
+    return json_success()
--- a/zproject/urls.py
+++ b/zproject/urls.py
@ -262,6 +262,7 @@ urls += [
    url(r'^api/v1/external/desk$',          'zerver.views.webhooks.deskdotcom.api_deskdotcom_webhook'),
    url(r'^api/v1/external/freshdesk$',     'zerver.views.webhooks.freshdesk.api_freshdesk_webhook'),
    url(r'^api/v1/external/github$',        'zerver.views.webhooks.github.api_github_landing'),
+    url(r'^api/v1/external/helloworld$',    'zerver.views.webhooks.helloworld.api_helloworld_webhook'),
    url(r'^api/v1/external/ifttt$',         'zerver.views.webhooks.ifttt.api_iftt_app_webhook'),
    url(r'^api/v1/external/jira$',          'zerver.views.webhooks.jira.api_jira_webhook'),
    url(r'^api/v1/external/newrelic$',      'zerver.views.webhooks.newrelic.api_newrelic_webhook'),