zilencer: Add automated signup system for push notifications.

Based on an initial version by Rishi Gupta.

Fixes #7325.

docs/production/mobile-push-notifications.md

@@ -18,28 +18,28 @@ support forwarding push notifications to a central push notification
 forwarding service.  You can enable this for your Zulip server as
 follows:
 
-1. First, contact support@zulipchat.com with the `zulip_org_id` and
+1. If you're running Zulip 1.8.1 or newer, you can run `manage.py
-   `zulip_org_key` values from your `/etc/zulip/zulip-secrets.conf` file, as
+   register_server` from `/home/zulip/deployments/current`.  This
-   well as a `hostname` and `contact email` address you'd like us to use in case
+   command will print the registration data it would send to the
-   of any issues (we hope to have a nice web flow available for this soon).
+   mobile push notifications service, ask you to accept the terms of
+   service, and if you accept, register your server.  Otherwise, see
+   the [legacy signup instructions](#legacy-signup).
 
-2. We'll enable push notifications for your server on our end. Look for a
+1. Uncomment the `PUSH_NOTIFICATION_BOUNCER_URL =
-   reply from Zulipchat support within 24 hours.
+   'https://push.zulipchat.com'` line in your `/etc/zulip/settings.py`
-
+   file (i.e. remove the `#` at the start of the line), and
-3. Uncomment the `PUSH_NOTIFICATION_BOUNCER_URL = "https://push.zulipchat.com"`
-   line in your `/etc/zulip/settings.py` file, and
    [restart your Zulip server](../production/maintain-secure-upgrade.html#updating-settings).
-   Note that if you installed Zulip older than 1.6, you'll need to add
+   If you installed your Zulip server with a version older than 1.6,
-   the line (it won't be there to uncomment).
+   you'll need to add the line (it won't be there to uncomment).
 
-4. If you or your users have already set up the Zulip mobile app,
+1. If you or your users have already set up the Zulip mobile app,
    you'll each need to log out and log back in again in order to start
    getting push notifications.
 
-That should be all you need to do!
+Congratulations!  You've successful setup the service.
 
-If you'd like to verify the full pipeline, you can do the following.
+If you'd like to verify that everything is working, you can do the
-Please follow the instructions carefully:
+following.  Please follow the instructions carefully:
 
 * [Configure mobile push notifications to always be sent][notification-settings]
   (normally they're only sent if you're idle, which isn't ideal for
@@ -57,9 +57,19 @@ in the Android notification area.
 
 [notification-settings]: https://zulipchat.com/help/configure-mobile-notifications
 
-Note that use of the push notification bouncer is subject to the
+## Updating your server's registration
-[Zulipchat Terms of Service](https://zulipchat.com/terms/). By using push
+
-notifications, you agree to those terms.
+Your server's registration includes the server's hostname and contact
+email address (from `EXTERNAL_HOST` and `ZULIP_ADMINISTRATOR` in
+`/etc/zulip/settings.py`, aka the `--hostname` and `--email` options
+in the installer).  You can update your server's registration data by
+running `manage.py register_server` again.
+
+If you'd like to rotate your server's API key for this service
+(`zulip_org_key`), you need to use `manage.py register_server
+--rotate-key` option; it will automatically generate a new
+`zulip_org_key` and store that new key in
+`/etc/zulip/zulip-secrets.conf`.
 
 ## Why this is necessary
 
@@ -77,7 +87,11 @@ notification forwarding service, which allows registered Zulip servers
 to send push notifications to the Zulip app indirectly (through the
 forwarding service).
 
-## Security and privacy implications
+## Security and privacy
+
+Use of the push notification bouncer is subject to the
+[Zulipchat Terms of Service](https://zulipchat.com/terms/). By using
+push notifications, you agree to those terms.
 
 We've designed this push notification bouncer service with security
 and privacy in mind:
@@ -110,6 +124,18 @@ and privacy in mind:
 If you have any questions about the security model, contact
 support@zulipchat.com.
 
+## Legacy signup
+
+Here are the legacy instructions for signing a server up for push
+notifications:
+
+1. First, contact support@zulipchat.com with the `zulip_org_id` and
+   `zulip_org_key` values from your `/etc/zulip/zulip-secrets.conf` file, as
+   well as a `hostname` and `contact email` address you'd like us to use in case
+   of any issues (we hope to have a nice web flow available for this soon).
+
+2. We'll enable push notifications for your server on our end. Look for a
+   reply from Zulipchat support within 24 hours.
 ## Sending push notifications directly from your server
 
 As we discussed above, it is impossible for a single app in their

zerver/management/commands/register_server.py

@@ -0,0 +1,88 @@
+from argparse import ArgumentParser
+import json
+import requests
+import subprocess
+from typing import Any
+
+from django.conf import settings
+from django.core.management.base import CommandError
+from django.utils.crypto import get_random_string
+
+from zerver.lib.management import ZulipBaseCommand, check_config
+from zilencer.models import RemoteZulipServer
+
+if settings.DEVELOPMENT:
+    SECRETS_FILENAME = "zproject/dev-secrets.conf"
+else:
+    SECRETS_FILENAME = "/etc/zulip/zulip-secrets.conf"
+
+class Command(ZulipBaseCommand):
+    help = """Register a remote Zulip server for push notifications."""
+
+    def add_arguments(self, parser: ArgumentParser) -> None:
+        parser.add_argument('--agree_to_terms_of_service',
+                            dest='agree_to_terms_of_service',
+                            action='store_true',
+                            default=False,
+                            help="Agree to the Zulipchat Terms of Service: https://zulipchat.com/terms/.")
+        parser.add_argument('--rotate-key',
+                            dest="rotate_key",
+                            action='store_true',
+                            default=False,
+                            help="Automatically rotate your server's zulip_org_key")
+
+    def handle(self, **options: Any) -> None:
+        check_config()
+
+        if not options['agree_to_terms_of_service'] and not options["rotate_key"]:
+            raise CommandError(
+                "You must agree to the Zulipchat Terms of Service: https://zulipchat.com/terms/.  Run as:\n"
+                "  python manage.py register_remote_server --agree_to_terms_of_service\n")
+
+        if not settings.ZULIP_ORG_ID:
+            raise CommandError("Missing zulip_org_id; run scripts/setup/generate_secrets.py to generate.")
+        if not settings.ZULIP_ORG_KEY:
+            raise CommandError("Missing zulip_org_key; run scripts/setup/generate_secrets.py to generate.")
+
+        request = {
+            "zulip_org_id": settings.ZULIP_ORG_ID,
+            "zulip_org_key": settings.ZULIP_ORG_KEY,
+            "hostname": settings.EXTERNAL_HOST,
+            "contact_email": settings.ZULIP_ADMINISTRATOR}
+        if options["rotate_key"]:
+            request["new_org_key"] = get_random_string(64)
+
+        print("The following data will be submitted to the push notification service:")
+        for key in sorted(request.keys()):
+            print("  %s: %s" % (key, request[key]))
+        print("")
+
+        if not options['agree_to_terms_of_service'] and not options["rotate_key"]:
+            raise CommandError(
+                "You must agree to the Terms of Service: https://zulipchat.com/terms/\n"
+                "  python manage.py register_remote_server --agree_to_terms_of_service\n")
+
+        registration_url = settings.PUSH_NOTIFICATION_BOUNCER_URL + "/api/v1/remotes/server/register"
+        try:
+            response = requests.post(registration_url, params=request)
+        except Exception:
+            raise CommandError("Network error connecting to push notifications service (%s)"
+                               % (settings.PUSH_NOTIFICATION_BOUNCER_URL,))
+        try:
+            response.raise_for_status()
+        except Exception:
+            content_dict = json.loads(response.content.decode("utf-8"))
+            raise CommandError("Error: " + content_dict['msg'])
+
+        if response.json()['created']:
+            print("You've successfully register for the Mobile Push Notification Service!\n"
+                  "To finish setup for sending push notifications:")
+            print("- Uncomment PUSH_NOTIFICATION_BOUNCER_URL in /etc/zulip/settings.py (remove the '#')")
+            print("- Restart the server, using /home/zulip/deployments/current/scripts/restart-server")
+            print("- Return to the documentation to learn how to test push notifications")
+        else:
+            if options["rotate_key"]:
+                print("Success! Updating %s with the new key..." % (SECRETS_FILENAME,))
+                subprocess.check_call(["crudini", '--set', SECRETS_FILENAME, "secrets", "zulip_org_key",
+                                       request["new_org_key"]])
+            print("Mobile Push Notification Service registration successfully updated!")

zerver/tests/test_push_notifications.py

@@ -9,12 +9,15 @@ from typing import Any, Dict, List, Optional, Union, SupportsInt, Text
 
 import base64
 import gcm
+import json
 import os
 import ujson
+import uuid
 
 from django.test import TestCase, override_settings
 from django.conf import settings
 from django.http import HttpResponse
+from django.utils.crypto import get_random_string
 
 from zerver.models import (
     PushDeviceToken,
@@ -1302,3 +1305,95 @@ class TestPushNotificationsContent(ZulipTestCase):
         for test in fixtures:
             actual_output = get_mobile_push_content(test["rendered_content"])
             self.assertEqual(actual_output, test["expected_output"])
+
+class PushBouncerSignupTest(ZulipTestCase):
+    def test_push_signup_invalid_host(self) -> None:
+        zulip_org_id = str(uuid.uuid4())
+        zulip_org_key = get_random_string(64)
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=zulip_org_key,
+            hostname="invalid-host",
+            contact_email="server-admin@example.com",
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_error(result, "invalid-host is not a valid hostname")
+
+    def test_push_signup_invalid_email(self) -> None:
+        zulip_org_id = str(uuid.uuid4())
+        zulip_org_key = get_random_string(64)
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=zulip_org_key,
+            hostname="example.com",
+            contact_email="server-admin",
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_error(result, "Enter a valid email address.")
+
+    def test_push_signup_success(self) -> None:
+        zulip_org_id = str(uuid.uuid4())
+        zulip_org_key = get_random_string(64)
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=zulip_org_key,
+            hostname="example.com",
+            contact_email="server-admin@example.com",
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_success(result)
+        server = RemoteZulipServer.objects.get(uuid=zulip_org_id)
+        self.assertEqual(server.hostname, "example.com")
+        self.assertEqual(server.contact_email, "server-admin@example.com")
+
+        # Update our hostname
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=zulip_org_key,
+            hostname="zulip.example.com",
+            contact_email="server-admin@example.com",
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_success(result)
+        server = RemoteZulipServer.objects.get(uuid=zulip_org_id)
+        self.assertEqual(server.hostname, "zulip.example.com")
+        self.assertEqual(server.contact_email, "server-admin@example.com")
+
+        # Now test rotating our key
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=zulip_org_key,
+            hostname="example.com",
+            contact_email="server-admin@example.com",
+            new_org_key=get_random_string(64),
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_success(result)
+        server = RemoteZulipServer.objects.get(uuid=zulip_org_id)
+        self.assertEqual(server.hostname, "example.com")
+        self.assertEqual(server.contact_email, "server-admin@example.com")
+        zulip_org_key = request["new_org_key"]
+        self.assertEqual(server.api_key, zulip_org_key)
+
+        # Update our hostname
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=zulip_org_key,
+            hostname="zulip.example.com",
+            contact_email="new-server-admin@example.com",
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_success(result)
+        server = RemoteZulipServer.objects.get(uuid=zulip_org_id)
+        self.assertEqual(server.hostname, "zulip.example.com")
+        self.assertEqual(server.contact_email, "new-server-admin@example.com")
+
+        # Now test trying to double-create with a new random key fails
+        request = dict(
+            zulip_org_id=zulip_org_id,
+            zulip_org_key=get_random_string(64),
+            hostname="example.com",
+            contact_email="server-admin@example.com",
+        )
+        result = self.client_post("/api/v1/remotes/server/register", request)
+        self.assert_json_error(result, "zulip_org_id and zulip_org_key do not match.")

zilencer/models.py

@@ -9,10 +9,14 @@ def get_remote_server_by_uuid(uuid: Text) -> 'RemoteZulipServer':
     return RemoteZulipServer.objects.get(uuid=uuid)
 
 class RemoteZulipServer(models.Model):
-    uuid = models.CharField(max_length=36, unique=True)  # type: Text
+    UUID_LENGTH = 36
-    api_key = models.CharField(max_length=64)  # type: Text
+    API_KEY_LENGTH = 64
+    HOSTNAME_MAX_LENGTH = 128
 
-    hostname = models.CharField(max_length=128)  # type: Text
+    uuid = models.CharField(max_length=UUID_LENGTH, unique=True)  # type: Text
+    api_key = models.CharField(max_length=API_KEY_LENGTH)  # type: Text
+
+    hostname = models.CharField(max_length=HOSTNAME_MAX_LENGTH)  # type: Text
     contact_email = models.EmailField(blank=True, null=False)  # type: Text
     last_updated = models.DateTimeField('last updated', auto_now=True)  # type: datetime.datetime
 

zilencer/urls.py

@@ -17,6 +17,9 @@ v1_api_and_json_patterns = [
         {'POST': 'zilencer.views.unregister_remote_push_device'}),
     url('^remotes/push/notify$', rest_dispatch,
         {'POST': 'zilencer.views.remote_server_notify_push'}),
+
+    # Push signup doesn't use the REST API, since there's no auth.
+    url('^remotes/server/register$', zilencer.views.register_remote_server),
 ]
 
 # Make a copy of i18n_urlpatterns so that they appear without prefix for English

zilencer/views.py

@@ -1,6 +1,9 @@
 
 from typing import Any, Dict, Optional, Text, Union, cast
 
+from django.core.exceptions import ValidationError
+from django.core.validators import validate_email, URLValidator
+from django.db import IntegrityError
 from django.http import HttpRequest, HttpResponse
 from django.utils import timezone
 from django.utils.translation import ugettext as _, ugettext as err_
@@ -15,7 +18,8 @@ from zerver.lib.push_notifications import send_android_push_notification, \
     send_apple_push_notification
 from zerver.lib.request import REQ, has_request_variables
 from zerver.lib.response import json_error, json_success
-from zerver.lib.validator import check_int
+from zerver.lib.validator import check_int, check_string, check_url, \
+    validate_login_email, check_capped_string, check_string_fixed_length
 from zerver.models import UserProfile, Realm
 from zerver.views.push_notifications import validate_token
 from zilencer.lib.stripe import STRIPE_PUBLISHABLE_KEY, count_stripe_cards, \
@@ -33,6 +37,49 @@ def validate_bouncer_token_request(entity: Union[UserProfile, RemoteZulipServer]
     validate_entity(entity)
     validate_token(token, kind)
 
+@csrf_exempt
+@require_post
+@has_request_variables
+def register_remote_server(
+        request: HttpRequest,
+        zulip_org_id: str=REQ(str_validator=check_string_fixed_length(RemoteZulipServer.UUID_LENGTH)),
+        zulip_org_key: str=REQ(str_validator=check_string_fixed_length(RemoteZulipServer.API_KEY_LENGTH)),
+        hostname: str=REQ(str_validator=check_capped_string(RemoteZulipServer.HOSTNAME_MAX_LENGTH)),
+        contact_email: str=REQ(str_validator=check_string),
+        new_org_key: Optional[str]=REQ(str_validator=check_string_fixed_length(
+            RemoteZulipServer.API_KEY_LENGTH), default=None),
+) -> HttpResponse:
+    # REQ validated the the field lengths, but we still need to
+    # validate the format of these fields.
+    try:
+        # TODO: Ideally we'd not abuse the URL validator this way
+        url_validator = URLValidator()
+        url_validator('http://' + hostname)
+    except ValidationError:
+        raise JsonableError(_('%s is not a valid hostname') % (hostname,))
+
+    try:
+        validate_email(contact_email)
+    except ValidationError as e:
+        raise JsonableError(e.message)
+
+    remote_server, created = RemoteZulipServer.objects.get_or_create(
+        uuid=zulip_org_id,
+        defaults={'hostname': hostname, 'contact_email': contact_email,
+                  'api_key': zulip_org_key})
+
+    if not created:
+        if remote_server.api_key != zulip_org_key:
+            raise JsonableError(err_("zulip_org_id and zulip_org_key do not match."))
+        else:
+            remote_server.hostname = hostname
+            remote_server.contact_email = contact_email
+            if new_org_key is not None:
+                remote_server.api_key = new_org_key
+            remote_server.save()
+
+    return json_success({'created': created})
+
 @has_request_variables
 def register_remote_push_device(request: HttpRequest, entity: Union[UserProfile, RemoteZulipServer],
                                 user_id: int=REQ(), token: bytes=REQ(),

zproject/dev_settings.py

@@ -87,3 +87,4 @@ SENDFILE_BACKEND = 'sendfile.backends.development'
 
 # Set this True to send all hotspots in development
 ALWAYS_SEND_ALL_HOTSPOTS = False  # type: bool
+PUSH_NOTIFICATION_BOUNCER_URL = EXTERNAL_URI_SCHEME + EXTERNAL_HOST

zproject/test_settings.py

@@ -158,3 +158,4 @@ SOCIAL_AUTH_GITHUB_SECRET = "secret"
 # By default two factor authentication is disabled in tests.
 # Explicitly set this to True within tests that must have this on.
 TWO_FACTOR_AUTHENTICATION_ENABLED = False
+PUSH_NOTIFICATION_BOUNCER_URL = None