upload: Support S3-compatible S3 hosting providers.

Previously, we were hardcoding the domain s3.amazonaws.com.  Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.

We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it.  Since this doesn't regress existing S3
backend support, it seems worth merging.

docs/production/upload-backends.md

@@ -12,8 +12,8 @@ running quickly. You can later migrate the uploads to S3 by
 [following the instructions here](#migrating-from-local-uploads-to-amazon-s3-backend).
 
 We also support an `S3` backend, which uses the Python `boto` library
-to upload files to Amazon S3 (and, with some work, it should be
-possible to use any other storage provider compatible with `boto`).
+to upload files to Amazon S3 (or an S3-compatible block storage
+provider supported by the `boto` library).
 
 ## S3 backend configuration
 
@@ -27,24 +27,26 @@ two buckets because the "user avatars" bucket is generally configured
 as world-readable, whereas the "uploaded files" one is not.
 
 1. Set `s3_key` and `s3_secret_key` in /etc/zulip/zulip-secrets.conf
-to be the S3 access and secret keys for the IAM account.
+   to be the S3 access and secret keys for the IAM account.
 
 1. Set the `S3_AUTH_UPLOADS_BUCKET` and `S3_AVATAR_BUCKET` settings in
-`/etc/zulip/settings.py` to be the names of the S3 buckets you
-created (e.g. `exampleinc-zulip-uploads`).
+   `/etc/zulip/settings.py` to be the names of the S3 buckets you
+   created (e.g. `exampleinc-zulip-uploads`).
 
 1. Comment out the `LOCAL_UPLOADS_DIR` setting in
-`/etc/zulip/settings.py` (add a `#` at the start of the line).
+   `/etc/zulip/settings.py` (add a `#` at the start of the line).
 
-1. In some AWS regions, you need to explicitly
-    [configure boto](http://boto.cloudhackers.com/en/latest/boto_config_tut.html)
-    to use AWS's SIGv4 signature format (because AWS has stopped
-    supporting the older v3 format in those regions).  You can do this
+1. If you are using a non-AWS block storage provider, or certain AWS
+   regions, you may need to explicitly
+   [configure boto](http://boto.cloudhackers.com/en/latest/boto_config_tut.html).
+   For AWS, you may need to use AWS's SIGv4 signature format (because AWS has stopped
+    supporting the older v3 format in those regions); for other
+   providers, you may just need to set the hostname.  You can do this
     by adding an `/etc/zulip/boto.cfg` containing the following:
     ```
     [s3]
     use-sigv4 = True
-    # Edit to provide your S3 bucket's AWS region here.
+    # Edit to provide your bucket's AWS region or hostname here.
     host = s3.eu-central-1.amazonaws.com
     ```
 

zerver/lib/upload.py

@@ -455,7 +455,8 @@ class S3UploadBackend(ZulipUploadBackend):
         bucket = settings.S3_AVATAR_BUCKET
         medium_suffix = "-medium.png" if medium else ""
         # ?x=x allows templates to append additional parameters with &s
-        return "https://%s.s3.amazonaws.com/%s%s?x=x" % (bucket, hash_key, medium_suffix)
+        return "https://%s.%s/%s%s?x=x" % (bucket, self.connection.DefaultHost,
+                                           hash_key, medium_suffix)
 
     def get_export_tarball_url(self, realm: Realm, export_path: str) -> str:
         bucket = settings.S3_AVATAR_BUCKET
@@ -490,7 +491,8 @@ class S3UploadBackend(ZulipUploadBackend):
     def get_realm_icon_url(self, realm_id: int, version: int) -> str:
         bucket = settings.S3_AVATAR_BUCKET
         # ?x=x allows templates to append additional parameters with &s
-        return "https://%s.s3.amazonaws.com/%s/realm/icon.png?version=%s" % (bucket, realm_id, version)
+        return "https://%s.%s/%s/realm/icon.png?version=%s" % (
+            bucket, self.connection.DefaultHost, realm_id, version)
 
     def upload_realm_logo_image(self, logo_file: File, user_profile: UserProfile,
                                 night: bool) -> None:
@@ -529,7 +531,8 @@ class S3UploadBackend(ZulipUploadBackend):
             file_name = 'logo.png'
         else:
             file_name = 'night_logo.png'
-        return "https://%s.s3.amazonaws.com/%s/realm/%s?version=%s" % (bucket, realm_id, file_name, version)
+        return "https://%s.%s/%s/realm/%s?version=%s" % (
+            bucket, self.connection.DefaultHost, realm_id, file_name, version)
 
     def ensure_medium_avatar_image(self, user_profile: UserProfile) -> None:
         file_path = user_avatar_path(user_profile)
@@ -599,7 +602,7 @@ class S3UploadBackend(ZulipUploadBackend):
         bucket = settings.S3_AVATAR_BUCKET
         emoji_path = RealmEmoji.PATH_ID_TEMPLATE.format(realm_id=realm_id,
                                                         emoji_file_name=emoji_file_name)
-        return "https://%s.s3.amazonaws.com/%s" % (bucket, emoji_path)
+        return "https://%s.%s/%s" % (bucket, self.connection.DefaultHost, emoji_path)
 
     def upload_export_tarball(self, realm: Optional[Realm], tarball_path: str) -> str:
         def percent_callback(complete: Any, total: Any) -> None: