storage: Fix static files storage reuse issues.

Zulip's previous model for managing static asset files via Django
pipeline had some broken behavior around upgrades.  In particular, it
was for some reason storing the information as to which static files
should be used in a memcached cache that was shared between different
deployments of Zulip.  This means that during the upgrade process,
some clients might be served a version of the static assets that does
not correspond to the server they were connected to.

We've replaced that model with using ManifestStaticFilesStorage, which
instead allows each Zulip deployment directory to have its own
complete copy of the mapping of files to static assets, as it should
be.

We have to do a little bit of hackery with the staticfiles.json path
to make this work, basically because Django expects staticfiles.json
to be under STATIC_ROOT (aka the path nginx is serving to users), but
doing that doesn't really make sense for Zulip, since that directory
is shared between different deployments.

.gitignore

@@ -21,6 +21,7 @@ static/generated/emoji
 static/generated/github-contributors.json
 static/locale/language_options.json
 /node_modules
+/staticfiles.json
 npm-debug.log
 *.mo
 var/*

tools/build-release-tarball

@@ -112,7 +112,7 @@ find prod-static/serve/generated/emoji/images/emoji/ -regex '.*\.[0-9a-f]+\.png'
 
 cd "$TMPDIR"
 
-tar --append -f "$TARBALL" "$prefix/prod-static" "$prefix/build_id" "$prefix/version"
+tar --append -f "$TARBALL" "$prefix/prod-static" "$prefix/build_id" "$prefix/version" "$prefix/staticfiles.json"
 
 rm -rf "$prefix"
 

tools/update-prod-static

@@ -46,6 +46,10 @@ subprocess.check_call(['./tools/minify-js'] +
 subprocess.check_call(['./manage.py', 'collectstatic', '--no-default-ignore',
                        '--noinput', '-i', 'assets'],
                       stdout=fp, stderr=fp)
+if not settings.PRODUCTION:
+    # When building a release tarball, we need to move staticfiles.json
+    subprocess.check_call(['mv', 'prod-static/serve/staticfiles.json', 'staticfiles.json'],
+                          stdout=fp, stderr=fp)
 
 # Compile translation strings to generate `.mo` files.
 subprocess.check_call(['./manage.py', 'compilemessages'],

zerver/storage.py

@@ -5,7 +5,7 @@ import shutil
 from typing import List, Any, Tuple
 
 from django.conf import settings
-from django.contrib.staticfiles.storage import CachedFilesMixin, StaticFilesStorage
+from django.contrib.staticfiles.storage import ManifestStaticFilesStorage
 from pipeline.storage import PipelineMixin
 
 class AddHeaderMixin(object):
@@ -76,8 +76,24 @@ class RemoveUnminifiedFilesMixin(object):
 
         return []
 
+if settings.PRODUCTION:
+    # This is a hack to use staticfiles.json from within the
+    # deployment, rather than a directory under STATIC_ROOT.  By doing
+    # so, we can use a different copy of staticfiles.json for each
+    # deployment, which ensures that we always use the correct static
+    # assets for each deployment.
+    ManifestStaticFilesStorage.manifest_name = os.path.join(settings.DEPLOY_ROOT,
+                                                            "staticfiles.json")
+    orig_path = ManifestStaticFilesStorage.path
+
+    def path(self, name):
+        # type: (Any, str) -> str
+        if name == ManifestStaticFilesStorage.manifest_name:
+            return name
+        return orig_path(self, name)
+    ManifestStaticFilesStorage.path = path
 
 class ZulipStorage(PipelineMixin,
                    AddHeaderMixin, RemoveUnminifiedFilesMixin,
-                   CachedFilesMixin, StaticFilesStorage):
+                   ManifestStaticFilesStorage):
     pass