6.9 KiB

Raw Blame History

Static asset pipeline

This page documents additional information that may be useful when developing new features for Zulip that require front-end changes, especially those that involve adding new files. For a more general overview, see the new feature tutorial.

Our dependencies documentation has useful relevant background as well.

Primary build process

Most of the existing JS in Zulip is written in IIFE-wrapped modules, one per file in the static/js directory. We will over time migrate these to Typescript modules. Stylesheets are written in the Sass extension of CSS (with the scss syntax), they are converted from plain CSS and we have yet to take full advantage of the features Sass offers. We use Webpack to transpile and build JS and CSS bundles that the browser can understand, one for each entry points specifed in tools/webpack.assets.json; source maps are generated in the process for better debugging experience.

In development mode, bundles are built and served on the fly using webpack-dev-server with live reloading. In production mode (and when creating a release tarball using tools/build-release-tarball), the tools/update-prod-static tool (called by both tools/build-release-tarball and tools/upgrade-zulip-from-git) is responsible for orchestrating the webpack build, JS minification and a host of other steps for getting the assets ready for deployment.

Adding static files

To add a static file to the app (JavaScript, TypeScript, CSS/Sass, images, etc), first add it to the appropriate place under static/.

Third-party packages from the NPM repository should be added to package.json for management by yarn, this allows them to be upgraded easily and not bloat our codebase. Run ./tools/provision for yarn to install the new packages and update its lock file. You should also update PROVISION_VERSION in version.py in the same commit. When adding modules to package.json, please pin specific versions of them (don't using carets ^, tildes ~, etc). We prefer fixed versions so that when the upstream providers release new versions with incompatible APIs, it can't break Zulip. We update those versions periodically to ensure we're running a recent version of third-party libraries.
Third-party files that we have patched should all go in static/third/. Tag the commit with "[third]" when adding or modifying a third-party package. Our goal is to the extent possible to eliminate patched third-party code from the project.
Our own JavaScript and TypeScript files live under static/js. Ideally, new modules should be written in TypeScript (details on this policy below).
CSS/Sass files lives under static/styles.
Portico JavaScript ("portico" means for logged-out pages) lives under static/js/portico.
Custom SVG graphics living under static/assets/icons are compiled into custom icon webfonts that live under static/generated/icons/fonts by tools/setup/generate-custom-icon-webfont according to the static/icons/fonts/template.hbs template.

For your asset to be included in a development/production bundle, it needs to be accessible from one of the entry points defined in tools/webpack.assets.json.

If you plan to only use the file within the app proper, and not on the login page or other standalone pages, put it in the app bundle by importing it in static/js/bundles/app.js.
If it needs to be available both in the app and all logged-out/portico pages, add it to the common entry. Note that you also need to import it to static/js/bundles/commons.js which in itself is imported to the app bundle (this duplication dates back to the transition from our legacy django-pipeline system and should be fixed).
If it's just used on a single standalone page (e.g. /stats), create a new entry point in tools/webpack.assets.json. Use the render_bundle function in the relevant Jinja2 template to inject the compiled JS and CSS.

If you want to test minified files in development, look for the PIPELINE_ENABLED = line in zproject/settings.py and set it to True -- or just set DEBUG = False.

Note that static/html/5xx.html will only render properly if minification is enabled, since they, by nature, hardcode the path static/min/portico.css.

How it works in production

You can learn a lot from reading about django-pipeline, but a few useful notes are:

Zulip installs static assets in production in /home/zulip/prod-static. When a new version is deployed, before the server is restarted, files are copied into that directory.
We use the VFL (Versioned File Layout) strategy, where each file in the codebase (e.g. favicon.ico) gets a new name (e.g. favicon.c55d45ae8c58.ico) that contains a hash in it. Each deployment, has a manifest file (e.g. /home/zulip/deployments/current/staticfiles.json) that maps codebase filenames to serving filenames for that deployment. The benefit of this VFL approach is that all the static files for past deployments can coexist, which in turn eliminates most classes of race condition bugs where browser windows opened just before a deployment can't find their static assets. It also is necessary for any incremental rollout strategy where different clients get different versions of the site.
Some paths for files (e.g. emoji) are stored in the rendered_content of past messages, and thus cannot be removed without breaking the rendering of old messages (or doing a mass-rerender of old messages).

CommonJS/Typescript modules

Webpack provides seemless interoperability between different module systems such as CommonJS, AMD and ES6. Our JS files are written in the CommonJS format, which specifies public functions and variables as properties of the special module.exports object. We also currently assign said object to the global window variable, which is a hack allowing us to use modules without importing them with the require() statement.

New modules should ideally be written in TypeScript (though in cases where one is moving code from an existing JavaScript module, the new commit should just move the code, not translate it to TypeScript).

TypeScript provides more accurate information to development tools, allowing for better refactoring, auto-completion and static analysis. TypeScript uses an ES6-like module system. Any declaration can be made public by adding the export keyword. Consuming variables, functions, etc exported from another module should be done with the import statement as oppose to accessing them from the global window scope. Internally our typescript compiler is configured to transpile TS to the ES6 module system.

Read more about these module systems here:

6.9 KiB Raw Blame History