zulip/docs/development/setup-advanced.md

Advanced Setup (non-Vagrant)

Contents:

• Installing directly on Ubuntu, Debian, Centos, or Fedora
• Installing manually on other Linux/UNIX
• Installing directly on cloud9
• Using Docker (experimental)

Installing directly on Ubuntu, Debian, Centos, or Fedora

If you'd like to install a Zulip development environment on a computer that's running one of:

• Ubuntu 18.10 Cosmic, 18.04 Bionic, 16.04 Xenial, 14.04 Trusty
• Debian 9 Stretch or 10 Buster
• Centos 7 (beta)
• Fedora 29 (beta)
• RHEL 7 (beta)

You can just run the Zulip provision script on your machine.

Note: you should not use the root user to run the installation. If you are using a remote server, see the section on creating appropriate user accounts.

Warning: there is no supported uninstallation process with this method. If you want that, use the Vagrant environment, where you can just do vagrant destroy to clean up the development environment.

Start by cloning your fork of the Zulip repository and connecting the Zulip upstream repository:

git clone --config pull.rebase https://github.com/YOURUSERNAME/zulip.git
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git

# On CentOS/RHEL, you must first install epel-release, and then python36,
# and finally you must run `sudo ln -nsf /usr/bin/python36 /usr/bin/python3`
# On Fedora, you must first install python3
# From a clone of zulip.git
./tools/provision
source /srv/zulip-py3-venv/bin/activate
./tools/run-dev.py  # starts the development server

Once you've done the above setup, you can pick up the documentation on using the Zulip development environment, ignoring the parts about vagrant (since you're not using it).

Installing manually on Unix

We recommend one of the other installation methods, since they are extremely well-tested and generally Just Work. But if you know what you're doing, these instructions can help you install a Zulip development environment on other Linux/UNIX platforms.

• Newer versions of supported distributions
• OpenBSD 5.8 (experimental)
• Common steps

Because copy-pasting the steps documented here can be error-prone, we prefer to extend tools/provision to support additional platforms over adding new platforms to this documentation (and likely will eventually eliminate this documentation section altogether).

Newer versions of supported distributions

You can use our provisioning tool to setup the Zulip development environment on current versions of these platforms reliably and easily, so we no long maintain manual installation instructions for these platforms.

If tools/provision doesn't yet support a newer release of Debian or Ubuntu that you're using, we'd love to add support for it. It's likely only a few lines of changes to tools/lib/provision.py and scripts/lib/setup-apt-repo if you'd like to do it yourself and submit a pull request, or you can ask for help in #development help on chat.zulip.org, and a core team member can help add support for you.

On OpenBSD 5.8 (experimental):

These instructions are experimental and may have bugs; patches welcome!

Start by cloning your fork of the Zulip repository and connecting the Zulip upstream repository:

git clone --config pull.rebase https://github.com/YOURUSERNAME/zulip.git
git remote add -f upstream https://github.com/zulip/zulip.git

doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \
    memcached libmemcached py-Pillow py-cryptography py-cffi

# Get tsearch_extras and build it (using a modified version which
# aliases int4 on OpenBSD):
git clone https://github.com/blablacio/tsearch_extras
cd tsearch_extras
gmake && sudo gmake install

# Point environment to custom include locations and use newer GCC
# (needed for Node modules):
export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl"
export CXX=eg++

# Create tsearch_data directory:
sudo mkdir /usr/local/share/postgresql/tsearch_data


# Hack around missing dictionary files -- need to fix this to get the
# proper dictionaries from what in debian is the hunspell-en-us
# package.
sudo touch /usr/local/share/postgresql/tsearch_data/english.stop
sudo touch /usr/local/share/postgresql/tsearch_data/en_us.dict
sudo touch /usr/local/share/postgresql/tsearch_data/en_us.affix

Finally continue with the Common steps instructions below.

Common steps

Make sure you have followed the steps specific for your platform:

• OpenBSD 5.8 (experimental)

For managing Zulip's python dependencies, we recommend using virtualenvs.

You must create a Python 3 virtualenv. You must also install appropriate python packages in it.

You should either install the virtualenv in /srv, or put a symlink to it in /srv. If you don't do that, some scripts might not work correctly.

You can run python3 tools/setup/setup_venvs.py. This script will create a virtualenv /srv/zulip-py3-venv.

If you want to do it manually, here are the steps:

sudo virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenv
sudo chown -R `whoami`:`whoami` /srv/zulip-py3-venv
source /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv
pip install --upgrade pip # upgrade pip itself because older versions have known issues
pip install --no-deps -r requirements/dev.txt # install python packages required for development

Now run these commands:

sudo ./scripts/lib/install-node
yarn install
sudo mkdir /srv/zulip-emoji-cache
sudo chown -R `whoami`:`whoami` /srv/zulip-emoji-cache
./tools/setup/emoji/build_emoji
./tools/inline-email-css
./tools/setup/generate-custom-icon-webfont
./tools/setup/build_pygments_data
./tools/setup/generate_zulip_bots_static_files.py
./scripts/setup/generate_secrets.py --development
if [ $(uname) = "OpenBSD" ]; then
    sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/tsearch_data/
else
    sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/*/tsearch_data/
fi
./scripts/setup/configure-rabbitmq
./tools/setup/postgres-init-dev-db
./tools/do-destroy-rebuild-database
./tools/setup/postgres-init-test-db
./tools/do-destroy-rebuild-test-database
./manage.py compilemessages

To start the development server:

./tools/run-dev.py

… and visit http://localhost:9991/.

If you're running your development server on a remote server, look at the remote development docs for port forwarding advice.

Proxy setup for by-hand installation

If you are building the development environment on a network where a proxy is required to access the Internet, you will need to set the proxy in the environment as follows:

• On Ubuntu, set the proxy environment variables using:

export https_proxy=http://proxy_host:port
export http_proxy=http://proxy_host:port

• And set the yarn proxy and https-proxy using:

yarn config set proxy http://proxy_host:port
yarn config set https-proxy http://proxy_host:port

Installing on cloud9

AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and terminal.

This section documents how to setup the Zulip development environment in a cloud9 workspace. If you don't have an existing cloud9 account, you can sign up here.

• Create a Workspace, and select the blank template.
• Resize the workspace to be 1GB of memory and 4GB of disk space. (This is under free limit for both the old Cloud9 and the AWS Free Tier).
• Clone the zulip repo: git clone --config pull.rebase https://github.com/<your-username>/zulip.git
• Restart rabbitmq-server since its broken on cloud9: sudo service rabbitmq-server restart.
• And run provision cd zulip && ./tools/provision, once this is done.
• Activate the zulip virtual environment by source /srv/zulip-py3-venv/bin/activate or by opening a new terminal.

Install zulip-cloud9

There's an NPM package, zulip-cloud9, that provides a wrapper around the Zulip development server for use in the Cloud9 environment.

Note: npm i -g zulip-cloud9 does not work in zulip's virtual environment. Although by default, any packages installed in workspace folder (i.e. the top level folder) are added to $PATH.

cd .. # switch to workspace folder if you are in zulip directory
npm i zulip-cloud9
zulip-dev start # to start the development server

If you get error of the form bash: cannot find command zulip-dev, you need to start a new terminal.

Your development server would be running at https://<workspace-name>-<username>.c9users.io on port 8080. You dont need to add :8080 to your url, since the cloud9 proxy should automatically forward the connection. You might want to visit zulip-cloud9 repo and it's wiki for more info on how to use zulip-cloud9 package.

Using Docker (experimental)

Start by cloning your fork of the Zulip repository and connecting the Zulip upstream repository:

git clone --config pull.rebase https://github.com/YOURUSERNAME/zulip.git
git remote add -f upstream https://github.com/zulip/zulip.git

The docker instructions for development are experimental, so they may have bugs. If you try them and run into any issues, please report them!

You can also use Docker to run a Zulip development environment. First, you need to install Docker in your development machine following the instructions. Some other interesting links for somebody new in Docker are:

• Get Started
• Understand the architecture
• Docker run reference
• Dockerfile reference

Then you should create the Docker image based on Ubuntu Linux, first go to the directory with the Zulip source code:

docker build -t user/zulipdev -f Dockerfile-dev .

Commit and tag the provisioned images. The below will install Zulip's dependencies:

docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash
$ /bin/bash sudo chown -R zulip:zulip /srv/zulip
$ /bin/bash /srv/zulip/tools/provision --docker
docker ps -af ancestor=user/zulipdev
docker commit -m "Zulip installed" <container id> user/zulipdev:v2

Now you can run the docker server with:

docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \
    /srv/zulip/tools/start-dockers

You'll want to read the guide for Zulip development to understand how to use the Zulip development. Note that start-dockers automatically runs tools/run-dev.py inside the container; you can then visit http://localhost:9991 to connect to your new Zulip Docker container.

To view the container's run-dev.py console logs to get important debugging information (and e.g. outgoing emails) printed by the Zulip development environment, you can use:

docker logs --follow <container id>

To restart the server use:

docker ps
docker restart <container id>

To stop the server use:

docker ps
docker kill <container id>

If you want to connect to the Docker instance to run commands (e.g. build a release tarball), you can use:

docker ps
docker exec -it <container id> /bin/bash
$ source /home/zulip/.bash_profile
$ <Your commands>
$ exit

If you want to run all the tests you need to start the servers first, you can do it with:

docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash
$ tools/test-all-docker

You can modify the source code in your development machine and review the results in your browser.

Currently, the Docker workflow is substantially less convenient than the Vagrant workflow and less documented; please contribute to this guide and the Docker tooling if you are using Docker to develop Zulip!