From 55ddc35b90ce236caca27ada875a6fd31ba8bd42 Mon Sep 17 00:00:00 2001 From: Tim Abbott Date: Tue, 3 Nov 2015 20:57:54 -0800 Subject: [PATCH] Move development installations instructions to a separate file. This will make the core README.md file cleaner. --- README.dev.md | 296 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 308 ++------------------------------------------------ 2 files changed, 307 insertions(+), 297 deletions(-) create mode 100644 README.dev.md diff --git a/README.dev.md b/README.dev.md new file mode 100644 index 0000000000..fa3ff675dd --- /dev/null +++ b/README.dev.md @@ -0,0 +1,296 @@ + +Installing the Zulip Development environment +============================================ + +You will need a machine with at least 2GB of RAM available (see +https://github.com/zulip/zulip/issues/32 for a plan for how to +dramatically reduce this requirement). + +Start by cloning this repository: `git clone https://github.com/zulip/zulip.git` + +Using Vagrant +------------- + +This is the recommended approach, and is tested on OS X 10.10 as well as Ubuntu 14.04. + +* The best performing way to run the Zulip development environment is + using an LXC container. If your host is Ubuntu 14.04 (or newer; + what matters is having support for LXC containers), you'll want to + install and configure the LXC Vagrant provider like this: + `sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir && vagrant plugin install vagrant-lxc` + +* If your host is OS X, [download VirtualBox](https://www.virtualbox.org/wiki/Downloads), + [download Vagrant](https://www.vagrantup.com/downloads.html), and install them both. + +Once that's done, simply change to your zulip directory and run +`vagrant up` in your terminal to install the development server. This +will take a long time on the first run because Vagrant needs to +download the Ubuntu Trusty base image, but later you can run `vagrant +destroy` and then `vagrant up` again to rebuild the environment and it +will be much faster. + +Once that finishes, you can run the development server as follows: + +``` +vagrant ssh -- -L9991:localhost:9991 +# Now inside the container +cd /srv/zulip +source /srv/zulip-venv/bin/activate +./tools/run-dev.py --interface='' +``` + +You can now visit in your browser. To get +shell access to the virtual machine running the server, use `vagrant ssh`. + +(A small note on tools/run-dev.py: the `--interface=''` option will make +the development server listen on all network interfaces. While this +is correct for the Vagrant guest sitting behind a NAT, you probably +don't want to use that option when using run-dev.py in other environments). + +The run-dev.py console output will show any errors your Zulip +development server encounters. It runs on top of Django's [manage.py +runserver](https://docs.djangoproject.com/en/1.8/ref/django-admin/#runserver-port-or-address-port) +tool, which will automatically restart the Zulip Django and Tornado +servers whenever you save changes to Python code. + +However, the Zulip queue workers will not automatically restart when +you save changes, so you will need to ctrl-C and then restart +`run-dev.py` manually if you are testing changes to the queue workers +or if a queue worker has crashed. + +Using provision.py without Vagrant +---------------------------------- + +If you'd like to install a Zulip development environment on a server +that's already running Ubuntu 14.04 Trusty, you can do that by just +running: + +``` +sudo apt-get update +sudo apt-get install -y python-pbs +python /srv/zulip/provision.py + +cd /srv/zulip +source /srv/zulip-venv/bin/activate +./tools/run-dev.py +``` + +By hand +------- +If you really want to install everything by hand, the below +instructions should work. + +Install the following non-Python dependencies: + * libffi-dev — needed for some Python extensions + * postgresql 9.1 or later — our database (also install development headers) + * memcached (and headers) + * rabbitmq-server + * libldap2-dev + * python-dev + * redis-server — rate limiting + * tsearch-extras — better text search + * libfreetype6-dev - needed before you pip install Pillow to properly generate emoji PNGs + +On Debian or Ubuntu systems: + +``` +sudo apt-get install libffi-dev memcached rabbitmq-server libldap2-dev python-dev redis-server postgresql-server-dev-all libmemcached-dev libfreetype6-dev + +# If on 12.04 or wheezy: +sudo apt-get install postgresql-9.1 +wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb +sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb + +# If on 14.04: +sudo apt-get install postgresql-9.3 +wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.3-tsearch-extras_0.1.2_amd64.deb +sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.2_amd64.deb + +# If on 15.04 or jessie: +sudo apt-get install postgresql-9.4 +wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb +sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb +``` + +Now continue with the "All systems" instructions below. + +On Fedora 22 (experimental): + +These instructions are experimental and may have bugs; patches welcome! + +``` +sudo dnf install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server postgresql-devel postgresql libmemcached-devel freetype-devel +wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz +tar xvzf tsearch-extras_0.1.3.tar.gz +cd ts2 +make +sudo make install + +# Hack around missing dictionary files -- need to fix this to get +# the proper dictionaries from what in debian is the hunspell-en-us package. +sudo touch /usr/share/pgsql/tsearch_data/english.stop +sudo touch /usr/share/pgsql/tsearch_data/en_us.dict +sudo touch /usr/share/pgsql/tsearch_data/en_us.affix + +# Edit the postgres settings: +sudo vi /var/lib/pgsql/data/pg_hba.conf + +# Add this line before the first uncommented line to enable password auth: +host all all 127.0.0.1/32 md5 + +# Start the services +sudo systemctl start redis memcached rabbitmq-server postgresql + +# Enable automatic service startup after the system startup +sudo systemctl enable redis rabbitmq-server memcached postgresql +``` + +Now continue with the All Systems instructions below. + +On CentOS 7 Core (experimental): + +These instructions are experimental and may have bugs; patches welcome! + +``` +# Add user zulip to the system (not necessary if you configured zulip as the administrator +# user during the install process of CentOS 7). +useradd zulip + +# Create a password for zulip user +passwd zulip + +# Allow zulip to sudo +visudo +# Add this line after line `root ALL=(ALL) ALL` +zulip ALL=(ALL) ALL + +# Switch to zulip user +su zulip + +# Enable EPEL 7 repo so we can install rabbitmq-server, redis and other dependencies +sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + +# Install dependencies +sudo yum install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server \ +postgresql-devel postgresql libmemcached-devel wget python-pip openssl-devel freetype-devel libjpeg-turbo-devel \ +zlib-devel nodejs + +# We need these packages to compile tsearch-extras +sudo yum groupinstall "Development Tools" + +cd && wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz +tar xvzf tsearch-extras_0.1.3.tar.gz +cd ts2 +make -j $(nproc) +sudo make install + +# Hack around missing dictionary files -- need to fix this to get +# the proper dictionaries from what in debian is the hunspell-en-us package. +sudo touch /usr/share/pgsql/tsearch_data/english.stop +sudo touch /usr/share/pgsql/tsearch_data/en_us.dict +sudo touch /usr/share/pgsql/tsearch_data/en_us.affix + +# clone Zulip's git repo and cd into it +cd && git clone https://github.com/zulip/zulip && cd zulip/ + +# Initialize the postgres db +sudo postgresql-setup initdb + +# Edit the postgres settings: +sudo vi /var/lib/pgsql/data/pg_hba.conf + +# Add these two lines after line 80: +local zulip zulip md5 +host all all 127.0.0.1/32 md5 + +# Change these lines: +host all all 127.0.0.1/32 ident +host all all ::1/128 ident +# to this: +host all all 127.0.0.1/32 md5 +host all all ::1/128 md5 + +# Start the services +sudo systemctl start redis rabbitmq-server memcached postgresql + +# Enable automatic service startup after the system startup +sudo systemctl enable redis rabbitmq-server memcached postgresql +``` + +Now continue with the All Systems instructions below. + +All Systems: + +``` +pip install -r requirements.txt +./tools/download-zxcvbn +./tools/emoji_dump/build_emoji +./scripts/setup/generate_secrets.py -d +sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/9.3/tsearch_data/ +./scripts/setup/configure-rabbitmq +./tools/postgres-init-dev-db +./tools/do-destroy-rebuild-database +./tools/postgres-init-test-db +./tools/do-destroy-rebuild-test-database +``` + +To start the development server: + +``` +./tools/run-dev.py +``` + +… and visit [http://localhost:9991/](http://localhost:9991/). + + +Running the test suite +====================== + +Run all tests: + +``` +./tools/test-all +``` + +This runs the linter (`tools/lint-all`) plus all of our test suites; +they can all be run separately (just read `tools/test-all` to see +them). You can also run individual tests which can save you a lot of +time debugging a test failure, e.g.: + +``` +./tools/test-backend zerver.test_bugdown.BugdownTest.test_inline_youtube +./tools/test-js-with-casper 10-navigation.js +./tools/test-js-with-node # Runs all node tests but is very fast +``` + +The above instructions include the first-time setup of test databases, +but you may need to rebuild the test database occasionally if you're +working on new database migrations. To do this, run: + +``` +./tools/postgres-init-test-db +./tools/do-destroy-rebuild-test-database +``` + +Possible testing issues +======================= + +- The Casper tests are flaky on the Virtualbox environment (probably + due to some performance-sensitive races; they work reliably in + Travis CI). Until this issue is debugged, you may need to rerun + them to get them to pass. + +- When running the test suite, if you get an error like this: + + ``` + sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not exist + LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l... + ^ + ``` + + … then you need to install tsearch-extras, described + above. Afterwards, re-run the `init*-db` and the + `do-destroy-rebuild*-database` scripts. + +- When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to `chown -R 1000:$(whoami) /path/to/zulip` on the host before running `vagrant up` in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run `id username` on the host where `username` is the user running Vagrant and the output is anything but 1000. + This seems to be caused by Vagrant behavior; more information can be found here https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner diff --git a/README.md b/README.md index 2da5eb8f61..5a2ae307d3 100644 --- a/README.md +++ b/README.md @@ -12,10 +12,20 @@ missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org. +Installing the Zulip Development environment +============================================ + +The Zulip development environment is the recommened option for folks +interested in trying out Zulip. This is documented in +[README.dev.md](README.dev.md). + Running Zulip in production =========================== -This is documented in https://zulip.org/server.html and [README.prod.md](README.prod.md). +Zulip in production only supports Ubuntu 14.04 right now, but work is +ongoing on adding support for additional platforms. The installation +process is documented in https://zulip.org/server.html and in more +detail in [README.prod.md](README.prod.md). Contributing to Zulip ===================== @@ -116,302 +126,6 @@ Feedback on how to make this development process more efficient, fun, and friendly to new contributors is very welcome! Just shoot an email to the Zulip Developers list with your thoughts. -Installing the Zulip Development environment -============================================ - -You will need a machine with at least 2GB of RAM available (see -https://github.com/zulip/zulip/issues/32 for a plan for how to -dramatically reduce this requirement). - -Start by cloning this repository: `git clone https://github.com/zulip/zulip.git` - -Using Vagrant -------------- - -This is the recommended approach, and is tested on OS X 10.10 as well as Ubuntu 14.04. - -* The best performing way to run the Zulip development environment is - using an LXC container. If your host is Ubuntu 14.04 (or newer; - what matters is having support for LXC containers), you'll want to - install and configure the LXC Vagrant provider like this: - `sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir && vagrant plugin install vagrant-lxc` - -* If your host is OS X, [download VirtualBox](https://www.virtualbox.org/wiki/Downloads), - [download Vagrant](https://www.vagrantup.com/downloads.html), and install them both. - -Once that's done, simply change to your zulip directory and run -`vagrant up` in your terminal to install the development server. This -will take a long time on the first run because Vagrant needs to -download the Ubuntu Trusty base image, but later you can run `vagrant -destroy` and then `vagrant up` again to rebuild the environment and it -will be much faster. - -Once that finishes, you can run the development server as follows: - -``` -vagrant ssh -- -L9991:localhost:9991 -# Now inside the container -cd /srv/zulip -source /srv/zulip-venv/bin/activate -./tools/run-dev.py --interface='' -``` - -You can now visit in your browser. To get -shell access to the virtual machine running the server, use `vagrant ssh`. - -(A small note on tools/run-dev.py: the `--interface=''` option will make -the development server listen on all network interfaces. While this -is correct for the Vagrant guest sitting behind a NAT, you probably -don't want to use that option when using run-dev.py in other environments). - -The run-dev.py console output will show any errors your Zulip -development server encounters. It runs on top of Django's [manage.py -runserver](https://docs.djangoproject.com/en/1.8/ref/django-admin/#runserver-port-or-address-port) -tool, which will automatically restart the Zulip Django and Tornado -servers whenever you save changes to Python code. - -However, the Zulip queue workers will not automatically restart when -you save changes, so you will need to ctrl-C and then restart -`run-dev.py` manually if you are testing changes to the queue workers -or if a queue worker has crashed. - -Using provision.py without Vagrant ----------------------------------- - -If you'd like to install a Zulip development environment on a server -that's already running Ubuntu 14.04 Trusty, you can do that by just -running: - -``` -sudo apt-get update -sudo apt-get install -y python-pbs -python /srv/zulip/provision.py - -cd /srv/zulip -source /srv/zulip-venv/bin/activate -./tools/run-dev.py -``` - -By hand -------- -If you really want to install everything by hand, the below -instructions should work. - -Install the following non-Python dependencies: - * libffi-dev — needed for some Python extensions - * postgresql 9.1 or later — our database (also install development headers) - * memcached (and headers) - * rabbitmq-server - * libldap2-dev - * python-dev - * redis-server — rate limiting - * tsearch-extras — better text search - * libfreetype6-dev - needed before you pip install Pillow to properly generate emoji PNGs - -On Debian or Ubuntu systems: - -``` -sudo apt-get install libffi-dev memcached rabbitmq-server libldap2-dev python-dev redis-server postgresql-server-dev-all libmemcached-dev libfreetype6-dev - -# If on 12.04 or wheezy: -sudo apt-get install postgresql-9.1 -wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb -sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb - -# If on 14.04: -sudo apt-get install postgresql-9.3 -wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.3-tsearch-extras_0.1.2_amd64.deb -sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.2_amd64.deb - -# If on 15.04 or jessie: -sudo apt-get install postgresql-9.4 -wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb -sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb -``` - -Now continue with the "All systems" instructions below. - -On Fedora 22 (experimental): - -These instructions are experimental and may have bugs; patches welcome! - -``` -sudo dnf install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server postgresql-devel postgresql libmemcached-devel freetype-devel -wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz -tar xvzf tsearch-extras_0.1.3.tar.gz -cd ts2 -make -sudo make install - -# Hack around missing dictionary files -- need to fix this to get -# the proper dictionaries from what in debian is the hunspell-en-us package. -sudo touch /usr/share/pgsql/tsearch_data/english.stop -sudo touch /usr/share/pgsql/tsearch_data/en_us.dict -sudo touch /usr/share/pgsql/tsearch_data/en_us.affix - -# Edit the postgres settings: -sudo vi /var/lib/pgsql/data/pg_hba.conf - -# Add this line before the first uncommented line to enable password auth: -host all all 127.0.0.1/32 md5 - -# Start the services -sudo systemctl start redis memcached rabbitmq-server postgresql - -# Enable automatic service startup after the system startup -sudo systemctl enable redis rabbitmq-server memcached postgresql -``` - -Now continue with the All Systems instructions below. - -On CentOS 7 Core (experimental): - -These instructions are experimental and may have bugs; patches welcome! - -``` -# Add user zulip to the system (not necessary if you configured zulip as the administrator -# user during the install process of CentOS 7). -useradd zulip - -# Create a password for zulip user -passwd zulip - -# Allow zulip to sudo -visudo -# Add this line after line `root ALL=(ALL) ALL` -zulip ALL=(ALL) ALL - -# Switch to zulip user -su zulip - -# Enable EPEL 7 repo so we can install rabbitmq-server, redis and other dependencies -sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm - -# Install dependencies -sudo yum install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server \ -postgresql-devel postgresql libmemcached-devel wget python-pip openssl-devel freetype-devel libjpeg-turbo-devel \ -zlib-devel nodejs - -# We need these packages to compile tsearch-extras -sudo yum groupinstall "Development Tools" - -cd && wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz -tar xvzf tsearch-extras_0.1.3.tar.gz -cd ts2 -make -j $(nproc) -sudo make install - -# Hack around missing dictionary files -- need to fix this to get -# the proper dictionaries from what in debian is the hunspell-en-us package. -sudo touch /usr/share/pgsql/tsearch_data/english.stop -sudo touch /usr/share/pgsql/tsearch_data/en_us.dict -sudo touch /usr/share/pgsql/tsearch_data/en_us.affix - -# clone Zulip's git repo and cd into it -cd && git clone https://github.com/zulip/zulip && cd zulip/ - -# Initialize the postgres db -sudo postgresql-setup initdb - -# Edit the postgres settings: -sudo vi /var/lib/pgsql/data/pg_hba.conf - -# Add these two lines after line 80: -local zulip zulip md5 -host all all 127.0.0.1/32 md5 - -# Change these lines: -host all all 127.0.0.1/32 ident -host all all ::1/128 ident -# to this: -host all all 127.0.0.1/32 md5 -host all all ::1/128 md5 - -# Start the services -sudo systemctl start redis rabbitmq-server memcached postgresql - -# Enable automatic service startup after the system startup -sudo systemctl enable redis rabbitmq-server memcached postgresql -``` - -Now continue with the All Systems instructions below. - -All Systems: - -``` -pip install -r requirements.txt -./tools/download-zxcvbn -./tools/emoji_dump/build_emoji -./scripts/setup/generate_secrets.py -d -sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/9.3/tsearch_data/ -./scripts/setup/configure-rabbitmq -./tools/postgres-init-dev-db -./tools/do-destroy-rebuild-database -./tools/postgres-init-test-db -./tools/do-destroy-rebuild-test-database -``` - -To start the development server: - -``` -./tools/run-dev.py -``` - -… and visit [http://localhost:9991/](http://localhost:9991/). - - -Running the test suite -====================== - -Run all tests: - -``` -./tools/test-all -``` - -This runs the linter (`tools/lint-all`) plus all of our test suites; -they can all be run separately (just read `tools/test-all` to see -them). You can also run individual tests which can save you a lot of -time debugging a test failure, e.g.: - -``` -./tools/test-backend zerver.test_bugdown.BugdownTest.test_inline_youtube -./tools/test-js-with-casper 10-navigation.js -./tools/test-js-with-node # Runs all node tests but is very fast -``` - -The above instructions include the first-time setup of test databases, -but you may need to rebuild the test database occasionally if you're -working on new database migrations. To do this, run: - -``` -./tools/postgres-init-test-db -./tools/do-destroy-rebuild-test-database -``` - -Possible testing issues -======================= - -- The Casper tests are flaky on the Virtualbox environment (probably - due to some performance-sensitive races; they work reliably in - Travis CI). Until this issue is debugged, you may need to rerun - them to get them to pass. - -- When running the test suite, if you get an error like this: - - ``` - sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not exist - LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l... - ^ - ``` - - … then you need to install tsearch-extras, described - above. Afterwards, re-run the `init*-db` and the - `do-destroy-rebuild*-database` scripts. - -- When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to `chown -R 1000:$(whoami) /path/to/zulip` on the host before running `vagrant up` in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run `id username` on the host where `username` is the user running Vagrant and the output is anything but 1000. - This seems to be caused by Vagrant behavior; more information can be found here https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner - License =======