Commit Graph

65 Commits

Author SHA1 Message Date
rht bf4eda7374 tools: Remove absolute_import in most tools.
Tweaked by tabbott to not remove it from lister.py, linter_lib, and
friends, since those are intended to support both Python 2 and 3
(we're planning to extract them from the repository).
2017-09-29 12:28:43 -07:00
Eeshan Garg bb34ba37b5 tools: Rename build_pygments_data.py -> build_pygments_data.
Dropping the file extension (.py) conforms to our naming
convention for such scripts.
2017-09-26 16:00:41 -07:00
Eeshan Garg 390a1fec92 zulip_bots: Generate static files during provisioning.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.

This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
2017-09-22 15:14:42 -07:00
Vishnu Ks f4c69bb126 tools: Create tool for inlining email CSS during provision. 2017-09-16 07:17:48 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Brock Whittaker 83184d5c60 /about/: Merge with /authors, redesign page.
This adds the authors to the Zulip repository on GitHub from
/authors/ along with re-styling the page to fit the same
aesthetic as /for/open-source/ and other product-pages.
2017-08-15 16:09:01 -07:00
Greg Price b6d90808a5 prod: Make `cp` copy a directory to its target, not into it.
This was causing `language_options.json` and the `django.mo` files
never to get updated by `upgrade-zulip-from-git` after first deploy.
The stale `django.mo` files meant that any message string which wasn't
in the original from-tarball installation of the server would never
get translated.

The same bug exists in several other spots in this file.  Some of
our repetitive-yet-subtly-different uses of `cp` should probably
turn into just a list of arguments to one `rsync -R` command.
Also this should probably really use `rsync --delete` -- for example,
in prod on zulipchat.com we still have lying around a `locale/zh_CN`
directory, even though in a freshly-built static tree there's only
`locale/zh_Hans` instead.
2017-08-07 19:23:27 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Harshit Bansal 92f6613bdd tools: Remove `tools/setup/emoji/download-emoji-data` tool.
This tool was used for downloading sprite sheets from iamcal's
repository. Since now we have moved to using `emoji-datasource`
npm package, this tool is no longer required.
2017-07-13 21:52:01 -07:00
Rafid Aslam ef2c0f806e update-prod-static: Add code to copy CSS from node_modules 2017-06-11 10:15:48 -07:00
Reid Barton 2e4078cbd2 update-prod-static: Redirect update-authors-json stderr to log file. 2017-05-24 16:44:02 -07:00
Cory Lynch 0965c43238 Add typeahead for syntax highlighting languages.
Modified composebox_typeahead.js to recognize the triple backtick
and tilde for code blocks, and added appropriate typeahead functions
in that file and in typeahead_helper.js.

Additionally, a new file pygments_data.js contains a dictionary of
the supported languages, mapping to relative popularity
rankings. These rankings determine the order of sort of the
languages in the typeahead.

This JavaScript file is actually in static/generated/pygments_data.js, as it
is generated by a Python script, tools/build_pymgents_data.py. This is
so that if Pygments adds support for new languages, the JavaScript file
will be updated appropriately. This python script uses a set of popularity
rankings defined in lang.json.

Corresponding unit tests were also added.

Fixes #4111.
2017-04-28 17:22:59 -07:00
Harshit Bansal 4470a3947b provision: Download emoji-data during provisioning.
Add code to download iamcal's emoji_pretty.json
(emoji-data/emoji.json) during provisioning.
2017-04-18 12:46:52 -07:00
Yago González 34a9e1ae11 markdown: Add TeX typesetting support.
Co-authored-by: Reid Barton <rwbarton@gmail.com>

Fixes #2056.
2017-03-21 16:40:00 -07:00
Tim Abbott 1005d70ff5 update-prod-static: Fix incorrect ordering of venv setup code.
We should check if we succeeded in using a venv only after we've
actually setup sys.path to use it.
2017-02-07 11:49:31 -08:00
sinwar eab355b0cd tools: Create more consistent checks for venv.
This helps make the Zulip development environment somewhat more robust
to new contributors, since it will give them a nice warning if they
try running any of our development tools outside the Zulip virtualenv.

Fixes #3468.
2017-02-06 21:50:32 -08:00
Tim Abbott d71f2e7b9b update-prod-static: Don't ship node_modules.
This saves more than 50MB of unnecessary duplicate content in release
tarballs (the static assets that will actually be served have already
been built at this point by tools/minify-js).
2017-02-06 17:21:34 -08:00
Tim Abbott e2ee1951e0 storage: Fix static files storage reuse issues.
Zulip's previous model for managing static asset files via Django
pipeline had some broken behavior around upgrades.  In particular, it
was for some reason storing the information as to which static files
should be used in a memcached cache that was shared between different
deployments of Zulip.  This means that during the upgrade process,
some clients might be served a version of the static assets that does
not correspond to the server they were connected to.

We've replaced that model with using ManifestStaticFilesStorage, which
instead allows each Zulip deployment directory to have its own
complete copy of the mapping of files to static assets, as it should
be.

We have to do a little bit of hackery with the staticfiles.json path
to make this work, basically because Django expects staticfiles.json
to be under STATIC_ROOT (aka the path nginx is serving to users), but
doing that doesn't really make sense for Zulip, since that directory
is shared between different deployments.
2017-02-06 16:10:24 -08:00
Tim Abbott f6f8168f4d emoji: Build emoji before minifying JS.
Since build_emoji will soon be generated
static/generated/emoji_data.js (containing the emoji data), we need to
generate those JavaScript files before minifying them.
2017-01-28 12:15:34 -08:00
Tim Abbott 126b1c4edc upgrade: Don't require authors updates deploying from Git.
Fixes #3392.
2017-01-23 23:00:09 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Tim Abbott 4f67b9802d travis: Use fixture for authors page in CI. 2017-01-17 13:42:37 -08:00
Tommy Ip c407919db3 Add /authors page.
Contributor visualization showing the avatar, user name and number
of commits for each contributors. The JSON data would be updated
upon deployment, triggered by the `update-prod-static` script.
2017-01-17 13:35:55 -08:00
Tim Abbott 246bdbd829 update-prod-static: Fix node_modules/.bin being unavailable in prod.
Apparently `manage.py collectstatic` by default strips files starting
with "." from what it collects.  This is a reasonably precaution,
though mostly irrelevant to us, since Zulip primarily runs that as
part of build-release tarball, which runs in a clean directory.

It also breaks our current approach for transferring node_modules to
prod machines via release tarballs; this change fixes that bug.
2017-01-06 20:23:46 -08:00
Tim Abbott 08dae5108e emoji: Move tools/setup/emoji_dump to tools/setup/emoji. 2016-12-27 19:58:41 -08:00
reallyly 94b28e0afb pep8: Fix E128 violations.
With some line-wrapping tweaks by tabbott.
2016-12-13 10:50:14 -08:00
Rafid Aslam ce94fb2eb1 deps: Change zxcvbn download method from download-zxcvbn to npm.
- Replace download-zxcvbn with downloading it from npm.
- Change zxcvbn.js path to node_modules (because npm put it to
`node_modules` directory.
- Bump `PROVISION_VERSION` in `version.py` to 2.4.

Fixes #2423.
2016-12-06 17:54:56 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Prabhakar Gupta bc6421fbfb tools: Remove empty wrapper script build_emoji.
This renames the old `emoji_dump.py` to `build_emoji`, removing the
old shell essentially empty shell script.  `emoji_dump.py` was always
a weird name, and this makes it a bit easier to read the code for this
system.
2016-10-20 22:57:45 -07:00
Umair Khan 4301d11b4a Use npm caching in tools/update-prod-static. 2016-09-16 12:40:48 -07:00
Gordon P. Hemsley aace1c5bc9 Switch tools/update-prod-static from optparse to argparse. 2016-09-12 12:00:16 -07:00
Tim Abbott b200c6cdc0 Move assets/ to live under static/.
This decreases the clutter in the root directory of the Zulip
repository.
2016-08-26 10:23:38 -07:00
Tim Abbott b50abc5131 npm: Split out production-only dependencies. 2016-08-04 22:17:07 -07:00
Taranjeet Singh 44d23975ed Move update-prod-static.log to var/log. 2016-07-20 18:27:44 -07:00
Tim Abbott 7348841e65 update-prod-static: Copy static/locale rather than moving.
The old behavior results in a large diff when deploying based on a git
commit.
2016-07-19 20:52:08 -07:00
Tim Abbott 07473f4007 update-prod-static: Use a virtualenv to find dependencies.
This is needed to support using update-deployment with a virtualenv.
2016-07-19 09:51:13 -07:00
Umair Khan b546391f0b Move locale to static/locale 2016-07-18 13:59:07 -07:00
Tim Abbott 599f12f94f update-prod-static: Fix deploy path for language_options.json. 2016-07-13 18:56:37 -07:00
Tim Abbott c0a6672471 Update path to language_options.json in production.
The previous code didn't correctly transport language_options.json to
the production environment.
2016-07-13 12:30:45 -07:00
Eklavya Sharma 149938d468 Change shebangs from python2.7 to python. 2016-05-29 05:03:08 -07:00
Umair Khan 1161862b07 Move tools/emoji_dump to tools/setup/. 2016-05-26 18:28:14 +05:00
Umair Khan d5f3a82284 Move tools/download-zxcvbn to tools/setup. 2016-05-26 16:05:23 +05:00
Umair Khan e56d3196ef Update provisioning/release scripts to build `.mo` files.
We need to update provision.py to compile the messages files, since
they are needed for the new i18n tests.  And of course we need to
include the .mo files in release tarballs; there's a bit of complexity
there around how the tarball archives are created.
2016-05-18 19:30:00 -07:00
Allie Jones 4de0325a9d Install node dependencies using npm.
The node packages 'jQuery' and 'jquery' are different--'jQuery' is the
legacy support package that is needed for Zulip so the require statements
in the tests were updated.

Travis uses node 4.0 by default and we are using 0.10, so the command to
install the correct version had to be added to the .travis.yml file.
2015-11-06 09:08:59 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Tim Abbott 9c0c5c57a7 Download zxcvbn.js rather than vendoring it.
(imported from commit 8d55b6b903dc2292852636dfb444219b643b5f64)
2015-09-25 02:24:07 -07:00
Tim Abbott cc33b68d73 Remove pre-built emoji images from version control.
Instead, build them automatically when provision the development
environment and in update-prod-static.

(imported from commit aac8dfeaafbe872c113e5f2b6bd8f655a1af36f2)
2015-09-25 00:22:15 -07:00
David Roe d702ddb3cf DEPLOYED => PRODUCTION
(imported from commit a7d365c07916c5103fc721c712db94f9af977640)
2015-08-21 10:33:35 -07:00
Tim Abbott 5f92ccb422 [manual] Move update-prod-static and update-deployments back to tools/.
update-deployment has been replaced by upgrade-zulip for local server
instances, since it won't be running off a git repository, and
update-prod-static won't be needed since we plan on shipping minified
javascript.

When we deploy this, the deployment will fail, and then we'll need to
update the git checkout from which post-receive runs on git.zulip.net.

(imported from commit 86aaedbab09c60ae86ac1d0ae492d0d1bc45569f)
2013-11-04 13:22:41 -05:00
Tim Abbott 56e9ad230e [manual] Move our deployment scripts to scripts/.
This will require updating the post-receive code on git.zulip.net to
work.

(imported from commit 2e51fa2d7b891c1138d3f22ae534cfb8a6cf174c)
2013-10-28 10:54:48 -04:00