Commit Graph

151 Commits

Author SHA1 Message Date
anirudhjain75 beaa62cafa mypy: Convert several directories to use typing.Text.
Specifically, these directories are converted: [analytics/, scripts/,
tools/, zerver/management/, zilencer/, zproject/]
2016-12-07 20:51:05 -08:00
Rafid Aslam ce94fb2eb1 deps: Change zxcvbn download method from download-zxcvbn to npm.
- Replace download-zxcvbn with downloading it from npm.
- Change zxcvbn.js path to node_modules (because npm put it to
`node_modules` directory.
- Bump `PROVISION_VERSION` in `version.py` to 2.4.

Fixes #2423.
2016-12-06 17:54:56 -08:00
Rafid Aslam 0951b838ca deps: Update zxcvbn to 4.4.1
Update zxcvbn to 4.4.1, and change `crack_time` to
`crack_times_seconds` because `crack_time` has been changed to
`crack_times_seconds` in 4.4.1.
2016-12-06 17:51:33 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Christie Koehler 391ff638d5 build_emoji: Add nice error message for symlink errors. 2016-11-15 21:10:39 -08:00
Tim Abbott 2b8324b778 emoji: Fix caching permissions issues.
Previously, you needed to be root to update the emoji cache, which
caused problems with how Zulip is upgraded in production.
2016-10-25 17:52:19 -07:00
Prabhakar Gupta bc6421fbfb tools: Remove empty wrapper script build_emoji.
This renames the old `emoji_dump.py` to `build_emoji`, removing the
old shell essentially empty shell script.  `emoji_dump.py` was always
a weird name, and this makes it a bit easier to read the code for this
system.
2016-10-20 22:57:45 -07:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Umair Khan e0c10ed1e8 Cache emoji dump output.
This saves a bunch of time building release tarballs, provisioning,
and upgrading Zulip from git that was spent regenerating the Zulip
emoji sprite sheet.

[commit message tweaked by tabbott]
2016-10-10 11:15:43 -07:00
Steve Howell 0bdc9fef5c Upgrade caspersjs to version 1.1.3. (w/acrefoot)
(Most of this work was done by acrefoot in an earlier branch.
I took over the branch to fix casper tests that were broken during
the upgrade (which were fixed in a different commit).  I also
made most of the changes to run-casper.)

This also upgrades phantomjs to 2.1.7.

The huge structural change here is that we no longer vendor casperjs
or download phantomjs with our own script.  Instead, we just use
casperjs and phantomjs from npm, via package.json.

Another thing that we do now is run casperjs tests individually, so
that we don't get strange test flakes from test interactions.  (Tests
can still influence each other in terms of changing data, since we
don't yet have code to clear the test database in between tests.)

A lot of this diff is just removing files and obsolete configurations.

The main new piece is in package.json, which causes npm to install the
new version.

Also, run-casper now runs files individually, as mentioned above.

We had vendored casperjs in the past.  I didn't bring over any of our
changes.  Some of the changes were performance-related (primarily
5fd58cf249), so the upgraded version may
be slower in some instances.  (I didn't do much measurement of that,
since most of our slowness when running tests is about the setup
environment, not casper itself.)  Any bug fixes that we may have
implemented in the past were either magically fixed by changes to
casper itself or by improvements we have made in the tests themselves
over the years.

Tim tested the Casper suite on his machine and running the full Casper
test suite is faster than it was before this change (1m30 vs. 1m50),
so we're at least not regressing overall performance.
2016-10-08 12:08:43 -07:00
Tomasz Kolek dbeab6aa6f Optimize checks of test database state by moving into Python.
Previously, the generate-fixtures shell script by called into Django
multiple times in order to check whether the database was in a
reasonable state.  Since there's a lot of overhead to starting up
Django, this resulted in `test-backend` and `test-js-with-casper`
being quite slow to run a single small test (2.8s or so) even on my
very fast laptop.

We fix this is by moving the checks into a new Python library, so that
we can avoid paying the Django startup overhead 3 times unnecessarily.
The result saves about 1.2s (~40%) from the time required to run a
single backend test.

Fixes #1221.
2016-10-05 10:40:19 -07:00
Umair Khan b6ad24b11b Fix names of sprite emojis. 2016-09-30 10:42:52 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
umkay 798e6faa9e provision: Use NVM to install node and npm.
NVM takes a specific node version and installs the node package and
a corresponding compatible npm package.

We use it in a somewhat hackish way to install node/npm globally with
a pinned version, since that's how we actually want to consume node in
our development environment.

Other details:
- Travis CI now is configured to use the version of node installed by
provision; the easiest way to do this was to sabotage the existing node
installation.
- jsdom is upgraded to a current version, which both requires recent
node and also is required for the tests to pass with recent node.
This fixes running the node tests on Xenial.

Fixes #1498.

[tweaked by tabbott]
2016-09-23 14:34:44 -07:00
Tim Abbott 5fcc8146c1 emoji_dump: Clean up spammy sprite creation output.
Glue apparently prints 1 line per file being processed.
2016-09-14 11:49:55 -07:00
Umair Khan 6c21d7a2c8 Annotate tools/setup/emoji_dump/emoji_dump.py. 2016-09-12 08:10:49 -07:00
Tim Abbott 24906f7678 emoji_dump: Update SIZE to 136x136 to reflect new emoji set.
I'm not entirely sure that this is correct, but it suffices to make
the spritesheet emoji have the right size.
2016-09-08 15:18:04 -07:00
Tim Abbott a4536cca1e emoji_dump: Generate a sprite sheet for the Zulip emoji. 2016-09-08 15:16:58 -07:00
Tim Abbott abf85e7fbd emoji_dump: Add warning for fallback to black/white.
I don't think the black/white fallback code is actually used by
default, and thus when it is used, it's usually a sign that something
is broken.  We should throw an error, but at the very least it makes
sense to print a warning.
2016-09-08 15:05:38 -07:00
Tim Abbott aca6636729 emoji_dump: Fix buggy path management.
In d583710f7c, I apparently broke the
color emoji handling, which was masked (for test purposes) by the fact
that we catch an expection if color doesn't work and in that case fall
back to black and white emoji.
2016-09-08 15:04:09 -07:00
Tim Abbott a6fffe5120 emoji_dump: Move size constants to top of file. 2016-09-08 14:30:20 -07:00
Tim Abbott d583710f7c emoji_dump: Refactor path code to be a bit cleaner. 2016-09-08 14:27:47 -07:00
Kouhei Sutou 683f49aa99 Support full text search for all languages using pgroonga.
This adds support for using PGroonga to back the Zulip full-text
search feature.  Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.

Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.

PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.

Fixes #615.

[docs and tests tweaked by tabbott]
2016-08-26 21:04:03 -07:00
Tim Abbott b200c6cdc0 Move assets/ to live under static/.
This decreases the clutter in the root directory of the Zulip
repository.
2016-08-26 10:23:38 -07:00
Taranjeet Singh d606b95242 zulip_tools.py: Move zulip_tools.py in scripts/lib.
This commit moves zulip_tools.py as part of cleaning the root directory
and organizing proejct into better directory structure.
2016-08-15 16:44:50 -07:00
Taranjeet Singh b82836a901 tools/setup/emoji_dump/emoji_dump.py: Move bitmaps & *.ttx to var/.
Also update .gitignore to remove the old .gitignore markings for
these files' old locations.
2016-08-02 15:18:36 -07:00
Steve Howell f3962994b5 Fix regression for running migrations during tests.
When running ./tools/test-backend, the script to generate
fixtures, ./tools/setup/generate-fixtures, looks for a file
called migration-status to determine whether it can short
circuit doing database migrations.  This file got moved as
part of the effort to put files in "var," but the existence
check was still looking for that file in its old location.
2016-07-29 08:57:56 -07:00
Taranjeet Singh 1328a10069 Move available-migration and migration-status to var/. 2016-07-25 14:39:35 -07:00
Taranjeet Singh b143d6ca6e Move test_credentials.js to var/casper. 2016-07-25 14:39:23 -07:00
Eklavya Sharma baa157344c Patch activate script only in development. 2016-07-21 14:11:41 -07:00
Eklavya Sharma 6548f1dd1c Factor out venv-installing code into a module.
Factor out the code in tools/provision.py which installs a python2
and python3 venv into a module (tools/setup/setup_venvs.py) which
can also be used as a script.
2016-07-20 18:20:37 -07:00
Eklavya Sharma 2930a769a9 Add __init__.py to tools/ and tools/setup/.
This will make it possible to import stuff from tools/setup/.
2016-07-20 18:20:37 -07:00
Tim Abbott a07eca2639 Revert "Factor out venv-installation into a script."
This reverts commit 852c49a44e.

I think this may have broken provisioning without a venv.
2016-07-20 14:55:45 -07:00
Eklavya Sharma 852c49a44e Factor out venv-installation into a script. 2016-07-20 14:18:51 -07:00
Tim Abbott 78a0c7c557 tools: Rewrite install-server to not hardcode configuration.
Now install-aws-server is a reusable script for setting up a Zulip
role server in AWS, without any hardcoded configuration.
2016-07-19 20:12:43 -07:00
Tim Abbott 5c92639f81 Move mypy requirements to requirements/ directory. 2016-06-18 16:41:40 -07:00
James Porter 2a8e8129d1 Make emoji_dump delete old ttx files on start. 2016-06-04 17:05:48 -07:00
James Porter f5f2d72178 Wrap main bit of emoji_dump.py in a main function. 2016-06-04 17:05:48 -07:00
James Porter 9d58624359 Minor reformatting of emoji_dump.py. 2016-06-04 17:05:48 -07:00
James Porter d008d96597 Update Emoji set. 2016-06-04 17:05:48 -07:00
Reid Barton 0b7852f081 Generate mypy coverage report for travis + coveralls. 2016-06-04 13:01:19 -07:00
Tim Abbott 5d5f1f46dc Upgrade mypy and typed_ast to latest versions. 2016-06-03 18:59:55 -07:00
Umair Khan 072551a94e Fix most redownloading in 'vagrant reload --provision'.
Fixes #264.
2016-06-02 22:32:56 -07:00
Eklavya Sharma 149938d468 Change shebangs from python2.7 to python. 2016-05-29 05:03:08 -07:00
Umair Khan 0278ce9102 Move tools/py3_test_reqs.txt to tools/setup/.
Fixes #708
2016-05-26 18:43:24 +05:00
Umair Khan 57f477dd8b Move tools/install-phantomjs to tools/setup/. 2016-05-26 18:28:14 +05:00
Umair Khan 1161862b07 Move tools/emoji_dump to tools/setup/. 2016-05-26 18:28:14 +05:00
Umair Khan b85526576a Move tools/postgres-init-dev-db to tools/setup/. 2016-05-26 18:28:10 +05:00
Umair Khan d425e05a02 Move tools/generate-fixtures to tools/setup/. 2016-05-26 17:54:32 +05:00
Umair Khan 8335bd672f Move tools/generate-test-credentials to tools/setup/. 2016-05-26 17:54:28 +05:00
Umair Khan d5f3a82284 Move tools/download-zxcvbn to tools/setup. 2016-05-26 16:05:23 +05:00