Commit Graph

2328 Commits

Author SHA1 Message Date
Vishnu Ks dfaa939af2 emails: Store the compiled emails in separate directory. 2017-09-27 16:48:18 -07:00
Greg Price 704e94c85c run-mypy: Deindent a bit of logic by moving an exit path up. 2017-09-27 12:22:03 -07:00
Greg Price 103178ffb2 run-mypy: Move "which mypy" noise into --version, and use in Travis.
Printing the version in Travis builds will help in debugging when we
get different results there from locally.  The new `--version` path
also gives us a handy place to put the "what mypy command are we running"
diagnostic, getting it out of the way of normal interactive use.
2017-09-27 12:22:03 -07:00
Greg Price 9ef68d6d42 run-mypy: Add --version to check the version of mypy used. 2017-09-27 12:22:03 -07:00
Greg Price c46755d48d run-mypy: Reorder CLI options for clearer --help. 2017-09-27 12:22:03 -07:00
Greg Price 6697e3620b run-mypy: Give CLI option help a consistent style.
Most CLI tools (including GNU tools and Mercurial) use lowercase
sentence fragments, with no period, for option glosses, so we
follow that style.  Also make the voice and wording consistent,
and the quote type in the Python source.
2017-09-27 12:22:03 -07:00
Greg Price 8c50f7f8a3 run-mypy: Remove extra exclude quirk for nonexistent "stubs" directory.
This exclusion was getting snuck in at the end bypassing --all (and so
giving the lie to the --help documentation).  There is no "stubs"
directory in the tree in any case.
2017-09-27 12:22:03 -07:00
Greg Price be5b26dc68 tools/run-mypy: Cut redundant data from CLI parsing.
argparse has reasonable default behavior for the `dest` argument,
and for `default` at least in these two obvious cases.
So cut those out where we're just doing the default thing.

Also rewrap a couple of calls to fit at least in 100 columns.
2017-09-27 12:22:03 -07:00
Robert Hönig fa8199819c linter: Remove unnecessary escape sequences in custom rules. 2017-09-27 09:02:44 -07:00
Robert Hönig 48696ac9a1 linter: Catch all malformatted string formatting clauses.
This also catches code like `"foo%s"%('bar')`, whereas before,
the first tuple entry could not be a string.
2017-09-27 09:02:44 -07:00
Robert Hönig 83eb22fc2f lister: Make excluding directories work on Windows. 2017-09-27 08:43:53 -07:00
Eeshan Garg bb34ba37b5 tools: Rename build_pygments_data.py -> build_pygments_data.
Dropping the file extension (.py) conforms to our naming
convention for such scripts.
2017-09-26 16:00:41 -07:00
Umair Khan 5d00f29c68 capitalization: Ignore LDAP. 2017-09-26 15:58:17 -07:00
Tim Abbott 8e2ae66fe6 provision: Update apt operations text. 2017-09-25 14:47:18 -07:00
Tim Abbott 16ed23414d provision: Fix spelling of apt_dependencies_hash.
Also use a variable to avoid duplicating the path.
2017-09-25 14:46:46 -07:00
Robert Hönig 41e6704b99 custom linter rules test: Add pattern test.
The pattern test method `test_rule_patterns` tests each rule by
fetching two strings from it: `test_good` and `test_bad`. Each
string is then presented as an input file to `custom_check_file`,
which should return True or False.
All lines in a string need to end with `\n`. Since the linter
expects an additional newline at the end of a file, the test case
adds `\n` to each string on top of that.

Fixes #6320.
2017-09-25 10:53:06 +02:00
Harshit Bansal 995a0f3cc0 build_emoji: Remove an unnecessary check for symlink support.
We already check for symlink support in provisioning script so no
need of doing so here.
2017-09-24 04:51:33 -07:00
Harshit Bansal 6ff7da04de emoji: Remove `NotoColorEmoji.ttf`.
We no longer use glyphs from `NotoColorEmoji.ttf` so removing this.
2017-09-24 04:51:33 -07:00
Harshit Bansal c8c1c8ef43 emoji: Remove `AndroidEmoji.ttf`. 2017-09-24 04:51:33 -07:00
Harshit Bansal 0d8e68d75e build_emoji: Some code cleanup. 2017-09-24 04:51:33 -07:00
Harshit Bansal 6b1525680e build_emoji: Remove now unused `MissingGlyphError` exception.
This exception was raised if there was no glyph available for a
codepoint. We no longer need this.
2017-09-24 04:51:33 -07:00
Harshit Bansal 898d7d1160 build_emoji: Remove now unused `bw_font()` function.
This function was used get a black and white glyph for an emoji if there
was no corresponding image file present in the `NotoColorEmoji.ttf` but
due to the new emoji farm setup code, we no longer need this.
2017-09-24 04:51:33 -07:00
Harshit Bansal b95fbbe29a build_emoji: Remove now unused `color_font()` function.
This function was used to color transparent number emojis. We no
longer need to do this since now we have remapped number emojis
to the corresponding colored emojis in the new emoji farm.
2017-09-24 04:51:33 -07:00
Harshit Bansal 6ebacba425 build_emoji: Remove `code_point_to_file_name_map()`.
We have symlinked the old emoji farm to the old emoji farm and hence
we don't use the images from the `NotoColorEmoji.ttf` file. This
function was used to generate a map from codepoint to filename by
parsing the ttx file passed to it. We no longer need this map.
2017-09-24 04:51:33 -07:00
Harshit Bansal e295bed20f build_emoji: Extract `generate_map_files()` function.
This commit extracts out the `generate_map_files()` function from
the `dump_emojis()` function. This function generates various data
files like `emoji_codes.js`, `name_to_codepoint.json` etc which are
used by webapp, bugdown etc.
2017-09-24 04:51:33 -07:00
Harshit Bansal ef96adf7e5 build_emoji: Remove old emoji farm setup code.
This commit removes the old emoji farm generation code  in favor of
`setup_old_emoji_farm()`. Instead of having individual images in old
emoji farm we now symlink them to the images in the new emoji farm.
2017-09-24 04:51:33 -07:00
Harshit Bansal 273e37b2f9 build_emoji: Add `setup_old_emoji_farm()` function.
This commit adds `setup_old_emoji_farm()` function to the build_emoji
script. This will change the way of setting up the old emoji farm.
Earlier we used to extract the glyphs corresponding to each emoji from
the `NotoColorEmoji.ttf` file. But since now we already have individual
images in the new emoji farm(from iamcal's 'emoji-datasource-google' npm
package) we can just symlink old emoji files to the new image files. This
apart from helping us in cleaning up the `build_emoji` script will also
help in reducing the increased size of the release tarball due to the
addition of new emoji farm.
2017-09-24 04:51:33 -07:00
Harshit Bansal eed7d4b8d7 build_emoji: Remove old code in favor of `generate_sprite_css_files()`. 2017-09-24 04:51:33 -07:00
Harshit Bansal beab00a5e1 build_emoji: Add `generate_sprite_css_files()` function. 2017-09-24 04:51:33 -07:00
Harshit Bansal dcf6f1e0f0 build_emoji: Build new emoji farm.
This commit adds the 'setup_emoji_farm()' which copies the individual
images from `emoji-datasource-google` packge to target emoji dump.
2017-09-24 04:51:33 -07:00
Eeshan Garg 390a1fec92 zulip_bots: Generate static files during provisioning.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.

This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
2017-09-22 15:14:42 -07:00
Vishnu Ks 61e48734a7 provision: Add moreutils in APT common dependencies.
This fixes an issue where `tools/update-locked-requirements` would fail.
2017-09-22 14:04:57 -07:00
Juliana Bacelar 928dd06cc8 linter: Add lint rule banning 'import os.path' 2017-09-22 10:32:21 -07:00
Tim Abbott b1dafe5d9d renumber-migrations: Fix last_correct_migration code. 2017-09-22 07:51:58 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Steve Howell 6ce68dfc3d minor: Remove obsolete lint check.
Some code that we had a lint exception for got cleaned up in a
recent commit.
2017-09-20 13:05:04 -07:00
Steve Howell f13cb94d84 Have exclude_topic_mutes() accept a stream id.
We want to convert stream names to stream ids as close
to the "edges" of our system as possible, so we let our
caller do the work of finding the stream id for a stream
narrow.
2017-09-20 10:31:33 -07:00
Tim Abbott 02d931a53e tools: Fix running full-stack tests from tools/ directory.
Previously these tests required you to run them with the root of the
Zulip repository as the current working directory, just due to
sloppiness.

We clean this up, while also making the path handling more consistent
and involving less fragile code.

Fixes #4169.
2017-09-17 11:12:51 -07:00
Vishnu Ks f4c69bb126 tools: Create tool for inlining email CSS during provision. 2017-09-16 07:17:48 -07:00
invinciblycool 5475c5cedb tests: Replace test_user_agent_parsing with a normal test.
Previously, this was its own separate test script; now it's a normal
part of the test suite.

Tweaked by tabbott to use a proper test method.

Fixes #6327.
2017-09-16 03:56:18 -07:00
Greg Price ceca96899a deps: Adjust the header comments in lock files to reflect our tools. 2017-09-15 00:23:20 -07:00
Greg Price 7c5f3d7be1 deps: Refactor tools/update-locked-requirements to save repeating the filenames.
My first version of this just replaced the repeated list of two output
files with an array variable, but I decided `"${outputs[@]}"` was too
much to ask people to understand, and the alternative of `$outputs`,
unquoted, encourages bad habits of shell programming.  So just handle
one file at a time; the only at all expensive part here is `pip-compile`.

I am tempted to move this to Python, but holding back.
2017-09-15 00:23:20 -07:00
Greg Price a171f23d63 deps: Remove a redundant `deactivate`.
This has no effect except on the running shell; so at the end
of a script, it has no meaningful effect at all.
2017-09-15 00:23:20 -07:00
Greg Price b362e21ebb deps: Simplify tools/update-locked-requirements slightly.
This makes the code a little more concise and also more uniform,
treating `future` the same in prod and in dev.  The non-uniformity
looks like an oversight in 2be8a793e, one of the commits that updated
this code for the Python 3-only world.
2017-09-15 00:23:20 -07:00
Greg Price bab2078705 deps: Explain the `-e` oddness, and handle it uniformly. 2017-09-15 00:23:20 -07:00
Greg Price 806220cafa deps: Organize tools/update-locked-requirements a bit further.
A couple of remarks still reflected the Python 2 world; and the
"Remove the editable flag" comment was confusingly above a line
that it looks like it could be talking about, but isn't.
2017-09-15 00:23:20 -07:00
Supermanu 184bfe7cfd docs: Expand the french translation style guide.
As the french word organisation is used, the related line is excluded
from the linter spell check.
2017-09-14 08:35:51 -07:00
Robert Hönig 5294e10801 custom linter rules test: Load rules in test setup. 2017-09-14 04:17:03 -07:00
Robert Hönig 402d7fca50 test_linter_custom_check.py: Fix typo in docstring. 2017-09-14 04:17:03 -07:00
Umair Khan df70fa962e linter: Improve i18n related regexes.
There are four regexes which try to ensure that the i18n strings are
properly captured.

1) The one which disallows multiline strings.
```
i18n\.t\([^)]+[^,\{\)]$

// Disallows:
i18n.t('some '
       + 'text');
```

2) The one which disallows concatenation within argument to i18n.t():
```
i18n\.t\([\'\"].+?[\'\"]\s*\+

// Disallows:
i18n.t("some " + "text");
```

3) There are two which disallow concatenation with i18n.t():
```
i18n\.t\(.+\).*\+

// Disallows:
i18n.t('some text') +

\+.*i18n\.t\(.+\)

// Disallows:
+ i18n.t('some text')
```

The ideal case is that you try to bring the string argument to the
i18n.t() on one line. In case this is not possible, you can do the
following:

```
var1 = i18n.t("Some text to be translated");
var2 = i18n.t("Some more text to be translated");
complete = var1 + var2;
2017-09-13 12:58:27 -07:00
Robert Hönig 53436d9ea7 linter: Add test case for custom_check.py
Fixes #5406.
2017-09-12 03:29:18 -07:00
Robert Hönig b5cf31efdc linter: Move custom_check_file to global namespace.
This is needed in order to mock the method when testing
`custom_check.py`. The diff for this commit is a bit broken;
all it really does is moving the method out of `build_custom_checkers`.
2017-09-12 03:24:59 -07:00
Robert Hönig 7a2234dfa2 linter: Remove redundant custom rule file delimitors. 2017-09-12 03:24:59 -07:00
Robert Hönig 2248c7ac0d linter: Add explanation for custom rule file delimitation operators. 2017-09-12 03:24:59 -07:00
Steve Howell 4dfe6bb320 Add migration to fix unread messages. 2017-09-07 07:06:03 -07:00
Robert Hönig 34e3329919 linter: Add rule against misspelled "Terms of Service". 2017-09-07 06:29:33 -07:00
Umair Khan bd43c447c3 travis: Log the Python version.
This allows us to ascertain the Python version under which we ran the
tests.
2017-09-06 13:48:50 -07:00
Umair Khan c039f73656 setup_venv: Link to the correct Python interpreter.
Travis enables different Python versions through virtual environments,
but it seems that there is a little caveat when we try to create Zulip's
virtual environment by referring Travis' virtual environment; Zulip's
virtual environment refers the system Python. We encountered this
behaviour when we tried to run our backend test suite under Python 3.5
in Travis. 'python3 --version' command before activating Zulip's
virtualenv showed 'Python 3.5.3' and after it showed 'Python 3.4.3'.
This happened when we created the virtual environment using
'virtualenv -p python3'.

The solution seems to be to explicitly give the path of the Python
interpreter in the Travis' virtual environment using 'which python3'.
2017-09-06 13:48:50 -07:00
Aditya Bansal d7f10bf3a1 profiling: Add snakeviz to dev tools.
This adds snakeviz to dev tools and also updates the message displayed
upon running `test-backend` with `--profile` option to say how to run
snakeviz correctly when using vagrant development environment.
2017-09-06 13:46:24 -07:00
Umair Khan ee6235d716 lint: Add regex to disallow i18n string concatenation.
Fixes #6398
2017-09-06 07:01:43 -07:00
Greg Price 1ab89b1d89 tools/test-all: Split `tools/lint` to match Travis scripts.
This keeps `test-all` visibly aligned with `travis/frontend` and
`travis/backend`, which helps us keep it substantively aligned.
2017-08-29 15:42:01 -07:00
Tim Abbott b087886d1c travis: Move i18n and capitalization linters to frontend.
Since these usually result from changes to HTML templates and other
frontend-side things, it seems better to group them with the frontend.

[Tweaked by gnprice in whitespace and comments.]
2017-08-29 15:42:01 -07:00
Tim Abbott 6ef55145f5 travis: Merge the static-analysis build into backend.
[Tweaked by gnprice to leave the use of --linecoverage-report
unmodified for now.]
2017-08-29 15:42:01 -07:00
Tim Abbott 6de77db93f lint: Fix problematic merging of pyflakes stdout/stderr.
We were getting pyflakes lint error output without line numbers like
this:

pyflakes  |     if user_profile.is_realm_admin and
pyflakes  |                                       ^
pyflakes  |

Apparently the cause was that stdout and stderr was getting mixed
badly, creating "unused import"s lines that had the first of that
error (containing the line number) just above.

As a result, printing out the lines of output from pyflakes' merged
stdout/stderr feed looked like this:

b"zproject/settings.py:95: 'from .prod_settings import *' used; unable to detect undefined nameszerver/views/users.py:49:39: invalid syntax\n"

Note the lack of newline in between the end of the first error at
"names" and the start of the second at "zerver".

This appears to be a change in Pyflakes behavior when we switched to
Python 3; probably they're missing a flush() somewhere.
2017-08-29 10:44:50 -07:00
Tim Abbott 130b5847ad pyflakes: Remove now-unnecessary exclude lines. 2017-08-29 10:27:29 -07:00
Tim Abbott 126171cb76 api: Remove root_domain_uri from API page links.
These pages should be references the current server's UI anyway.
2017-08-28 14:13:37 -07:00
Tim Abbott a0a1fe1512 settings: Rename SERVER_URI to ROOT_DOMAIN_URI.
This should be a lot less confusing.

See #6013 for discussion.
2017-08-28 14:09:28 -07:00
Brock Whittaker 04888101e2 Replace old portico footer with new sectioned one.
This replaces the old footer that has one section with a small list of
items. This expands the footer to have multiple sections.

Actual content tweaked and tagged for i18n by tabbott.
2017-08-28 11:07:02 -07:00
Tim Abbott b49448de7d message_edit: Eliminat whitespace in recipient row controls.
This is hacky, but I can't figure out another way to do it that
doesn't cause other problems.

Ideally, we'd add some sort of exclude rule to our HTML template
linter so we can check the rest of the file.
2017-08-27 22:18:29 -07:00
Harshit Bansal e5361b4403 clean_emoji_cache: Expand `clean-emoji-cache`.
Expands `clean-emoji-cache` so that it can be used in production
environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-27 17:51:24 -07:00
Harshit Bansal facb5dbe85 zulip_tools.py: Extract `generate_sha1sum_emoji()`.
Given the path of a zulip installation, it returns a hash corresponding
to the emoji infrastructure of that installation.
2017-08-27 17:51:24 -07:00
Harshit Bansal 948cf54ee3 clean-npm-cache: Expand `clean-npm-cache`.
Expands `clean-npm-cache` so that it can be used in production
environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-27 17:51:24 -07:00
Tim Abbott 90012f8109 find_add_class: Don't check frontend tests. 2017-08-27 14:04:46 -07:00
Tim Abbott ce1a8bf5b2 find_add_class: Include full path names in error messages. 2017-08-27 14:04:46 -07:00
Tim Abbott b1d98c1d0f lint: Only run find_add_class when JS files change. 2017-08-27 13:51:06 -07:00
Tim Abbott 79ae262463 lint: Only run check-urls after python files change. 2017-08-27 13:51:06 -07:00
Robert Hönig 3f01907d3b travis: Distribute linting over frontend and backend suites. 2017-08-27 13:51:06 -07:00
Robert Hönig c5da0ada31 linter: Replace --pep8 with new --frontend/--backend options.
Adds the --frontend and --backend options to replace
--pep8.

Significantly modified by tabbott to use a cleaner framework.
2017-08-27 13:50:53 -07:00
Robert Hönig 2186d53cf5 linter: Migrate lint from optparse to argparse. 2017-08-27 13:18:55 -07:00
Tim Abbott 1560646144 renumber-migrations: Fix construction of new number. 2017-08-27 12:01:06 -07:00
Tim Abbott 96dcb8f745 renumber-migrations: Fix unnecessary prompt.
The tool already knows the answer anyway.

Fixes #6157.
2017-08-27 12:01:06 -07:00
rht 437b2a3146 Update "MacOS" text to "macOS" 2017-08-26 09:00:42 -07:00
neiljp (Neil Pilgrim) 92985f888b mypy: Adjust check of json from github in update-authors-json. 2017-08-25 12:04:15 -07:00
neiljp (Neil Pilgrim) a059872856 mypy: Clarify xml.find use in build_emoji:code_point_to_file_name_map. 2017-08-25 12:04:15 -07:00
Tim Abbott 4b709de9c9 python3: Remove six.PY2/six.PY3 checks. 2017-08-25 11:06:43 -07:00
Tim Abbott b8e7369dee mypy: Remove type: ignores not needed in Python 3. 2017-08-25 11:04:20 -07:00
Tim Abbott b4d698f0db js-dep-visualizer: Report the PNG file as well. 2017-08-25 10:43:37 -07:00
Tim Abbott 3ceba5ef1e test-backend: Require coverage on most webhook integrations. 2017-08-24 11:33:14 -07:00
Yago González 7d5d8ac61e lint: Add rule for spacing inside Handlebars translation tags.
This commit also includes the fixes to make all Handlebars templates
compliant with the new rule.
2017-08-23 13:08:57 -07:00
Yago González 3a4c6f83fc lint: Ban trailing spaces in Handlebars translation tags. 2017-08-23 13:08:57 -07:00
Yago González aa5f57fda9 lint: Ban leading spaces in Handlebars translation tags. 2017-08-23 13:08:57 -07:00
rht e455284bb4 requirements: Remove pathlib2.
`pathlib2` is a backport of pathlib to Python 2.x. Ni!
This dependency can be removed since:
- Zulip 1.6.0+git has been on Python 3 nowadays.
  Ekki-ekki-ekki-ptang-zoom-boing.
- As stated in ticket #6211, having this removed, prevents the need to
  have lockfiles for each of 3.4, 3.5, (and maybe 3.6).

This fixes #6211.
2017-08-23 10:00:21 -07:00
Harshit Bansal 3534e328c5 clean-venv-cache: Expand `clean-venv-cache` tool.
Expands `clean-venv-cache` tool so that it can be used in prod.
Also moves it from `tools/` to `scripts/`.
2017-08-23 00:03:25 -07:00
Tim Abbott 50ad679342 test-backend: Recommend snakeviz over runsnake.
It has support for Python 3.
2017-08-22 15:44:30 -07:00
Tim Abbott 8dff14430e tools: Remove old python 3 compatibility checker.
Now that we've migrated over to Python 3, this tool is no longer required.

Fixes #6217.
2017-08-22 14:58:59 -07:00
Rishi Gupta b32fe9f2a0 features: Update screenshot and add tool to generate.
This adds a new management command that should make generating or
updating screenshots a bit easier in the future.
2017-08-22 11:32:22 -07:00
rht 0a469fd4c8 requirements: Lock prod.txt instead of its py3_common dependency.
`py3_common.txt` is no longer locked.
2017-08-22 10:10:57 -07:00
rht 80f4ba7a76 requirements: Replace py3_dev.txt with dev_lock.txt. 2017-08-22 10:10:57 -07:00
rht 2be8a793e4 Lock the recursive requirements in dev.txt via pip-compile.
First, all the lines of py3_dev.txt except for mypy.txt are mv-ed to
dev.txt. Then dev_lock.txt is generated from dev.txt to be used by
py3_dev.txt. `click` is removed from moto.txt since it is already a
dependency of several libraries and will appear as an autogenerated
dependency.
2017-08-21 18:24:55 -07:00
rht 6baef37564 requirements: Remove Python 2-specific files. 2017-08-21 18:24:55 -07:00