Commit Graph

2379 Commits

Author SHA1 Message Date
rht 34bafa654c mypy: Fix future syntax errors and other minor mistakes.
When we move to the Python 3 mypy syntax, we can't reference a class
before its definition.
2017-10-10 22:27:16 -07:00
Tim Abbott 416f34c80e setup-production: Try to fix Travis failures by blocking mongodb repo.
It appears the mongodb repo is not accessible by Travis CI right now.
This is sadly our problem, because Travis puts a bunch of crap in
their apt `sources.list` file, so `apt-get update` starts failing.
2017-10-08 20:06:42 -07:00
Steve Howell 810532641e activity.js: Extract get_filtered_and_sorted_user_ids().
This function was extracted from build_user_sidebar().  We
also slightly streamlined it to not unnecessarily call
filter() when the filter text was blank. This extraction
also eliminated the need for us to have the two-line
filter_and_sort() function.

Also, we get to 100% coverage in this commit.
2017-10-08 12:01:59 -07:00
Steve Howell 86d372f9ce lint: Prevent "userid" in code.
We should always use user_id for consistency sake.
2017-10-07 12:16:45 -07:00
Greg Price 9cf44a77da lint: Narrow the space-around-% patterns, remove exclusions.
This lint rule has bitten me a couple of times in working on logging.
These regex rules will inevitably be heuristic, but we can make it a bit more
specific so that the heuristic mainly means it could occasionally miss
something, rather than get in the way with an obviously wrong complaint.
2017-10-06 19:21:40 -07:00
Tim Abbott a9a22a3651 lint: Use new exclude_pattern system to clean up inline style lint. 2017-10-06 13:16:32 -07:00
derAnfaenger 7ff697d053 linter: Add rule against using `pk` instead of `id`.
There's one exception here, for model._meta.pk.  To support this
nicely, we added this exclude-pattern feature.
2017-10-06 12:56:26 -07:00
derAnfaenger 64b8930225 linter: Add rules against improper shebangs. 2017-10-06 12:46:19 -07:00
rht dcd80e6650 travis/setup-backend: Remove the '--travis' flag in tools/provision.
Whether the env is Travis or not, this has been detected via $TRAVIS env var.
Proof:
6fbf41bdbc/tools/provision (L38)
2017-10-06 11:20:25 -07:00
derAnfaenger c022b7a8fa linter: Add rule against using the `style` attribute.
This has a ton of exclude rules, for two reasons:

(1) We haven't been particularly systematic about avoiding unnecessary
inline style in the past, so there's a lot of code we need to fix.

(2) There are cases where one wants to dynamically compute style
rules. For the latter category, ideally we'd figure out a way to
exclude these automatically (e.g. checking for mustache tags in the
style tag).
2017-10-06 08:33:10 -07:00
Tim Abbott 930eef3caa tools: Add new script to sync translations.
We just learned we should be using the "onlytranslated" mode of
Transifex.  Since the command is getting a bit complex (and you need
to remember to run `makemessages` first), it makes sense to have a
tool for it.
2017-10-05 23:07:16 -07:00
Tim Abbott 4b24b9f0e7 install-aws-server: Add code for transferring an SSH key.
For quick iteration, sometimes you want a server to have an SSH key
(e.g. a GitHub deploy key for your fork).
2017-10-05 21:59:36 -07:00
Tim Abbott d9c26c307c install-aws-server: Improve handling of various corner cases.
This script is pretty far from being polished, but it is useful, so we
might as well make it a bit more robust.
2017-10-05 21:57:18 -07:00
derAnfaenger 2cdde8b168 linter: Add rule against verbose $(document).ready() calls. 2017-10-05 10:11:34 -07:00
derAnfaenger 6b99022a02 linter: Add rule against using inline event handlers. 2017-10-05 10:08:40 -07:00
derAnfaenger ec54cb5f72 replacer: Run as script with python 3. 2017-10-05 10:07:54 -07:00
derAnfaenger c01981a65f linter: Extend no-space-before-bracket linting rule. 2017-10-05 10:05:27 -07:00
Harshit Bansal ea5cb23c8e build_emoji: Extend emoji mappings to use new emojis from iamcal dataset.
Emojis which are represented by a sequence of codepoints or emojis
with ZWJ are not included until we implement a mechanism for dealing
with their unicode versions.

Fixes: #6279.
2017-10-04 23:09:14 -07:00
Harshit Bansal 1836ac2ff5 emoji_setup_utils: Add 'get_extended_codepoint_to_name()` function. 2017-10-04 23:09:14 -07:00
Harshit Bansal 3fdf25940f emoji_setup_utils: Add `get_extended_name_to_codepoint()` function. 2017-10-04 23:09:14 -07:00
Harshit Bansal f6e113a8a3 emoji_setup_utils: Add 'get_extended_names_list()` function. 2017-10-04 23:09:14 -07:00
Harshit Bansal 95b17217b2 emoji_setup_utils: Add `get_new_emoji_dicts()` function. 2017-10-04 23:09:14 -07:00
Harshit Bansal dc4ddb1578 emoji_setup_utils.py: Add `emoji_can_be_included()` function. 2017-10-04 23:09:14 -07:00
Tim Abbott 55136cec68 Disabled 100% coverage requirement for activity.js for now. 2017-10-04 22:50:04 -07:00
Harshit Bansal 64795adef3 tools/test-all: Pass the `--force` arg to run-mypy script.
While running the mypy script we were not passing the `--force`
argument correctly to the run-mypy which was causing it complain
about provision status.
2017-10-04 12:27:43 -07:00
derAnfaenger 6b879a2b04 docs: Split bots guide into running and writing guides. 2017-10-04 11:49:20 -07:00
Tim Abbott 1d72629dc4 subdomains: Hardcode REALMS_HAVE_SUBDOMAINS=True. 2017-10-02 16:42:43 -07:00
Tim Abbott 95a0a4a06f subdomains: Update AuthBackendTest for subdomains always on.
This is separate from the main subdomains commit mainly for
readability of the history.
2017-10-02 16:32:11 -07:00
Tim Abbott f05719e335 test-api: Fix tests to work with subdomains. 2017-10-02 13:31:22 -07:00
Tim Abbott f69523e0c8 travis: Modify success-http-headers.
Travis CI seems to have changed their version of wget such that it no
longer prints this.
2017-10-02 10:35:48 -07:00
rht 45f30c7f54 tools: Remove unused imports. 2017-10-02 10:13:48 -07:00
rht 2c2adbf647 tools/lib: Remove unused imports. 2017-10-02 10:13:46 -07:00
rht 8a631a2fde tools: Replace optparse with argparse. 2017-10-02 10:09:49 -07:00
derAnfaenger 6c8b405c7e linter: Add rule against using the deprecated Logging.warn.
Tweaked by tabbott to improve the test cases.
2017-10-02 09:31:11 -07:00
rht e9d35be2bd run-dev: Replace optparse with argparse.
Tweaked by tabbott to preserve the usage information correctly and
remove some duplication.
2017-09-30 09:40:19 -07:00
rht f6e6e8ba44 html-grep: Replace optparse with argparse.
Tweated by tabbott to still have a useful description.
2017-09-30 09:35:40 -07:00
rht 3ac8ab8be3 check-provision: Replace optparse with argparse. 2017-09-30 09:32:46 -07:00
rht f15bdce90e tools: Remove print_function.
Tweaked by tabbott to exclude the linter libraries.
2017-09-29 15:44:56 -07:00
Rhea Parekh 0e5fdcea3f provision: Run apt-get update on catching an error.
For broken apt cache, we run the command apt-get update on catching
the error.
2017-09-29 15:38:23 -07:00
rht bf4eda7374 tools: Remove absolute_import in most tools.
Tweaked by tabbott to not remove it from lister.py, linter_lib, and
friends, since those are intended to support both Python 2 and 3
(we're planning to extract them from the repository).
2017-09-29 12:28:43 -07:00
Harshit Bansal 521ac31d3d build_emoji: Remove the redundant `codepoints` list.
This list contained a list of codepoints of all the emojis. We now have
`codepoint_to_name` in emoji_codes.js and hence this is not required.
2017-09-29 11:17:06 -07:00
Harshit Bansal f6c621dab5 build_emoji: Remove the hack used for displaying number emojis correctly.
This hack was used to fix the broken number emojis in the emoji picker.
It was broken because of the partial migration to the iamcal dataset.
See issue #4775 for more details.
2017-09-29 11:17:06 -07:00
Harshit Bansal 0940d2cf85 build_emoji: Remove the hack used for displaying flag emojis correctly.
This hack was used to fix the broken flag emojis in emoji-picker.
It was broken due to the incomplete migration to iamcal dataset.
See issue #4775 for more details.
2017-09-29 11:17:06 -07:00
Harshit Bansal 5b5bcce098 emoji: Migrate bugdown emoji to use sprite sheets.
This commit switches to use sprite sheets for rendering emojis
in all the remaining places, i.e., message bodies and composebox
typeahead. This commit also includes some changes to notifications.py
file so that the spans used for rendering emojis can be converted
to corresponding image tags so that we don't break the emoji rendering
in missed message emails since we can't use sprite sheets there.

As part of switching the bugdown system to use sprite sheets, we need
to switch the name_to_codepoint mappings to match the new sprite
sheets.  This has the side effect of fixing a bunch of emoji like
numbers and flag emoji in the emoji pickers.

Fixes: #3895.
Fixes: #3972.
2017-09-29 11:14:34 -07:00
Tim Abbott 43f967bccd settings: Remove obsolete VOYAGER setting. 2017-09-29 10:19:43 -07:00
Greg Price 0d5f77cf86 logging: Abbreviate log-level names to 4 characters.
These are long enough to still be self-explanatory (the only one I'm
at all in doubt about there is DEBG; I avoided "DBUG" because it reads
"BUG" which suggests a high-priority message, and those are the
opposite of that), while saving a good bit of horizontal space
vs. padding everything to the 8 characters of "CRITICAL".

Also add a linter exception to allow easy-to-read alignment here,
similar to several existing exceptions for other alignment cases.
2017-09-28 18:26:39 -07:00
Greg Price 0163920577 logging: Make our own Formatter class.
This doesn't yet do much, but it gives us a suitable place to
add code to customize how log messages are displayed, beyond what
a format string passed to the default formatter can do.
2017-09-28 18:26:39 -07:00
rht f22daeba1a Cleanup rabbitmq queues when doing dev db rebuild.
Modified by tabbott to cleanup the argument handling in the modified
purge_queue management command.

Fixes #1335.
2017-09-28 18:18:54 -07:00
Robert Hönig f6f601f22b linter: Add tests for first 10 python custom rules. 2017-09-27 18:27:43 -07:00
rht b937e1fede Remove the rest of `import six` outside zerver.lib.
We no longer need the six module, now that we're Python 3 only.
2017-09-27 17:09:15 -07:00
Rishi Gupta d0a9e92f1d emails: Fix duplicate html structure added to compiled emails. 2017-09-27 16:48:18 -07:00
Vishnu Ks dfaa939af2 emails: Store the compiled emails in separate directory. 2017-09-27 16:48:18 -07:00
Greg Price 704e94c85c run-mypy: Deindent a bit of logic by moving an exit path up. 2017-09-27 12:22:03 -07:00
Greg Price 103178ffb2 run-mypy: Move "which mypy" noise into --version, and use in Travis.
Printing the version in Travis builds will help in debugging when we
get different results there from locally.  The new `--version` path
also gives us a handy place to put the "what mypy command are we running"
diagnostic, getting it out of the way of normal interactive use.
2017-09-27 12:22:03 -07:00
Greg Price 9ef68d6d42 run-mypy: Add --version to check the version of mypy used. 2017-09-27 12:22:03 -07:00
Greg Price c46755d48d run-mypy: Reorder CLI options for clearer --help. 2017-09-27 12:22:03 -07:00
Greg Price 6697e3620b run-mypy: Give CLI option help a consistent style.
Most CLI tools (including GNU tools and Mercurial) use lowercase
sentence fragments, with no period, for option glosses, so we
follow that style.  Also make the voice and wording consistent,
and the quote type in the Python source.
2017-09-27 12:22:03 -07:00
Greg Price 8c50f7f8a3 run-mypy: Remove extra exclude quirk for nonexistent "stubs" directory.
This exclusion was getting snuck in at the end bypassing --all (and so
giving the lie to the --help documentation).  There is no "stubs"
directory in the tree in any case.
2017-09-27 12:22:03 -07:00
Greg Price be5b26dc68 tools/run-mypy: Cut redundant data from CLI parsing.
argparse has reasonable default behavior for the `dest` argument,
and for `default` at least in these two obvious cases.
So cut those out where we're just doing the default thing.

Also rewrap a couple of calls to fit at least in 100 columns.
2017-09-27 12:22:03 -07:00
Robert Hönig fa8199819c linter: Remove unnecessary escape sequences in custom rules. 2017-09-27 09:02:44 -07:00
Robert Hönig 48696ac9a1 linter: Catch all malformatted string formatting clauses.
This also catches code like `"foo%s"%('bar')`, whereas before,
the first tuple entry could not be a string.
2017-09-27 09:02:44 -07:00
Robert Hönig 83eb22fc2f lister: Make excluding directories work on Windows. 2017-09-27 08:43:53 -07:00
Eeshan Garg bb34ba37b5 tools: Rename build_pygments_data.py -> build_pygments_data.
Dropping the file extension (.py) conforms to our naming
convention for such scripts.
2017-09-26 16:00:41 -07:00
Umair Khan 5d00f29c68 capitalization: Ignore LDAP. 2017-09-26 15:58:17 -07:00
Tim Abbott 8e2ae66fe6 provision: Update apt operations text. 2017-09-25 14:47:18 -07:00
Tim Abbott 16ed23414d provision: Fix spelling of apt_dependencies_hash.
Also use a variable to avoid duplicating the path.
2017-09-25 14:46:46 -07:00
Robert Hönig 41e6704b99 custom linter rules test: Add pattern test.
The pattern test method `test_rule_patterns` tests each rule by
fetching two strings from it: `test_good` and `test_bad`. Each
string is then presented as an input file to `custom_check_file`,
which should return True or False.
All lines in a string need to end with `\n`. Since the linter
expects an additional newline at the end of a file, the test case
adds `\n` to each string on top of that.

Fixes #6320.
2017-09-25 10:53:06 +02:00
Harshit Bansal 995a0f3cc0 build_emoji: Remove an unnecessary check for symlink support.
We already check for symlink support in provisioning script so no
need of doing so here.
2017-09-24 04:51:33 -07:00
Harshit Bansal 6ff7da04de emoji: Remove `NotoColorEmoji.ttf`.
We no longer use glyphs from `NotoColorEmoji.ttf` so removing this.
2017-09-24 04:51:33 -07:00
Harshit Bansal c8c1c8ef43 emoji: Remove `AndroidEmoji.ttf`. 2017-09-24 04:51:33 -07:00
Harshit Bansal 0d8e68d75e build_emoji: Some code cleanup. 2017-09-24 04:51:33 -07:00
Harshit Bansal 6b1525680e build_emoji: Remove now unused `MissingGlyphError` exception.
This exception was raised if there was no glyph available for a
codepoint. We no longer need this.
2017-09-24 04:51:33 -07:00
Harshit Bansal 898d7d1160 build_emoji: Remove now unused `bw_font()` function.
This function was used get a black and white glyph for an emoji if there
was no corresponding image file present in the `NotoColorEmoji.ttf` but
due to the new emoji farm setup code, we no longer need this.
2017-09-24 04:51:33 -07:00
Harshit Bansal b95fbbe29a build_emoji: Remove now unused `color_font()` function.
This function was used to color transparent number emojis. We no
longer need to do this since now we have remapped number emojis
to the corresponding colored emojis in the new emoji farm.
2017-09-24 04:51:33 -07:00
Harshit Bansal 6ebacba425 build_emoji: Remove `code_point_to_file_name_map()`.
We have symlinked the old emoji farm to the old emoji farm and hence
we don't use the images from the `NotoColorEmoji.ttf` file. This
function was used to generate a map from codepoint to filename by
parsing the ttx file passed to it. We no longer need this map.
2017-09-24 04:51:33 -07:00
Harshit Bansal e295bed20f build_emoji: Extract `generate_map_files()` function.
This commit extracts out the `generate_map_files()` function from
the `dump_emojis()` function. This function generates various data
files like `emoji_codes.js`, `name_to_codepoint.json` etc which are
used by webapp, bugdown etc.
2017-09-24 04:51:33 -07:00
Harshit Bansal ef96adf7e5 build_emoji: Remove old emoji farm setup code.
This commit removes the old emoji farm generation code  in favor of
`setup_old_emoji_farm()`. Instead of having individual images in old
emoji farm we now symlink them to the images in the new emoji farm.
2017-09-24 04:51:33 -07:00
Harshit Bansal 273e37b2f9 build_emoji: Add `setup_old_emoji_farm()` function.
This commit adds `setup_old_emoji_farm()` function to the build_emoji
script. This will change the way of setting up the old emoji farm.
Earlier we used to extract the glyphs corresponding to each emoji from
the `NotoColorEmoji.ttf` file. But since now we already have individual
images in the new emoji farm(from iamcal's 'emoji-datasource-google' npm
package) we can just symlink old emoji files to the new image files. This
apart from helping us in cleaning up the `build_emoji` script will also
help in reducing the increased size of the release tarball due to the
addition of new emoji farm.
2017-09-24 04:51:33 -07:00
Harshit Bansal eed7d4b8d7 build_emoji: Remove old code in favor of `generate_sprite_css_files()`. 2017-09-24 04:51:33 -07:00
Harshit Bansal beab00a5e1 build_emoji: Add `generate_sprite_css_files()` function. 2017-09-24 04:51:33 -07:00
Harshit Bansal dcf6f1e0f0 build_emoji: Build new emoji farm.
This commit adds the 'setup_emoji_farm()' which copies the individual
images from `emoji-datasource-google` packge to target emoji dump.
2017-09-24 04:51:33 -07:00
Eeshan Garg 390a1fec92 zulip_bots: Generate static files during provisioning.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.

This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
2017-09-22 15:14:42 -07:00
Vishnu Ks 61e48734a7 provision: Add moreutils in APT common dependencies.
This fixes an issue where `tools/update-locked-requirements` would fail.
2017-09-22 14:04:57 -07:00
Juliana Bacelar 928dd06cc8 linter: Add lint rule banning 'import os.path' 2017-09-22 10:32:21 -07:00
Tim Abbott b1dafe5d9d renumber-migrations: Fix last_correct_migration code. 2017-09-22 07:51:58 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Steve Howell 6ce68dfc3d minor: Remove obsolete lint check.
Some code that we had a lint exception for got cleaned up in a
recent commit.
2017-09-20 13:05:04 -07:00
Steve Howell f13cb94d84 Have exclude_topic_mutes() accept a stream id.
We want to convert stream names to stream ids as close
to the "edges" of our system as possible, so we let our
caller do the work of finding the stream id for a stream
narrow.
2017-09-20 10:31:33 -07:00
Tim Abbott 02d931a53e tools: Fix running full-stack tests from tools/ directory.
Previously these tests required you to run them with the root of the
Zulip repository as the current working directory, just due to
sloppiness.

We clean this up, while also making the path handling more consistent
and involving less fragile code.

Fixes #4169.
2017-09-17 11:12:51 -07:00
Vishnu Ks f4c69bb126 tools: Create tool for inlining email CSS during provision. 2017-09-16 07:17:48 -07:00
invinciblycool 5475c5cedb tests: Replace test_user_agent_parsing with a normal test.
Previously, this was its own separate test script; now it's a normal
part of the test suite.

Tweaked by tabbott to use a proper test method.

Fixes #6327.
2017-09-16 03:56:18 -07:00
Greg Price ceca96899a deps: Adjust the header comments in lock files to reflect our tools. 2017-09-15 00:23:20 -07:00
Greg Price 7c5f3d7be1 deps: Refactor tools/update-locked-requirements to save repeating the filenames.
My first version of this just replaced the repeated list of two output
files with an array variable, but I decided `"${outputs[@]}"` was too
much to ask people to understand, and the alternative of `$outputs`,
unquoted, encourages bad habits of shell programming.  So just handle
one file at a time; the only at all expensive part here is `pip-compile`.

I am tempted to move this to Python, but holding back.
2017-09-15 00:23:20 -07:00
Greg Price a171f23d63 deps: Remove a redundant `deactivate`.
This has no effect except on the running shell; so at the end
of a script, it has no meaningful effect at all.
2017-09-15 00:23:20 -07:00
Greg Price b362e21ebb deps: Simplify tools/update-locked-requirements slightly.
This makes the code a little more concise and also more uniform,
treating `future` the same in prod and in dev.  The non-uniformity
looks like an oversight in 2be8a793e, one of the commits that updated
this code for the Python 3-only world.
2017-09-15 00:23:20 -07:00
Greg Price bab2078705 deps: Explain the `-e` oddness, and handle it uniformly. 2017-09-15 00:23:20 -07:00
Greg Price 806220cafa deps: Organize tools/update-locked-requirements a bit further.
A couple of remarks still reflected the Python 2 world; and the
"Remove the editable flag" comment was confusingly above a line
that it looks like it could be talking about, but isn't.
2017-09-15 00:23:20 -07:00
Supermanu 184bfe7cfd docs: Expand the french translation style guide.
As the french word organisation is used, the related line is excluded
from the linter spell check.
2017-09-14 08:35:51 -07:00
Robert Hönig 5294e10801 custom linter rules test: Load rules in test setup. 2017-09-14 04:17:03 -07:00
Robert Hönig 402d7fca50 test_linter_custom_check.py: Fix typo in docstring. 2017-09-14 04:17:03 -07:00