Commit Graph

2277 Commits

Author SHA1 Message Date
Robert Hönig b5cf31efdc linter: Move custom_check_file to global namespace.
This is needed in order to mock the method when testing
`custom_check.py`. The diff for this commit is a bit broken;
all it really does is moving the method out of `build_custom_checkers`.
2017-09-12 03:24:59 -07:00
Robert Hönig 7a2234dfa2 linter: Remove redundant custom rule file delimitors. 2017-09-12 03:24:59 -07:00
Robert Hönig 2248c7ac0d linter: Add explanation for custom rule file delimitation operators. 2017-09-12 03:24:59 -07:00
Steve Howell 4dfe6bb320 Add migration to fix unread messages. 2017-09-07 07:06:03 -07:00
Robert Hönig 34e3329919 linter: Add rule against misspelled "Terms of Service". 2017-09-07 06:29:33 -07:00
Umair Khan bd43c447c3 travis: Log the Python version.
This allows us to ascertain the Python version under which we ran the
tests.
2017-09-06 13:48:50 -07:00
Umair Khan c039f73656 setup_venv: Link to the correct Python interpreter.
Travis enables different Python versions through virtual environments,
but it seems that there is a little caveat when we try to create Zulip's
virtual environment by referring Travis' virtual environment; Zulip's
virtual environment refers the system Python. We encountered this
behaviour when we tried to run our backend test suite under Python 3.5
in Travis. 'python3 --version' command before activating Zulip's
virtualenv showed 'Python 3.5.3' and after it showed 'Python 3.4.3'.
This happened when we created the virtual environment using
'virtualenv -p python3'.

The solution seems to be to explicitly give the path of the Python
interpreter in the Travis' virtual environment using 'which python3'.
2017-09-06 13:48:50 -07:00
Aditya Bansal d7f10bf3a1 profiling: Add snakeviz to dev tools.
This adds snakeviz to dev tools and also updates the message displayed
upon running `test-backend` with `--profile` option to say how to run
snakeviz correctly when using vagrant development environment.
2017-09-06 13:46:24 -07:00
Umair Khan ee6235d716 lint: Add regex to disallow i18n string concatenation.
Fixes #6398
2017-09-06 07:01:43 -07:00
Greg Price 1ab89b1d89 tools/test-all: Split `tools/lint` to match Travis scripts.
This keeps `test-all` visibly aligned with `travis/frontend` and
`travis/backend`, which helps us keep it substantively aligned.
2017-08-29 15:42:01 -07:00
Tim Abbott b087886d1c travis: Move i18n and capitalization linters to frontend.
Since these usually result from changes to HTML templates and other
frontend-side things, it seems better to group them with the frontend.

[Tweaked by gnprice in whitespace and comments.]
2017-08-29 15:42:01 -07:00
Tim Abbott 6ef55145f5 travis: Merge the static-analysis build into backend.
[Tweaked by gnprice to leave the use of --linecoverage-report
unmodified for now.]
2017-08-29 15:42:01 -07:00
Tim Abbott 6de77db93f lint: Fix problematic merging of pyflakes stdout/stderr.
We were getting pyflakes lint error output without line numbers like
this:

pyflakes  |     if user_profile.is_realm_admin and
pyflakes  |                                       ^
pyflakes  |

Apparently the cause was that stdout and stderr was getting mixed
badly, creating "unused import"s lines that had the first of that
error (containing the line number) just above.

As a result, printing out the lines of output from pyflakes' merged
stdout/stderr feed looked like this:

b"zproject/settings.py:95: 'from .prod_settings import *' used; unable to detect undefined nameszerver/views/users.py:49:39: invalid syntax\n"

Note the lack of newline in between the end of the first error at
"names" and the start of the second at "zerver".

This appears to be a change in Pyflakes behavior when we switched to
Python 3; probably they're missing a flush() somewhere.
2017-08-29 10:44:50 -07:00
Tim Abbott 130b5847ad pyflakes: Remove now-unnecessary exclude lines. 2017-08-29 10:27:29 -07:00
Tim Abbott 126171cb76 api: Remove root_domain_uri from API page links.
These pages should be references the current server's UI anyway.
2017-08-28 14:13:37 -07:00
Tim Abbott a0a1fe1512 settings: Rename SERVER_URI to ROOT_DOMAIN_URI.
This should be a lot less confusing.

See #6013 for discussion.
2017-08-28 14:09:28 -07:00
Brock Whittaker 04888101e2 Replace old portico footer with new sectioned one.
This replaces the old footer that has one section with a small list of
items. This expands the footer to have multiple sections.

Actual content tweaked and tagged for i18n by tabbott.
2017-08-28 11:07:02 -07:00
Tim Abbott b49448de7d message_edit: Eliminat whitespace in recipient row controls.
This is hacky, but I can't figure out another way to do it that
doesn't cause other problems.

Ideally, we'd add some sort of exclude rule to our HTML template
linter so we can check the rest of the file.
2017-08-27 22:18:29 -07:00
Harshit Bansal e5361b4403 clean_emoji_cache: Expand `clean-emoji-cache`.
Expands `clean-emoji-cache` so that it can be used in production
environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-27 17:51:24 -07:00
Harshit Bansal facb5dbe85 zulip_tools.py: Extract `generate_sha1sum_emoji()`.
Given the path of a zulip installation, it returns a hash corresponding
to the emoji infrastructure of that installation.
2017-08-27 17:51:24 -07:00
Harshit Bansal 948cf54ee3 clean-npm-cache: Expand `clean-npm-cache`.
Expands `clean-npm-cache` so that it can be used in production
environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-27 17:51:24 -07:00
Tim Abbott 90012f8109 find_add_class: Don't check frontend tests. 2017-08-27 14:04:46 -07:00
Tim Abbott ce1a8bf5b2 find_add_class: Include full path names in error messages. 2017-08-27 14:04:46 -07:00
Tim Abbott b1d98c1d0f lint: Only run find_add_class when JS files change. 2017-08-27 13:51:06 -07:00
Tim Abbott 79ae262463 lint: Only run check-urls after python files change. 2017-08-27 13:51:06 -07:00
Robert Hönig 3f01907d3b travis: Distribute linting over frontend and backend suites. 2017-08-27 13:51:06 -07:00
Robert Hönig c5da0ada31 linter: Replace --pep8 with new --frontend/--backend options.
Adds the --frontend and --backend options to replace
--pep8.

Significantly modified by tabbott to use a cleaner framework.
2017-08-27 13:50:53 -07:00
Robert Hönig 2186d53cf5 linter: Migrate lint from optparse to argparse. 2017-08-27 13:18:55 -07:00
Tim Abbott 1560646144 renumber-migrations: Fix construction of new number. 2017-08-27 12:01:06 -07:00
Tim Abbott 96dcb8f745 renumber-migrations: Fix unnecessary prompt.
The tool already knows the answer anyway.

Fixes #6157.
2017-08-27 12:01:06 -07:00
rht 437b2a3146 Update "MacOS" text to "macOS" 2017-08-26 09:00:42 -07:00
neiljp (Neil Pilgrim) 92985f888b mypy: Adjust check of json from github in update-authors-json. 2017-08-25 12:04:15 -07:00
neiljp (Neil Pilgrim) a059872856 mypy: Clarify xml.find use in build_emoji:code_point_to_file_name_map. 2017-08-25 12:04:15 -07:00
Tim Abbott 4b709de9c9 python3: Remove six.PY2/six.PY3 checks. 2017-08-25 11:06:43 -07:00
Tim Abbott b8e7369dee mypy: Remove type: ignores not needed in Python 3. 2017-08-25 11:04:20 -07:00
Tim Abbott b4d698f0db js-dep-visualizer: Report the PNG file as well. 2017-08-25 10:43:37 -07:00
Tim Abbott 3ceba5ef1e test-backend: Require coverage on most webhook integrations. 2017-08-24 11:33:14 -07:00
Yago González 7d5d8ac61e lint: Add rule for spacing inside Handlebars translation tags.
This commit also includes the fixes to make all Handlebars templates
compliant with the new rule.
2017-08-23 13:08:57 -07:00
Yago González 3a4c6f83fc lint: Ban trailing spaces in Handlebars translation tags. 2017-08-23 13:08:57 -07:00
Yago González aa5f57fda9 lint: Ban leading spaces in Handlebars translation tags. 2017-08-23 13:08:57 -07:00
rht e455284bb4 requirements: Remove pathlib2.
`pathlib2` is a backport of pathlib to Python 2.x. Ni!
This dependency can be removed since:
- Zulip 1.6.0+git has been on Python 3 nowadays.
  Ekki-ekki-ekki-ptang-zoom-boing.
- As stated in ticket #6211, having this removed, prevents the need to
  have lockfiles for each of 3.4, 3.5, (and maybe 3.6).

This fixes #6211.
2017-08-23 10:00:21 -07:00
Harshit Bansal 3534e328c5 clean-venv-cache: Expand `clean-venv-cache` tool.
Expands `clean-venv-cache` tool so that it can be used in prod.
Also moves it from `tools/` to `scripts/`.
2017-08-23 00:03:25 -07:00
Tim Abbott 50ad679342 test-backend: Recommend snakeviz over runsnake.
It has support for Python 3.
2017-08-22 15:44:30 -07:00
Tim Abbott 8dff14430e tools: Remove old python 3 compatibility checker.
Now that we've migrated over to Python 3, this tool is no longer required.

Fixes #6217.
2017-08-22 14:58:59 -07:00
Rishi Gupta b32fe9f2a0 features: Update screenshot and add tool to generate.
This adds a new management command that should make generating or
updating screenshots a bit easier in the future.
2017-08-22 11:32:22 -07:00
rht 0a469fd4c8 requirements: Lock prod.txt instead of its py3_common dependency.
`py3_common.txt` is no longer locked.
2017-08-22 10:10:57 -07:00
rht 80f4ba7a76 requirements: Replace py3_dev.txt with dev_lock.txt. 2017-08-22 10:10:57 -07:00
rht 2be8a793e4 Lock the recursive requirements in dev.txt via pip-compile.
First, all the lines of py3_dev.txt except for mypy.txt are mv-ed to
dev.txt. Then dev_lock.txt is generated from dev.txt to be used by
py3_dev.txt. `click` is removed from moto.txt since it is already a
dependency of several libraries and will appear as an autogenerated
dependency.
2017-08-21 18:24:55 -07:00
rht 6baef37564 requirements: Remove Python 2-specific files. 2017-08-21 18:24:55 -07:00
Steve Howell c26e000d4e Fix tools/review for python3. 2017-08-17 17:40:24 -04:00
Umair Khan cec45b0ae5 tools: Add inline string replacer for files. 2017-08-17 09:03:35 -07:00
Greg Price b8dcac1b42 run-mypy: Stop running `mypy --py2`.
Also unconditionally use the `mypy` from our virtualenv --
that's how we ensure we use a common version across different
Zulip developers and in CI.

And as a side effect of cutting some Python 2 vs. Python 3 logic,
fix a bug where `--all` was having no effect.
2017-08-16 17:54:43 -07:00
Greg Price 6346861ccd py3: Cut out the `build-release-tarball --py3` flag.
This is now redundant, as we've applied this transformation to the
source tree itself.
2017-08-16 17:54:43 -07:00
Greg Price 33430f83eb py3: Rely on the shebang to invoke provision too.
We had been forcing provision to Python 3 in dev.  Now that everything
is Python 3 and the `tools/lib/provision.py` shebang reflects that, we
can just invoke the script directly like everything else.
2017-08-16 17:54:43 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Greg Price b7ec1b55dd py3: Cut Python 2 codepaths from provision. 2017-08-16 17:54:43 -07:00
Greg Price d968179e54 py3: Stop running on Python 2 in Travis.
This clears the way to dropping logic in lots of places
for supporting Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott debfe708d0 test-all: Fix buggy PYTHONWARNINGS invocation.
Shell only sets variables if they're the first thing on a line.
2017-08-16 11:56:24 -07:00
Tim Abbott 37eb6a9863 run-dev: Remove now-unnecessary checks for Google/GitHub auth. 2017-08-16 10:05:19 -07:00
Anirudh Jain 28944b6c94 provision: Partially add zesty to supported systems.
We can't fully support it until we fix the tsearch_extras availability
issue, but for now, this is an improvement.

Tweaked by tabbott to cover the outstanding tsearch_extras issue.
2017-08-15 21:51:19 -07:00
Tim Abbott 9b109b4b73 build-release-tarball: Cleanup Git version sanity check.
Checking for whether tests appear is going to be a more stable
long-term check than looking for zilencer, which we might eventually
include in releases.
2017-08-15 20:44:41 -07:00
neiljp (Neil Pilgrim) 7aacda1883 mypy: Mark internet_guard on requests.request as 'type: ignore'
[gnprice: tweaked comment]
2017-08-15 17:50:18 -07:00
neiljp (Neil Pilgrim) 1a3d8ced78 mypy: Allow Optional parameters in html_branches.py. 2017-08-15 17:50:18 -07:00
Greg Price 73cb361896 install-aws-server: Cut out use of /root/zulip .
Our previous dependencies on the `/root/zulip` path should all be
long gone at this point.  Use a fresh temporary directory instead;
that's cleaner.
2017-08-15 17:41:07 -07:00
Greg Price 835c2ca1ce travis: Cut /root/zulip out of prod test path, just to be sure.
Our previous dependencies on the `/root/zulip` path should all be
long gone at this point.  Run our production-install test suite through
a fresh temporary path instead, mainly just to avoid causing any confusion
over whether that's quite the case.
2017-08-15 17:41:07 -07:00
Greg Price ceccd0c0f6 lint: Accept mentions of specific Python versions.
We're about to do this (a) in a number of places mentioning
system packages like `python3-dev` to install, and (b) in the
shebangs of every script in the tree.
2017-08-15 17:30:31 -07:00
Greg Price 61666a9262 zulip_ops: Delete the long-disused `stats1.zulip.net` config and its dependencies.
This consists of the `zulip_ops::stats` Puppet class, which has apparently
not been used since 2014, and a number of files that I believe were
only used for that.  Also a couple of tiny loose ends in other files.
2017-08-15 17:30:31 -07:00
Greg Price e469578a55 py3: Fix up (almost) all script invocations to rely on shebangs.
This follows up on 207cf6302 from last year to clean up cases that
have apparently popped up since then.  Invoking the scripts directly
makes a cleaner command line in any case, and moreover is essential
to how we control running a Zulip install as either Python 2 or 3
(soon, how we always ensure it runs as Python 3.)

One exception: we're currently forcing `provision` in dev to run
Python 3, while still running both Python 2 and Python 3 jobs in CI.
We use a non-shebang invocation to do the forcing of Python 3.
2017-08-15 17:30:31 -07:00
Brock Whittaker 83184d5c60 /about/: Merge with /authors, redesign page.
This adds the authors to the Zulip repository on GitHub from
/authors/ along with re-styling the page to fit the same
aesthetic as /for/open-source/ and other product-pages.
2017-08-15 16:09:01 -07:00
Steve Howell f0de6bd461 node tests: Add "Starting node tests..." message. 2017-08-15 14:50:10 -04:00
Tim Abbott 84da22da67 travis: Improve production test for HTTP success to not check length.
This will make life a lot easier when iterating on the login page.
2017-08-15 10:51:29 -07:00
Tim Abbott ffc1ceeaa4 lint: Ban use of get_stream in the rest of messages.py.
This will help avoid potential future security bugs.
2017-08-15 10:40:02 -07:00
Robert Hönig c5f0478374 linter: Add trailing whitespace check for json files. 2017-08-15 09:28:11 -07:00
Vaida Plankyte 902d2715a5 run-dev.py: Fix typo in GitHub auth warning message. 2017-08-14 14:35:15 -07:00
Greg Price 6e9f8b90dc provision: Remove the last reference to `install-mypy`, and delete it.
This script has been redundant for some time now.
2017-08-10 18:00:19 -07:00
Jack Zhang ffe14e339d Extract /integrations code from landing-page into separate bundle. 2017-08-10 10:21:53 -07:00
Jack Zhang b85d3e28ce portico: Update display text for integrations categories dropdown.
Category 'All' -> text 'Filter by category'; icon chevron right when
the dropdown is closed, icon chevron down when the dropdown is open

All other categories -> text CATEGORIES[state.category]; icon chevron down
2017-08-10 10:21:53 -07:00
Jack Zhang c546016964 portico: Add translations bundle to portico pages. 2017-08-10 10:21:53 -07:00
Greg Price a32a3cfe23 provision: Fix some shell hygiene.
An array expansion should nearly always be double-quoted, like `"$@"`.
See http://mywiki.wooledge.org/Quotes for some discussion.
2017-08-09 14:03:07 -07:00
Greg Price d3a69da829 py3: Fix or remove (almost) all references to Python 2 venv in docs.
I think soon we'll put the Python 3 venv at `/srv/zulip-venv` and
make `/srv/zulip-py3-venv` just an alias to that (then remove the
alias too), but for now the old name is helpful for spotting places
that need an update.

The `/srv/zulip-venv` name still appears in codepaths used by Travis
tests.
2017-08-08 15:48:12 -07:00
rht b0e4233d3f Dev environment setup: default to Python 3.
From the line in tools/provision it should trickle to the rest of the
scripts. This works since almost all the python scripts have been linted
to be generic.

Proof that the setup is python3 only: with this commit, within the
vagrant container env, /srv/zulip-venv is no longer present and
`./tools/run-dev.py` runs just fine.

[gnprice: Added `rm -f` and warning message, and made small edits.]
2017-08-08 15:28:11 -07:00
Tim Abbott 9bbcb712ec travis: Fix content-length test. 2017-08-08 13:22:22 -07:00
neiljp (Neil Pilgrim) b782db48e1 mypy: Remove superfluous older 'type: ignore' annotations. 2017-08-08 11:27:51 -07:00
neiljp (Neil Pilgrim) 982c0481d0 mypy: Mark log_file as Optional[str] in test_server.py. 2017-08-07 21:27:50 -07:00
neiljp (Neil Pilgrim) b7103752dc mypy: Add assertions relating to Popen.stdout with subprocess.PIPE use. 2017-08-07 21:27:50 -07:00
Greg Price b6d90808a5 prod: Make `cp` copy a directory to its target, not into it.
This was causing `language_options.json` and the `django.mo` files
never to get updated by `upgrade-zulip-from-git` after first deploy.
The stale `django.mo` files meant that any message string which wasn't
in the original from-tarball installation of the server would never
get translated.

The same bug exists in several other spots in this file.  Some of
our repetitive-yet-subtly-different uses of `cp` should probably
turn into just a list of arguments to one `rsync -R` command.
Also this should probably really use `rsync --delete` -- for example,
in prod on zulipchat.com we still have lying around a `locale/zh_CN`
directory, even though in a freshly-built static tree there's only
`locale/zh_Hans` instead.
2017-08-07 19:23:27 -07:00
Pweaver (Paul Weaver) a20e1298ab webpack: jQuery and underscore should be compilied normally to make errors readable.
We will use the normal compliation process so we can understand errors in development and production.
2017-08-05 19:53:56 -07:00
neiljp (Neil Pilgrim) 6ac6d6ff5d mypy: Add note re mypy supporting only global type aliases.
https://github.com/python/mypy/issues/3145
2017-08-05 17:53:18 -07:00
Tim Abbott 0d5cd2d5b7 tools: Remove obsolete print-all script.
This is an old script that could be used to literally print the Zulip
codebase to a PDF.  It hasn't been run since ~2013, and probably never
will be used again.  Since it's probably not interesting enough to be
its own open source project, we should just delete it.
2017-08-05 17:53:17 -07:00
Tim Abbott 52100e0b3b run-mypy: Remove requirement to be inside a Zulip venv.
It can be nice to just install+run mypy from outside.
2017-08-05 16:49:51 -07:00
Tim Abbott f7cfd76403 run-mypy: Check PROVISION_VERSION.
Since new mypy versions frequently break old versions, this check is a
useful way to help prevent problems where mypy output is wrong.

We could probably tighten this by checking explicitly the expected
mypy version from requirements.txt, but that's work.

Replaces #5026.
2017-08-05 16:18:48 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
neiljp (Neil Pilgrim) 353f6fdb88 mypy: Hoist RuleList type alias out of function. 2017-08-05 12:08:59 -07:00
Rishi Gupta 4bf8ac2498 generate_test_data: Replace source with non-Gutenberg text.
It's hard to find literature with the community tone we're going for, that
is consistent with the Zulip code of conduct, etc.

This commit removes the special tooling for Gutenberg plays, and changes the
text to be some mixture of scigen, Communications From Elsewhere,
chat.zulip.org, and various books from the public domain.
2017-08-05 11:04:35 -07:00
Pweaver (Paul Weaver) 21cad9cc24 webpack: Convert webpack config to typescript.
This helps catch bugs in the configuration.
2017-08-04 11:01:37 -07:00
Pweaver (Paul Weaver) d0b8e3f0c8 webpack: Merge webpack configuration into one file.
This fixes the confusing duplication between our various webpack config files.
2017-08-04 11:01:37 -07:00
Pweaver (Paul Weaver) 42a2e5dfa3 webpack: Turn webpack.config.js into a function.
This is a prerequisite for refactoring this.
2017-08-04 11:01:37 -07:00
neiljp (Neil Pilgrim) f96f71e439 mypy: assert Popen.stdout based on API, in tools/lint. 2017-08-03 11:09:50 -07:00
neiljp (Neil Pilgrim) e5c446c9b3 mypy: Set get_next_page_url in check-issue-labels to return Optional. 2017-08-03 11:03:14 -07:00
Rishi Gupta 3e9a85ecb2 onboarding: Add turtle reaction to an initial message. 2017-08-01 22:51:05 -07:00