Commit Graph

2197 Commits

Author SHA1 Message Date
rht b0e4233d3f Dev environment setup: default to Python 3.
From the line in tools/provision it should trickle to the rest of the
scripts. This works since almost all the python scripts have been linted
to be generic.

Proof that the setup is python3 only: with this commit, within the
vagrant container env, /srv/zulip-venv is no longer present and
`./tools/run-dev.py` runs just fine.

[gnprice: Added `rm -f` and warning message, and made small edits.]
2017-08-08 15:28:11 -07:00
Tim Abbott 9bbcb712ec travis: Fix content-length test. 2017-08-08 13:22:22 -07:00
neiljp (Neil Pilgrim) b782db48e1 mypy: Remove superfluous older 'type: ignore' annotations. 2017-08-08 11:27:51 -07:00
neiljp (Neil Pilgrim) 982c0481d0 mypy: Mark log_file as Optional[str] in test_server.py. 2017-08-07 21:27:50 -07:00
neiljp (Neil Pilgrim) b7103752dc mypy: Add assertions relating to Popen.stdout with subprocess.PIPE use. 2017-08-07 21:27:50 -07:00
Greg Price b6d90808a5 prod: Make `cp` copy a directory to its target, not into it.
This was causing `language_options.json` and the `django.mo` files
never to get updated by `upgrade-zulip-from-git` after first deploy.
The stale `django.mo` files meant that any message string which wasn't
in the original from-tarball installation of the server would never
get translated.

The same bug exists in several other spots in this file.  Some of
our repetitive-yet-subtly-different uses of `cp` should probably
turn into just a list of arguments to one `rsync -R` command.
Also this should probably really use `rsync --delete` -- for example,
in prod on zulipchat.com we still have lying around a `locale/zh_CN`
directory, even though in a freshly-built static tree there's only
`locale/zh_Hans` instead.
2017-08-07 19:23:27 -07:00
Pweaver (Paul Weaver) a20e1298ab webpack: jQuery and underscore should be compilied normally to make errors readable.
We will use the normal compliation process so we can understand errors in development and production.
2017-08-05 19:53:56 -07:00
neiljp (Neil Pilgrim) 6ac6d6ff5d mypy: Add note re mypy supporting only global type aliases.
https://github.com/python/mypy/issues/3145
2017-08-05 17:53:18 -07:00
Tim Abbott 0d5cd2d5b7 tools: Remove obsolete print-all script.
This is an old script that could be used to literally print the Zulip
codebase to a PDF.  It hasn't been run since ~2013, and probably never
will be used again.  Since it's probably not interesting enough to be
its own open source project, we should just delete it.
2017-08-05 17:53:17 -07:00
Tim Abbott 52100e0b3b run-mypy: Remove requirement to be inside a Zulip venv.
It can be nice to just install+run mypy from outside.
2017-08-05 16:49:51 -07:00
Tim Abbott f7cfd76403 run-mypy: Check PROVISION_VERSION.
Since new mypy versions frequently break old versions, this check is a
useful way to help prevent problems where mypy output is wrong.

We could probably tighten this by checking explicitly the expected
mypy version from requirements.txt, but that's work.

Replaces #5026.
2017-08-05 16:18:48 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
neiljp (Neil Pilgrim) 353f6fdb88 mypy: Hoist RuleList type alias out of function. 2017-08-05 12:08:59 -07:00
Rishi Gupta 4bf8ac2498 generate_test_data: Replace source with non-Gutenberg text.
It's hard to find literature with the community tone we're going for, that
is consistent with the Zulip code of conduct, etc.

This commit removes the special tooling for Gutenberg plays, and changes the
text to be some mixture of scigen, Communications From Elsewhere,
chat.zulip.org, and various books from the public domain.
2017-08-05 11:04:35 -07:00
Pweaver (Paul Weaver) 21cad9cc24 webpack: Convert webpack config to typescript.
This helps catch bugs in the configuration.
2017-08-04 11:01:37 -07:00
Pweaver (Paul Weaver) d0b8e3f0c8 webpack: Merge webpack configuration into one file.
This fixes the confusing duplication between our various webpack config files.
2017-08-04 11:01:37 -07:00
Pweaver (Paul Weaver) 42a2e5dfa3 webpack: Turn webpack.config.js into a function.
This is a prerequisite for refactoring this.
2017-08-04 11:01:37 -07:00
neiljp (Neil Pilgrim) f96f71e439 mypy: assert Popen.stdout based on API, in tools/lint. 2017-08-03 11:09:50 -07:00
neiljp (Neil Pilgrim) e5c446c9b3 mypy: Set get_next_page_url in check-issue-labels to return Optional. 2017-08-03 11:03:14 -07:00
Rishi Gupta 3e9a85ecb2 onboarding: Add turtle reaction to an initial message. 2017-08-01 22:51:05 -07:00
Darren Fix 69767a43cb Create complex sample messages for dev VM.
Create a generator script to pull lines from a play, enhancing
random lines with emoji, Markdown and other flair.

With numerous contributions from Rein Zustand and Tim Abbott to finish
the project.

Fixes: #1666.
2017-08-01 20:54:39 -07:00
rht 532bd6eaa6 Ensure virtualenv is run throughout the entire Travis dev provision. 2017-08-01 15:29:13 -07:00
Steve Howell 30646f2a3b test-js-with-node: Add "Success:" to success message. 2017-08-01 13:43:35 -04:00
Greg Price 2d534e7397 Remove local_settings.py, as it grows stale with disuse.
This file hasn't reflected the actual configuration of any live
installation for some time, nor been part of any tests or other
mechanism to regularly validate it, so it's naturally fallen
behind as we make changes to the set of settings and typically
don't update this file accordingly.  Just remove it; all the
documentation functions it serves are already served just as
well by prod_settings_template.py and its ample comments.
2017-07-31 21:27:32 -07:00
Greg Price 5784724d00 api: Update lint and analysis code to forget about now-gone files.
These files don't exist in this repository anymore, so there's
no need to mention them in exclude lists for lint and mypy.
2017-07-31 21:24:52 -07:00
Tim Abbott c638ea8d11 docs: Fix spelling of Travis CI. 2017-07-31 09:36:22 -07:00
Greg Price 81bfab373d clean-repo: Revert optimization to .pyc removal.
Fixes #5887.  It seems there's some sort of issue in CPython where it
can get confused into thinking a `.pyc` file that's actually stale is
up to date -- perhaps when they date from the same second, say from
the middle of a rebase.

For now, rather than dig further to fix this properly and be sure of
having really done so, just go back to wiping out all `.pyc` files.
The impact is about 1s; that's noticeable when running a short test
file on a quad-core machine (about 8s), but not so much on a more
typical dev machine or on a larger set of tests.
2017-07-30 17:24:55 -07:00
Jack Zhang 5a8b1e6253 hotspots: Implement hotspots frontend. 2017-07-28 16:34:15 -07:00
Vaida Plankyte 02d8f62bb2 run-dev.py: Add Google auth test message & link to docs. 2017-07-27 17:35:14 -07:00
Tim Abbott 53e4d8562b lint: Add a lint check for bare `type: ignore`s. 2017-07-27 16:31:55 -07:00
Tim Abbott 27e1891706 mypy: Add flag to warn for unused ignores. 2017-07-27 16:12:26 -07:00
Tim Abbott d7766d7b6e lint: Fix colors being randomized on every file with custom checks.
Now, each custom check language gets a consistent color.
2017-07-27 16:03:12 -07:00
Pweaver (Paul Weaver) 1ae02cce97 Frontend debug: script-loader should load sourceURL. 2017-07-27 14:36:51 -07:00
Steve Howell bc0761b22b Extract topic_data.js.
This new module tracks the recent topic names for any given
stream.

The code was pulled over almost verbatim from stream_data.js,
with minor renames to the function names.

We introduced a minor one-line function called stream_has_topics.
2017-07-27 14:26:22 -07:00
Greg Price 360c27dedd tests: Bring all the top-level test scripts into alignment on what to test.
Notably, this adds our checks on translated message strings to
`tools/test-all`, so that they don't cause surprise failures in
CI after a branch is pushed.  (Alternately they could have gone
in `tools/lint` to accomplish the same goal, but `makemessages`
which they depend on is quite slow -- on my machine it takes 7s,
compared to 10s for all of `tools/lint`.)
2017-07-25 15:31:32 -07:00
Greg Price cd7cd73538 tests: Reorder tests in top-level test scripts to agree with each other.
This will make it easier to get the scripts to contain the same set of
tests (with explicit, conscious exceptions), and to keep it that way.
2017-07-25 15:31:32 -07:00
Vaida Plankyte 388acbbfcb frontend: Remove old required div & css from registration pages.
Removes class="required" divs that used to contain an asterisk
for valid/invalid input, and their associated css.
2017-07-25 13:39:34 -07:00
Robert Hönig 500ec48215 testing: Properly exclude files for linting in Windows.
For files to be excluded in Windows, their paths have to
be normalized, so that the string matching works.
2017-07-25 11:30:58 -07:00
Greg Price 7cd621bc9d webhooks/updown: Fix a JsonableError to provide a real error message. 2017-07-24 16:41:22 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Tim Abbott 8523c24a17 node_modules: Move success stamp to root of cache dir.
This location makes more sense, since it's the entire directory we're
certifying, not just the `node_modules` subdirectory.
2017-07-21 17:28:20 -07:00
Tim Abbott 4c37f93238 clean-npm-cache: Rename npm_cache to current_cache_dir_base. 2017-07-21 17:28:20 -07:00
Tim Abbott a3fd3fe1eb clean-npm-cache: Rename cached_node_modules to current_. 2017-07-21 17:28:20 -07:00
Tim Abbott 6bbd08eca3 clean-npm-cache: Rename success_stamp to current_success_stamp. 2017-07-21 17:28:20 -07:00
Tim Abbott 9f57532c3d clean-npm-cache: Rename npm_cache_dir to node_modules_cache_dir. 2017-07-21 17:28:20 -07:00
Tim Abbott fe6925b341 clean-npm-cache: Swap the order of the cleaning logic. 2017-07-21 17:28:20 -07:00
Tim Abbott 169fdd6e69 tools: Remove remaining deprecated tools.
It seems unlikely that we're going to get more future value out of the
old stress test than the cost of trying to keep it.
2017-07-21 17:28:20 -07:00
Tim Abbott 35a041c3bd clean-npm-cache: Move up NODE_MODULES_CACHE_PATH for Travis.
This makes the code more readable and also will help simplify the
patch when we switch to yarn.
2017-07-21 17:28:20 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Jack Zhang 1c9426fdce registration: Divide form into user and org sections. 2017-07-21 13:09:06 -07:00
Jack Zhang 47b4056948 register.html: Fix indentiation via check-templates. 2017-07-21 13:09:06 -07:00
Jack Zhang 8e2ec1cddc portico: Update registration form placeholders. 2017-07-21 13:09:06 -07:00
Jack Zhang e3ef056991 portico: Update registration form help messages. 2017-07-21 13:09:06 -07:00
Vishnu Ks fdafe69ba1 tools: Use get_user in test-api. 2017-07-20 16:50:23 -07:00
rht af430fe24d tools/linter_lib: Clean unused imports told by F401 pyflakes.
The '  # NOQA' comment is added so that the unused imports check (F401)
can still be run while still having the Mypy type check.
2017-07-20 16:46:07 -07:00
Aditya Bansal 7840ae0fe4 template_parser: Add 'with' to Django/Jinja block elements.
Fixes #5863.
2017-07-19 16:39:54 -04:00
Pweaver (Paul Weaver) 01b0ec69d6 Move the webpack stats files to var/ and deploy root.
static/ serves static files which get copied around per deploy. Since
the webpack stats files need a consistent name and change per deploy,
they can't live in static/.

This fixes a bug that preventing downgrading a Zulip server to an old
version.
2017-07-18 15:02:58 -07:00
Pweaver (Paul Weaver) d3ffc81726 Enable Hot Module Replacement in webpack.
This allow the webbpack dev server to properly reload JavaScript modules
while running in dev without restarting the server. We need to connect
to webpack-dev-server directly because SockJS doesn't support more than
one connection on the same host/port.
2017-07-18 11:02:05 -07:00
Tim Abbott 0970fb1a8f run-dev: Remove old node_modules/.bin/handlebars deletion hack.
This hack saved a lot of time debugging weird issues back in 2016, but
it's no longer needed.

Anyone rebasing a branch from 2016 will need to provision afterwards
regardless, which will fix this issue automatically, and more
importantly, these changes were made obsolete when we moved to the
cached `node_modules` model.
2017-07-17 17:57:38 -07:00
Tim Abbott 29c5913bb8 travis: Update success-http-headers. 2017-07-17 16:41:50 -07:00
Harshit Bansal d7617085e6 build_emoji: Update `get_success_stamp()` function.
Updates `get_success_stamp()` function to use the `emoji-datasource`
package's version while calculating success stamp so that an emoji
cache rebuild gets triggered automatically if the version is changed.
2017-07-16 12:36:19 -07:00
Tim Abbott 680381c9d0 setup_venv: Add virtualenv as a dependency for Debian.
Apparently, this was missing on Xenial, and we just never noticed
because the package was installed by default on Ubuntu.

It doesn't exist yet in Trusty.
2017-07-14 17:22:20 -07:00
Tim Abbott 4906a47c1a provision: Disable tsearch_extras for Debian. 2017-07-14 17:22:20 -07:00
Tim Abbott 9efc017e8c provision: Add postgres versions for Debian stretch. 2017-07-14 17:22:20 -07:00
Yago González 2079d84018 lint: Add mobile stores to the capitalization exceptions. 2017-07-14 16:27:17 -07:00
Tim Abbott 5cdfd61f2b lint: Require translation tags on aria-label tags.
Since these are, by their nature, directly user-facing strings, they
should always be tagged for translation.
2017-07-14 14:48:17 -07:00
Tim Abbott 67b30aa231 clean-repo: Reorder pycache removal to make output cleaner. 2017-07-14 12:34:18 -07:00
Harshit Bansal 92f6613bdd tools: Remove `tools/setup/emoji/download-emoji-data` tool.
This tool was used for downloading sprite sheets from iamcal's
repository. Since now we have moved to using `emoji-datasource`
npm package, this tool is no longer required.
2017-07-13 21:52:01 -07:00
Harshit Bansal ebdd8c9e3d emoji: Switch to using `emoji-datasource` npm package.
This commit does the following things:
* Instead of using a manual tool for downloading sprite sheets, use
  `emoji-datasource` npm package.
* Modify the `build_emoji` script to use sprite sheets from the npm
  package.

Bumps PROVISION_VERSION.

Fixes: #4730.
2017-07-13 21:52:01 -07:00
Harshit Bansal 420327b558 provision: Install NPM packages in the beginning of provisioning process.
NPM packages should be installed at the beginning of the provisioning
process so that later in the provisioning process if a script requires
any NPM package it can use it. Earlier, we were installing NPM packages
in the last as the installation process can fail due to network issues
but since we now retry in case the installation fails, they can be
installed safely at the beginning of the process as well just like apt
packages.
2017-07-13 21:52:01 -07:00
Steve Howell 7e88fb25b3 Move sent_messages callbacks into transmit_message().
This mostly sets the stage for a subsequent commit to start
using client_message_id as the key into sent_messages.

It has the nice side effect of making it more explicit that
certain things should always happen when transmit_message()
succeeds.

This commit does regress our node test coverage a bit.
2017-07-13 23:42:27 -04:00
Steve Howell f6d670ae3d Extract sent_messages.js.
This is mostly straightforward moving of code out of compose.js.

The code that was moved currently supports sending time
reports for sent messages, but we intend to grow out the new
module to track more state about sent messages.

The following function names in this commit are new, but their
code was basically pulled over verbatim:

    process_success (was process_send_time)
    set_timer_for_restarting_event_loop
    clear
    initialize

All the code in the new module is covered by previous tests that
had been written for compose.js.  This commit only modifies
a few things to keep those tests.

The new module has 100% node coverage, so we updated `enforce_fully_covered`.
2017-07-13 23:42:27 -04:00
Jack Zhang e29893a420 Fix indentation for templates/zerver/integrations/index.html. 2017-07-13 20:37:09 -07:00
Brock Whittaker 1cf7ee966b help: Redesign /help/ pages to be a single page app.
This redesigns the /help/ page sets to be a single page app that uses
history.pushState to work the same as the old app.

The big new feature is that now we have the index in a nicely designed
left sidebar.
2017-07-13 15:43:44 -07:00
Tim Abbott 344af2a7c9 travis: Use a wrapper script for installing moreutils.
This works around a frequent problem we've been having with Travis
CI's apt repository being busted on startup.  This, in turn, caused
`apt-get install moreutils` to fail.

We attempt to fix this by adding our own retry logic.
2017-07-12 13:43:47 -07:00
Aditya Bansal de1dc50204 linter/pep8: Remove code infra to ignore the E261 violations. 2017-07-11 15:17:32 -07:00
Aditya Bansal 84e53f960d pep8: Add compliance with rule E261 to tests/test_upload.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal 1b5e3190e3 pep8: Add compliance with rule E261 to tests/test_subs.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal 2643a40176 pep8: Add compliance with rule E261 to tests/test_signup.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal 59b26e582a pep8: Add compliance with rule E261 to tests/test_realm.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal 0970d29ec1 pep8: Add compliance with rule E261 to tests/test_narrow.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal ce70ded488 pep8: Add compliance with rule E261 to tests/test_messages.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal 8c233198f8 pep8: Add compliance with rule E261 to tests/test_events.py. 2017-07-11 15:17:32 -07:00
Aditya Bansal ee97ea1356 pep8: Add compliance with rule E261 to tests/test_bugdown.py. 2017-07-11 15:17:32 -07:00
Abdul Qabiz 7d7f2f78c8 clean-repo: Optimize .pyc removal code.
This optimizes .pyc files removal.

Instead of removing all pyc files, it will only remove pyc files where
corresponding py file is not found.  This saves about 1s in the
startup time of `run-dev.py` (and thus also `test-js-with-casper`).

Fixes #5760.
2017-07-11 14:40:56 -07:00
Aditya Bansal 0c4165a5bd pep8: Add compliance with rule E261 to populate_db.py. 2017-07-11 11:56:50 -07:00
Aditya Bansal 856e005d64 linter_lib/pep8.py: Remove files which don't have E261 conflicts. 2017-07-11 11:55:30 -07:00
Aditya Bansal d46cf59b0d pep8: Add compliance with rule E261 to worker/queue_processors.py. 2017-07-11 11:55:02 -07:00
Aditya Bansal 1e3b57570e pep8: Add compliance with rule E261 to tornado/websocket_client.py. 2017-07-11 11:55:02 -07:00
Aditya Bansal 94b790c28d pep8: Add compliance with rule E261 to tornado/socket.py. 2017-07-11 11:55:02 -07:00
Aditya Bansal b9bffaa446 pep8: Add compliance with rule E261 to prod_settings_template.py. 2017-07-11 11:55:01 -07:00
Aditya Bansal 5f87a0f8f8 pep8: Add compliance with rule E261 zproject/settings.py. 2017-07-11 11:53:33 -07:00
Aditya Bansal 7c6ad4a770 pep8: Add compliance with rule E261 zproject/dev_settings.py. 2017-07-11 11:53:33 -07:00
Aditya Bansal 44edab1cff pep8: Add compliance with rule E261 tools/run-dev.py. 2017-07-11 11:53:33 -07:00
Aditya Bansal 73ad834129 pep8: Add compliance with rule E261 bugdown/__init__.py. 2017-07-11 11:53:33 -07:00
Aditya Bansal f5d5c48831 pep8: Add compliance with rule E261 models.py. 2017-07-11 11:53:33 -07:00
Tim Abbott f75b72ef07 test-migrations: Fix error code bug.
Previously, this was failing unconditionally, because of `set -e`.
2017-07-10 16:20:47 -07:00
Aditya Bansal c493a0f02a test-migrations: Add a check to avoid auto named migrations. 2017-07-10 16:02:52 -07:00
Aditya Bansal 0f2c208a5b tools: Add new tool to pretty-print-html. 2017-07-09 07:01:29 -04:00
neiljp (Neil Pilgrim) 33a6a633eb mypy: Amend tools/check-css functions from Text to str. 2017-07-08 10:49:42 -07:00