Commit Graph

231 Commits

Author SHA1 Message Date
Tim Abbott 335c383bbb lint: Add some excludes to capitalization linter. 2018-03-02 11:55:46 -08:00
Aditya Bansal 6fce1d7834 template_parser: Add parsing support for self closing tags as per HTML5.
In this commit we add support for some tags which are also called
void-elements according to
http://w3c.github.io/html/syntax.html#void-elements to be parsed by
our template parser and get tagged as singleton_html_tags.

Fixes: #8387.
2018-02-15 17:47:43 -05:00
Shubham Padia e1f943913a lint: Allow revert commit messages in gitlint.
Fixes #8145.
2018-02-13 09:21:01 -08:00
Aditya Bansal d5736ef65a provision.py: Remove duplicate install candidated from APT_DEPENDENCIES.
This is necessary, since we now have some dependencies in common
between the Thumbor and main Zulip dependency lists.
2018-02-12 14:40:26 -08:00
Umair Khan c415cc74d7 validate_stream_message_address_info: Add i18n tags.
Fixes #7076
2018-02-05 16:26:24 -08:00
rht 396a8b644d provision.py: Colorize error messages for fail condition. 2018-02-05 11:44:33 -08:00
Eeshan Garg c158869096 Rename tools/lib/api_tests.py to zerver/lib/api_test_helpers.py.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
2018-01-31 07:30:54 -05:00
Eeshan Garg a1a69a0ac2 api/render-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_example to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Aditya Bansal 0fcf0c5052 thumbor: Add thumbor on port 9995 in development.
For now, this does nothing in a production environment, but it should
simplify the process of doing testing on the Thumbor implementation,
by integrating a lot of dependency management logic.
2018-01-29 13:10:29 -08:00
Greg Price ff29fe07be provision: Install virtualenv on xenial too.
As the commit message on 680381c9d which added this for stretch says,
we need this dependency when on xenial as well.  Add it there.
2018-01-22 19:36:52 -08:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Steve Howell 3cb6871f90 Improve messaging in test_server_running().
We now make it a bit easier to see that you are waiting for
the server, as well as indicating when the server is up.
2018-01-16 13:25:19 -05:00
Cynthia Lin 83706e1bcd org settings: Add User groups content and design. 2018-01-06 11:50:52 -05:00
rht 185fd99816 mypy: Use Python 3 type syntax in several files. 2017-12-30 07:34:51 -05:00
Rhea Parekh 6e374d30e1 tools/lib/provision: Clean the code to directly read files in binary mode. 2017-12-26 09:09:31 -05:00
Greg Price e5c6c446b5 travis + circleci: Start postgresql during provision.
Otherwise, in CircleCI it isn't running when we need it later.
Travis must special-case starting it automatically.
2017-12-18 22:05:27 -08:00
Greg Price c83c118c6c circleci: Crudely clone a bit of `is_travis` logic.
This isn't really the right way to do this -- commit
  dcd80e665 "travis/setup-backend: Remove the '--travis' flag"
took us in the wrong direction by introducing more magic, deeper in
the stack.

But it's the same way we do it for Travis.  For now, just copy that.

[Thanks to hackerkid for cleaning up my original crude hack.]
2017-12-18 22:04:39 -08:00
Greg Price e8bf7d23bf provision: Remove transitional Python 2->3 logic in setup_venvs.
At this point if we were accidentally using `/srv/zulip-venv` for
anything, we'd have run into it by now.  So just drop the bit of
historical logic that we had to ensure that.
2017-12-18 21:59:51 -08:00
Greg Price 137c0e65bb tools: Revert to Python 2 typing syntax for now.
This reverts commit 66261f1cc.  See parent commit for reason; here,
provision worked but `tools/run-dev.py` would give errors.

We need to figure out a test that reproduces these issues, then make a
version of these changes that keeps that test working, before we
re-merge them.
2017-12-13 10:38:15 -08:00
rht 66261f1cc3 tools: Use Python 3 syntax for typing in many files. 2017-12-12 17:42:57 -08:00
rht bff736868e Generate custom-icon-webfont on each provision or update-prod-static.
Fixes #7354.
2017-11-20 16:36:49 -08:00
Greg Price 13aa23eb40 provision: Run clean-unused-caches.
This prevents the caches in /srv from growing to fill up the disk --
e.g., on my laptop after 6 months of regular development the venv cache
was 12G and the NPM cache 5G, making them by far the largest disk hogs
on the machine.

It costs about 0.4s, apart from any time spent actually removing
things.  This is a little annoyingly slow to be adding to every
provision, and seems like it could be optimized, but I think already
worth it as is.
2017-11-18 15:04:08 -08:00
Tim Abbott 638eb7a8e4 docs: Update links to ReadTheDocs to always use https.
This is better security practice.

We also add a lint rule to enforce this for the future.
2017-11-16 10:59:24 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
Tim Abbott 774380dcb8 capitalization: Allow enabled/disabled to be lower-case. 2017-11-15 17:39:09 -08:00
rht 72ce1c6501 Use a more concise `mkdir -p` when the situation permits. 2017-11-14 09:47:09 -08:00
Steve Howell ba51078418 Simplify CSS linter and clean up CSS.
The CSS linter was pretty hard to reason about.  It was
pretty flexible about certain things, but then it would
prevent seemingly innocuous code from getting checked in.

This commit overhauls the pretty-printer to be more composable,
where every object in the AST knows how to render itself.  It
also cleans up a little bit of the pre_fluff/post_fluff logic
in the parser itself, so comments are more likely to be "attached"
to the AST node that make sense.

The linter is actually a bit more finicky about newlines, but
this is mostly a good thing, as most of the variations before
this commit were pretty arbitrary.
2017-11-13 12:43:43 -08:00
rht 7a96b8c7ad tools/lib: Text-wrap long lines exceeding 110. 2017-11-10 16:24:09 -08:00
rht e3eebf3be0 Remove inheritance from object. 2017-11-06 08:53:48 -08:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht 9c5e0971f4 sanity_check: Check a few packages.
This just makes our check for whether we're actually in a Zulip
development environment better.
2017-10-17 22:21:33 -07:00
Tim Abbott 508686bed4 provision: Fix buggy bash_profile management code.
Apparently, the previous version would just append to .bash_profile
every single time :(.
2017-10-17 21:22:32 -07:00
Tim Abbott 3fed995a41 provision_version: Move to under UUID_VAR_PATH. 2017-10-17 21:15:58 -07:00
Tim Abbott 3983ebce9a test_fixtures: Move test_db_status state under UUID_VAR_PATH.
This should mean that maintaining two Zulip development environments
using the same Git checkout no longer has caching problems keeping
track of the migration status.
2017-10-17 21:15:58 -07:00
Tim Abbott a7c639cff7 provision: Use UUID in path to compilemessages hash. 2017-10-17 21:15:58 -07:00
Tim Abbott 2ae2a94444 provision: Stop using shared var/ for caching apt state.
This didn't work at all when one did a `vagrant destroy` and then
`vagrant up`, because the cache state would be preserved even though
the machine is gone.

Fixes #5981.
2017-10-17 21:15:58 -07:00
derAnfaenger df948c7ea5 Remove useless shebang lines. 2017-10-11 12:52:36 -07:00
rht 34bafa654c mypy: Fix future syntax errors and other minor mistakes.
When we move to the Python 3 mypy syntax, we can't reference a class
before its definition.
2017-10-10 22:27:16 -07:00
rht dcd80e6650 travis/setup-backend: Remove the '--travis' flag in tools/provision.
Whether the env is Travis or not, this has been detected via $TRAVIS env var.
Proof:
6fbf41bdbc/tools/provision (L38)
2017-10-06 11:20:25 -07:00
rht 2c2adbf647 tools/lib: Remove unused imports. 2017-10-02 10:13:46 -07:00
rht f15bdce90e tools: Remove print_function.
Tweaked by tabbott to exclude the linter libraries.
2017-09-29 15:44:56 -07:00
Rhea Parekh 0e5fdcea3f provision: Run apt-get update on catching an error.
For broken apt cache, we run the command apt-get update on catching
the error.
2017-09-29 15:38:23 -07:00
rht bf4eda7374 tools: Remove absolute_import in most tools.
Tweaked by tabbott to not remove it from lister.py, linter_lib, and
friends, since those are intended to support both Python 2 and 3
(we're planning to extract them from the repository).
2017-09-29 12:28:43 -07:00
Eeshan Garg bb34ba37b5 tools: Rename build_pygments_data.py -> build_pygments_data.
Dropping the file extension (.py) conforms to our naming
convention for such scripts.
2017-09-26 16:00:41 -07:00
Umair Khan 5d00f29c68 capitalization: Ignore LDAP. 2017-09-26 15:58:17 -07:00
Tim Abbott 8e2ae66fe6 provision: Update apt operations text. 2017-09-25 14:47:18 -07:00
Tim Abbott 16ed23414d provision: Fix spelling of apt_dependencies_hash.
Also use a variable to avoid duplicating the path.
2017-09-25 14:46:46 -07:00
Eeshan Garg 390a1fec92 zulip_bots: Generate static files during provisioning.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.

This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
2017-09-22 15:14:42 -07:00
Vishnu Ks 61e48734a7 provision: Add moreutils in APT common dependencies.
This fixes an issue where `tools/update-locked-requirements` would fail.
2017-09-22 14:04:57 -07:00
Vishnu Ks f4c69bb126 tools: Create tool for inlining email CSS during provision. 2017-09-16 07:17:48 -07:00
Tim Abbott 126171cb76 api: Remove root_domain_uri from API page links.
These pages should be references the current server's UI anyway.
2017-08-28 14:13:37 -07:00
Tim Abbott a0a1fe1512 settings: Rename SERVER_URI to ROOT_DOMAIN_URI.
This should be a lot less confusing.

See #6013 for discussion.
2017-08-28 14:09:28 -07:00
Tim Abbott 90012f8109 find_add_class: Don't check frontend tests. 2017-08-27 14:04:46 -07:00
Tim Abbott ce1a8bf5b2 find_add_class: Include full path names in error messages. 2017-08-27 14:04:46 -07:00
rht 437b2a3146 Update "MacOS" text to "macOS" 2017-08-26 09:00:42 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Greg Price b7ec1b55dd py3: Cut Python 2 codepaths from provision. 2017-08-16 17:54:43 -07:00
Anirudh Jain 28944b6c94 provision: Partially add zesty to supported systems.
We can't fully support it until we fix the tsearch_extras availability
issue, but for now, this is an improvement.

Tweaked by tabbott to cover the outstanding tsearch_extras issue.
2017-08-15 21:51:19 -07:00
neiljp (Neil Pilgrim) 1a3d8ced78 mypy: Allow Optional parameters in html_branches.py. 2017-08-15 17:50:18 -07:00
Greg Price d3a69da829 py3: Fix or remove (almost) all references to Python 2 venv in docs.
I think soon we'll put the Python 3 venv at `/srv/zulip-venv` and
make `/srv/zulip-py3-venv` just an alias to that (then remove the
alias too), but for now the old name is helpful for spotting places
that need an update.

The `/srv/zulip-venv` name still appears in codepaths used by Travis
tests.
2017-08-08 15:48:12 -07:00
neiljp (Neil Pilgrim) b782db48e1 mypy: Remove superfluous older 'type: ignore' annotations. 2017-08-08 11:27:51 -07:00
neiljp (Neil Pilgrim) 982c0481d0 mypy: Mark log_file as Optional[str] in test_server.py. 2017-08-07 21:27:50 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
rht 532bd6eaa6 Ensure virtualenv is run throughout the entire Travis dev provision. 2017-08-01 15:29:13 -07:00
Jack Zhang 5a8b1e6253 hotspots: Implement hotspots frontend. 2017-07-28 16:34:15 -07:00
Greg Price 7cd621bc9d webhooks/updown: Fix a JsonableError to provide a real error message. 2017-07-24 16:41:22 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Jack Zhang e3ef056991 portico: Update registration form help messages. 2017-07-21 13:09:06 -07:00
Aditya Bansal 7840ae0fe4 template_parser: Add 'with' to Django/Jinja block elements.
Fixes #5863.
2017-07-19 16:39:54 -04:00
Tim Abbott 680381c9d0 setup_venv: Add virtualenv as a dependency for Debian.
Apparently, this was missing on Xenial, and we just never noticed
because the package was installed by default on Ubuntu.

It doesn't exist yet in Trusty.
2017-07-14 17:22:20 -07:00
Tim Abbott 4906a47c1a provision: Disable tsearch_extras for Debian. 2017-07-14 17:22:20 -07:00
Tim Abbott 9efc017e8c provision: Add postgres versions for Debian stretch. 2017-07-14 17:22:20 -07:00
Yago González 2079d84018 lint: Add mobile stores to the capitalization exceptions. 2017-07-14 16:27:17 -07:00
Harshit Bansal ebdd8c9e3d emoji: Switch to using `emoji-datasource` npm package.
This commit does the following things:
* Instead of using a manual tool for downloading sprite sheets, use
  `emoji-datasource` npm package.
* Modify the `build_emoji` script to use sprite sheets from the npm
  package.

Bumps PROVISION_VERSION.

Fixes: #4730.
2017-07-13 21:52:01 -07:00
Harshit Bansal 420327b558 provision: Install NPM packages in the beginning of provisioning process.
NPM packages should be installed at the beginning of the provisioning
process so that later in the provisioning process if a script requires
any NPM package it can use it. Earlier, we were installing NPM packages
in the last as the installation process can fail due to network issues
but since we now retry in case the installation fails, they can be
installed safely at the beginning of the process as well just like apt
packages.
2017-07-13 21:52:01 -07:00
Tim Abbott f2379d66a2 i18n: Fix capitalization of Zulip Botserver download code. 2017-07-07 17:02:39 -07:00
Tim Abbott 919c610abf capitalization: Add APNS to list of valid acronyms. 2017-07-07 10:43:02 -07:00
Tim Abbott 774216713b provision: Document more clearly the is_production_travis block. 2017-07-06 17:39:49 -07:00
sinwar 49db0855d2 provision: Avoid spending ~3s on RabbitMQ in no-op case.
Tweaked by tabbott to reorganize the logic more clearly.

Fixes #5180.
2017-07-06 17:38:39 -07:00
sinwar 3d0b5b14be provision: Avoid spending ~1.5s on compilemessages in no-op case.
This brings the total no-op provision time down to around 7.3s!

Tweaked by tabbott to clean up the code, close file descriptions, and
avoid issues with a somehow corrupted file.

Fixes #5185.
2017-07-01 11:27:54 -07:00
Yago González 7e3154f7bc lint: Ignore tildes in mustache tag template checker.
Handlebars allows putting tildes (~) by the braces in mustache tags to
strip whitespaces on that side of the tag.

    {{#if foo~}}
        <p>Bar</p>
    {{~/if}}

This way, the linter ignores any potential tilde at the ends of Handlebars
tags so we can use this feature without lint fails.
2017-06-26 19:33:25 -04:00
sinwar 0e24930770 provision: Avoid spending ~3s on apt operations in no-op provision.
Fixes #5186
2017-06-22 09:37:40 -04:00
Tommy Ip e9fd41bb66 lint: Check if the first word of a commit message is imperative.
Only check the first word using a black list approach to reduce
force positive as shown in the current implementation.
2017-06-19 07:01:52 -04:00
Aditya Bansal a2b5d6efcb provision: Add wrapper to retry apt-get install if it fails first time. 2017-06-14 21:49:15 -07:00
Aditya Bansal d2f0e17454 linter: Fix bug resulting in <pre> being indented cyclically. 2017-06-13 08:39:59 -04:00
Aditya Bansal a9e0dbe085 pep8: Add compliance with rule E261 pretty_print.py. 2017-06-04 15:05:51 +05:30
Tim Abbott cecf7f1740 provision: Don't regenerate dev database unnecessarily.
This reuses the work we did some time ago to avoid regenerating the
test database unnecessarily.

In addition to being a nice convenience for developers (since any
accomulated test data is still available), this also saves about half
the time consumed in a no-op provision.

Fixes #5182.
2017-06-02 13:27:34 -07:00
Steve Howell 5f48b51b60 provision: Add `cd /srv/zulip` to profile.
I didn't bump the provision version on this change, as it
won't break anything if somebody fails to make this change.
2017-05-29 08:40:49 -07:00
Tim Abbott 9b8641d9eb lint: Add iOS to list of valid weird capitalizations. 2017-05-26 20:59:32 -07:00
Umair Khan ad5780c03e lint: Mark alt argument for translation. 2017-05-26 15:28:46 -07:00
rht a1f82e02d6 python: Replace os.system with subprocess.call.
Generally, we avoid os.system, since it shells out and thus can be
a cause of security issues.
2017-05-26 15:03:16 -07:00
Tim Abbott 8aaf9b1426 Revert "Fix emoji cache setup when not using provision.py."
This reverts commit ab2cfadcaa.

This broke production installation.
2017-05-24 10:35:41 -07:00
Eklavya Sharma 7636972a6c mypy: Wrap return value in Optional. 2017-05-23 21:56:50 -07:00
Eklavya Sharma cc1937c8d5 mypy: Use Optional with strings where required. 2017-05-23 21:56:50 -07:00
Jordan Gedney a9f4d56a95 mypy: Add type to info dictionary. 2017-05-22 11:19:06 -07:00
Carey Metcalfe ab2cfadcaa Fix emoji cache setup when not using provision.py.
Moves creating the emoji folder from the provisioning script to
the build_emoji script.

Fixes the fact that the emoji cache directory wasn't being created
when not using the provision.py script.
2017-05-22 11:08:43 -07:00
Aditya Bansal 9fb6d976ca pep8: Add compliance with rule E261 to test_server.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 46b3c14876 pep8: Add compliance with rule E261 to html_grep.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 47e7437932 pep8: Add compliance with rule E261 to html_branches.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal ffb8440eef pep8: Add compliance with rule E261 to graph.py. 2017-05-07 23:21:50 -07:00