Commit Graph

196 Commits

Author SHA1 Message Date
Tim Abbott 2ae2a94444 provision: Stop using shared var/ for caching apt state.
This didn't work at all when one did a `vagrant destroy` and then
`vagrant up`, because the cache state would be preserved even though
the machine is gone.

Fixes #5981.
2017-10-17 21:15:58 -07:00
derAnfaenger df948c7ea5 Remove useless shebang lines. 2017-10-11 12:52:36 -07:00
rht 34bafa654c mypy: Fix future syntax errors and other minor mistakes.
When we move to the Python 3 mypy syntax, we can't reference a class
before its definition.
2017-10-10 22:27:16 -07:00
rht dcd80e6650 travis/setup-backend: Remove the '--travis' flag in tools/provision.
Whether the env is Travis or not, this has been detected via $TRAVIS env var.
Proof:
6fbf41bdbc/tools/provision (L38)
2017-10-06 11:20:25 -07:00
rht 2c2adbf647 tools/lib: Remove unused imports. 2017-10-02 10:13:46 -07:00
rht f15bdce90e tools: Remove print_function.
Tweaked by tabbott to exclude the linter libraries.
2017-09-29 15:44:56 -07:00
Rhea Parekh 0e5fdcea3f provision: Run apt-get update on catching an error.
For broken apt cache, we run the command apt-get update on catching
the error.
2017-09-29 15:38:23 -07:00
rht bf4eda7374 tools: Remove absolute_import in most tools.
Tweaked by tabbott to not remove it from lister.py, linter_lib, and
friends, since those are intended to support both Python 2 and 3
(we're planning to extract them from the repository).
2017-09-29 12:28:43 -07:00
Eeshan Garg bb34ba37b5 tools: Rename build_pygments_data.py -> build_pygments_data.
Dropping the file extension (.py) conforms to our naming
convention for such scripts.
2017-09-26 16:00:41 -07:00
Umair Khan 5d00f29c68 capitalization: Ignore LDAP. 2017-09-26 15:58:17 -07:00
Tim Abbott 8e2ae66fe6 provision: Update apt operations text. 2017-09-25 14:47:18 -07:00
Tim Abbott 16ed23414d provision: Fix spelling of apt_dependencies_hash.
Also use a variable to avoid duplicating the path.
2017-09-25 14:46:46 -07:00
Eeshan Garg 390a1fec92 zulip_bots: Generate static files during provisioning.
This commit implements support for copying over static files
for all bots in the zulip_bots package to
static/generated/bots/ during provisioning. This directory
isn't tracked by Git. This allows us to have access to files
stored in an arbitrary zulip_bots package directory somewhere
on the system. For now, logo.* and doc.md files are copied over.

This commit should act as a starting point for extending our
macro-based Markdown framework to our bots/API packages'
documentation and eventually rendering these static files
alongside our webhooks' documentation.
2017-09-22 15:14:42 -07:00
Vishnu Ks 61e48734a7 provision: Add moreutils in APT common dependencies.
This fixes an issue where `tools/update-locked-requirements` would fail.
2017-09-22 14:04:57 -07:00
Vishnu Ks f4c69bb126 tools: Create tool for inlining email CSS during provision. 2017-09-16 07:17:48 -07:00
Tim Abbott 126171cb76 api: Remove root_domain_uri from API page links.
These pages should be references the current server's UI anyway.
2017-08-28 14:13:37 -07:00
Tim Abbott a0a1fe1512 settings: Rename SERVER_URI to ROOT_DOMAIN_URI.
This should be a lot less confusing.

See #6013 for discussion.
2017-08-28 14:09:28 -07:00
Tim Abbott 90012f8109 find_add_class: Don't check frontend tests. 2017-08-27 14:04:46 -07:00
Tim Abbott ce1a8bf5b2 find_add_class: Include full path names in error messages. 2017-08-27 14:04:46 -07:00
rht 437b2a3146 Update "MacOS" text to "macOS" 2017-08-26 09:00:42 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Greg Price b7ec1b55dd py3: Cut Python 2 codepaths from provision. 2017-08-16 17:54:43 -07:00
Anirudh Jain 28944b6c94 provision: Partially add zesty to supported systems.
We can't fully support it until we fix the tsearch_extras availability
issue, but for now, this is an improvement.

Tweaked by tabbott to cover the outstanding tsearch_extras issue.
2017-08-15 21:51:19 -07:00
neiljp (Neil Pilgrim) 1a3d8ced78 mypy: Allow Optional parameters in html_branches.py. 2017-08-15 17:50:18 -07:00
Greg Price d3a69da829 py3: Fix or remove (almost) all references to Python 2 venv in docs.
I think soon we'll put the Python 3 venv at `/srv/zulip-venv` and
make `/srv/zulip-py3-venv` just an alias to that (then remove the
alias too), but for now the old name is helpful for spotting places
that need an update.

The `/srv/zulip-venv` name still appears in codepaths used by Travis
tests.
2017-08-08 15:48:12 -07:00
neiljp (Neil Pilgrim) b782db48e1 mypy: Remove superfluous older 'type: ignore' annotations. 2017-08-08 11:27:51 -07:00
neiljp (Neil Pilgrim) 982c0481d0 mypy: Mark log_file as Optional[str] in test_server.py. 2017-08-07 21:27:50 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
rht 532bd6eaa6 Ensure virtualenv is run throughout the entire Travis dev provision. 2017-08-01 15:29:13 -07:00
Jack Zhang 5a8b1e6253 hotspots: Implement hotspots frontend. 2017-07-28 16:34:15 -07:00
Greg Price 7cd621bc9d webhooks/updown: Fix a JsonableError to provide a real error message. 2017-07-24 16:41:22 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Jack Zhang e3ef056991 portico: Update registration form help messages. 2017-07-21 13:09:06 -07:00
Aditya Bansal 7840ae0fe4 template_parser: Add 'with' to Django/Jinja block elements.
Fixes #5863.
2017-07-19 16:39:54 -04:00
Tim Abbott 680381c9d0 setup_venv: Add virtualenv as a dependency for Debian.
Apparently, this was missing on Xenial, and we just never noticed
because the package was installed by default on Ubuntu.

It doesn't exist yet in Trusty.
2017-07-14 17:22:20 -07:00
Tim Abbott 4906a47c1a provision: Disable tsearch_extras for Debian. 2017-07-14 17:22:20 -07:00
Tim Abbott 9efc017e8c provision: Add postgres versions for Debian stretch. 2017-07-14 17:22:20 -07:00
Yago González 2079d84018 lint: Add mobile stores to the capitalization exceptions. 2017-07-14 16:27:17 -07:00
Harshit Bansal ebdd8c9e3d emoji: Switch to using `emoji-datasource` npm package.
This commit does the following things:
* Instead of using a manual tool for downloading sprite sheets, use
  `emoji-datasource` npm package.
* Modify the `build_emoji` script to use sprite sheets from the npm
  package.

Bumps PROVISION_VERSION.

Fixes: #4730.
2017-07-13 21:52:01 -07:00
Harshit Bansal 420327b558 provision: Install NPM packages in the beginning of provisioning process.
NPM packages should be installed at the beginning of the provisioning
process so that later in the provisioning process if a script requires
any NPM package it can use it. Earlier, we were installing NPM packages
in the last as the installation process can fail due to network issues
but since we now retry in case the installation fails, they can be
installed safely at the beginning of the process as well just like apt
packages.
2017-07-13 21:52:01 -07:00
Tim Abbott f2379d66a2 i18n: Fix capitalization of Zulip Botserver download code. 2017-07-07 17:02:39 -07:00
Tim Abbott 919c610abf capitalization: Add APNS to list of valid acronyms. 2017-07-07 10:43:02 -07:00
Tim Abbott 774216713b provision: Document more clearly the is_production_travis block. 2017-07-06 17:39:49 -07:00
sinwar 49db0855d2 provision: Avoid spending ~3s on RabbitMQ in no-op case.
Tweaked by tabbott to reorganize the logic more clearly.

Fixes #5180.
2017-07-06 17:38:39 -07:00
sinwar 3d0b5b14be provision: Avoid spending ~1.5s on compilemessages in no-op case.
This brings the total no-op provision time down to around 7.3s!

Tweaked by tabbott to clean up the code, close file descriptions, and
avoid issues with a somehow corrupted file.

Fixes #5185.
2017-07-01 11:27:54 -07:00
Yago González 7e3154f7bc lint: Ignore tildes in mustache tag template checker.
Handlebars allows putting tildes (~) by the braces in mustache tags to
strip whitespaces on that side of the tag.

    {{#if foo~}}
        <p>Bar</p>
    {{~/if}}

This way, the linter ignores any potential tilde at the ends of Handlebars
tags so we can use this feature without lint fails.
2017-06-26 19:33:25 -04:00
sinwar 0e24930770 provision: Avoid spending ~3s on apt operations in no-op provision.
Fixes #5186
2017-06-22 09:37:40 -04:00
Tommy Ip e9fd41bb66 lint: Check if the first word of a commit message is imperative.
Only check the first word using a black list approach to reduce
force positive as shown in the current implementation.
2017-06-19 07:01:52 -04:00
Aditya Bansal a2b5d6efcb provision: Add wrapper to retry apt-get install if it fails first time. 2017-06-14 21:49:15 -07:00
Aditya Bansal d2f0e17454 linter: Fix bug resulting in <pre> being indented cyclically. 2017-06-13 08:39:59 -04:00
Aditya Bansal a9e0dbe085 pep8: Add compliance with rule E261 pretty_print.py. 2017-06-04 15:05:51 +05:30
Tim Abbott cecf7f1740 provision: Don't regenerate dev database unnecessarily.
This reuses the work we did some time ago to avoid regenerating the
test database unnecessarily.

In addition to being a nice convenience for developers (since any
accomulated test data is still available), this also saves about half
the time consumed in a no-op provision.

Fixes #5182.
2017-06-02 13:27:34 -07:00
Steve Howell 5f48b51b60 provision: Add `cd /srv/zulip` to profile.
I didn't bump the provision version on this change, as it
won't break anything if somebody fails to make this change.
2017-05-29 08:40:49 -07:00
Tim Abbott 9b8641d9eb lint: Add iOS to list of valid weird capitalizations. 2017-05-26 20:59:32 -07:00
Umair Khan ad5780c03e lint: Mark alt argument for translation. 2017-05-26 15:28:46 -07:00
rht a1f82e02d6 python: Replace os.system with subprocess.call.
Generally, we avoid os.system, since it shells out and thus can be
a cause of security issues.
2017-05-26 15:03:16 -07:00
Tim Abbott 8aaf9b1426 Revert "Fix emoji cache setup when not using provision.py."
This reverts commit ab2cfadcaa.

This broke production installation.
2017-05-24 10:35:41 -07:00
Eklavya Sharma 7636972a6c mypy: Wrap return value in Optional. 2017-05-23 21:56:50 -07:00
Eklavya Sharma cc1937c8d5 mypy: Use Optional with strings where required. 2017-05-23 21:56:50 -07:00
Jordan Gedney a9f4d56a95 mypy: Add type to info dictionary. 2017-05-22 11:19:06 -07:00
Carey Metcalfe ab2cfadcaa Fix emoji cache setup when not using provision.py.
Moves creating the emoji folder from the provisioning script to
the build_emoji script.

Fixes the fact that the emoji cache directory wasn't being created
when not using the provision.py script.
2017-05-22 11:08:43 -07:00
Aditya Bansal 9fb6d976ca pep8: Add compliance with rule E261 to test_server.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 46b3c14876 pep8: Add compliance with rule E261 to html_grep.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 47e7437932 pep8: Add compliance with rule E261 to html_branches.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal ffb8440eef pep8: Add compliance with rule E261 to graph.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal c0e1b49b68 pep8: Add compliance with rule E261 to find_add_class.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal fef732d85b pep8: Add compliance with rule E261 to css_parser.py. 2017-05-07 23:21:50 -07:00
Brock Whittaker c8902d5d71 settings: Make settings overlay more responsive.
This doesn't completely fix settings responsiveness, but it's a big
step along the way.  Outstanding issues include:

1. When switching tabs from settings to organization, it will launch
the first item which is more annoying in this view since it brings you
into that tab. Haven’t decided on an elegant solution to this yet.

2. Sidebar scrolling doesn’t work. I have to restructure how the top
section and bottom sections of content are displayed to fix this.
Likely by enforcing min-height of 100% - bottom height on the top piece.

3. Most of it is actually reasonably responsive but some isn’t, and
should be fixed on a case-by-case.
2017-05-05 13:42:09 -07:00
Tommy Ip 19adf2e327 requirements: Upgrade gitlint to 0.8.2.
As of Gitlint v0.8.2, the --extra-path option now accepts both
directories and standalone python modules.
2017-05-05 09:59:39 -07:00
Steve Howell e3edc4d829 Send welcome messages for new streams. 2017-05-01 16:23:38 -07:00
Cory Lynch 0965c43238 Add typeahead for syntax highlighting languages.
Modified composebox_typeahead.js to recognize the triple backtick
and tilde for code blocks, and added appropriate typeahead functions
in that file and in typeahead_helper.js.

Additionally, a new file pygments_data.js contains a dictionary of
the supported languages, mapping to relative popularity
rankings. These rankings determine the order of sort of the
languages in the typeahead.

This JavaScript file is actually in static/generated/pygments_data.js, as it
is generated by a Python script, tools/build_pymgents_data.py. This is
so that if Pygments adds support for new languages, the JavaScript file
will be updated appropriately. This python script uses a set of popularity
rankings defined in lang.json.

Corresponding unit tests were also added.

Fixes #4111.
2017-04-28 17:22:59 -07:00
Neeraj Wahi 90a154e451 Add mobile auth redirect to custom URI scheme (zulip://).
This makes it possible for the Zulip mobile apps to use the normal web
authentication/Oauth flows, so that they can support GitHub, Google,
and other authentication methods we support on the backend, without
needing to write significant custom mobile-app-side code for each
authentication backend.

This PR only provides support for Google auth; a bit more refactoring
would be needed to support this for the GitHub/Social backends.

Modified by tabbott to use the mobile_auth_otp library to protect the
API key.
2017-04-28 11:47:35 -07:00
Steve Howell 7326971380 Extract stream_edit.js.
This code makes the right pane work in "Manage Streams" when
you are editing a stream subscription.  It handles basic
functionality (submitting forms, etc.), live updates, and
showing the pane as needed.

Most of the code here was simply moved from subs.js, but some
functions were pulled out of larger functions:

    live update:
        add_me_to_member_list
        update_stream_name
        update_stream_description

    collapse/show:
        collapse
        show_sub

We also now export subs.show_subs_pane.

We eventually want stream_edit not to call into subs.js, and
this should be fairly easy--we just need to move some shared
methods to a new module.
2017-04-25 09:57:32 -07:00
Tim Abbott 145fa0e5b2 capitalization: Flag Emoji One as allowed. 2017-04-24 23:15:39 -07:00
Tommy Ip 6ec32b37c3 lint: Fix lint rule error due to unordered list.
Tweaked to automatically sort by tabbott.
2017-04-22 11:38:36 -07:00
Tommy Ip 680e39ff55 linter: Add custom gitlint rule to check for imperative mood. 2017-04-21 15:43:19 -07:00
Joshua Pan 64984f5bf4 Rename tools/lint-all to tools/lint.
Fixes #4574.
2017-04-21 14:57:47 -07:00
Harshit Bansal 4470a3947b provision: Download emoji-data during provisioning.
Add code to download iamcal's emoji_pretty.json
(emoji-data/emoji.json) during provisioning.
2017-04-18 12:46:52 -07:00
Yago González e710110a9e i18n: Automatically strip Handlebars strings.
Some Handlebars strings contained whitespaces characters at their ends.
With this, such characters are removed, as well as multiple spaces
(like the ones produced by code indentation).

This also includes a couple of fixes that removes spaces that were
intentionally placed before/after the string to translate.
2017-04-14 17:37:25 -07:00
sinwar 4296d608ab test_server: Improve warning in venv check.
This (1) changes test_server to use the common `check_venv` method and
(2) improves check_venv to provide a clearer error message in the case
that you're inside Vagrant but not in a venv.

Tweaked by tabbott to borrow logic from run_dev.py.
2017-04-05 16:51:23 -07:00
Brock Whittaker 34f9ccb87c alerts: Change sidebar alerts to be at top of the screen.
This changes the alerts to be individual boxes that slide down from
the top of the screen for a better UI experience.
2017-04-03 16:22:05 -07:00
Steve Howell 2d52463b61 Extract ui_report.js.
This moves the implementations of error/report/message from
ui.js to ui_report.js.  They had been shimmed before, so calling
modules still use the same names to call the functions, but we
no longer need the shims.
2017-04-03 07:13:25 -07:00
Steve Howell c0a6038a95 css: Enforce one selector per line.
While it's sometimes nice to put a few selectors on the same line,
it is generally better to have a consistent way of formatting our
selectors, and most of our code up until now lists them vertically.
This change fixes the linter to enforce one selector per line, and
it cleans up the places in the CSS where we had multiple selectors
on the same line.

The advantages of one-per-line are as followers:
    * cleaner diffs
    * easier to see when multiple areas of the app may have the
      same format
    * less likely to go over 80 cols
    * makes it more clear where we have deep nesting in the
      individual selectors
    * makes it easier for our linting tools to enforce
      whitespace violations

This also fixed an old bug where we had ".landing_page h2, h4", which
sets "h4" styles outside of the landing page.
2017-03-26 16:57:33 -07:00
Steve Howell 3d76088f59 tools: Make CSS style more consistent with opening brace.
We now always put the opening brace of a CSS section on
the same line as the last selector (with a space in front).
2017-03-26 16:57:33 -07:00
Tim Abbott c7f1a7aa19 test_script: Use LooseVersion for version conparisons.
Fixes #4292.
2017-03-25 19:49:03 -07:00
wizsid11 95789eb879 Add Slack webhook.
Adds a new webhook integration for Slack to receive messages
from one's Slack team's public channels.
Contains negative tests for broken, missing or invalid data.

Allows two different option for integration:
1. Receive notification on a single stream with different topics
for each of Slack's public channels.
2. Receive notification on different streams for each of Slack's
public channels.

Steps to choose between the two options is described in the documentation.

Fixes #3569.
2017-03-23 23:32:38 -07:00
adnrs96 b3cbb13a79 linter: Add support for automatic checking for 4 space indents in CSS.
In this commit we modify our CSS parser not only to render the text from
a given CSS tokens produced but also enforce 4 space indentation on it.
Also we enforce some basic rules we would like our CSS to follow such as
* Always have "\n" in between the starting of body({) and body itself
  and ending of the body and the closing of body(}).
* Use 4 space indents while having but something within the block
  structure ( { .... } ).
* Have single space after ',' in between multiple selectors.
* Have only a single space in between selector and the starting of
  block structure ({ ... }) if block structure starts on same line as
  of selector.
  eg. body {
          body content here
      }
  Notice single space between 'body' and '{'.

Fixes: #1659.
2017-03-21 13:40:05 -07:00
Steve Howell 36b29a966e Add roadmap feature to js-dep-visualizer.
The js-dep-visualizer tool now attempts to find a set of edges
to remove from a call graph that would reduce it to having only
trivial mutual dependencies, and it produces a roadmap of the
changes that need to happen.

If the tool can't reduce the graph all the way, it still produces
a DOT file that can be visualized.

This fix also has some significant code cleanup.
2017-03-21 07:39:30 -07:00
Tim Abbott 24c2e62ae1 capitalization: Add exception for selected message.
This fixes a test that's been failing overnight.
2017-03-20 07:52:58 -07:00
Steve Howell edb8b0cb5e Improve js-dep-visualizer.
This adds a report of nodes, handles some errors better, adds
some helpful output, cleans up some abspath calls, and
updates which modules and/or dependencies we temporarily are
ignoring for the report.
2017-03-19 21:03:45 -07:00
Tim Abbott 9866124b78 mypy: Fix some new errors flagged by latest mypy master.
Mostly list -> List bugs in annotations.
2017-03-19 21:03:45 -07:00
Steve Howell 623a486c5c tools: Extract is_exterior_node(). 2017-03-17 20:53:27 -07:00
Steve Howell e04a9d38b5 Add tools/lib/graph.py.
This library helps us produce nice graphs for documentation.
2017-03-17 16:00:50 -07:00
adnrs96 dd52291fa5 linter: Add support for automatic checking for 4 space indents in HTML. 2017-03-14 14:50:16 -07:00
Steve Howell 1e5ec689b7 tools/css: Trim whitespace around CSS values. 2017-03-14 09:29:56 -07:00
Steve Howell 2ffea94bd8 tools/css: Require semicolons after CSS declarations. 2017-03-14 09:29:56 -07:00
adnrs96 a75c0dd248 Enhance template parser for comments, Fix deep nesting in Jinja2.
In this commit we add the ability of recognizing comments in
handlebar and Jinja2 templates to the template parser. This
fixes issue of template parser picking up code tags which are
commented out in files.
Also we fix the problem of too much deep nesting in the Jinja2
logic statement. Now only nested Jinja2 logic statements will get
a deeper indentation level.With this another fix was introduced
relating with the tags before a nested handlebar or django tag getting
wrong indent.
We also fix the issue with wrong offsets with closing tags in
cases if they were not matching indent level of the starting
tag intially.
Also we also start to ignore any tags occuring in between 'pre'
tags for deeper indent levels. As well we start to filter out django
non block tags from getting deeper indent level.
2017-03-11 15:16:44 -08:00
Umair Khan 90ee06bd89 capitalization: Make it easier to ignore phrases.
This commit allows us to add the errors shown by the
tools/check-capitalization in the IGNORED_PHRASES list
without any modification.
2017-03-10 11:21:49 -08:00
Umair Khan 83dd901ecf Add capitalization checker tool.
Initial rules significantly by modified by tabbott, who also added the
hacky list of excludes that the tool can't handle correctly yet.

Fixes: #3899.
2017-03-09 00:44:57 -08:00
Rishi Gupta 3797fa657e Change bot domains to string_id.EXTERNAL_HOST.
Change applies to both subdomains and non-subdomains case, though we use
just the EXTERNAL_HOST in the non-subdomains case if there is only 1 realm.

Fixes #3903.
2017-03-08 16:26:13 -08:00