Commit Graph

3211 Commits

Author SHA1 Message Date
Anders Kaseorg b5bfe77844 build-release-tarball: Fix shellcheck warnings.
In tools/build-release-tarball line 50:
for i in `cat "$TMPDIR/$prefix/tools/release-tarball-exclude.txt"`; do
         ^-- SC2013: To read lines rather than words, pipe/redirect to a 'while read' loop.
         ^-- SC2006: Use $(..) instead of legacy `..`.

In tools/build-release-tarball line 51:
    rm -r --interactive=never "$TMPDIR/$prefix/$i";
                              ^-- SC2115: Use "${var:?}" to ensure this never expands to / .

In tools/build-release-tarball line 97:
    echo; echo -ne "\033[33mRunning update-prod-static failed. "
                    ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/build-release-tarball line 98:
    echo -e "Check $TMPDIR/update-prod-static.log for more information.\033[0m"
                                                                       ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg a5280d7d02 build-docs: Fix shellcheck warnings.
In tools/build-docs line 3:
cd "$(dirname "$0")"/../docs
^-- SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Rishi Gupta d76f728dda api docs: Rename webhook-walkthrough to incoming-webhooks-walkthrough. 2018-10-16 21:07:00 -07:00
Eeshan Garg 0292ffe04b webhooks/taiga: Add test for when a task is moved between stories.
This commit adds a test for the payload that is generated when
a Task is moved from one user story to another on Taiga's Sprint
Taskboard UI.

This commit also gets up this webhook's test coverage up to 100%.
2018-10-16 13:06:20 -07:00
Aditya Bansal 75ae94e459 font-awesome: Drop support for legacy font awesome icons.
We drop support for usage of `icon-vector` as base class when
including icons from font awesome icons package.
Now on, only icons as specified in font awesome v4.7.0 can be used
in the code base.
2018-10-15 20:14:55 +05:30
Aditya Bansal 7eeed1413c icons: Remove base class from comments and test templates. 2018-10-15 19:55:18 +05:30
Steve Howell b7f764aa29 settings: Add confirm_dialog module.
This module makes it really easy to create are-you-sure
dialogs for dangerous operations.

Basically it's one function with five parameters.  You
give three chunks of HTML, a callback function, and
a parent container.

The first use of this will be in settings_user_groups,
coming up in a couple commits.
2018-10-12 10:37:06 -07:00
Eeshan Garg ba929508e2 webhooks/ifttt: Get test coverage up to 100%.
IFTTT allows custom templating for their payloads, so the onus is
on the user to ensure that their custom templates conform to the
expectations outlined in our IFTTT webhook docs. For that reason,
these payloads weren't generated, but were manually edited.
2018-10-04 12:16:06 -07:00
Eeshan Garg 316f9aa78b webhooks/hellosign: Rewrite the integration from scratch.
After discovering a couple of bugs, I decided to thoroughly test
and rewrite this integration from scratch. The older code wasn't
generating coherent messages.

This also commit gets this integration up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 5039f6dfb5 webhooks/gitlab: Get test coverage up to 100%.
Test coverage was improved by removing an unused function and
removing some code (written by me) that was actually handling
Test Hook event types incorrectly.
2018-10-04 12:16:06 -07:00
Eeshan Garg 6e2e2b9125 webhooks/github: Test commit status payloads with target_url.
It was a painful amount of work to generate the actual payload.
Since the only difference was a small build URL, I manually
edited the payload and used that for testing.

This commit gets our GitHub webhook up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg b40aec3a09 webhooks/freshdesk: Improve test coverage.
Note that Freshdesk allows custom templating for outgoing payloads
in their webhook UI. Therefore, the payloads added in this commit
did not have to be official payloads from Freshdesk.
2018-10-04 12:16:06 -07:00
Eeshan Garg d8101ca139 webhooks/bitbucket2: Improve test coverage.
The lack of coverage was due to:

* An unused function that was never used anywhere.
* get_commit_status_changed_body was using a regex where it didn't
  really need to use one. And there was an if statement that
  assumed that the payload might NOT contain the URL to the commit.
  However, I checked the payload and there shouldn't be any instances
  where a commit event is generated but there is no URL to the commit.
* get_push_tag_body had an `else` condition that really can't happen
  in any payload. I verified this by checking the BitBucket webhook
  docs.
2018-10-04 12:16:06 -07:00
Eeshan Garg 185a023745 webhooks/beanstalk: Improve test coverage.
We shouldn't just ignore exceptions when encoding the incoming
auth credentials. Even if the incoming credentials are properly
encoded, it is better to know when that is the case or if
something else fails.
2018-10-04 12:16:06 -07:00
Harshit Bansal fc7e654d7b emoji: Remove single-digit names for number emojis.
Names like `:1:`, `:2:` etc don't seem to be of much benefit rather
they seem to get rendered at places not intended.
2018-10-02 11:30:11 -07:00
Vishnu Ks 59865797ac billing: Move billing tests from zilencer to corporate. 2018-10-01 15:44:02 -07:00
Vishnu Ks 93b398bc0a billing: Move zilencer/lib/stripe to corporate. 2018-10-01 15:43:44 -07:00
Vishnu Ks daf3b46262 billing: Move billing related templates from zilencer to corporate. 2018-10-01 15:43:39 -07:00
Vishnu Ks 6914ee126c billing: Move billing-related views and urls to corporate. 2018-10-01 15:43:21 -07:00
Tim Abbott 00821d67bb travis: Remove no-longer-present packages from apt-mark hold list.
Apparently, Travis removed the Heroku bundle of packages from their
servers, which made the build start failing when trying to configure
apt to hold their versions (sigh).  This commit removes the
problematic packages.
2018-09-28 14:20:04 -07:00
Greg Price 3b646c5d28 push-to-pull-request: Update local tracking ref, if any.
The companion tool `tools/reset-to-pull-request` has a handy feature
to maintain a local ref tracking the PR: e.g., pr/1234 for PR 1234.
If this were a normal remote-tracking branch maintained by `git fetch`,
it'd get updated on `git push`.  Do the same thing here.

This helps keep a view like `gitk --all @` a bit tidier, by causing
merged PRs to stop pointing at side branches of the main history.
2018-09-25 13:15:02 -07:00
Vishnu Ks 5a6b2ebb1f billing: Add ability for users to change their card. 2018-09-24 19:25:26 -07:00
Anders Kaseorg 5e88b3013a zanitizer_config.pm.sample: keep_file → scrub_filename
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-09-23 20:42:27 -07:00
Eeshan Garg ecd4f821be user docs: Automatically activate correct tab for OS-specific instructions. 2018-09-18 13:49:34 -07:00
Vishnu Ks d6b548bd57 styles: Extract billing SCSS into its own stylesheet. 2018-09-11 00:08:38 -07:00
Steve Howell 06225d1424 tests: Clean up calls to tools/webpack.
Before this change, the way we loaded
webpack for various tools was brittle.

First, I addressed test-api and test-help-documentation.

These tools used to be unable to run standalone on a
clean provision, because they were (indirectly)
calling tools/webpack without the `--test` option.

The problem was a bit obscure, since running things
like `./tools/test-backend` or `./tools/test-all` in
your workflow would create `./var/webpack-stats-test.json`
for the broken tools (and then they would work).

The tools themselves weren't broken; they were the
only relying on the common `test_server_running` helper.
And even that helper wasn't broken; it was just that
`run-dev.py` wasn't respecting the `--test` option.

So I made it so that `./tools/run-dev` passes in `--test` to
`./tools/webpack`.

To confuse matters even more, for some reason Casper
uses `./webpack-stats-production.json` via various
hacks for its webpack configuration, so when I fixed
the other tests, it broke Casper.

Here is the Casper-related hack in zproject/test_settings.py,
which was in place before my change and remains
after it:

    if CASPER_TESTS:
        WEBPACK_FILE = 'webpack-stats-production.json'
    else:
        WEBPACK_FILE = os.path.join('var', 'webpack-stats-test.json')

I added similar logic in tools/webpack:

    if "CASPER_TESTS" in os.environ:
        build_for_prod_or_casper(args.quiet)

I also made the helper functions in `./tools/webpack` have
nicer names.

So, now tools should all be able to run standalone and not
rely on previous tools creating webpack stats files for
them and leaving them in the file system.  That's good.

Things are still a bit janky, though.  It's not completely
clear to me why `test-js-with-casper` should work off of
a different webpack configuration than the other tests.

For now most of the jankiness is around Casper, and we have
hacks in two different places, `zproject/test_settings.py` and
`tools/webpack` to force it to use the production stats
file instead of the "test" one, even though Casper uses
test-like settings for other things like which database
you're using.
2018-09-07 11:39:55 -04:00
Steve Howell 25ad755914 Fix usage for test-api. 2018-09-07 11:39:55 -04:00
Steve Howell 336c2cee20 Revert unused code for creating non-editable pills.
We don't use input.create_non_editable_pill() in our
code yet.  If we add this back, we'll want to have node
tests on it.

Removing this unused code brings us to 100% line
coverage for input_pill.js.

This directly reverts 5c11ab85 with the small addition
of adding input_pill to our list of fully covered
modules.
2018-08-31 16:26:05 -07:00
Harshit Bansal 88bfb9778b build_emoji: Add infrastructure for google-blob and twitter emojisets. 2018-08-31 11:52:24 -07:00
Marco Burstein 25e624eab1 team: Include contributions from legacy repositories in the total.
These repositories (`zulip-ios-legacy` and `zulip-android`) are
deprecated, and as such should not have their own tabs, but still
should be included in the total contributions count.
2018-08-28 16:45:00 -07:00
Tim Abbott 2c7f9ce0fc puppet: Fix puppet-lint warnings in various manifests.
Apparently, `puppet-lint` on Ubuntu trusty throws warnings for certain
quoting patterns that are OK in modern `puppet-lint`.  I believe the
old Zulip code was actually correct (i.e. the old `puppet-lint`
implementation was the problem), but it seems worth changing anyway to
suppress the warnings.

We also exclude more of puppet-apt from linting, since it's
third-party code.
2018-08-28 13:46:31 -07:00
Lyla Fischer 1efcdfdb5c templates: Merge Help and API doc main into documentation_main.
With minor fixes by eeshangarg.
2018-08-27 20:19:24 -02:30
Harshit Bansal e204fe3948 minor: Fix for failing tests. 2018-08-27 06:16:16 +00:00
Harshit Bansal d96d6a96f7 build_emoji: Add a missing check for image existence. 2018-08-26 22:16:23 -07:00
Harshit Bansal f5454dff51 build_emoji: Automatically calculate sprite dimensions.
Instead of using a hardcoded value for spritesheet dimensions,
automatically calculate it using `emoji_data`. This will free
us from updating it only emoji datasource update as well as
allow us to add google blob emojiset.
2018-08-26 22:16:23 -07:00
Harshit Bansal dbc3774c2f build_emoji: Refactor `get_emoji_code()` to not fail for old datasources.
If `non_qualified` field is not present then return the value of
`unified` field instead of failing.
2018-08-26 22:16:23 -07:00
Harshit Bansal ec2ebd1cc9 build_emoji: Change sprite CSS files to use hyphens in names.
This is more consistent with our other names in emoji infra.
2018-08-26 22:16:23 -07:00
Harshit Bansal 9bd56d0ce4 build_emoji: Change emoji sprite sheets to use hyphens in names.
Use of hyphens is more consistent with our other emoji infra names.
2018-08-26 22:16:23 -07:00
Harshit Bansal aa6f098496 build_emoji: Extract `setup_emoji_farm()`. 2018-08-26 22:16:23 -07:00
Harshit Bansal dc9ff928f3 build_emoji: Refactor `ensure_emoji_image()` to take paths as args. 2018-08-26 22:16:23 -07:00
Harshit Bansal ed694001da build_emoji: Rename `setup_emoji_farm()` to `setup_emoji_farms()`. 2018-08-26 22:16:23 -07:00
Steve Howell ca272ef49d tests: Test how we handle extra data for zform messages. 2018-08-24 10:00:25 -07:00
Steve Howell bee592b44e node tests: Bring list_cursor to 100% coverage.
We had some indirect coverage of this already, so
this mostly focuses on error cases and corner
cases.
2018-08-24 10:00:04 -07:00
Steve Howell 44756a326d tests: Test duplicate default stream/groups.
This also gets us to 100% line coverage
for actions.py.
2018-08-22 16:52:30 -07:00
Steve Howell 4318f75718 Remove `code` from JsonableError.__init__.
Since this class was built, folks have always chosen
to subclass JsonableError for situations where
the default of ErrorCode.BAD_REQUEST is insufficient.

So now we simplify the use cases, which also gets
us 100% coverage on this core module.
2018-08-22 16:51:40 -07:00
Tim Abbott b93d5bd5e8 test-js-with-node: Temporarily remove server_events_dispatch.js.
This fixes master failing tests since 2aeabf24a6.
2018-08-21 14:40:05 -07:00
Joshua Pan 793ec9e266 tests: Test all update_display_settings events. 2018-08-19 14:51:31 -04:00
xificurC 9e053c74cf scripts: Allow configuring a custom CA bundle for build process.
For building Zulip in an environment where a custom CA certificate is
required to access the public Internet, one needs to be able to
specify that CA certificate for all network access done by the Zulip
installer/build process.  This change allows configuring that via the
environment.
2018-08-13 15:45:28 -07:00
Tim Abbott 90828297e4 puppet-lint: Enforce double_quoted_strings check.
This makes our puppet codebase more consistent by using single-quoted
strings consistently.
2018-08-13 12:31:19 -07:00
Tim Abbott d0b51b70f4 puppet-lint: Enforce 2sp_soft_tables puppet-lint check.
This cleans up the puppet codebase's whitespace formatting to be more
consistent.
2018-08-13 12:31:16 -07:00
Tim Abbott b26e0a957d puppet-lint: Enforce arrow_alignment check.
This fixes all exceptions in our puppet codebase to this lint rule.
2018-08-13 12:30:57 -07:00
Aditya Bansal a3b8ab46b3 puppet-lint: Include in linter tool chain to run by default.
We start to use puppet-lint to lint puppet modules by default by
adding it to tools/lint (which controls our linter tool chain).

We also define a few puppet-lint rules to exclude.

Fixes: #9185.
2018-08-13 12:30:57 -07:00
Aditya Bansal 56d49980de puppet-lint: Add as a dependancy to install during provision process. 2018-08-13 12:30:57 -07:00
Harshit Bansal 29daf76420 emoji: Don't generate emoji farms corresponding to now banned emojisets.
This will help us in reducing the size of the release tarball
significantly. I have refrained from changing the `EMOJISETS`
constant in the `emoji_setup_utils.py` as that controls the
emojisets that we want to support. Since we want to re-enable
the feature of changing emojisets sometime again in the future
that variable should be kept as it is as it controls several
other things like emoji scripts that we use to generate emoji
names. Changing it might cause hard to catch bugs.
2018-08-13 12:09:06 -07:00
Harshit Bansal 36b977c382 emoji: Refactor `generate_sprite_css_files()` to take emojiset in args. 2018-08-13 12:09:06 -07:00
Harshit Bansal 67bbbbb007 emoji: Move logic for hard-coding octopus emoji.
This logic was incorrectly placed in `setup_old_emoji_farm()` while
it should have been placed in `setup_emoji_farm()`.
2018-08-13 12:09:06 -07:00
Harshit Bansal 1eb3825048 emoji: Remove now-unnecessary `get_remapped_emojis_map()`. 2018-08-13 12:08:14 -07:00
Harshit Bansal 99e1a81055 emoji: Fix some emoji images not loading in missed message emails.
`emoji-datasource` package v4.0.4 introduced the concept of qualified
and non-qualified emoji codes. As chat programs don't need to use
emoji representation selector, so we used migrated our infrastructure
to use non-qualified emoji codes. But we missed the fact that the
emoji file names in emoji farm are based on emoji data's 'unified'
field and the value of this field has changed. Consequently the image
file names must also have been changed. We used `emoji_code` while
converting the span tags to img tags while processing notifications.
But since now `emoji_code` refers to non-qualified code while image
file names are based on qualified code, we need to rename images
to correctly do the conversion. This commit just fixes this.
2018-08-13 12:03:16 -07:00
Steve Howell 413a0174f4 Extract a zephyr.py library.
Right now it only has one function, but the function
we removed never really belonged in actions.py, and
now we have better test coverage on actions.py, which
is an important module to get to 100%.
2018-08-11 14:51:26 -07:00
Max Nussenbaum 570b9515c6 billing: Add location for errors on billing page.
This adds a location for error messages to the billing page,
using the standard error styling.
2018-08-10 22:48:10 -07:00
Vishnu Ks 0c84260faa billing: Enforce full coverage in zilencer/tests. 2018-08-10 13:52:35 -07:00
Vishnu Ks 9946202148 coverage: Enforce full coverage for zilencer/lib/stripe.py. 2018-08-10 13:52:35 -07:00
Tim Abbott aaedec1fdb tools: Move a few i18n scripts to tools/i18n.
These are likely to only be run by Zulip maintainers.
2018-08-09 15:46:27 -07:00
Tim Abbott ba626dcad6 tools: Move optimize-svg to tools/setup/. 2018-08-09 15:39:57 -07:00
Tim Abbott c2e5cc99a4 tools: Move generate-custom-icon-webfont to a subdirectory. 2018-08-09 15:39:57 -07:00
Tim Abbott 2bb04fc720 tools: Remove send-github-payloads.
This ancient tool predates our practice of collecting test fixtures
for third-party integrations, which is a better general system for the
problem this solved.
2018-08-09 15:39:57 -07:00
Tim Abbott 74dc3e9a3e tools: Remove unused munge-postgres-logs.
This hasn't been used in several years.
2018-08-09 15:39:57 -07:00
Tim Abbott bd945333a3 tools: Remove replace-tarball-shebang.
This was a temporary helper tool for part of
tools/build-release-tarball during our Python 3 migration, which
finished some time ago.
2018-08-09 15:27:09 -07:00
Tim Abbott 78539e7819 tools: Remove unused deployment-lock-ctl.
This interacted with a deployment locking system that is no longer
part of the project.
2018-08-09 15:25:15 -07:00
Rhea Parekh 26300110ca import: Fix rendered_content in imported messages.
After the messages have been imported, set the rendered_content of the
messages instead of leaving its value to be 'None'.

This is important to ensure that:
(1) Performance for users is good after completing the import.
(2) The database's full-text indexes have all of the imported messages
(which only happens properly when Message rows have their
rendered_content field edited).

Fixes #9168.
2018-08-09 15:12:53 -07:00
Yago González f6219745de users: Get all API keys via wrapper method.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.

Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.

This commit is the first step towards allowing users have multiple API
keys.
2018-08-08 16:35:17 -07:00
Yago González a0d8cbc9cb test-backend: Fix typo. 2018-08-08 16:34:29 -07:00
Cynthia Lin ecbc58c586 settings: Fix color styling for nested inputs to work in night mode. 2018-08-08 11:29:48 -07:00
Rhea Parekh 30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh 1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00
Vishnu Ks 119d108da0 droplets: Update code and remove nvm settings from bashrc.
* Synced zulip repo to master and provisioned
* Synced python-zulip-api repo to master and provisioned.
* Removed unnecessary nvm config from bashrc
2018-08-07 13:38:18 -07:00
Tim Abbott 4300683bca lint: Fix mypy error in check_pyflakes annotations.
Introduced in 78a93b8d9e.
2018-08-07 09:59:14 -07:00
Tim Abbott e67fa36803 lint: Update config to split files more evenly. 2018-08-07 09:42:37 -07:00
Tim Abbott 78a93b8d9e lint: Refactor pyflakes to use more consistent style.
In particular, we no longer pass in the full `by_lang` object and
expect it to pull out the Python piece.
2018-08-07 09:42:37 -07:00
Tim Abbott 458c17868f zulint: Move pycodestyle runner code to library.
This way, only the Zulip project's specific configuration ends up
living in pep8.py.
2018-08-04 19:54:18 -07:00
Tim Abbott 3b53ddf779 zulint: Stop importing color codes from zulip_tools.py.
This is I think the final dependency that zulint/ has on the rest of
the project.
2018-08-04 19:53:53 -07:00
Tim Abbott c0beeebc3e zulint: Move printer.py module to tools/zulint.
At this point, tools/linter_lib is intended to be the directory of
Zulip-specific code.
2018-08-04 19:53:53 -07:00
Tim Abbott 55ba434ecb zulint: Extract LinterConfig class to zulint. 2018-08-04 19:53:53 -07:00
Tim Abbott f74bb8a9b2 zulint: Extract run_parallel and do_lint. 2018-08-04 19:53:53 -07:00
Tim Abbott acf8ec492d zulint: Extract common linter argument parsing logic.
The remaining code is Zulip-specific business logic.
2018-08-04 19:53:53 -07:00
Tim Abbott 1fcc2a6ea4 zulint: Move lister.py to tools/zulint.
This is preparatory refactoring work for being able to extract the
linter as an external project called "zulint".
2018-08-04 19:53:53 -07:00
Joshua Pan 5303281dfb tests: Fully cover hash_util.js. 2018-08-04 17:56:02 -07:00
Abhilash Verma fd9652a9d7 settings_emoji: Make strings more user-friendly. 2018-08-04 09:37:48 -07:00
Steve Howell c8898e1dc8 refactor: Move by_conversation_and_time_uri to hash_util.
This removes the 100% coverage on hash_util, but we are
pretty careful about not messing with this code.
2018-08-04 09:32:27 -07:00
Anders Kaseorg d79555e9be update-locked-requirements: Fix shellcheck warnings.
In tools/update-locked-requirements line 66:
compile_requirements requirements/prod.in $OUTPUT_BASE_DIR/prod.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 67:
compile_requirements requirements/dev.in $OUTPUT_BASE_DIR/dev.txt
                                         ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 68:
compile_requirements requirements/mypy.in $OUTPUT_BASE_DIR/mypy.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 69:
compile_requirements requirements/docs.in $OUTPUT_BASE_DIR/docs.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 70:
compile_requirements requirements/thumbor.in $OUTPUT_BASE_DIR/thumbor.txt py2
                                             ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg 3e7bf9be9c production-helper: Fix shellcheck warnings.
In tools/travis/production-helper line 24:
if ! apt-get dist-upgrade -y $APT_OPTIONS; then
                             ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/travis/production-helper line 26:
    apt-get dist-upgrade -y $APT_OPTIONS
                            ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e05a56aa8b test-migrations: Fix shellcheck warnings.
In tools/test-migrations line 18:
    echo "$new_auto_named_migrations" | sed 's/\[[x ]\] /  /'
    ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/test-migrations line 27:
    echo 'ERROR: Migrations are not consistent with models!  Fix with `./tools/renumber-migrations`.'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c4ed4bc021 test-install: Fix shellcheck warnings.
In tools/test-install/destroy-all line 31:
  | while read c
          ^-- SC2162: read without -r will mangle backslashes.

In tools/test-install/install line 57:
    installer_dir="$(readlink -f $INSTALLER)"
                                 ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/test-install/lxc-wait line 30:
for i in {1..60}; do
^-- SC2034: i appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e1d770d451 test-documentation: Fix shellcheck warnings.
In tools/test-documentation line 6:
    echo -e "\e[${color_code}m${message}\e[0m" >&2
             ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                        ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".

In tools/test-documentation line 41:
scrapy crawl_with_status documentation_crawler $loglevel
                                               ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg bb51503907 test-all-docker: Fix shellcheck warnings.
In tools/test-all-docker line 7:
source /home/zulip/.bash_profile
^-- SC1091: Not following: /home/zulip/.bash_profile: openBinaryFile: does not exist (No such file or directory)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c93f53b463 test-all: Fix shellcheck warnings.
In tools/test-all line 7:
TEMP=`getopt -o f --long force -- "$@"`
     ^-- SC2006: Use $(..) instead of legacy `..`.

In tools/test-all line 24:
    echo "Running $@"
                  ^-- SC2145: Argument mixes string and array. Use * or separate argument.

In tools/test-all line 26:
        printf "\n\e[31;1mFAILED\e[0m $@\n"
               ^-- SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
                ^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\\n".
                  ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                      ^-- SC2145: Argument mixes string and array. Use * or separate argument.
                                        ^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\\n".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e9656564c2 start-dockers: Fix shellcheck warnings.
In tools/start-dockers line 7:
source /home/zulip/.bash_profile
^-- SC1091: Not following: /home/zulip/.bash_profile: openBinaryFile: does not exist (No such file or directory)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg f4c4e857ff generate-test-credentials: Fix shellcheck warnings.
In tools/setup/generate-test-credentials line 10:
password=$(./manage.py print_initial_password "$email" | fgrep "$email" | awk '{ print $2 }')
                                                         ^-- SC2197: fgrep is non-standard and deprecated. Use grep -F instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg af64bff9d4 reset-to-pull-request: Fix shellcheck warnings.
In tools/reset-to-pull-request line 25:
    git fetch "$remote" +"pull/$request_id/head":"$target_ref"
                                                ^-- SC2140: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c669626666 provision: Fix shellcheck warnings.
In tools/provision line 13:
FAIL="\033[91m"
      ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 14:
WARNING="\033[93m"
         ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 15:
ENDC="\033[0m"
      ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 19:
PARENT_PATH=$( cd "$(dirname "${BASH_SOURCE}")" ; pwd -P )
                              ^-- SC2128: Expanding an array without an index only gives the first element.

In tools/provision line 32:
if [ $failed = 1 ]; then
     ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/provision line 49:
    echo 'or just close this shell and start a new one (with Vagrant, `vagrant ssh`).'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg d7c7e8dd7c optimize-svg: Fix shellcheck warnings.
In tools/optimize-svg line 3:
if [ `node_modules/.bin/svgo -f static/images/integrations/logos | grep -o '\.[0-9]% = ' | wc -l` -ge 1 ]
     ^-- SC2046: Quote this to prevent word splitting.
     ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg cd451491ff find-unused-css: Fix shellcheck warnings.
In tools/find-unused-css line 5:
    if [ $(git grep "$n" | grep -v '^static/styles/zulip.css' | wc -l) -eq 0 ]; then
         ^-- SC2046: Quote this to prevent word splitting.
                           ^-- SC2126: Consider using grep -c instead of grep|wc -l.

In tools/find-unused-css line 6:
        echo $n
             ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg b920d19358 fetch-rebase-pull-request: Fix shellcheck warnings.
In tools/fetch-rebase-pull-request line 15:
git checkout -B "review-${request_id}" $remote/master
                                       ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg e7e25fb70b do-destroy-rebuild-test-database: Fix shellcheck warnings.
In tools/do-destroy-rebuild-test-database line 6:
"`dirname "$0"`/../tools/setup/generate-fixtures" --force
 ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Rhea Parekh ee37866687 import: Add gitter import file in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Rhea Parekh b8e1e8b31d import: Add slack import files in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Anders Kaseorg de5e3d3fbd tools/review: Don’t pretend to emulate shell=True either.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Anders Kaseorg 9a99d76296 tools: Remove pointless uses of shell=True.
Which, not coincidentally, is all of them.  Don’t use shell=True.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:06 -07:00
Steve Howell 95d136ca5e Extract search_util.js module.
We probably should have done this a while ago, even
though these functions are pretty tiny.  The goal here
is to make it easier to have more consistent search
semantics.

Our first use case is subs.js.  In this case we
are able to decouple a bit of generic string
matching from the subs-specific code.
2018-07-30 11:25:32 -07:00
Greg Price 4ba8015002 tools/reset...: Add experimental option to record the PR as a ref.
I often find myself looking manually through the reflog of `master` to
find a commit I previously reset to with tools/reset-to-pull-request .
Sometimes I want to see a previous version of a PR I'm reviewing a
revised version of; sometimes to look at two related PRs together.

So, here's a feature to automate that by saving each PR branch in its
own ref, with a name like `refs/remotes/pr/1234` -- or `pr/1234`, as
you'd normally refer to it.

To enable this, set the new config option:
  $ git config zulip.prPseudoRemote pr
(Or you can pick another name.)

The reason I hesitate to just make this the behavior for everyone
immediately is that the resulting `pr/1234` refs will naturally
accumulate and may clutter up the view -- and because with the
`refs/remotes/` style of name I've chosen, it requires a bit of
Git plumbing to clean them up.  (Use `git update-ref -d`.)
I'll play with it and iterate; comments welcome from other willing
early adopters.
2018-07-27 23:47:32 -07:00
Harshit Bansal cf5b2b4815 emoji: Change emoticon mapping for `:)`, `(:` and `:(`.
See discussion on CZO:
https://chat.zulip.org/#narrow/stream/101-design/subject/emoji.20picker/near/617811
2018-07-26 11:17:03 -07:00
Anders Kaseorg 19f37fb3d4 tools/test-queue-worker-reload: Avoid shelling out for touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg d3f8f74ac4 tools/test-locked-requirements: Avoid shelling out for cp, mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg dfed7da911 tools/lib/provision.py: Avoid shelling out for mkdir, touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Aditya Bansal 0042694e24 stylelint: Start using stylelint to lint our css.
Tweaked by tabbott to document it.

Fixes #8894.
2018-07-25 15:03:35 -07:00
Priyank Patel 8a012b9e7f Revert "webpack: Combine archive-style bundle into archive."
This reverts commit d1d9d63841, and fixes
the fonts in webapp being overwritten by bootstarap due to webpack bundling.
2018-07-24 16:00:26 -07:00
Greg Price 79dfe6e0ae push-to-pull-request: Fix shell usage in an error path.
Errors go to stderr, not stdout.

While we're here, match the message itself to the others in
this script.
2018-07-23 23:19:27 -07:00
Greg Price 26d0904436 push-to-pull-request: Give a nice error if `jq` missing. 2018-07-23 23:19:27 -07:00
Priyank Patel d1d9d63841 webpack: Combine archive-style bundle into archive. 2018-07-23 15:19:47 -07:00
Priyank Patel f8398238a9 webpack: Migrate archive bundle to webpack.
Tested by visiting localhost:9991/archive/streams/1/topics/denmark3.
2018-07-23 15:19:44 -07:00
Harshit Bansal c0b0fb7cce emoji: Move `EMOTICON_CONVERSIONS` mapping to build_emoji infra.
This commit closes a long pending issue which involved moving the
`EMOTICON_CONVERSION` mapping to build_emoji infrastructure so
that there is only one source of truth. This was pending from the
time when this feature was implemented.
2018-07-23 12:35:08 -07:00
Joshua Pan 565fd75661 tests: Test closing alert word status msg. 2018-07-23 14:53:49 -04:00
Tim Abbott 2918cc92f3 search: Duplicate HTML template content for search area.
This is preparation for being able to work on the search pills feature
without making any user-facing changes until we're ready to enable it.
2018-07-23 11:29:10 -07:00
Rhea Parekh 3a6921e54b gitter import: Add gitter data conversion script. 2018-07-23 08:36:30 -07:00
Harshit Bansal bf70955c0c emoji: Update `emoji-datasource` packages.
This commit updates the `emoji-datasource` packages to version 4.0.4.
This update brings following changes to emoji infra:

1: Fix for the bleeding sprite sheets.

2: The category of some emojis has been changed. Categorywise breakup of
   net gain or loss is as follows:
    Travel & Places: 58 (gain)
    Symbols: 47 (loss)
    Smileys & People: 52 (gain)
    Objects: 11 (loss)
    Food & Drink: 3 (gain)
    Animals and Nature: 46 (gain)
    Activities: 9 (loss)

3: There were some changes in the image farm of the package which were
   breaking our old emoji farm. I fixed them by modifying the remapped
   emoji map.

Fixes: #8235.
2018-07-23 08:04:58 -07:00
Harshit Bansal 6c28a60db0 emoji: Hardcode octopus emoji.
Google emojiset's octopus is really cute and whole Zulip community
loves it. So using a CSS hack, we hardcode octopus emoji to use image
from Google's emojiset only irrespective of the choosen emojiset.
2018-07-23 08:04:58 -07:00
Harshit Bansal 3f73826ffa emoji: Extract `get_emoji_code()`. 2018-07-23 08:04:58 -07:00
Harshit Bansal 69eaa2de67 emoji: Move `remapped_emojis` list to emoji_setup_utils.py. 2018-07-23 08:04:58 -07:00
Joshua Pan 015bc8b01e tests: Reach 100% coverage for keydown_util.
This commit pretty much just gets line coverage.
2018-07-23 08:42:11 -04:00
Joshua Pan 0596632457 lint: Exclude frontend_tests/node_tests from $().text rule. 2018-07-20 12:13:56 -07:00
Steve Howell b9c9dd9b5a lint: Turn off the CSS checker (officially).
Our CSS checker globs for .css files.  Since the
SCSS cutover, it has been a no-op, so there's no
sense launching it.  See #8894 for details on
future plans.
2018-07-20 10:03:28 -04:00
Anders Kaseorg 8698198452 tools/update-authors-json: Avoid shelling out for cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 6a8f0165ee tools/test-help-documentation: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg f01942e510 tools/test-emoji-name-scripts: Avoid shelling out for cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 07f5b81406 tools/minify-js: Avoid shelling out for mkdir, cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 2d12b5b3d9 tools/diagnose: Avoid shelling out for pwd.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Tim Abbott 811ac718d6 build_emoji: Document a quirk of generate_sprite_css_files. 2018-07-13 21:19:26 +05:30
Harshit Bansal ec1ce08523 emoji_setup_utils: Migrate to python3 type annotations. 2018-07-13 21:19:26 +05:30
Harshit Bansal f6fe5d6266 build_emoji: Migrate to python3 type annotations. 2018-07-13 21:19:25 +05:30
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Tim Abbott 7a86e83a6f build_emoji: Use with for writing to files.
This is a nice code cleanup.
2018-07-13 21:07:08 +05:30
Tim Abbott 85243c525d build_emoji: Pass around emoji_map data structure less.
It wasn't used in either of these functions.
2018-07-13 21:04:37 +05:30
Harshit Bansal b9f974ba61 tools: Add a tool to validate emoji name listing scripts.
This tools emoji name import/export tooling by verifying that no
information gets modified in a round-trip conversion.
2018-07-13 20:53:25 +05:30
Harshit Bansal 628e868d1e tools: Add `export_emoji_names_to_csv`.
This tool is used for exporting `emoji_names.py` to a csv file.
2018-07-13 20:53:08 +05:30
Harshit Bansal 05f85eb94d emoji: Add emoji_names.py file.
Credits to @rishig, Alice Lai, and @rntharu for naming all the emoji.

Names are inspired by iamcal, gemoji, and unicode names, sources like
emojipedia and iemoji, google search results for articles about emoji,
and emoji usage on twitter.
2018-07-13 20:53:07 +05:30
Harshit Bansal ae17a15b29 tools: Add generate_emoji_names_table tool.
This tool generates a listing of the emojis as per the current cache
in use.
2018-07-13 20:53:07 +05:30
Harshit Bansal fef5b43e22 minor: Fix directory name label in 'import_emoji_names_from_csv'. 2018-07-13 20:53:02 +05:30
Max Nussenbaum 8fd900626d billing: Style billing page.
This adds styles to /billing.
2018-07-13 18:30:38 +05:30
Tim Abbott 3cc93fd721 setup_venv: Fix missing libssl-dev dependency.
We were already correctly including libssl-dev in Zulip's dependencies
in development environment provisioning, but (at least now) it's
needed to build certain Python packages like pycurl when building a
Zulip virtualenv in production.  I haven't investigated why we didn't
need this on Ubuntu, but one possible reason would be that some other
library in our dependencies list happens to depend on it on Ubuntu.

We fix this by moving the dependency over to the shared
VENV_DEPENDENCIES list.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Harshit Bansal 8ada7cfe5b tools: Add import_emoji_names_from_csv tool. 2018-07-12 18:05:02 +05:30
Greg Price 36cf898589 mypy: Add a place to put type stubs.
This will allow us to begin to add our own stubs for external
libraries.  Writing stubs can be surprisingly little work to do, and
can have high leverage in keeping our type annotations high-quality.
2018-07-12 14:10:09 +05:30
Yago González a3d42d9901 test-api: Pass non-admin client to the test helpers. 2018-07-11 23:54:00 +05:30
Cynthia Lin 7d9281fa6c simplebar: Add simplebar dependency for help page scrolling. 2018-07-11 20:04:55 +05:30
Puneeth Chaganti 3dace27ed6 tools: Only files starting with same migration number conflict. 2018-07-10 21:09:34 +05:30
Shayan Toqraee 0757d022f5 messages: Add support for right-to-left messages.
This implements right-to-left message automatic detection support in
the compose box as well as the message feed.  Full unit tests and
support in the message-editing UI are for future work (as are
potentially more fancy things like supporting things like
right-to-left multi-word names for users/streams/etc.).

Fixes #3123.
2018-07-10 10:47:56 +05:30
Rishi Gupta 16334a1ba7 billing: Update billing system. 2018-07-09 14:33:07 +05:30
Aditya Bansal b6ba162769 generate-fixtures: Extract zulip_test_template creation as a function.
This will be helpful in the upcoming changes which will make use
of this extracted function to re-create zulip_test_template after
migrating zulip_test db so that we have latest schema in tests.
2018-07-09 09:11:48 +05:30
Armaan Ahluwalia ba6543b850 webpack: Fix missing exposed modules left out of transition.
This commit fixes some modules that were erroneously left out while
transitioning app.js to webpack. This commit exposes them using
expose-loader or setting them directly to window.
2018-07-07 08:00:34 +02:00
Shubham Dhama ced6f967bc voting_widget: Increase node test coverage to 100%. 2018-07-06 11:30:12 -04:00
Tim Abbott 0bd139af0c webpack: Fix missing expose for toMarkdown. 2018-07-05 13:24:19 +02:00
Armaan Ahluwalia 54d3d8e8b3 webpack: Transition app.js to be compiled by webpack.
This commit moves all files previously under the 'app' bundle in
the Django pipeline to being compiled by webpack under the 'app'
entry point. In the process, it moves assets under the app entry
to a file called app.js that consumes all relevant css and js files.

This commit also edits the webpack config to be able to expose certain
variables for third party libraries that are currently required by
some modules. This is bad coding form and should be refactored to
requiring whatever dependencies a module may have; we're just
deferring that to the future to simplify the series of transitions we
need to do here. The variable exposure is done using expose-loader in
webpack.

The app/index.html template is edited to override the newly introduced
'commonjs' block in the base template. This is done as a temporary
measure so as not to disrupt other pages on the app during the transition.

It also fixes the value of the 'this' context that was being inferred
as window by third party libraries. This is done using imports-loader
in the webpack config.  This is also messy and probably isn't how we
want things to work long term.
2018-07-05 11:03:08 +02:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Shubham Dhama b938523592 widgets: Increase node tests coverage to 100%. 2018-07-03 08:48:49 -04:00
Yashashvi Dave 2b478268f3 third: Add sortablejs library. 2018-07-01 02:05:00 -07:00
Vishnu Ks e1bf749d3c provision: Run build_pygments_data only if required. 2018-06-23 00:15:39 +05:30
Vishnu Ks 109fa85614 provision: Rename file_hash_updated to file_or_package_hash_updated.
Check for changes in package version as well along
with the files.
2018-06-22 23:40:31 +05:30
Vishnu Ks 431d9d6076 provision: Run generate-custom-icon-webfont only if required. 2018-06-22 23:40:31 +05:30
Shubham Padia 6853530eba search: Add search_pill.js with basic operations.
Adds search_pill.js to the static asset pipeline. The items
for search pill contain 2 keys, display_value and search_string.
Adding all the operator information i.e the operator, operand and
negated fields along with the search_string and description was tried out.
It was dropped because it didn't provide any advantage as one had to
always calculate the search_string and the description from the operator.
2018-06-22 09:05:36 -04:00
Shubham Dhama c167227ddc search: Add 100% coverage for search.js. 2018-06-22 07:58:06 -04:00
Greg Price 0462e85d02 tools: Make remote configurable in {reset,push}-to-pull-request.
The appropriate name for the remote pointing at the repo we maintain
may be `upstream` for most of our repos... but not when we're
downstream of someone else, e.g. for react-native.  So, make it easy
to configure per-repo.
2018-06-21 17:28:21 -07:00
Vishnu Ks 59169cc302 provision: Run inline-email-css only if required.
This saves a few hundred milliseconds on provision time.

Edited by tabbott to include email.css in the dependencies.

Fixes #9468.
2018-06-18 06:58:55 -07:00
Vishnu Ks 7b8e79ae48 provision: Refactor hashing of compilemessages into a function.
This allows it to be reused for other tools.

Edited by tabbott to remove the use of "compilemessages" in variable
names.
2018-06-18 06:55:36 -07:00
Raymond Akornor e82581e5bd provision: Close opened files automatically. 2018-06-12 13:38:58 -04:00
Aditya Bansal 1205e02c64 provisioning: Don't rebuild DB if running migrations is sufficient.
This results in a significant optimization in the performance of
re-provisioning Zulip if all that you're doing is rebasing onto a
newer version of master (which just adds new migrations).

The change carries some risk of generating unpleasant-to-debug
situations, because if we merge a buggy migration and then later fix
it, some clients may not have a properly migrated database (and also,
this changes how populate_db commutes with migrations).  But it seems
worth it, given how much time is currently wasted by not having this.

Fixes: #9512.
2018-06-06 13:43:59 -07:00
Aditya Bansal 65dc80fe9d test_fixtures: Add run_generate_fixtures_if_required function.
In this commit we are adding run_generate_fixtures_if_required,
a new function which is meant to de-duplicate a bit of code
between test-server and test-backend which is essentially
responsible for rebuilding the test database if that was required.
2018-06-06 13:37:35 -07:00
Aditya Bansal f7c11d1747 test_fixtures: Refactor to have template_database_status API.
In this commit we are essentially just refactoring the function
is_template_database_current to be called template_database_status
and adjusting the return values accordingly.
This is essentially a preparatory commit for the upcoming commits
which will essentially enable us to not throw away entire DB and
rebuild from scratch if only running migrations could do the job.
2018-06-06 13:37:35 -07:00
Tim Abbott e9312b921c tests: Disable test-queue-worker-reload test.
This test is low value (it's for testing a very specific feature/bug),
is slow, and apparently flakes in CI occasionally.
2018-06-05 09:24:03 -07:00
Tim Abbott 4cdcb7e0d3 webpack: Use more consistent bundle naming scheme.
This way, both our JS and CSS files using .chunkhash.ext as their
naming scheme.
2018-06-03 16:49:59 -07:00
Tim Abbott 7813376934 webpack: Use filenames based on hashes.
This should avoid us creating duplicate webpack bundles every time we
do a deployment, even if none of the files in the bundles themselves
have changed at all.
2018-06-03 16:49:43 -07:00
Tim Abbott 714ad67db1 storage: Move library code into zerver/lib.
There's no particularly good reason for this to have been a top-level
file.
2018-06-03 16:02:02 -07:00
Shubham Padia e32dd53cd9 lint: Check for occurrences of `.includes` except in `frontend_tests/`.
Adds a custom check to js_rules in `/tools/lint/lib/custom_check.py`.
2018-06-03 14:30:22 -07:00
Greg Price f686d55fa0 push-to-pull-request: Broaden one error message.
I discovered this case just now while testing other changes.
2018-06-01 16:52:18 -07:00
Greg Price 954c71e90c push-to-pull-request: Use `jq -r` to simplify a test slightly.
This option (aka `--raw-output`) prints a string as itself, rather
than JSON-encoded; which makes it fit a bit better in a shell script,
saving us a layer of quoting.
2018-06-01 16:52:18 -07:00
Greg Price 8e81ca0fb2 push-to-pull-request: Fix regex usage in parsing remote.
This replaces ad4617c95 with a different fix for the same issue:
instead of stripping the `.git` off separately, we can just correct
the regex, using `+?` to fix our stepping in a classic regex pitfall.
2018-06-01 16:52:18 -07:00
Tim Abbott 751c602a0b update-prod-static: Don't copy/minify templates and styles.
This is a performance optimization: Rather than copying these files
into the `prod-static` directory and then deleting them, we just don't
copy them over in the first place.

For styles, it might have once been the case that this did something,
but we've moved them all to being managed by webpack some time ago.

For the js directory, I think it was never useful to copy and then
delete them; these files were always compiled via tools/minify-js,
and the raw JS files weren't needed, anyway.
2018-06-01 16:39:35 -07:00
Tim Abbott cefdf6f6d4 update-authors-json: Pretty-print output with sorted keys.
This makes working with this file for debugging/etc. much nicer.
2018-06-01 16:39:04 -07:00
Tim Abbott 60eae53327 push-to-pull-request: Provide better output for invalid PR.
Previously, we were very confusingly showing the error message for
permissions when the issue was a not-found error.
2018-06-01 16:00:27 -07:00
Tim Abbott ad4617c953 push_to_pull_request: Fix handling of .git.
This was throwing the "you can't access the repo" error for this
reason.
2018-06-01 16:00:27 -07:00
Tim Abbott 95f1f1d363 mypy: Remove logic for linecoverage report. 2018-06-01 11:14:32 -07:00
Vishnu Ks 69f8b7e8df circleci: Add a base image for Ubuntu Bionic. 2018-05-31 18:51:32 -07:00
Tim Abbott 5b5d1f3e87 update-prod-static: Rewrite to use the run() scripting tool.
This provides nice `set -x` style logging of which commands are
running when.
2018-05-31 16:45:51 -07:00
Sampriti Panda e95f972e24 migrate: Add do_batch_update method for running batch updates.
* Refactor pgroonga_0002 migration to use new method.
2018-05-31 13:00:34 -07:00
Tim Abbott a2c9517f8d test_server: Don't continuously recompile handlebars templates.
This changes run-dev.py to ensure that we have in fact compiled
handlebars templates before running webpack, which is the right model.

Future work will likely include running the handlebars compiler from
webpack, and thus eliminating this extra process.
2018-05-30 20:16:06 -07:00
Tim Abbott 5a35507a7a success-http-headers: Add X-Frame-Options. 2018-05-30 09:41:50 -07:00
Tim Abbott 0fabff6dda setup_venv: Clean up VENV_DEPENDENCIES using parsed lsb_release.
This is mostly a cleanup, but it should also save 50ms in the runtime
of create-production-venv.
2018-05-29 10:57:36 -07:00
Raymond Akornor 5d39a0f0fc scripts: Replace calls to lsb release with our own parsing.
This improves the performance of these operations, by saving a ~50ms
Python process startup.  While not a major performance improvement, it
seems worth it, given how often these commands get run.

Fixes #9571.
2018-05-29 10:57:36 -07:00
Robert Hönig afa806a0ce linter: Add rule to favor Botserver over botserver or bot server. 2018-05-29 07:27:36 -07:00
Tim Abbott 2f571f9a45 webpack: Move font and pygments CSS to common.css.
First, it's silly that these weren't in common.css in the first place,
since that meant these were a bunch of duplicated code, but
additionally, that meant that these weren't available on the
`/activity` page (or other pages that don't include the portico styles).

Fixes #9561.
2018-05-27 23:19:40 -07:00
Yago González b4084d30d2 emails: Move user role logic for followup_day1 to notifications.py. 2018-05-26 07:29:58 -07:00
Yago González 5b0b9feb7d i18n: Atomize translation tags.
Makes the i18n strings in this file much easier to translate by splitting
them into smaller chunks (which avoids having a lot of code in the tagged
strings), and adds a string that was missing as well.
2018-05-26 07:06:43 -07:00
Tim Abbott 20aa3233b0 lint: Improve the check for realm->organization to support identifiers. 2018-05-25 12:03:28 -07:00
Tim Abbott 07b4da7659 Revert "node tests: Change timezone for node tests."
This reverts commit 311ddefd32.

This was causing tests to fail in some but not other environments.
2018-05-25 08:52:13 -07:00
Vishnu Ks abade460a9 tools: Add tool for listing outdated pip packages. 2018-05-24 15:16:02 -07:00
Aditya Bansal f9aa012486 linter: Fix issue with check-templates output.
We fix the issue of check-templates spitting out diff between
expected and found indentation of a file before mentioning the
error message and the file name. Basically stuff was being in the
wrong order despite the fact that in code stuff was happening in the
correct order ie, first print the error message along with the filename
and then the actual diff between expected and found file indentation.

Fixes: #9533.
2018-05-24 12:42:32 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 1b57e568ff Add "zform" functionality to the web client.
A "zform" knows how to render data that follows our
schema for widget messages with form elements like
buttons and choices.

This code won't be triggered until a subsequent
server-side commit takes widget_content from
API callers such as the trivial chat bot and
creates submessages for us.
2018-05-24 09:30:22 -07:00
Steve Howell 435719c65b Add a schema checking concept to the web app.
This starts the concept of a schema checker, similar to
zerver/lib/validator.py on the server.  We can use this
to validate incoming data.  Our server should filter most
of our incoming data, but it's useful to have client-side
checking to defend against things like upgrade
regressions (i.e. what if we change the name of the field
on the server side without updating all client uses).
2018-05-24 09:30:22 -07:00
Steve Howell d018995409 node tests: Make the failure message more obvious.
I mistakenly pushed a PR when my tests failed.  I ran with
the coverage option, so I saw this brightly colored summary
report that distracted me from the failure message.

This adds a couple newlines and some all caps.
2018-05-24 09:30:22 -07:00
Sivakar Sithamparanathan 311ddefd32 node tests: Change timezone for node tests.
The timezone environment variable was set to UTC initially. It was
changed to something other than UTC so that any local vs UTC
conversion issues will manifest in the tests.

Fixes: #5105.
2018-05-23 13:29:19 -07:00
Aditya Bansal 8cfb437677 provision: Don't run create_realm_internal_bots management command.
We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.

Fixes: #9467.
2018-05-23 13:34:52 +05:30
Aditya Bansal a27c9a08dd generate-fixtures: Export DJANGO_SETTINGS_MODULE=zproject.test_settings.
This is required because the --settings=zproject.test_settings param
doesn't work with migrate or the dumpdata management commands. Thus
untill now if one ran just this tool ended up with test database not
properly setup. We never noticed this because test-backend ran this
tool again (after exporting DJANGO_SETTINGS_MODULE) thus making the
tool work this time.
2018-05-23 11:32:40 +05:30
Sampriti Panda 51c71fd3fd provision: Add libfontconfig1 to bionic apt dependencies. 2018-05-22 22:37:04 -07:00
Shubham Dhama ec4e5e73ab org settings: Make time limit input elements inline to dropdown.
(To make the capitalization pass we added `N` to ignored phrases.)
2018-05-22 19:32:44 -07:00
Greg Price 5e89c33e7f tools: Add push-to-pull-request tool.
I've often done this by hand -- basically typed out the last line,
with the variables found from looking at the PR page in a browser.
Seems nicer for both us maintainers and the contributor, in particular
because the PR gets marked as merged, instead of closed.  But it's a
bit of a pain, and I do it maybe half the time or less; plus it's kind
of a subtle GitHub feature, and as a result I think other maintainers
of Zulip repos do this approximately never.

I've always figured this couldn't be hard to automate; today I decided
to take the 45 minutes to look up how, write out the script, QA it,
write up a nice usage message and some comments, and commit it. :)
2018-05-22 18:33:51 -07:00
Armaan Ahluwalia 1525e92058 logging/errors/webpack: Improve error stack traces with blueslip.
This commit improves the output that blueslip produces while
showing error stack traces on the front-end. This is done by
using a library called error-stack-parser to format the stack
traces.

This commit also edits the webpack config to use a different
devtool setting since the previous one did not support sourcemaps
within stack traces. It also removes a plugin that was obviated
by this change.
2018-05-22 14:36:41 -07:00
Eeshan Garg e0ef831993 webhooks: Migrate to UnexpectedWebhookEventType.
This improves test coverage for a lot of our webhooks that relied
on ad-hoc methods to handle unexpected event types.

Note that I have deliberately skipped github_legacy, it isn't
advertised and is officially deprecated.

Also, I have refrained from making further changes to Trello, I
believe further improvements to test coverage should be covered
in separate per-webhook commits/PRs.
2018-05-22 08:30:19 -07:00
Eeshan Garg 3ed20589f2 webhooks: Add generic exception for unexpected webhook events.
UnexpectedWebhookEventType is a generic exception that we may
now raise when we encounter a webhook event that is new or one
that we simply aren't aware of.
2018-05-22 08:30:19 -07:00
Michael J. Sullivan c81b276c1c mypy: Add basic support for invoking dmypy to run-mypy. 2018-05-21 22:41:08 -07:00
Aditya Bansal e41d32a15a reminders: Add various design improvements for the UI/reminder msg. 2018-05-21 09:07:37 -07:00
Tim Abbott c6fda61227 block_internet: Make compatible with httpretty library.
If httpretty is enabled, we can let it handle HTTP requests.
2018-05-20 23:31:42 -07:00
Tim Abbott a170bd70af provision: Don't block installing with Ubuntu bionic.
Now that we have tsearch_extras packages uploaded, this mostly works.

There's a few issues being debugged in #9460; they should be fixed
soon, and regardless, merging this will make it easier to develop.
2018-05-20 21:21:11 -07:00
Tim Abbott 06ed55e45a webpack: Add ALLOWED_HOSTS list for zulipdev.com and friends.
This makes it possible to again use the *.zulipdev.com domains in the
development environment.

Ideally, we'd also read REALM_HOSTS to make this more flexible.
2018-05-20 18:12:28 -07:00
Max Nussenbaum 0ff2051982 portico: Add tour section to homepage.
This adds a tour of Zulip to the bottom of the homepage.

In order to get the carousel nave, we use Bootstrap 2 from a CDN on
this page; this isn't ideal in the medium term, but upgrading
Bootstrap across the project is too much work for now.
2018-05-20 15:04:23 -07:00
Tim Abbott 4f8e09d5af provision: Fix buggy management of apt_dependencies_hash.
Apparently, we were incorrectly appending each new hash onto the end
of the file, basically resulting in every run of provision being
treated as a miss for this cache.

Fixing this saves about 4s (over 1/3) of the no-op provision time.
2018-05-20 14:22:32 -07:00
Tim Abbott 235002a549 provision: Don't install lsb-release if already installed.
This early bootstrapping step should be rearely required, and it saves
about 0.5s in our no-op provision time.
2018-05-20 14:22:32 -07:00
Shubham Padia 0824308a7a tools: Allow optional arguments after file arguments in test_backend.
Fixes #9233.
Uses nargs='*' instead of nargs='argparse.REMAINDER'.
nargs='argparse.REMAINDER' gathers remaining terms as arguments
even if it is an option e.g --coverage, while '*' gathers all the
command-line arguments until the next option is encountered.
2018-05-18 17:08:40 +05:30
Tim Abbott 0dcf873370 third: Rename thirdparty-fonts.css for better clarity.
Now it's clear that this is just legacy fontawesome styles.
2018-05-17 04:58:30 -07:00
Aman Ghebreweldi b05a702285 third: Use font awesome css file from NPM package. 2018-05-17 04:58:30 -07:00
Steve Howell b48f052b0c Add basic widgets framework (JS side). 2018-05-16 15:13:33 -07:00
Steve Howell 1f0ffdc415 Add widget.py to enable widgets on the server side.
The only slash command implemented in this initial
version is an extremely crippled version of a
"/stats" slash command that reports that you are
running 1 server.
2018-05-16 15:13:33 -07:00
Joshua Pan a67ba617d6 minor: Fix typo in tools/lint. 2018-05-16 13:19:28 -04:00
Tim Abbott 00861b910d test_presence: Add tests for seconds_usage_between.
This isn't widely used, but is a nice way to verify the logic further.
2018-05-15 18:37:49 -07:00
Tim Abbott e39a3e07d7 coverage: Mark zerver/lib/debug.py as not needing coverage. 2018-05-15 17:19:14 -07:00
Tim Abbott fd24eac8bb coverage: Exclude zerver/lib/parallel.py from coverage. 2018-05-15 17:19:14 -07:00
Tim Abbott c46149efcb migrate: Remove obsolete act_on_message_ranges.
This has a cool structure, but it's written against the long-dead
South API, and we can always pull it out of the Git history if we want
to use this approach in the future.
2018-05-15 17:19:14 -07:00
Tim Abbott 3e04aa99fa fonts: Use source-sans-pro CSS from upstream. 2018-05-15 16:22:44 -07:00
Greg Price 6dbd90835b mypy: Move remaining exclusion list to config file.
This leaves the wrapper script with very little left to do!

The main thing left is finding scripts by searching for shebang lines;
mypy itself would happily do the search for importable Python files.
2018-05-15 18:14:33 -04:00
Tim Abbott badfec2b49 coverage: Don't include settings files in test coverage. 2018-05-15 13:55:01 -07:00
Tim Abbott 191e6dc13a create_user: Remove user_profile_id option.
This hasn't been used in years, had no test coverage, and doesn't have
a clear use case.
2018-05-15 13:51:56 -07:00
RobbieClarken f81b936727 zerver/tests: Require 100% test coverage of zerver/lib/upload.py.
The last line here is impossible to test.

Fixes #4489.
2018-05-15 10:52:20 -07:00
Aman Ghebreweldi 83ee8211a8 third: Get sorttable package from npm.
We didn't have any local changes, so there's no need to have it
vendored in the repository.
2018-05-15 09:25:48 -07:00
Steve Howell 4f0b3fc1b6 test-js-with-node: Exit more cleanly for errors.
We don't need a Python stack trace when the node
piece fails, so just give a non-distracting error
message.
2018-05-15 08:24:44 -07:00
Joshua Pan df84e1d7eb tests: Reach 100% coverage for zerver/liv/management.py.
Cleaned up add_user_list_args(). The "help" and
"all_users_help" have all default values. As noted in
an earlier commit, "all_users_help" is always passed in,
so we can get rid of "all_users_arg". We keep the default
for "all_users_help" so we don't have to change variable order
in function definition.
2018-05-14 10:46:21 -07:00
Darshan Markandaiah 607cab2a53 tests: Add 100% test coverage to zerver/apps.py.
Tweaked by tabbott to rename to test_cache.py and remove the
sender_name argument (Since it was kinda confusing).
2018-05-14 10:36:10 -07:00
Vishnu Ks b73603a97c droplets: Do shutdown command along with clear history.
Otherwise shutdown command would be still present in the
history.
2018-05-14 08:12:43 -07:00
Vishnu Ks 861001f7b3 droplets: Update snapshot id.
Upgraded zulip and python-zulip-api repos.
Installed flask in python-zulip-api.
2018-05-14 08:12:43 -07:00
Tim Abbott f5ea661c1f tools: Upgrade pycodestyle.
We have to add a few rules that we're ignoring for now.
2018-05-14 06:15:13 -07:00
Aditya Bansal 64678b459c linter: Add rule to lint against use of typing.Text and promote str.
This commit is the last in a series of commits which migrated our
entire codebase to use str instead of typing.Text.

Fixes: #9203.
2018-05-13 17:17:32 -07:00
Aditya Bansal addfde7374 tools: Change use of typing.Text to str. 2018-05-13 17:17:32 -07:00