Commit Graph

3900 Commits

Author SHA1 Message Date
Anders Kaseorg 82828bdba4 HTML validation: Remove invalid <button href> attribute.
For .start-button, Bootstrap carousel already supports <button
data-target> as a valid alternative to <button href>.  For
.call-to-action, the margin is decreased to exactly offset the lack of
margin collapsing with display: inline-block.  There should be no
visual change.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-23 16:16:22 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Anders Kaseorg f54a63e2f9 webpack: Transpile JS code with Babel.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-22 17:55:32 -07:00
Anders Kaseorg ecfb7c6a7f lint: Add TypeScript compiler as a linter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-22 17:55:18 -07:00
Anders Kaseorg 41209dd32e build-release-tarball: Run with zulip-py-venv symlink present.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-21 18:43:52 -07:00
Rishi Gupta 7d8d0b2284 settings: Update upgrade text and styling.
When we add Plus, the first sentence should change to "Available on Zulip
Standard and Plus".

I copied the styling of .tip out of expediency, but it's also possible that
long term we'll want only 1 tip-like box styling.

The hover styling is a bit random, but I tried to copy other hover styles I
found in settings.scss.

Note that this renames .upgrade_realm_plan_type_suggestion to .upgrade-tip.
2019-07-21 14:32:36 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Tim Abbott c931e76cf2 docs: Rewrite docs on writing API documentation.
This had gotten badly out of date, since it wasn't updated when we did
the big migration to the OpenAPI documentation system.

Fixes part of #12571.
2019-07-19 18:00:01 -07:00
Anders Kaseorg e80b57a18a provision_inner: Remove unused lsb_release run.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 16:36:11 -07:00
Anders Kaseorg c5a4b0501b provision: Start a new Python process after activating the virtualenv.
Mismatching imports from outside and inside the virtualenv in the same
process was causing segfaults after apparently benign changes to the
script!

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 12:16:55 -07:00
Anders Kaseorg a7eb3faf96 storage: Move the staticfiles.json hack to ZulipStorage.
There’s no reason to monkey-patch something that we were already
subclassing.

Removing the PRODUCTION conditional causes us to generate
staticfiles.json in the right place to begin with so we don’t need to
move it later.  It also allows Django to find staticfiles.json if
running the dev server with PIPELINE_ENABLED.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Anders Kaseorg 695b79c5d8 generate_zulip_bots_static_files: Write to static, not STATIC_ROOT.
Otherwise the files aren’t processed by collectstatic and don’t end up
in the staticfiles.json manifest.

Signed-off-by: Anders Kaseorg <andersk@zulipchat.com>
2019-07-19 11:33:08 -07:00
Anders Kaseorg 5f712b960f update-authors-json: Write to static, not STATIC_ROOT.
Otherwise the file isn’t processed by collectstatic and doesn’t end up
in the staticfiles.json manifest.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:33:08 -07:00
Anders Kaseorg 1d05a71b5d install, provision: Treat all nonzero exit codes as failure, not just 1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:22:11 -07:00
Wyatt Hoodes 89df6e9425 check-templates: Add fix option to tools/check_templates.
`valid_indent_html` allows for replacing the incorrectly
indented file with the correct pretty-printed version
if `--fix` is passed to `tools/lint`.

Fixes #12641.
2019-07-19 11:12:11 -07:00
Anders Kaseorg ab89f40a66 generate-custom-icon-webfont: Replace with webpack webfonts-loader.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-18 12:00:00 -07:00
Anders Kaseorg 586aaddcc0 documentation_crawler: Stop treating all portico URLs as external.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-18 11:56:15 -07:00
Wyatt Hoodes 1418b8b05f test_server: Set LOCAL_UPLOADS_DIR in set_up_django.
For the Casper and API tests, we need to allow setting
LOCAL_UPLOADS_DIR, which is normally managed within test_runner.py.
2019-07-17 16:14:20 -07:00
Anders Kaseorg 54304b4008 webpack: Remove partial from handlebars-loader knownHelpers.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-17 16:07:17 -07:00
Anders Kaseorg b1aa304c4a templates: Suppress Handlebars automatic partial indentation.
Fixes #12795.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-17 16:07:17 -07:00
Sumanth V Rao 2c9f5e3980 tools: Add tool to update API field of local zuliprc file.
This tool can be used to update the API field of local
zuliprc files for dummy users of development server
(iago, prospero, etc) with the correct API key from database.

This tool can be run after provisioning (or similar tools) which change
the API keys in the database.
2019-07-17 16:00:21 -07:00
Tim Abbott 7c71e987ac openapi: Fix endpoints incorrectly tagged as documentation_pending. 2019-07-15 13:39:29 -07:00
Amal Rajan 09049f1678 provision: Add provision support for Ubuntu 19.04
This commit adds support for Ubuntu 19.04 in the development
environment.
2019-07-13 16:44:09 -07:00
Anders Kaseorg aa36e2f5e1 check-templates: Exclude static/icons/fonts/template.hbs.
This file was unchecked until the .handlebars ↦ .hbs rename, so this
is the easiest way to get tests passing again.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-12 22:52:41 -07:00
Anders Kaseorg 0c565f50be templates: Use upstream Handlebars partials syntax.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-12 21:11:14 -07:00
Anders Kaseorg db0b33842c templates: Replace templates.render with require calls.
This removes an unnecessary layer of indirection and allows webpack to
catch filename mistakes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-12 21:11:14 -07:00
Anders Kaseorg 3c3471b720 templates: Rename *.handlebars ↦ *.hbs and - ↦ _.
Tweaked by tabbott to avoid accidentally disabling the linter for
handlebars templates.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-12 21:11:03 -07:00
Wyatt Hoodes 0e3fddbc6e test_fixtures: Add logic for removing stale test directories.
Similarly to how stale database removal is handled, we add a check for
stale test run directories at the end of the `test-backend` script.
2019-07-12 17:42:18 -07:00
Wyatt Hoodes 16abd3d1e7 test-backend: Remove long-term clean up code.
With the refactored file structure approach, the clean up block at the
end of `test-backend` is no longer necessary.
2019-07-12 17:42:18 -07:00
Rishi Gupta 0f3c2748dd notification bot: Update stream announcement message. 2019-07-11 15:22:36 -07:00
Rishi Gupta 7de7b6872b notification bot: Modify initial stream creation message.
Discussion at
https://chat.zulip.org/#narrow/stream/137-feedback/topic/hello.20topic

The "by @**X**" part in the message content is in a later commit.
2019-07-11 14:16:56 -07:00
Anders Kaseorg 218c60ae86 third: Upgrade spectrum-colorpicker to 1.8.0 from NPM.
We had several patches to spectrum, but the only essential one 
(0ea770fc18) had already been fixed upstream,
and another was just handling jQuery deprecation warnings for not yet removed features.

See #12749 for details.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-10 10:07:34 -07:00
Wyatt Hoodes 886932fd23 provision: Update error output on failed `yarn install` reattempt. 2019-07-08 12:13:34 -07:00
Vishnu Ks 1f250a3fd2 casper: Add support for exporting results as XUnit XML file. 2019-07-07 22:30:01 -07:00
Vishnu Ks 4f65b50c44 requirements: Upgrade django-otp from 0.5.2 to 0.6.0.
This indirect dependency was upgraded separately since
it contained a migration with autogenerated name.
2019-07-07 22:28:54 -07:00
Anders Kaseorg 9f08513acb bundles: Rename commons.js to common.js.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 15:03:22 -07:00
Aman Agrawal 05aa3bcbc1 lint: Make `--fix` a default arg for zulint.
Since the check for `--fix` is in `zulint/command.py`,
we should keep it as an arg for zulint.
2019-07-03 14:57:31 -07:00
Aman Agrawal aa5a1cb509 lint: Make zulint independent of extra lint options in `./tools/lint`.
Check `options.full` in `pyflakes.py` instead of `linters.py` to
make zulint independent of `./tools/lint`.
2019-07-03 14:57:31 -07:00
Anders Kaseorg 1868e96850 tools/webpack: Don’t import Django.
It added about 150ms to the startup time of running webpack, for no
benefit.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:52:11 -07:00
Wyatt Hoodes 40227171e6 provision: Improve error messaging when attempting to use Ubuntu Trusty.
As part of dropping support, we add appropriate error messaging when a
user attempts to provision while using trusty.  If the user is running
in Vagrant we append information on how to proceed.
2019-07-03 14:32:25 -07:00
Anders Kaseorg 3ded6f53c3 storage: Transfer responsibility for hash-naming webpack files to webpack.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:16:17 -07:00
Anders Kaseorg b0a30aa01a storage: Remove RemoveUnminifiedFilesMixin.
This can be done much more easily with the right options to
collectstatic.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:16:07 -07:00
Anders Kaseorg 0cd6c75ab1 update-prod-static: Remove unused pre-webpack files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Anders Kaseorg 5f3d36ae87 tex: Bundle KaTeX CLI using Webpack.
We don’t need a hacked copy anymore.  We run the installed version out
of node_modules in development, and a Webpack-bundled version of that
in production.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Anders Kaseorg 9f074fc071 webpack: Fix production setting of publicPath.
This is needed by webpack asynchronous require.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:49:38 -07:00
Anders Kaseorg 05663aa3fa webpack: Rectify "common" bundle with bundles/commons.js.
Moving bootstrap-typeahead from bundles/commons.js to bundles/app.js
and csrf.js from bundles/app.js to bundles/commons.js makes
bundles/commons.js equivalent to the "common" bundle, so we can
replace the latter with the former.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:38:26 -07:00
Anders Kaseorg 1b94733953 webpack: Remove resolve.modules override.
The minimal syntactic sugar it might provide isn’t worth the
unexpected side effects (including side effects on third party
modules).

For now, we allow zrequire to emulate the previous syntax in the Node
test suite, even though stealing part of the NPM namespace is
confusing.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:38:26 -07:00
Thomas Ip 8c199fd44c webpack: Use handlebars-loader to handle frontend templates.
And remove the compile-handlebars-templates system.
2019-07-02 16:23:29 -07:00
Anders Kaseorg fc3305b623 webpack: Trim resolve.extensions.
We don’t have any .tsx files, and nobody expects to be able to omit
the extension when importing .json, .scss, or .css files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:09:29 -07:00
Anders Kaseorg 7b2a605978 provision: Sort system dependencies deterministically.
set iteration order is randomized in Python ≥ 3.3.  That might or
might not have had the potential for causing rare probabilistic bugs,
but if nothing else, it made build logs harder to compare.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 15:04:44 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Anders Kaseorg a1cf77ca54 test-help-documentation: Bind vnu.jar to 127.0.0.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-28 18:18:29 -07:00
Anders Kaseorg 2d50dcf7cc test-help-documentation: Validate HTML with vnu.jar.
The VNU_IGNORE whitelist lets in some crazy-invalid preexisting HTML,
but hopefully this will stop the problem from getting much larger.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-27 14:53:21 -07:00
Anders Kaseorg f407a12ba2 test-documentation: Validate HTML with vnu.jar.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-27 14:53:21 -07:00
Aman Agrawal 7a3e8c4868 zulint: Add option to list groups using `--list-groups`. 2019-06-27 14:42:33 -07:00
Aman Agrawal 4424867adc zulint: Show errors in `--verbose` in a more readable format. 2019-06-27 14:42:33 -07:00
Aman Agrawal e275457387 zulint: Avoid changing the color of next line by using ENDC.
ENDC was not being applied due to missing `{}` in the end.
2019-06-27 14:42:33 -07:00
Anders Kaseorg 23cd064c86 webpack: Elide node_modules when importing JS modules.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 16:49:32 -07:00
Anders Kaseorg caecd1c2ad install: Disable installation and provisioning on Ubuntu 14.04 Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 15:58:53 -07:00
Anders Kaseorg bfa82a70c8 ci: Remove Ubuntu 14.04 (trusty) from CI.
Tweaked by tabbott to improve our CI docs discussion of these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:01:13 -07:00
Anders Kaseorg b4204e7cfd Remove legacy (pre-Vagrant) Docker development environment setup.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 10:58:02 -07:00
Anders Kaseorg bac63f5a3e documentation_crawler: Reimplement crawl_with_status so it wraps crawl.
This way we inherit more of the upstream command’s behavior.
Importantly, this means we pass everything in `opts.spargs` to the
spider, not just `opts.spargs.skip_external`.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-24 23:20:17 -07:00
Anders Kaseorg 0f30c8c4ba dev-vagrant-docker: Upgrade docker-systemctl-replacement.
This fixes the 45 second boot delay bug.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-24 22:06:32 -07:00
Thomas Ip 80f972d42c webpack: Use correct types in config.
Cache-loader is used as an item in the use member so the correct
type should be RuleSetUseItem not RuleSetRule.
See: DefinitelyTyped/DefinitelyTyped/webpack/index.d.ts#L498
2019-06-24 17:58:51 -07:00
Thomas Ip 637b2b784a dependencies: Upgrade mini-css-extract-plugin to 0.6.0 and use HMR.
Given mini-css-extract-plugin can now do hot module replacement,
this commit also removed css-hot-loader. Not upgrading to 0.7.0
as that cause webpack to crash.
2019-06-24 17:58:51 -07:00
Yashashvi Dave 5c88475378 static/js/stream_data: Deduplicate notification setting updates.
Commit also add `stream_data.js` to full test covered
enforced files for `test-js-with-node` tests.
2019-06-24 14:46:45 -07:00
Thomas Ip a2de8ab44f lint: Add --fix option to ./tools/lint.
Linters with the fix option are stylelint, eslint and puppet-lint.
2019-06-24 14:13:04 -07:00
Wyatt Hoodes 6670b15234 provision: Clean up `var/` for the move to `var/<uuid>/test-backend`. 2019-06-24 12:42:00 -07:00
Wyatt Hoodes b1fe1ef42b test-backend: Clean up leak data import files after test-suite run.
This is a simple, non-intrusive way of removing the bulk of the
clutter from `var/<uuid>/test-backend` after running `test-backend`.

Ideally, we'll replace this logic with proper tearDown methods.
2019-06-24 12:39:56 -07:00
Aman Agrawal 8fc7c89874 zulint: Add `--verbose` arg to print good/bad lines in error.
This argument probably still needs some iteration, but it's already
useful.
2019-06-23 22:24:46 -07:00
Aman Agrawal 8eee19a160 lint: Add descriptions for all the linters. 2019-06-23 22:23:15 -07:00
Aman Agrawal 6d42b2392b zulint: Print description of linter with it in `--list`. 2019-06-23 22:23:15 -07:00
Aman Agrawal e236d39ef1 zulint: Add option to specify description of linter functions.
Extract description of function from __doc__ if used as wrapper.
2019-06-23 22:23:15 -07:00
Aman Agrawal e9ff9e34b3 lint: Use --groups to specify specific groups to run.
This helps generalize the use of groups inside zulint.
Introduce list_files to return `by_lang` files dict.
Add feature to create custom groups.
Make custom groups for backend and frontend files.
2019-06-23 22:23:15 -07:00
Aman Agrawal b5b3d9bd5f zulint: Refactor duplicate error printing code into print_error. 2019-06-23 21:59:00 -07:00
Aman Agrawal 7b7e887386 lint: Pass args to LinterConfig when initializing it.
This makes it easy to use args inside LinterConfig.
2019-06-23 21:55:02 -07:00
Aman Agrawal 0b12ec1417 lint: Initialize logger inside zulint.
This also helps move verbose_timing logic inside zulint.
2019-06-23 21:55:02 -07:00
Aman Agrawal b6d4438726 lint: Pass `args` instead of passing individual attributes to do_lint.
Pass the args to linter_config.do_lint.
2019-06-23 21:55:02 -07:00
Aman Agrawal b2b49089fd tools: Extract get_provisioning_status check logic.
Move get_provisioning_status check logic into
assert_provisioning_status_ok and use it instead of duplicating the
check code.
2019-06-23 21:55:02 -07:00
Tim Abbott f8b0727e5a test-backend: Check for venv before importing modules like ujson.
Previously, we were giving the wrong error message when running
test-backend not inside the Zulip venv if ujson wasn't installed on
the machine.
2019-06-23 16:12:21 -07:00
Aman Agrawal aa217efdbe lint: Add rule to ban rgb(a) in css/scss files, use hsl(a) instead. 2019-06-20 11:34:59 -07:00
Tim Abbott 4818011199 zulint: Add a basic README. 2019-06-18 12:25:46 -07:00
Aman Agrawal 6d040d330c lint: Add --list arg which lists all the registered linters. 2019-06-18 11:32:04 -07:00
Aman Agrawal 6457c1f7c6 lint: Generalize --skip arg and move logic to zulint/command. 2019-06-18 11:32:04 -07:00
Aman Agrawal 995b357cde lint: Add --only agr to run only the specified linters. 2019-06-18 11:32:04 -07:00
Aman Agrawal cf8653945f lint: Rename commit_messages to gitlint.
Use `gitlint` name to register gitlint linter since it is the commonly
used name for it.
2019-06-18 11:32:04 -07:00
Aman Agrawal 6b73926e93 lint: Add --skip arg to replace --no-gitlint/mypy.
Use --skip=gitlint,mypy instead of --no-gitlint/mypy.
2019-06-18 11:32:04 -07:00
Aman Agrawal 426a222b7e lint: Rename --verbose to --verbose-timing. 2019-06-18 11:32:04 -07:00
Aman Agrawal dcd46f1c11 lint: Move shebang_rules out of `RuleList` class to generalize it.
shebang_rules was moved to custom_check.py. Also add shebang_rules only
to those rules which need it.
2019-06-18 11:27:59 -07:00
Aman Agrawal db25c0c2ca lint: Combine functions in custom_rules into RuleList class.
This makes linting rules in zulint more general. Make necessary
changes in tools/lint and tools/custom_check.py to run with the new
RuleList class.

Modify tests for `RuleList` class. Tests only include minor changes to
test with the new class.
2019-06-18 11:27:50 -07:00
Tim Abbott 9760c8cfc9 tools: Fix postgres-init-*-db cache handling.
Previously, it didn't properly update the stamp files that determine
our caching behavior, so if one ran test-backend afterwards, nothing
would happen.

A secondary issue that this commit does not fix is that provision will
end up rerunning the whole thing.
2019-06-17 16:24:13 -07:00
Tim Abbott efe5e53118 docs: Update list of links to not check in tests. 2019-06-17 12:24:22 -07:00
Thomas Ip 0253ef12c6 popovers: Encode brackets in URI to avoid conflict with markdown links.
This fixes an issue where one could end up with a `(` in the markdown
syntax for a link after copy-pasting this, which doesn't work in
markdown.

Fixes #12579.
2019-06-16 15:18:33 -07:00
Vishnu Ks fdcae3b0b7 provision: Add cd /srv/zulip to .bashrc only if /srv/zulip exists. 2019-06-14 15:49:56 -07:00
Vishnu Ks d703e3638e droplets: Update snapshot id.
The new snapshot has zulip/zulip commit till
ac0d60f577. I have also included
the custom motd from b594708915
in base image.
2019-06-14 15:49:56 -07:00
Wyatt Hoodes 6b033c7909 test-backend: Add steps to deal with potential database leaks.
A function was written in `test_fixtures.py` to drop a test database
template if the corresponding database id doesn't belong to a file.
Alongside this fact, every file that is written is removed after 60
minutes.  Meaning any potential database template can never exist
longer than one hour.

This follow-up work was added to deal with the potential race
conditions when running `test-backend`.  Ensuring that all templates
are properly dealt with.

Essentially rewritten by tabbott for cleanliness.

Fixes the remainder of #12426.
2019-06-14 15:23:25 -07:00
Vishnu Ks 44d16d76c1 activity: Extract most js from realm_summary_table to activity.js. 2019-06-14 10:20:59 -07:00
Vishnu Ks 14e582fb59 support: Add functionality to copy admin emails.
Also renamed a bunch of functions in test_views for better
readability.
2019-06-14 10:19:50 -07:00
Puneeth Chaganti 6f4414e3f0 version: Fail gracefully if git describe cannot get version.
When running the `./tools/cache-zulip-git-version` script on Travis, the script
fails because Travis gets a shallow clone of the repository, and not a full
clone. This commit changes the script to fail gracefully, if we are unable to
get the version information using `git describe`.

When the command fails, it still writes an empty `zulip-git-version` and that
has not been changed to keep creation of the release tarball simple, and
avoiding a check for whether the file has any content. The code that sets
`ZULIP_VERSION` checks whether the contents of the `zulip-git-version` file are
empty, before setting `ZULIP_VERSION`. So, the version should never be set to an
empty string.
2019-06-14 13:47:27 +05:30
Tim Abbott 05e0e99b6e install-aws-server: Use our settings.py if available. 2019-06-13 14:39:25 -07:00
Tim Abbott 2d1c2d4802 install-aws-server: Stop using ssh -t without a real terminal.
More modern Linux versions like Bionic will block this, and what we
actually want to do is just run the code in our <<EOF block via bash,
so we should do that explicitly.
2019-06-13 14:39:25 -07:00
Alexandra Ciobica fbfd5d577a integrations: Move the css for the integrations page into a separate file.
The integrations page had css in both `landing-page.scss` and
`portico.scss`.  With this commit, the styles are mostly unified into
a single separate file.
2019-06-12 17:42:22 -07:00
Vaibhav 9096affe77 css: Move stylesheets not included in the app to another directory.
This moves all the stylesheets like stats, billing etc. to another
directory called `static/styles/portico/`, matching the directory
structure of our JavaScript.
2019-06-12 16:59:52 -07:00
ppreethi 86840adda5 version: Show number of commits and commit sha in ZULIP_VERSION.
We use `git describe --tags` to get information about the number of commit since
the last major version, and the sha of the current HEAD. This is added to the
ZULIP_VERSION when a deploy is done from `git`.

Modified heavily by punchagan to:
* to use git describe instead of `git log` and `wc`
* use a separate script to run the git describe command
* write the file with version info to var/ and remove it from the repo

Fixes #4685.
2019-06-07 13:39:12 -07:00
Vishnu Ks 8261c394b9 webpack: Include channel.js to email-log bundle.
During the refactoring in c27d927663
$.post was replaced with channel.post without including
channel.js in email-log webpack bundle.
2019-06-07 11:34:17 -07:00
Tim Abbott 9be3f30340 webpack: Don't import pyinotify in production context.
We only added the dependency for the development environment, but were
imported it unconditionally.
2019-06-04 18:10:58 -07:00
Tim Abbott 57a748ad63 node: Remove stream_ui_updates.js from modules expecting coverage.
This module is a UI module, which are harder to test, and in any case,
doesn't actually have any tests.
2019-06-04 16:45:54 -07:00
Priyank Patel 01f8c89294 webpack: Use cache-loader for various loaders.
Profiling shows that using cache-loader saves ~6-7 seconds of time take
by webpack-dev-server on subsequent runs. The overhaul this adds when
nothing is cached (when running first time) is around 1-2 seconds. We don't
use cache loader for ts-loader since webpack docs says it will slow it down
and file-loader since it just copies files over and caching it would just
was disk space.

This is the second merge of this commit.  It fixes the issue with the
previous one by placingn cache-loader after mini-css-loader because it
just extracts css and caching that will make file-loader not run which
in turn makes developement enviorment break.
2019-06-04 16:43:04 -07:00
Tim Abbott 5d0e144d62 Revert "webpack: Use cache-loader for various loaders."
This reverts commit eb53b5e8de.

This appeared to not have the right logic yet for handling branch
switches.

See https://chat.zulip.org/#narrow/stream/3-backend/topic/frontend.20hot.20reloading/near/749259 for details.
2019-06-03 22:11:47 -07:00
Priyank Patel eb53b5e8de webpack: Use cache-loader for various loaders.
Profiling shows that using cache-loader saves ~6-7 seconds of time take
by webpack-dev-server on subsequent runs. The overhaul this adds when
nothing is cached (when running first time) is around 1-2 seconds. We don't
use cache loader for ts-loader since webpack docs says it will slow it down
and file-loader since it just copies files over and caching it would just
was disk space.

Profiling data:

-------- Master ---------

~/zulip (master) $ tools/webpack --watch | ts -s '%.S' # master
03.995825 ℹ 「wds」: Project is running at http://127.0.0.1:9994/
03.996161 ℹ 「wds」: webpack output is served from /webpack/
03.996289 ℹ 「wds」: Content not from webpack is served from ...
19.284477 ℹ 「wdm」:
19.285371 ℹ 「wdm」: Compiled successfully.

-------- cache-loader ---------

~/zulip (cache-loader)$ tools/webpack --watch | ts -s '%.S'
04.107913 ℹ 「wds」: Project is running at http://127.0.0.1:9994/
04.108646 ℹ 「wds」: webpack output is served from /webpack/
04.109068 ℹ 「wds」: Content not from webpack is served from ...
12.633782 ℹ 「wdm」:
12.634083 ℹ 「wdm」: Compiled successfully.
2019-06-03 20:15:51 -07:00
Priyank Patel 4bb6c29d59 webpack: Use webpack's type defination.
Before we used to defined our own type Loader which was
partly incorrect because the use property can only
be string which is incorrect. We use the RuleSetRule type
provided by webpack instead.
2019-06-03 20:15:51 -07:00
Thomas Ip 537014ee47 dependencies: Upgrade typescript to 3.5.1. 2019-06-03 12:24:00 -07:00
Anders Kaseorg 5c734f7048 dev-vagrant-docker: Clear redis-server ExecStart before overriding it.
Real systemd requires this.  docker-systemctl-replacement currently
doesn’t but maybe it will later.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-01 16:55:56 -07:00
Anders Kaseorg faa4cf629a webpack: Close potential race condition leaking webpack-dev-server process.
This exchanges a race condition where webpack-dev-server might not be
stopped on a poorly timed KeyboardInterrupt for a less bad race
condition where we might get an UnboundLocalError.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-31 17:08:48 -07:00
Priyank Patel 2a864ebad3 webpack: Restart webpack-dev-server on config file changes.
This should make the run-dev.py user experience a lot nicer when
switching branches away from a branch that is at least as new as this
commit, since we won't need to manually restart run-dev.py to restart
webpack.

Fixes #9042.
2019-05-31 16:10:47 -07:00
YashRE42 5ff27c130a info-overlay: Fix "message formating" styling.
This reverts the temporary solution from 04d9d1d and introduces a
better, more permanent solution for the same bug.
2019-05-30 16:18:54 -07:00
Vaibhav 3462db3ecc node: Add test for updating bot_owner_id and enable user_events test suite.
Reverts the commit 4442509ec3
2019-05-30 15:43:18 -07:00
Anders Kaseorg d9f12cfb49 documentation_crawler: They’re called fragments, not permalinks.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 4f28b88746 documentation_crawler: Set a 15 second download timeout.
By default it just hangs for as long as the server keeps the
connection open.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg c59747360a documentation_crawler: Set the default log level to warning.
This doesn’t seem to add any noise in the normal case, but if anything
shows up here we might want to see it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 2f547eaee4 documentation_crawler: Improve error handling.
* Remove the custom has_error logic in favor of checking whether any
  errors were logged, which gives us a much better chance at catching
  unanticipated exceptions.
* Use our error_callback for the initial requests of start_urls too.
* Clean up mypy types.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 4c60e994c8 test-documentation: Fail on any nonzero exit code.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 0ca9c260ae test-documentation: Use getopt; don’t silently ignore multiple arguments.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Aman Agrawal a1fadc28ba lint: Make `custom_rules.py` pass all lint checks.
The functions here weren't checked by all linters when they were in
 `custom_check.py`. Hence, modified to pass all lint checks.
2019-05-29 16:53:59 -07:00
Aman Agrawal 90c3578a88 lint: Move `custom_check_file` from `custom_check.py` to `custom_rules.py`. 2019-05-29 16:53:59 -07:00
Aman Agrawal cf038f0880 lint: Allow shebang_rules to use `include_only/exclude`.
Make shebang rules generic to be later extracted out of the
`custom_check_file` function.
2019-05-29 16:53:59 -07:00
Aman Agrawal 42efb60665 lint: Extract general funcs from custom_check.py to zulint/custom_rules.py. 2019-05-29 16:53:59 -07:00
Aman Agrawal 337060de01 lint: Move runnable code form `pyflakes.py` to `zulint/linters`.
Extract runnable code from `linter_lib/pyflakes.py` and move it to
`zulint/linters`. This keeps zulip specific pyflakes config separate
form zulint.
2019-05-29 16:53:59 -07:00
Anders Kaseorg 630f0fa761 Vagrantfile: Add UBUNTU_MIRROR configuration option.
The default http://archive.ubuntu.com/ubuntu/ is pretty slow in some
locations.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-29 16:19:39 -07:00
Anders Kaseorg 03cfe10da7 coveragerc: Omit blub.
The jedi package exec()s some code in the context of the fake module
blub, causing errors when generating the coverage report.  See
https://github.com/davidhalter/jedi/issues/1122.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-29 15:04:47 -07:00
Tim Abbott 095ff28277 test-backend: Avoid dropping/rebuiling zulip_test database.
The test-backend parallel test runner system doesn't actually use the
zulip_test database; instead, it creates its own databases off the
zulip_test_template database.

We were accidentally running `tools/generate_fixtures` even when there
are no changes, because this function is shared with the
tools/lib/test_server.py codebase, which needs us to do the work of
creating a test database for it off the zulip_test_template database.

Fixing this saves about 1.5s / 4s of the runtime of a single test.
2019-05-28 16:38:45 -07:00
Tim Abbott e0afdb675f test_fixtures: Rename and document update_test_databases_if_required.
The previous name was confusing because `generate_fixtures` isn't
actually a good name in the first place.
2019-05-28 16:38:45 -07:00
Tim Abbott 7b6f219663 test-backend: Flush stdout when starting testing.
This makes the output more properly ordered.
2019-05-28 16:38:45 -07:00
Tim Abbott 4442509ec3 node: Disable a test suite that's missing a line of coverage. 2019-05-28 16:36:30 -07:00
Anders Kaseorg f21d3be5fb dev-vagrant-docker: Copy the host user’s uid for the vagrant user.
This allows /srv/zulip to be writable from the guest even if the host
user’s uid is not 1000.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 19:21:51 -07:00
Anders Kaseorg 5231c8e575 dev-vagrant-docker: Unminimize the minimized Ubuntu cloud image.
This restores man pages and other documentation that have been
stripped from the default Ubuntu cloud image and installs
ubuntu-minimal and ubuntu-standard.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 12:53:11 -07:00
Rohitt Vashishtha eb44b7d69d linter: Do not lint scripts parsed with tail (like dev-motd). 2019-05-27 12:40:02 -07:00
Anders Kaseorg b594708915 Vagrantfile: Be nicer to /etc/update-motd.d.
Some of its information is helpful.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:33:06 -07:00
Anders Kaseorg 407564086d provision: Move apt-get update from retry handler to setup-apt-repo.
This avoids unnecessarily alarming error messages if the apt cache is
missing.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:38 -07:00
Anders Kaseorg ae524b677d ci: Move backend and production tests to Ubuntu 16.04 (xenial).
This is preparation for dropping support for Trusty in CI.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-24 17:07:15 -07:00
Anders Kaseorg 684ebc2a5e Vagrantfile: Support docker provider.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-05-24 15:43:23 -07:00
Hemanth V. Alluri c47521401f devtools: Move the integrations devtool JS to the portico folder. 2019-05-22 21:28:11 -07:00
Mateusz Mandera 58da8fd5f1 test-backend: Remove email_mirror from not_yet_fully_covered.
The preceding commits get the email mirror to 100% coverage, so we can
now remove zerver/lib/email_mirror.py from the list of not-fully-covered
files.
2019-05-20 19:35:32 -07:00
Tim Abbott 136a0da30b provision: Update unsupported architecture logging messages.
We don't use zulip-devel@ anymore, and also add a comment that it's
probably not a huge project to support e.g. arm.
2019-05-20 15:23:40 -07:00
Rishi Gupta fbc2239f73 keyboard shortcuts: Update description for S. 2019-05-20 15:10:11 -07:00
Kanishk Kakar 77ab9a0eb0 docs: Update desktop repo URL. 2019-05-20 11:01:11 -07:00
Anders Kaseorg 141088586b Completely replace perfect-scrollbar with SimpleBar.
perfect-scrollbar replaces both the appearance and the behavior of the
scrollbar, and its emulated behavior will never feel native on most
platforms.  SimpleBar customizes the appearance while preserving the
native behavior.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-05-17 12:06:51 -07:00
Vishnu Ks e7507a6eb1 tools: Avoid upgrading stripe automatically.
Upgrades to the stripe library can sometimes break semantics for our
billing system, and so we should make sure to use our documented 
testing process before doing them.
2019-05-17 11:36:35 -07:00
Vishnu Ks 56259c7acd lint: Catch exit(1) in management commands.
Using sys.exit(1) in a management command makes it impossible
to unit test the code in question.  The correct approach to
do the same thing in Django management commands is to raise
CommandError.
2019-05-17 11:34:00 -07:00
Tim Abbott 68255fe1df lint: Fix exclusion for development directory.
I forgot about running `test-tools` when making linter changes.
2019-05-15 13:40:33 -07:00
Tim Abbott 7f2fcea794 lint: Ban i18n usage under zerver/views/development. 2019-05-15 13:11:04 -07:00
Hemanth V. Alluri bae8295c52 devtools: Add integrations dev panel.
This commit adds a new developer tool: The "integrations dev panel"
which will serve as a replacement for the send_webhook_fixture_message
management command as a way to test integrations with much greater ease.
2019-05-15 13:07:44 -07:00
Vishnu Ks 06983298ba export: Add support for exporting realm with member consent.
This lets us handle directly in our tooling the user experience that
we document for exporting a realm with member consent (before, it
required unpleasant manual work).
2019-05-15 12:35:32 -07:00
Tim Abbott 5ec8f6e812 test_migrations: Disable migrations test.
Our migrations test suite only really works when no future migrations
conflict with it.
2019-05-12 22:06:17 -07:00
Tim Abbott 7c94d350d4 ci: Run test-backend with limited parallelism.
See the comment for details; this should fix timeouts we've been
seeing in CI.
2019-05-08 18:26:26 -07:00
Harshit Bansal b553507412 subscriptions: Migrate notification setting defaults model.
This commit migrates the Subscription's notification fields from a
BooleanField to a NullBooleanField where a value of None means to
inherit the value from user's profile.

Also includes a migrations to set the corresponding settings to None
if they match the user profile's values. This migration helps us in
getting rid of the weird "Apply to all" widget that we offered on
subscription settings page.

The mobile apps can't handle None appearing as the stream-level
notification settings, so for backwards-compatibility we arrange to
only send True/False to the mobile apps by applying those defaults
server-side.  We introduce a notification_settings_null value within a
client_capabilities structure that newer versions of the mobile apps
can use to request the new model.

This mobile compatibility code is pretty effectively tested by the
existing test_events tests for the subscriptions subsystem.
2019-05-08 17:45:10 -07:00
Wyatt Hoodes ececf0a209 provision.py: Add a conditional to remove '.eslintcache'.
It was discovered that the '.eslintcache' file was causing eslint to
throw a TypeError after a recent update/addition to the dependencies.
It makes sense to remove this file as part of the provisioning process
to avoid such exceptions.
2019-05-08 17:06:17 -07:00
Vishnu Ks 6c58603eaf support: Add support for scrubbing realm. 2019-05-06 20:12:54 -07:00
Mayank Madan 99414e2d96 provision: Add support for Debian 10 Buster. 2019-05-05 17:57:19 -07:00
Raymond Akornor e01d3be1ba tools: Remove check-urls linter.
This commit removes `tools/check-urls`. It was added as
a useful tool in preparation for the Django 1.10 migration.
Since we completed that migration, it is no longer needed.

Fixes #12180.
2019-05-05 17:47:41 -07:00
Wyatt Hoodes 21c67ea1c2 test-backend: Change --parallel default to use `cpu_count`.
The number of processes to run the backend tests is currently a
hardcoded value, this commit transistions the default to be based on the
number of logical CPUs available.
2019-05-02 10:48:20 -07:00
Yashashvi Dave 3649a9f15c subs: Add `stream_ui_updates.js` module for managing ui elements.
This commit adds `stream_ui_updates.js` module. This module
will includes functions which will update different ui elements
(i.e. subscription button, subscriber count).
2019-04-30 14:36:52 -07:00
Eeshan Garg a73e8109b7 webhooks: Remove the legacy GitHub integration.
The github-services model for how GitHub would send requests to this
legacy integration is no longer available since earlier in 2019.
Removing this integration also allows us to finally remove
authenticated_api_view, the legacy authentication model from 2013 that
had been used for this integration (and other features long since
upgraded).

A few functions that were used by the Beanstalk webhook are moved into
that webhook's implementation directly.
2019-04-27 15:13:44 -07:00
Vishnu Ks 16394e53a3 tools: Make dependency upgrader upgrade indirect dependencies separately. 2019-04-26 16:22:00 -07:00
Anders Kaseorg 122ad5be58 lint: Lint against calls to _() on computed strings.
These don't actually work from an i18n perspective.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:23:18 -07:00
Anders Kaseorg 44dd7c2d95 lint: Strengthen lint checks for string % non-tuple.
This is really a job for an AST parser rather than a pile of regexes;
among other issues, these will still miss violations that span
multiple lines.  But, you know, I tried.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:22:51 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Rafid Aslam d983611919 template_parser: Add support for jinja2 whitespace markers.
Fixes #11516.
2019-04-23 11:52:13 -07:00
Anders Kaseorg 5290519a62 scripts: Always use ON_ERROR_STOP=1 when running psql.
Also use psql -e (--echo-queries) in scripts that use ‘set -x’, so
errors can be traced to a specific query from the output.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:19 -07:00
Anders Kaseorg efd22fd2ae postgres-init-dev-db: Catch exception when recreating extant users.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:18 -07:00
Anders Kaseorg 25798c8605 linter_lib: Remove sudo exemption for terminate-psql-sessions.
terminate-psql-sessions no longer uses sudo as of #11771.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:17 -07:00
Anders Kaseorg 12e2e0d658 tools/webpack: Run webpack in the foreground.
`tools/run-dev.py` already backgrounds `tools/webpack` (and deals with
cleaning it up on exit), so there’s no need for `tools/webpack` to
also background the actual `webpack` process.  But when running
`tools/webpack` by itself, it’s annoying to clean up the backgrounded
process manually.

Run `webpack` in the foreground, using `os.execvp` so we don’t waste
memory on an intermediate wrapper process.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 13:59:11 -07:00
Anders Kaseorg dd3ccbc897 webpack: Fix node_modules path resolution.
Webpack applies special logic to relative paths provided in
`resolve.modules`, and this logic is expected to be used for
`node_modules`.  One case where this is important is when
`node_modules/foo` wants to import a different version of package
`bar` than the one at `node_modules/bar`, and so yarn gives it its own
copy at `node_modules/foo/node_modules/bar`.

It would probably be better to avoid screwing with `resolve.modules`
at all, but this at least brings us one step closer to the default of
just `["node_modules"]`.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 13:49:23 -07:00
Anders Kaseorg 9fc0aa7ffa scripts: Fix exec invocation for in-process virtualenv activation.
activate_this.py has always documented that it should be exec()ed with
locals = globals, and in virtualenv 16.0.0 it raises a NameError
otherwise.

As a simplified demonstration of the weird things that can go wrong
when locals ≠ globals:

    >>> exec('a = 1; print([a])', {}, {})
    [1]
    >>> exec('a = 1; print([a for b in [1]])', {}, {})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
      File "<string>", line 1, in <listcomp>
    NameError: name 'a' is not defined
    >>> exec('a = 1; print([a for b in [1]])', {})
    [1]

Top-level assignments go into locals, but from inside a new scope like
a list comprehension, they’re read out of globals, which doesn’t work.

Fixes #12030.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-16 17:42:44 -07:00
Tim Abbott 31f6ace084 lint: Fix typo in eslint command line. 2019-04-13 22:45:07 -07:00
Thomas Ip 90478fc4b2 typescript: Fix violations of new typescript rules. 2019-04-13 11:42:48 -07:00
Thomas Ip 02cb85a8a9 typescript: Use ESLint instead of TSLint. 2019-04-13 11:42:47 -07:00
Thomas Ip ea9a74fe24 typescript: Fix violations of existing eslint rules. 2019-04-13 11:38:19 -07:00
Anders Kaseorg 53879c4673 scripts: Rename DEFAULT_USER to POSTGRES_USER.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Marco Burstein ce7d2fde70 tools: Add TypeScript to the dependency visualizer.
The dependency visualizer currently only supports JavaScript files,
such as in the `get_js_edges` function, where only the ".js" extension
is supported. Update the visualizer to support ".ts" files as well and
to output modules without their extensions.
2019-04-12 11:14:42 -07:00
Marco Burstein 57ffc3fe74 tests: Update JS Node tests to support TypeScript.
Currently, the `test-js-with-node` tests append ".js" to filenames
without an extension. Since Typescript is now also supported, it can
produce results such as "dict.ts.js". To remedy this, check for ".ts"
files as well.
2019-04-12 11:14:42 -07:00
sumanthvrao c5de6d0e18 spiders: Set error status for exceptions generated by spider.
Spider raises exceptions when errors like FileNotFound
are detected. However, these did not set error state
before exiting causing spider to fail silently.

This patch sets the status causing exceptions to exit with
non-zero exit status.
2019-04-08 17:04:29 -07:00
Abhinav Singh c27d927663 refactor: Remove inline javascript code in email_log.html.
All the inline javascript code present in email_log.html(which is
rendered when the user visits "/emails" in development mode) is
transferred to a new file: email_log.js in portico/ directory.

Fixes #11608.
2019-04-05 17:28:23 -07:00
Anders Kaseorg 11f18042fe run-dev-queue-processors: Remove
As of #367, `tools/run-dev-queue-processors` has evolved into nothing
more than an unnecessarily elaborate wrapper around `manage.py
process_queue --all`.  Remove it (mostly to make it marginally easier
to Tab-complete `tools/run-dev.py`, if I’m being honest).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-05 15:18:38 -07:00
Thomas Ip b8e1b8d6ff coverage: Bring dict.ts line coverage back to 100%.
The delete operator could throw a TypeError when attempting to
remove a non-configurable property, which is rare in practice since
they can only be created using `Object.defineProperty()` and
`Object.freeze()`. We also never uses the output of `del()` anyway.
2019-03-30 17:12:50 -07:00
Thomas Ip bec31c6791 coverage: Get Istanbul to recognise typescript modules. 2019-03-30 17:12:50 -07:00
Abhinav Singh d52e3b0d70 refactor: Remove inline javascript code in dev_login.html.
A new javascript file "dev-login.js" is created in static/js/portico/
and the inline javascipt code present in dev_login.html is transferred
to that file. An empty div element is added in dev_login.html with
unique data-page-id attribute to make it more easy to find in which
page we are, while working with the javascript code.
2019-03-25 15:14:24 -07:00
Thomas Ip a2872c107e typescript: Move TS files into JS directory.
This is just a code reorganization to avoid making it difficult to
find things as we migrate more file to TypeScript.
2019-03-25 12:11:37 -07:00
Rhythm Sharma e1212c04d1 emoji: Add a few useful aliases. 2019-03-17 13:38:31 -07:00
Tim Abbott 50dc317466 notifications: Rename notifications.py to email_notifications.py.
This library is entirely about email notifications specifically, and
this rename should help make the codebase more readable.
2019-03-15 11:02:17 -07:00
Steve Howell ba0418c989 stream_data: Remove untested useless code.
This code was actually dead, due to the .empty() check above.  Fixing
this lets us restore 100% coverage.
2019-03-15 09:27:41 -07:00
Raymond Akornor ea112828dc mypy: Use python3 type syntax in test-backend. 2019-03-15 09:22:30 -07:00
Raymond Akornor 193de819b8 test-backend: Refactor test-backend to be more pythonic. 2019-03-15 09:22:30 -07:00
Tim Abbott 76891f6a02 node: Disable a failing coverage check. 2019-03-14 22:22:06 -07:00
Tim Abbott cdf987fb14 push-translations: Run manage.py makemessages first.
This saves an extra step in pushing translations to transifex, with no
real negative side effects.
2019-03-13 14:11:27 -07:00
Vishnu Ks 8eeb8280b4 activity: Create interface for doing support operations.
This should grow into a tool that makes it much easier to do common
organization management tasks without using a manage.py shell.
2019-03-11 12:01:11 -07:00
Mayank Madan 264c2e9803 provision: Add provision support for Ubuntu 18.10.
Note that this is only support for Ubuntu 18.10 in the development
environment.
2019-03-09 16:01:46 -08:00
vinitS101 4523c8d7ee typo: Change "generage" to "generate". 2019-03-09 15:49:38 -08:00
Tim Abbott 0ab8cd9ca3 provision: Refactor handling of dependencies for building pgroonga.
This lets us leave build-pgroonga platform-agnostic.
2019-03-08 15:12:56 -08:00
Tim Abbott 5c20ee998c build-tsearch-extras: Add support for Debian/Ubuntu. 2019-03-08 15:12:56 -08:00
Tim Abbott 32d26c8aad provision: Clean up building search extensions from source.
This generalizes the provision logic for deciding whether to build our
tsearch_extras and pgroonga search extensions from source to support
Ubuntu cosmic as well (and evenutally, other future platforms).
2019-03-08 15:12:56 -08:00
Tim Abbott 8469c7774f scripts: Remove create-zulip-admin.
This was used only by ancient versions of the Docker project.
2019-03-04 15:33:51 -08:00
Tim Abbott 500b5cdaf1 tools: Fix missing comma in collectstatic configuration.
It appears that this code did the right thing despite being written
wrong, probably due to whatever `manage.py collectstatic` does in its
argument parsing.  But in any case, we should make the code read how
it's intended.
2019-03-04 08:31:12 -08:00
Rohitt Vashishtha 3d427d02cc scripts/zulip_tools: Use run_as_root instead of subprocess.check_call. 2019-03-01 11:21:16 -08:00
Rohitt Vashishtha ac48925977 scripts: Use run_as_root instead of run([sudo, ...]). 2019-03-01 11:21:16 -08:00
Wyatt Hoodes 5231b27dea popovers: Add a clear status option in user popover.
Accomplished by adding a function to clear the status message with
an empty string. The html is then updated to reflect changes without a
refresh.

Currently, it's a small hassle to clear a status message.  This option
makes things a bit easier.

Fixes #11630.
2019-02-26 14:40:17 -08:00
Anders Kaseorg 9faa009f66 lint: Prohibit unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:50 -08:00
Anders Kaseorg f1ec67c614 python: Mark intentionally unused imports with noop statements.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:47 -08:00
Wyatt Hoodes 2580965284 test-backend: Rename '--process' option to '--parallel'.
This change is being made to simply match up our naming conventions
with django's test framework.
2019-02-20 13:25:49 -08:00
Raymond Akornor 4dc7f5354d tests: Optimize test-backend performance.
This optimizes test-backend by skipping webhook
tests when run in default mode.

Tweaked by tabbott to extend the documentation and update the CI
commands.
2019-02-19 14:50:41 -08:00
Tim Abbott fe4a0d0fcd capitalization: Fix night logo lint errors. 2019-02-18 15:53:55 -08:00
Steve Howell e67cf30dfd private messages: Add user circles to top left.
This is mostly adding markup, calling some convenient
functions in buddy_data.js, and adjusting CSS.

To make the circles update dynamically, I mostly
orchestrate this though activity.js for now.  It's
possible we'll want to adjust that eventually to
happen through something like a `presence_events`
dispatcher, but that's essentially what
a good part of `activity.js` does now.
2019-02-18 14:22:37 -08:00
Anders Kaseorg dec8c7dba4 provision: Run generate_secrets earlier.
Commit 7d12e2019d (#10994) broke fresh
provisions by importing zproject.settings before we were done
modifying settings.  Fix it by moving the generate_secrets invocation
to the earliest reasonable place.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-08 10:13:18 -08:00
Tim Abbott a2373a63cd capitalization: Add GCM to list of proper nouns.
This fixes failures in master from the recent GCM merge.
2019-02-08 10:11:55 -08:00
Anders Kaseorg 8fdb62a6bc tools: Create DEPLOY_ROOT/zulip-py3-venv symlink in development too.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:12:33 -08:00
Vishnu Ks b62bd83083 invites: Add generate multiuse invite button. 2019-02-07 15:41:00 -08:00
Rishi Gupta 36472413e4 hotspots: Add hotspot for gear menu. 2019-02-05 12:19:21 -08:00
Vishnu Ks 4b1229c072 billing: Add node tests for upgrade.js. 2019-02-04 22:37:05 -08:00
Anders Kaseorg ee8ff4df66 tools: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:10:31 -08:00
Hemanth V. Alluri 73d26c8b28 streams: Render and store the stream description from the backend.
This commit does the following three things:
    1. Update stream model to accomodate rendered description.
    2. Render and save the stream rendered description on update.
    3. Render and save stream descriptions on creation.

Further, the stream's rendered description is also sent whenever the
stream's description is being sent.

This is preparatory work for eliminating the use of the
non-authoritative marked.js markdown parser for stream descriptions.
2019-02-01 22:24:18 -08:00
Anders Kaseorg 893e1475bb setup_virtualenv: Do not activate the virtualenv on creation.
Instead, manually activate it in the one place where this
functionality was used (tools/lib/provision.py).  This way we avoid
trying to activate the Python 2 thumbor virtualenv from Python 3.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-31 17:55:04 -08:00
Greg Price 6613f4c22c test-install/install: Print usage when run without arguments.
The `-z "$INSTALLER"` was intended to do this -- but we don't get that
far, because the `shift` fails.
2019-01-31 16:15:51 -08:00
Steve Howell c665f809f2 run-dev: Make address-in-use errors more obvious.
This seems to be a common enough pitfall to justify
a bit of extra handling.  Example output:

    $ ./tools/run-dev.py
    Clearing memcached ...
    Flushing memcached...
    OK
    Starting Zulip services on ports: web proxy: ...
    Note: only port 9991 is exposed to the host in a Vagrant environment.

    ERROR: You probably have another server running!!!

    Traceback (most recent call last):
      File "./tools/run-dev.py", line 421, in <module>
        app.listen(proxy_port, address=options.interface)
      File "/srv/zulip-py3-venv/lib/python3.5/...
        server.listen(port, address)
      File "/srv/zulip-py3-venv/lib/python3.5/...
        sockets = bind_sockets(port, address=address)
      File "/srv/zulip-py3-venv/lib/python3.5/...
        sock.bind(sockaddr)
    OSError: [Errno 98] Address already in use
    Terminated
2019-01-29 16:03:47 -08:00
Steve Howell 1228e541d7 dispatch test: Restore 100% line coverage (user_status).
We now have 100% coverage on server_events_dispatch.js.

The last piece were was covering "status_text" updates
in the "user_status" event.
2019-01-26 10:48:57 -08:00
Steve Howell bacf896228 poll widget: Clean up code and add edit controls.
NOTE: If you revert this commit, you want to revert
the immediately prior commit as well.  The history
is that Ishan made some improvements to the widget,
but there were some minor bugs.  I decided not
to squash the commits together so that the git
history is clear who did what.  (In particular, I
want questions about the JS code to come to me if
somebody does `git blame`.)

Anyway...

This is a fairly significant rewrite of the polling
widget, where I clean up the overall structure of
the code (including things from before the prior
fix) and try to polish the prior commit a bit as
well.

There are a few new features:

    * We tell "other" users to wait for the poll
      to start (if there's no question yet).
    * We tip the author to say "/poll foo" (as
      needed).
    * We add edit controls for the question.
    * We don't allow new choices until there's
      a question.
2019-01-22 10:27:39 -08:00
Harshit Bansal 2944075f70 lint: Respect `--force` flag while running mypy. 2019-01-17 10:34:43 -08:00
Anders Kaseorg 25d6107a4f docs: Avoid hardcoded /tmp paths in miscellaneous documentation.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:16:12 -08:00
Anders Kaseorg 79d888223f profiling: Use mkstemp for profile.data filename.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:08:48 -08:00
Anders Kaseorg f3c9b87c30 test-locked-requirements: Avoid hardcoded paths in /var/tmp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:51 -08:00
Anders Kaseorg b4e1403cf9 test-install: Avoid hardcoded paths in /tmp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:51 -08:00
Anders Kaseorg c17ddceeae test-emoji-name-scripts: Avoid hardcoded paths in /var/tmp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:50 -08:00
Anders Kaseorg df7b63cd5e tools: Remove check-css.
As of commit 0042694e24 (#10017), this
is unused.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:50 -08:00
Anders Kaseorg f099fd9e28 pretty_print: Replace hardcoded path in /var/tmp with a pipe.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Anders Kaseorg 6678a95b20 test-queue-worker-reload: Use a pipe rather than polling a log file.
This avoids unnecessary polling delays, quadratic string operations,
and hardcoded paths in /tmp.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Anders Kaseorg 19429c3ad7 test-run-dev: Use a pipe rather than polling a log file.
This avoids unnecessary polling delays, quadratic string operations,
and hardcoded paths in /tmp.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Tim Abbott e29533df0f emails: Fix duplicate <html> blocks in missed-message emails.
Apparently, the missed-message templates have a slightly different
structure from our other email templates, which triggered a latent,
subtle bug in inline-email-css's effort to remove duplicate <html>
blocks from emails that had been generated by premailer.  Fix this
bug, and add appropriate assertions to prevent similar issues in the
future.

Fixes #11249.
2019-01-11 14:23:30 -08:00
Anders Kaseorg 2ad1a27b28 create-production-venv: Remove Python 2 support.
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:05 -08:00
Vishnu Ks c263674507 billing: Add node test for billing.js. 2019-01-08 10:47:35 -08:00
rht 15763f8545 provision: Include DISTRIB_FAMILY in parse_lsb_release output. 2019-01-07 18:52:09 -08:00
Vishnu Ks 13de959121 tools: Add script to automate python dependency upgrade. 2019-01-07 10:25:35 -08:00
Marco Burstein 9ddadd39f4 compose: Add support for using Zoom as the video chat provider.
This adds Zoom call properties to the `Realm` model, creates endpoints
for creating calls, adds a frontend and tests.

Fixes #10979.
2019-01-07 10:00:02 -08:00
Mateusz Mandera 36714342a3 lint: Allow usage of the word subject in test_email_mirror.py. 2019-01-05 15:59:19 -08:00
Tim Abbott 99721755e8 node: Temporarily remove composebox_typeahead from fully covered. 2019-01-05 15:31:30 -08:00
Tim Abbott 453b65bde3 provision: Clean up comments and unused variables.
This has some changes inspired by #8514.
2019-01-04 15:16:19 -08:00
rht 7e1520b01f provision: Ensure `python3` exists in PATH for CentOS. 2019-01-04 14:11:03 -08:00
Vishnu Ks d94b0da3c8 billing: Separate billing.js into multiple modules. 2019-01-04 23:52:00 +05:30
Vishnu Ks 81e6a022e6 emails: Inline CSS in missed_message.
As part of this change, we port into the .messages class the work in
4e8e7348da to change overflow-y to auto,
not scroll (skipping that would result in a regression).
2019-01-03 17:55:29 -08:00
Vishnu Ks 4e1c0589c2 emails: Refactor if condition from style tag in missed_message_email. 2019-01-03 17:55:29 -08:00
Tim Abbott 4df618f584 portico: Order jquery.validate before signup.js.
This is important, because the signup.js module uses jquery.validate.
2019-01-03 15:22:57 -08:00
Steve Howell d8d703af45 frontend: Add basic user_status module.
So far this processes page_params, but it's otherwise
an unused internal API.
2019-01-02 09:16:31 -08:00
Steve Howell 4d5ce0b285 lint: Exclude "subject" from JS files (except util.js).
We also exempt the frontend tests for now.
2019-01-01 20:52:07 -08:00
Steve Howell b4bc6b4445 emails: Fix subject -> email_subject in send_email.py. 2019-01-01 20:49:00 -08:00
Steve Howell 4da28efc90 emails: Fix subject -> email_subject in error_notify.py. 2019-01-01 20:48:59 -08:00
Steve Howell 16328732d4 refactor: Remove "subject" from stream_topic.py. 2019-01-01 20:48:56 -08:00
rht d3139266c8 setup_venv: Uninstall typing on Python >3.4.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
2019-01-01 19:34:48 -08:00
rht c190959b58 provision: Move CentOS & Fedora venv deps construction to setup_venv.py.
They will be reused in production setup.
2019-01-01 19:34:46 -08:00
rht 17ffbdabe2 provision: Use virtualenv instead of python-virtualenv on Fedora.
The package name changed at some point.
2019-01-01 19:34:24 -08:00
rht 75134ef614 provision: Use generic python3 on Fedora dependencies.
It might be cleaner to add new variables in setup_venv.py, but we can
do that later.
2019-01-01 19:34:11 -08:00
Tim Abbott 5a8ad84381 test-documentation: Exclude the 500ing Citizen Code of Conduct.
We might want to eventually remove this, but I'm holding out hope that
it'll be fixed.
2018-12-29 15:30:19 -08:00
Vishnu Ks 6f5b1903c4 docs: Update code-reviewing docs to reflect CI changes. 2018-12-29 15:28:26 -08:00
Steve Howell 3fc8597119 popovers: Avoid passing message to actions popover template.
We instead get the specific fields from message
that we use.  This is particularly helpful
for subject -> topic migration; we no longer
have to account for "subject" fields in
client-side templates.
2018-12-29 14:19:18 -08:00
Sumanth V Rao ccd0badeed test-documentation: Simplify --skip-external-links code path.
The crawler used to be called directly for checking external links.
Now the scrapy command calls the crawl_with_status wrapper.
Crawl_with_status has been modified to pass the external parameter in
the previous commit, so we can now use this simpler approach.
2018-12-29 09:09:18 -08:00
Sumanth V Rao 776b981dde test-help-documentation: Skips checking external links.
This option causes test-help-documentation to verify internal links only.
This prevents documentation from flaking in CI due to links from
external websites having failed in the documentation.

Based on conversation:
https://chat.zulip.org/#narrow/stream/43-automated-testing/subject/quora.20404/near/674997
2018-12-29 20:15:30 +05:30
Rishi Gupta e7220fd71f billing: Do subscription management in-house instead of with Stripe Billing.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.

To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
2018-12-22 13:39:30 -08:00
Rishi Gupta f8720fddd6 emails: Update text of notify_new_login. 2018-12-20 16:26:19 -08:00
Vishnu Ks f9e4d5d28e emails: Tag password_reset for translation. 2018-12-19 09:21:09 -08:00
Vishnu Ks 0960e654ab emails: Tag notify_new_login email for translation. 2018-12-19 09:21:09 -08:00
rht acbb174100 provision: Add RHEL 7 support. 2018-12-18 17:13:56 -08:00
rht 84c968ab36 provision: Include scripts/lib/setup-* and scripts/lib/build-* for sha1 sum check. 2018-12-18 16:56:27 -08:00
Tim Abbott dd946fff1f lint: Improve ui_report.error linter rule.
We now allow passing "" as the first argument.
2018-12-18 12:33:15 -08:00
rht a093d45064 provision: Check for pg_hba.conf existence for doing postgres initdb.
Apparently, an empty pg_datadir is initialized after postgresql-10 is
installed.
2018-12-18 12:07:45 -08:00
rht 3060ee9397 provision: Fix typo when checking that vendor is Fedora. 2018-12-18 12:07:03 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
rht 5e12eb8d34 provision: Fix typo in list of services on RedHat. 2018-12-17 16:30:48 -08:00
rht 42cbf2071b provision: Skip RedHat postgresql initdb if already initialized.
We only need to initialize a postgres database once in the lifetime of
a system.
2018-12-17 16:29:50 -08:00
rht 295cbe7b9c provision: Add build-pgroonga and use it for Fedora. 2018-12-17 16:23:57 -08:00
rht 80ba105be5 provision: Abstract out SYSTEM_DEPENDENCIES. 2018-12-17 16:23:46 -08:00
rht b732fe819e provision: Add Fedora support. 2018-12-17 16:23:44 -08:00
Tim Abbott d469c14d11 docs: Advertise arbitrary org GitHub patterns for linkifiers.
Now that we support this, we should make it really obvious that we do.

Also, link to the Help Center article.
2018-12-17 12:58:52 -08:00
rht 2dc1dc8ec2 provision: Replace overwrite_symlink with `sudo ln -sf`.
overwrite_symlink doesn't work for a non-root user when accessing root
content.
2018-12-17 11:50:15 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott 209df75ffa lint: Enforce use of Python 2 style typing in zulint.
This is intended to be extracted as an external project that supports
Python 2+3.
2018-12-17 11:00:08 -08:00
Tim Abbott 35901cc955 lint: Fix a tooling test failure in last commit.
This is what I get for copy-pasting lint rule blocks.
2018-12-17 10:53:08 -08:00
Tim Abbott bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
Tim Abbott 6b69cc0b39 lint: Run mypy as part of main linter.
To support this, we add a pass_targets option to the main linter
library, because with current mypy, it's generally counterproductive
to pass the list of files in (can produce spurious errors; isn't
faster).
2018-12-16 21:52:53 -08:00
Tim Abbott 3003517430 docs: Move mypy documentation from contributing to testing.
The testing section is more appropriate, since it's fundamentally part
of our CI system.

While we're at it, fix the fact that we were linking to GitHub, not
ReadTheDocs, in the run-mypy output.
2018-12-16 21:52:53 -08:00
Tim Abbott 8637bdeb30 mypy: Provide a clear message when starting daemon.
Previously, when the mypy daemon was starting up, it was very unclear
from the linter context what was happening.
2018-12-16 21:52:53 -08:00
Tim Abbott f3c6d91e69 mypy: Switch default to daemon mode.
There isn't any real advantage to running the non-daemon mode at this
point.
2018-12-16 21:37:58 -08:00
Tim Abbott 7485cb2a50 widgets: Rename voting_widget to poll_widget.
This ensures greater consistency with our other widgets' naming
convention.
2018-12-16 19:46:48 -08:00
Tim Abbott adebe1bd4e js: Extract csrf.js and include in common bundle.
This should make it possible to use this AJAX setup code in logged-out
code as well, which is necessary to use blueslip from portico pages.
2018-12-16 16:18:42 -08:00
Tim Abbott ed9430f3a6 Revert "js: Extract csrf.js and include in common bundle."
This reverts commit dbffa66b6d.

This broke the logged-in app, due to some webpack issue.
2018-12-16 16:16:06 -08:00
Tim Abbott dbffa66b6d js: Extract csrf.js and include in common bundle.
This should make it possible to use this AJAX setup code in logged-out
code as well, which is necessary to use blueslip from portico pages.
2018-12-16 15:35:54 -08:00
Tim Abbott 9059c4d223 provision: Make apt/yum retry logic more coherent.
This changes a few things:
* Deplicates deps_to_install logic.
* Has a retry flag, under which we can guard the apt retry print statements.
* Makes the install_system_deps flow more parallel.
2018-12-16 13:41:12 -08:00
rht c6cde1cf5c provision: Rename install_apt_deps to a more generic install_system_deps. 2018-12-16 13:39:00 -08:00
rht 389c46b4e5 provision: Extract yum-specific part of install_apt_deps. 2018-12-16 13:39:00 -08:00
rht 2be3065249 provision.py: Abstract out postgres version for CentOS. 2018-12-16 13:26:28 -08:00
rht 43bedc0909 provision: Use vendored pg_hba.conf on CentOS. 2018-12-16 13:21:54 -08:00
Tim Abbott f2173bc2a7 check-capitalization: Add a few exclude rules for new strings. 2018-12-16 11:34:31 -08:00
rht 022e926d64 provision: Setup services for CentOS. 2018-12-16 10:14:51 -08:00
rht 9c776f4fcf provision: Provide tsearch stopwords path for CentOS. 2018-12-16 10:14:08 -08:00
rht 8b5a86b992 provision: Initialize Postgres db on CentOS. 2018-12-16 10:11:55 -08:00
rht 41a5f6a49d provision: Create symlink for pg_* on CentOS. 2018-12-16 10:11:54 -08:00
rht 10786375db provision: Do not use apt-specific retry when a failure happens. 2018-12-16 10:04:27 -08:00
rht 25a8dbef11 provision: Build tsearch_extras from source for CentOS support. 2018-12-16 10:04:27 -08:00
rht 448303b3f0 provision: Initialize setup-yum-repo. 2018-12-16 10:04:27 -08:00
rht 85b2627191 provision: Add experimental support for CentOS. 2018-12-16 10:04:27 -08:00
rht 79bcb4af47 provision: Remove unused comment. 2018-12-16 10:04:27 -08:00
Steve Howell d75ff80eb2 muting: Add stream_id to markup for mute settings.
We also prefer `attr` over `data` (it's more greppable).
2018-12-14 16:05:35 -08:00
Steve Howell bf6f5e7bc5 muting: Pass stream_id to muting_ui.mute and unmute.
We temporarily allow settings_muting to have
incomplete line coverage--we will fix this soon.
2018-12-14 16:02:48 -08:00
rht 501ae0c3d6
provision: Add centos7 postgresql dependencies. 2018-12-14 04:59:48 +00:00
rht e8c602ec58
provision: Rename APT_DEPENDENCIES -> SYSTEM_DEPENDENCIES. 2018-12-14 04:59:48 +00:00
rht d54fb5f40d
provision: Add venv dependencies for Centos 7. 2018-12-14 04:59:46 +00:00
Vishnu Ks 0fd6ff722b billing: Migrate /upgrade endpoint to JSON.
The fixture changes are because self.upgrade formerly used to cause a page load
of /billing, which in turn calls Customer.retrieve.

If we ran the full test suite with GENERATE_STRIPE_FIXTURES=True, we would
likely see several more Customer.retrieve.N.json's being deleted. But
keeping them there for now to keep the diff small.
2018-12-13 17:01:12 -08:00
Anders Kaseorg 149132348f lint: Update ShellCheck to 0.6.0.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-12 08:24:01 -08:00
rht a0b0ccf972 provision: Do not require lsb_release for centos. 2018-12-11 13:08:27 -08:00
rht c9d54f7854 puppet: Remove vendored puppetlabs apt and stdlibs dependencies.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.

Fixes #7423.
2018-12-11 13:03:26 -08:00
Vishnu Ks a86faecff5 tools: Rename tools/travis directory to tools/ci. 2018-12-10 11:07:26 -08:00
konraddeka e6be407e34 provision: Avoid spending time on update-authors-json.
This optimizes tools/provision by not running
`tools/update-authors-json --use-fixture` unless either the script
itself or its fixtures file (zerver/tests/fixtures/authors.json) was
changed.

Fixes #10991.
2018-12-09 22:21:51 -08:00
rht 9bfea2adba provision.py: Translate COMMON_APT_DEPENDENCIES to yum. 2018-12-09 21:54:02 -08:00
Sumanth V Rao 7d12e2019d provision: Generate static bots files using a function call.
This shaves about 350ms from the no-op provision runtime.

Fixes #10990.
2018-12-09 21:44:45 -08:00
Sumanth V Rao cae718f184 provision: Rename generate_zulip_bots_static_files to .py.
This should make it possible to call this script as a function without
starting up a new Python interpreter.
2018-12-09 21:42:40 -08:00
Tim Abbott 9e48b2f17f provision: Update comment for why we need puppet in development.
This should help avoid confusion like #10972.
2018-12-06 11:13:04 -08:00
Tim Abbott 9de1bd44e2 compatibility: Clean up linter logic for too-old client. 2018-12-05 15:04:21 -08:00
Greg Price fb7bfbe9ab compatibility: Require version 16.2.96+ for our Android app.
This release is from 2018-08-22, a little over 100 days ago.

It was the first release with the important fix so that when the
server advises it to stop displaying a notification because the user
has read the message (as the SEND_REMOVE_PUSH_NOTIFICATIONS server
setting enables), the app doesn't instead replace the notification
with a broken one reading "null".  We have that setting running now
on chat.zulip.org, and intend to roll it out more broadly soon.

The `# take 0` thing is a slightly absurd workaround for the fact
that our funky out-of-line way of marking lines to ignore doesn't
work right if there are multiple such lines in a given file that
are equal modulo leading and trailing whitespace.
2018-12-05 14:58:13 -08:00
Tim Abbott 6689913f77 ci: Enable test-documentation --skip-external-links in CI.
Fixes #10942.
2018-12-05 12:39:59 -08:00
Sumanth V Rao e1f33e0f6b test-documentation: Add --skip-external-links option.
This option causes test-documentation to only verify internal links
that we control and are important to be correct.  This prevents
test-documentation from flaking in CI due to issues with the dozens of
third-party blocks that we link to from various parts of our
documentation.

Tweaked by tabbott for comment clarity, and to also include
github.com/zulip links.

Fixes #10942.
2018-12-05 12:39:59 -08:00
Tim Abbott f5f320d5fc mypy: Print out link to documentation in error output.
This should help make details about how mypy works discoverable for
developers who are new to it.
2018-12-05 11:23:55 -08:00
Tim Abbott f694d01a1b test-backend: Expand desired files with full test coverage.
This should help catch a few classes of files that we weren't
correctly considering for potentially having full test coverage.
2018-12-05 11:01:55 -08:00
Steve Howell 67fba69b0c hashchange: Go to home page for bogus narrows.
This was the original intent of the code, and I think
it's the right behavior.
2018-12-04 17:16:38 -08:00
Steve Howell bcb142e68e hashchange: Move parse_narrow to hash_util.js.
The goal here was to enforce 100% coverage on
parse_narrow, but the code has an unreachable line
and is overly tolerant of bogus urls.  This will
be fixed in the next commit.
2018-12-04 17:16:32 -08:00
Anders Kaseorg 9ba860b4f2 Avoid double redirects to /login and then to /login/.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-04 16:12:58 -08:00
Tim Abbott 86536059b4 tools: Fix linter tests after SCSS rename.
This was introduced in 815388b023.
2018-12-04 09:43:56 -08:00
Vishnu Ks 788b98d041 portico: Add page for redirecting to a realm subdomain. 2018-12-04 09:35:35 -08:00
Tim Abbott 815388b023 lint: Fix CSS linter rules not running.
Apparently, our CSS linter rules haven't been running since we
migrated to SCSS.
2018-12-04 09:35:34 -08:00
Vishnu Ks 6de6b0ed31 emails: Extract CSS to email.css in digest.html. 2018-12-03 17:43:53 -08:00
Tim Abbott 07adda84da Revert "ci: Re-enable testing documentation links."
This reverts commit d8e5e2d960.

See https://github.com/zulip/zulip/issues/10942 for a plan for making
this something we can re-enable.
2018-12-02 18:34:35 -08:00
Tim Abbott 1303f8df20 docs: Fix missing quotes in `su zulip -c` documentation.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
2018-11-30 12:12:27 -08:00
Tim Abbott a3b38d8402 docs: Document nginx configuration for development SSL.
This is a useful technique for developing OAuth integrations with Zulip.
2018-11-29 20:45:02 -08:00
Steve Howell ba243416e3 stream_color: Make stream color assignment more efficient.
We now let color_data keep its own state for
unused_colors, so that we longer have to pass in
a large list of unused_colors every time we want
to assign a new stream color.

This mostly matters at startup, where we might
be cycling through 5000 streams.  We claim all
the unused colors up front.

Each operation now has an upper bound of expensiveness,
where the worst case scenario is basically popping
off the first element of a list of <= 24 colors.

The algorithm is now deterministic, too, to make
it easier to test.  It's unclear whether random color
assignment ever had much benefit, and it made unit
testing the algorithm difficult.  Now we have 100%
line coverage.

Fixes part of #10902.
2018-11-29 13:00:25 -08:00
Anders Kaseorg 0bfcd2294e tools/update-prod-static: Avoid shelling out for mkdir, cp, mv, rm.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:28:17 -08:00
Anders Kaseorg 19bdf54f33 tools/setup/emoji/build_emoji: Avoid shelling out for touch, rm, mkdir, cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:28:17 -08:00
Tim Abbott d8e5e2d960 ci: Re-enable testing documentation links.
We've had a few unpleasant bugs with real documentation links being
broken, so we're going to make this more aggressive for now.

I think we instead want a more subtle option for suppressing failures
in some places but not others.
2018-11-28 09:52:04 -08:00
Tim Abbott 723fb60476 docs: Fix up some broken links output spam. 2018-11-28 09:49:39 -08:00
Rishi Gupta dd0126ff1b settings: Improve error message when deactivating the last user.
This PR was originally started by Rishi Gupta (see #10383).
2018-11-27 12:49:43 -08:00
Tim Abbott adf27aae4c python: Remove now-unnecessary str_utils library.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.

Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.

Fixes #10807.
2018-11-27 11:57:54 -08:00
Tim Abbott 58a6dafa68 lint: Exclude outbox from linter checks on "subject". 2018-11-27 11:05:25 -08:00
Tim Abbott d5f53164b4 docs: Suppress markdown line length linter errors. 2018-11-18 18:56:44 -08:00
Steve Howell eb4d279bbc Extract get_turtle_message().
This seems like kind of a silly function to extract
to topic.py, but it will theoretically help us sweep
"subject" if we change the DB.

It had test coverage.
2018-11-14 23:24:06 -08:00
Steve Howell cc33e4cd0c digest: Eliminate unneeded queries for hot convos.
We can easily keep track of messages by bucket from the
original loop through messages.
2018-11-14 23:24:06 -08:00
Steve Howell 5a539d44b0 subject -> topic: Fix notification tests.
The URLs changed here are artificial, so we can convert
subject -> topic without breaking anything.
2018-11-14 23:24:06 -08:00
Steve Howell 27d79352da topic -> subject: Extract get_topic_from_message_info().
This changes files where it's safe to just assume caller
may use either "topic" or "subject", and we prefer "topic"
but support "subject".
2018-11-14 23:24:06 -08:00
Steve Howell 44cbcdf00c minor: Remove txt file from lint exception.
It's already ignored because it's not a .py file.
2018-11-14 23:24:06 -08:00
Steve Howell 2505a2f3cc lint: Build exceptions for subject. 2018-11-14 23:24:06 -08:00
Steve Howell a3460d1ef6 lint: Remove obsolete exemptions.
These two files have been cleaned up since the original
lint rule exemption.
2018-11-14 23:24:06 -08:00
Steve Howell c69c40a55a lint: Extract check_file_for_pattern(). 2018-11-14 23:24:06 -08:00
Steve Howell fd0b249a29 lint: Calculate strip_rule outside the loop.
This adds up over a million lines. :)
2018-11-14 23:24:06 -08:00
Steve Howell f386c64759 lint: Speed up by doing fewer regex compiles.
Even though you'd think these regexes would be
cached, compiling the regex outside of looping
through lines makes a difference.

My timings are 8.4s -> 6.0s.  (You need to hack
on the linter to isolate the custom checks.)
2018-11-14 23:24:06 -08:00
Steve Howell 08e44aac0d lint: Extract get_rules_applying_to_fn(). 2018-11-14 23:24:06 -08:00
Steve Howell eda138343b lint: Extract get_line_info_from_file(). 2018-11-14 23:24:06 -08:00
Steve Howell 6d88210110 lint: Extract check_file_for_long_lines().
In passing this removes some unnecessary string
decoding that's a relic of python2.
2018-11-14 23:24:06 -08:00
Steve Howell a942732d83 lint: Get rid of unused skip_rules.
We also go to the modern annotation style for mypy.
2018-11-12 16:00:52 -08:00
Steve Howell 3eb2b2be5f lint: Change lastLine logic slightly. 2018-11-12 15:47:11 -08:00
Steve Howell a86e7fbd9f subject -> topic: Fix message tests. 2018-11-12 15:47:11 -08:00
Steve Howell c90085b28e lint: Extend "subject" prohibition to zerver/tests.
There are several exemptions here, so I triaged
them into various buckets.
2018-11-12 15:47:11 -08:00
Steve Howell d86dd165da gitter/slack/hipchat: Remove "subject" from conversions.
We (lexically) remove "subject" from the conversion code.  The
`build_message` helper calls `set_topic_name` under the hood,
so things still have "subject" in the JSON.

There was good code coverage on `build_message`.
2018-11-12 15:47:11 -08:00
Steve Howell ea98a44db3 webhooks: Replace SUBJECT_WITH_* with TOPIC_WITH_*. 2018-11-12 15:47:11 -08:00
Steve Howell 97062c4017 lint: Prevent use of "subject" in zerver/views. 2018-11-12 15:47:11 -08:00
Eeshan Garg 10570d508f tools/lint: Fix JSON lint pattern for spaces after ':'.
Previously, a string ending in "... 😄" was reported as an
error and the linter complained that there should be a space
after the last ':'. This commit changes the pattern so that the
linter only checks for colons that are preceded by an opening
double-quote (").
2018-11-08 22:07:56 -08:00
Steve Howell cd4d447d90 Minor: change message.subject -> message.topic_name().
This also allows us to lock down
push_notifications.py from using "subject"
in the future.
2018-11-08 16:34:30 +00:00
Steve Howell ff9a6c5ced minor: Rename subject -> topic_name in bugdown. 2018-11-08 16:21:14 +00:00
Steve Howell 70c16ddbcc lint: Prevent "subject" more aggressively.
We now prevent adding "subject" to any code in
zerver/lib, unless you specifically exempt it.

The new set called `FILES_WITH_LEGACY_SUJECT`
is also has comments that give a roadmap of what
to fix.
2018-11-08 15:33:36 +00:00
Steve Howell e802be0fde Remove "subject" in notifications.py.
These last few are all pretty safe.
2018-11-08 15:12:10 +00:00
Steve Howell 226a2826b5 Remove "subject" from zerver/lib/message.py.
We also lock this down file in the linter, so
that you can no longer introduce "subject"
to `message.py`.
2018-11-08 14:45:16 +00:00
Steve Howell 6b6001c46c Remove "subject" from test_classes.py.
We now use "topic" lingo.
2018-11-07 10:03:53 -08:00
Steve Howell 79d5e36ca3 Extract topic_match_sa() helper.
We'll also use this in zerver/views/messages.py, but
that's a bigger change.
2018-11-07 10:03:53 -08:00
Steve Howell df743e8948 lint: Start to deprecate "subject" references. 2018-11-07 10:03:53 -08:00
Rishi Gupta 69e9a8f196 user docs: Add a few portico URLs to documentation crawler test. 2018-11-01 11:59:50 -07:00
Tim Abbott d9887759b4 plans: Fix broken link from FAQ to import docs and add tests. 2018-10-31 11:44:21 -07:00
zegervdv 6dd2218c50 provision: Pass proxy settings through sudo via environment.
This makes provision not fail when installing node behind a proxy.

See #10505 for details.
2018-10-29 13:30:45 -07:00
Vishnu Ks 0cf4a75fb5 coverage: Enforce full test coverage for corporate/views.py. 2018-10-24 19:39:23 -07:00
Anders Kaseorg c3bd293eaf Add shellcheck as a linter.
Fixes #10162.

Tweaked by tabbott to cover documentation as well.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-18 15:32:26 -07:00
Anders Kaseorg 49ae9c1e44 pre-commit: Fix shellcheck warnings.
In tools/pre-commit line 18:
if [ -z "$VIRTUAL_ENV" ] && `which vagrant > /dev/null` && [ -e .vagrant ]; then
                            ^-- SC2092: Remove backticks to avoid executing output.
                            ^-- SC2006: Use $(..) instead of legacy `..`.
                             ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.

In tools/pre-commit line 23:
    ./tools/lint --no-gitlint --force $changed_files || true
                                      ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-18 15:26:21 -07:00
Anders Kaseorg fc24d2e147 install-aws-server: Fix shellcheck warnings.
In tools/setup/install-aws-server line 25:
zulip_root=${ZULIP_ROOT:-$HOME/zulip}
^-- SC2034: zulip_root appears unused. Verify use (or export if used externally).

In tools/setup/install-aws-server line 40:
if [ -n "$zulip_confdir" ]; then
         ^-- SC2154: zulip_confdir is referenced but not assigned.

In tools/setup/install-aws-server line 55:
VIRTUALENV_NEEDED=$(if $(echo "$type" | grep -q app_frontend); then echo -n yes; else echo -n no; fi)
                       ^-- SC2091: Remove surrounding $() to avoid executing output.

In tools/setup/install-aws-server line 60:
SSH_OPTS=(-o HostKeyAlgorithms=ssh-rsa)
             ^-- SC2191: The = here is literal. To assign by index, use ( [index]=value ) with no spaces. To keep as literal, quote it.

In tools/setup/install-aws-server line 69:
ssh "${SSH_OPTS[@]}" "$server" -t -i "$amazon_key_file" -lroot <<EOF
                                                                 ^-- SC2087: Quote 'EOF' to make here document expansions happen on the server side rather than on the client.

In tools/setup/install-aws-server line 86:
ssh "${SSH_OPTS[@]}" "$server" -t -i "$amazon_key_file" -lroot <<EOF
                                                                 ^-- SC2087: Quote 'EOF' to make here document expansions happen on the server side rather than on the client.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:51:41 -07:00
Anders Kaseorg b15063155c postgres-init-dev-db: Fix shellcheck warnings.
In tools/setup/postgres-init-dev-db line 10:
ROOT_POSTGRES="sudo -i -u "$DEFAULT_USER" psql"
                           ^-- SC2027: The surrounding quotes actually unquote this. Remove or escape them.

In tools/setup/postgres-init-dev-db line 46:
    echo 'ERROR: Try `sudo service postgresql start`?'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

In tools/setup/postgres-init-dev-db line 64:
PGPASS_ESCAPED_PREFIX="*:\*:\*:$USERNAME:"
                         ^-- SC1117: Backslash is literal in "\*". Prefer explicit escaping: "\\*".
                            ^-- SC1117: Backslash is literal in "\*". Prefer explicit escaping: "\\*".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:48:57 -07:00
Anders Kaseorg 7f35ad916e django-template-graph: Fix shellcheck warnings.
In tools/django-template-graph line 10:
    for t in $(find -name '*.html' -printf '%P\n'); do
             ^-- SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
               ^-- SC2185: Some finds don't have a default path. Specify '.' explicitly.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg c63d852b55 deploy-branch: Fix shellcheck warnings.
In tools/deploy-branch line 17:
[ $? -ne 0 ] && error_out "Unknown branch: $branch"
  ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In tools/deploy-branch line 23:
    if [ $? -eq 0 ]; then
         ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In tools/deploy-branch line 35:
[ $? -ne 0 ] && error_out "Rebase onto origin/master failed"
  ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In tools/deploy-branch line 39:
[ $? -ne 0 ] && error_out "Push of master to origin/master failed"
  ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg 4a8074000c commit-msg: Fix shellcheck warnings.
In tools/commit-msg line 9:
if [ $(grep '^[^#]' .git/COMMIT_EDITMSG --count) -ne 0 ]; then
     ^-- SC2046: Quote this to prevent word splitting.

In tools/commit-msg line 10:
    lint_cmd="cd ~/zulip && cat \"$1\" | python -m gitlint.cli"
              ^-- SC2089: Quotes/backslashes will be treated literally. Use an array.

In tools/commit-msg line 11:
    if [ -z "$VIRTUAL_ENV" ] && `which vagrant > /dev/null` && [ -e .vagrant ]; then
                                ^-- SC2092: Remove backticks to avoid executing output.
                                ^-- SC2006: Use $(..) instead of legacy `..`.
                                 ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.

In tools/commit-msg line 14:
        $lint_cmd
        ^-- SC2090: Quotes/backslashes in this variable will not be respected.

In tools/commit-msg line 17:
    if [ $? -ne 0 ]; then
         ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg 8b38f61176 clean-branches: Fix shellcheck warnings.
In tools/clean-branches line 33:
                echo -n "Deleting local branch $(echo "$ref" | sed 's!^refs/heads/!!')"
                                                 ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/clean-branches line 41:
                echo -n "Deleting local branch $(echo "$ref" | sed 's!^refs/heads/!!')"
                                                 ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/clean-branches line 49:
                remote_name="$(echo "$ref" | sed 's!^refs/remotes/origin/!!')"
                               ^-- SC2001: See if you can use ${variable//search/replace} instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg b5bfe77844 build-release-tarball: Fix shellcheck warnings.
In tools/build-release-tarball line 50:
for i in `cat "$TMPDIR/$prefix/tools/release-tarball-exclude.txt"`; do
         ^-- SC2013: To read lines rather than words, pipe/redirect to a 'while read' loop.
         ^-- SC2006: Use $(..) instead of legacy `..`.

In tools/build-release-tarball line 51:
    rm -r --interactive=never "$TMPDIR/$prefix/$i";
                              ^-- SC2115: Use "${var:?}" to ensure this never expands to / .

In tools/build-release-tarball line 97:
    echo; echo -ne "\033[33mRunning update-prod-static failed. "
                    ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/build-release-tarball line 98:
    echo -e "Check $TMPDIR/update-prod-static.log for more information.\033[0m"
                                                                       ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg a5280d7d02 build-docs: Fix shellcheck warnings.
In tools/build-docs line 3:
cd "$(dirname "$0")"/../docs
^-- SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Rishi Gupta d76f728dda api docs: Rename webhook-walkthrough to incoming-webhooks-walkthrough. 2018-10-16 21:07:00 -07:00
Eeshan Garg 0292ffe04b webhooks/taiga: Add test for when a task is moved between stories.
This commit adds a test for the payload that is generated when
a Task is moved from one user story to another on Taiga's Sprint
Taskboard UI.

This commit also gets up this webhook's test coverage up to 100%.
2018-10-16 13:06:20 -07:00
Aditya Bansal 75ae94e459 font-awesome: Drop support for legacy font awesome icons.
We drop support for usage of `icon-vector` as base class when
including icons from font awesome icons package.
Now on, only icons as specified in font awesome v4.7.0 can be used
in the code base.
2018-10-15 20:14:55 +05:30
Aditya Bansal 7eeed1413c icons: Remove base class from comments and test templates. 2018-10-15 19:55:18 +05:30
Steve Howell b7f764aa29 settings: Add confirm_dialog module.
This module makes it really easy to create are-you-sure
dialogs for dangerous operations.

Basically it's one function with five parameters.  You
give three chunks of HTML, a callback function, and
a parent container.

The first use of this will be in settings_user_groups,
coming up in a couple commits.
2018-10-12 10:37:06 -07:00
Eeshan Garg ba929508e2 webhooks/ifttt: Get test coverage up to 100%.
IFTTT allows custom templating for their payloads, so the onus is
on the user to ensure that their custom templates conform to the
expectations outlined in our IFTTT webhook docs. For that reason,
these payloads weren't generated, but were manually edited.
2018-10-04 12:16:06 -07:00
Eeshan Garg 316f9aa78b webhooks/hellosign: Rewrite the integration from scratch.
After discovering a couple of bugs, I decided to thoroughly test
and rewrite this integration from scratch. The older code wasn't
generating coherent messages.

This also commit gets this integration up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 5039f6dfb5 webhooks/gitlab: Get test coverage up to 100%.
Test coverage was improved by removing an unused function and
removing some code (written by me) that was actually handling
Test Hook event types incorrectly.
2018-10-04 12:16:06 -07:00
Eeshan Garg 6e2e2b9125 webhooks/github: Test commit status payloads with target_url.
It was a painful amount of work to generate the actual payload.
Since the only difference was a small build URL, I manually
edited the payload and used that for testing.

This commit gets our GitHub webhook up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg b40aec3a09 webhooks/freshdesk: Improve test coverage.
Note that Freshdesk allows custom templating for outgoing payloads
in their webhook UI. Therefore, the payloads added in this commit
did not have to be official payloads from Freshdesk.
2018-10-04 12:16:06 -07:00
Eeshan Garg d8101ca139 webhooks/bitbucket2: Improve test coverage.
The lack of coverage was due to:

* An unused function that was never used anywhere.
* get_commit_status_changed_body was using a regex where it didn't
  really need to use one. And there was an if statement that
  assumed that the payload might NOT contain the URL to the commit.
  However, I checked the payload and there shouldn't be any instances
  where a commit event is generated but there is no URL to the commit.
* get_push_tag_body had an `else` condition that really can't happen
  in any payload. I verified this by checking the BitBucket webhook
  docs.
2018-10-04 12:16:06 -07:00
Eeshan Garg 185a023745 webhooks/beanstalk: Improve test coverage.
We shouldn't just ignore exceptions when encoding the incoming
auth credentials. Even if the incoming credentials are properly
encoded, it is better to know when that is the case or if
something else fails.
2018-10-04 12:16:06 -07:00
Harshit Bansal fc7e654d7b emoji: Remove single-digit names for number emojis.
Names like `:1:`, `:2:` etc don't seem to be of much benefit rather
they seem to get rendered at places not intended.
2018-10-02 11:30:11 -07:00
Vishnu Ks 59865797ac billing: Move billing tests from zilencer to corporate. 2018-10-01 15:44:02 -07:00
Vishnu Ks 93b398bc0a billing: Move zilencer/lib/stripe to corporate. 2018-10-01 15:43:44 -07:00
Vishnu Ks daf3b46262 billing: Move billing related templates from zilencer to corporate. 2018-10-01 15:43:39 -07:00
Vishnu Ks 6914ee126c billing: Move billing-related views and urls to corporate. 2018-10-01 15:43:21 -07:00
Tim Abbott 00821d67bb travis: Remove no-longer-present packages from apt-mark hold list.
Apparently, Travis removed the Heroku bundle of packages from their
servers, which made the build start failing when trying to configure
apt to hold their versions (sigh).  This commit removes the
problematic packages.
2018-09-28 14:20:04 -07:00
Greg Price 3b646c5d28 push-to-pull-request: Update local tracking ref, if any.
The companion tool `tools/reset-to-pull-request` has a handy feature
to maintain a local ref tracking the PR: e.g., pr/1234 for PR 1234.
If this were a normal remote-tracking branch maintained by `git fetch`,
it'd get updated on `git push`.  Do the same thing here.

This helps keep a view like `gitk --all @` a bit tidier, by causing
merged PRs to stop pointing at side branches of the main history.
2018-09-25 13:15:02 -07:00
Vishnu Ks 5a6b2ebb1f billing: Add ability for users to change their card. 2018-09-24 19:25:26 -07:00
Anders Kaseorg 5e88b3013a zanitizer_config.pm.sample: keep_file → scrub_filename
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-09-23 20:42:27 -07:00
Eeshan Garg ecd4f821be user docs: Automatically activate correct tab for OS-specific instructions. 2018-09-18 13:49:34 -07:00
Vishnu Ks d6b548bd57 styles: Extract billing SCSS into its own stylesheet. 2018-09-11 00:08:38 -07:00
Steve Howell 06225d1424 tests: Clean up calls to tools/webpack.
Before this change, the way we loaded
webpack for various tools was brittle.

First, I addressed test-api and test-help-documentation.

These tools used to be unable to run standalone on a
clean provision, because they were (indirectly)
calling tools/webpack without the `--test` option.

The problem was a bit obscure, since running things
like `./tools/test-backend` or `./tools/test-all` in
your workflow would create `./var/webpack-stats-test.json`
for the broken tools (and then they would work).

The tools themselves weren't broken; they were the
only relying on the common `test_server_running` helper.
And even that helper wasn't broken; it was just that
`run-dev.py` wasn't respecting the `--test` option.

So I made it so that `./tools/run-dev` passes in `--test` to
`./tools/webpack`.

To confuse matters even more, for some reason Casper
uses `./webpack-stats-production.json` via various
hacks for its webpack configuration, so when I fixed
the other tests, it broke Casper.

Here is the Casper-related hack in zproject/test_settings.py,
which was in place before my change and remains
after it:

    if CASPER_TESTS:
        WEBPACK_FILE = 'webpack-stats-production.json'
    else:
        WEBPACK_FILE = os.path.join('var', 'webpack-stats-test.json')

I added similar logic in tools/webpack:

    if "CASPER_TESTS" in os.environ:
        build_for_prod_or_casper(args.quiet)

I also made the helper functions in `./tools/webpack` have
nicer names.

So, now tools should all be able to run standalone and not
rely on previous tools creating webpack stats files for
them and leaving them in the file system.  That's good.

Things are still a bit janky, though.  It's not completely
clear to me why `test-js-with-casper` should work off of
a different webpack configuration than the other tests.

For now most of the jankiness is around Casper, and we have
hacks in two different places, `zproject/test_settings.py` and
`tools/webpack` to force it to use the production stats
file instead of the "test" one, even though Casper uses
test-like settings for other things like which database
you're using.
2018-09-07 11:39:55 -04:00
Steve Howell 25ad755914 Fix usage for test-api. 2018-09-07 11:39:55 -04:00
Steve Howell 336c2cee20 Revert unused code for creating non-editable pills.
We don't use input.create_non_editable_pill() in our
code yet.  If we add this back, we'll want to have node
tests on it.

Removing this unused code brings us to 100% line
coverage for input_pill.js.

This directly reverts 5c11ab85 with the small addition
of adding input_pill to our list of fully covered
modules.
2018-08-31 16:26:05 -07:00
Harshit Bansal 88bfb9778b build_emoji: Add infrastructure for google-blob and twitter emojisets. 2018-08-31 11:52:24 -07:00
Marco Burstein 25e624eab1 team: Include contributions from legacy repositories in the total.
These repositories (`zulip-ios-legacy` and `zulip-android`) are
deprecated, and as such should not have their own tabs, but still
should be included in the total contributions count.
2018-08-28 16:45:00 -07:00
Tim Abbott 2c7f9ce0fc puppet: Fix puppet-lint warnings in various manifests.
Apparently, `puppet-lint` on Ubuntu trusty throws warnings for certain
quoting patterns that are OK in modern `puppet-lint`.  I believe the
old Zulip code was actually correct (i.e. the old `puppet-lint`
implementation was the problem), but it seems worth changing anyway to
suppress the warnings.

We also exclude more of puppet-apt from linting, since it's
third-party code.
2018-08-28 13:46:31 -07:00
Lyla Fischer 1efcdfdb5c templates: Merge Help and API doc main into documentation_main.
With minor fixes by eeshangarg.
2018-08-27 20:19:24 -02:30
Harshit Bansal e204fe3948 minor: Fix for failing tests. 2018-08-27 06:16:16 +00:00
Harshit Bansal d96d6a96f7 build_emoji: Add a missing check for image existence. 2018-08-26 22:16:23 -07:00
Harshit Bansal f5454dff51 build_emoji: Automatically calculate sprite dimensions.
Instead of using a hardcoded value for spritesheet dimensions,
automatically calculate it using `emoji_data`. This will free
us from updating it only emoji datasource update as well as
allow us to add google blob emojiset.
2018-08-26 22:16:23 -07:00
Harshit Bansal dbc3774c2f build_emoji: Refactor `get_emoji_code()` to not fail for old datasources.
If `non_qualified` field is not present then return the value of
`unified` field instead of failing.
2018-08-26 22:16:23 -07:00
Harshit Bansal ec2ebd1cc9 build_emoji: Change sprite CSS files to use hyphens in names.
This is more consistent with our other names in emoji infra.
2018-08-26 22:16:23 -07:00
Harshit Bansal 9bd56d0ce4 build_emoji: Change emoji sprite sheets to use hyphens in names.
Use of hyphens is more consistent with our other emoji infra names.
2018-08-26 22:16:23 -07:00
Harshit Bansal aa6f098496 build_emoji: Extract `setup_emoji_farm()`. 2018-08-26 22:16:23 -07:00
Harshit Bansal dc9ff928f3 build_emoji: Refactor `ensure_emoji_image()` to take paths as args. 2018-08-26 22:16:23 -07:00
Harshit Bansal ed694001da build_emoji: Rename `setup_emoji_farm()` to `setup_emoji_farms()`. 2018-08-26 22:16:23 -07:00
Steve Howell ca272ef49d tests: Test how we handle extra data for zform messages. 2018-08-24 10:00:25 -07:00
Steve Howell bee592b44e node tests: Bring list_cursor to 100% coverage.
We had some indirect coverage of this already, so
this mostly focuses on error cases and corner
cases.
2018-08-24 10:00:04 -07:00
Steve Howell 44756a326d tests: Test duplicate default stream/groups.
This also gets us to 100% line coverage
for actions.py.
2018-08-22 16:52:30 -07:00
Steve Howell 4318f75718 Remove `code` from JsonableError.__init__.
Since this class was built, folks have always chosen
to subclass JsonableError for situations where
the default of ErrorCode.BAD_REQUEST is insufficient.

So now we simplify the use cases, which also gets
us 100% coverage on this core module.
2018-08-22 16:51:40 -07:00
Tim Abbott b93d5bd5e8 test-js-with-node: Temporarily remove server_events_dispatch.js.
This fixes master failing tests since 2aeabf24a6.
2018-08-21 14:40:05 -07:00
Joshua Pan 793ec9e266 tests: Test all update_display_settings events. 2018-08-19 14:51:31 -04:00
xificurC 9e053c74cf scripts: Allow configuring a custom CA bundle for build process.
For building Zulip in an environment where a custom CA certificate is
required to access the public Internet, one needs to be able to
specify that CA certificate for all network access done by the Zulip
installer/build process.  This change allows configuring that via the
environment.
2018-08-13 15:45:28 -07:00
Tim Abbott 90828297e4 puppet-lint: Enforce double_quoted_strings check.
This makes our puppet codebase more consistent by using single-quoted
strings consistently.
2018-08-13 12:31:19 -07:00
Tim Abbott d0b51b70f4 puppet-lint: Enforce 2sp_soft_tables puppet-lint check.
This cleans up the puppet codebase's whitespace formatting to be more
consistent.
2018-08-13 12:31:16 -07:00
Tim Abbott b26e0a957d puppet-lint: Enforce arrow_alignment check.
This fixes all exceptions in our puppet codebase to this lint rule.
2018-08-13 12:30:57 -07:00
Aditya Bansal a3b8ab46b3 puppet-lint: Include in linter tool chain to run by default.
We start to use puppet-lint to lint puppet modules by default by
adding it to tools/lint (which controls our linter tool chain).

We also define a few puppet-lint rules to exclude.

Fixes: #9185.
2018-08-13 12:30:57 -07:00
Aditya Bansal 56d49980de puppet-lint: Add as a dependancy to install during provision process. 2018-08-13 12:30:57 -07:00
Harshit Bansal 29daf76420 emoji: Don't generate emoji farms corresponding to now banned emojisets.
This will help us in reducing the size of the release tarball
significantly. I have refrained from changing the `EMOJISETS`
constant in the `emoji_setup_utils.py` as that controls the
emojisets that we want to support. Since we want to re-enable
the feature of changing emojisets sometime again in the future
that variable should be kept as it is as it controls several
other things like emoji scripts that we use to generate emoji
names. Changing it might cause hard to catch bugs.
2018-08-13 12:09:06 -07:00
Harshit Bansal 36b977c382 emoji: Refactor `generate_sprite_css_files()` to take emojiset in args. 2018-08-13 12:09:06 -07:00
Harshit Bansal 67bbbbb007 emoji: Move logic for hard-coding octopus emoji.
This logic was incorrectly placed in `setup_old_emoji_farm()` while
it should have been placed in `setup_emoji_farm()`.
2018-08-13 12:09:06 -07:00
Harshit Bansal 1eb3825048 emoji: Remove now-unnecessary `get_remapped_emojis_map()`. 2018-08-13 12:08:14 -07:00
Harshit Bansal 99e1a81055 emoji: Fix some emoji images not loading in missed message emails.
`emoji-datasource` package v4.0.4 introduced the concept of qualified
and non-qualified emoji codes. As chat programs don't need to use
emoji representation selector, so we used migrated our infrastructure
to use non-qualified emoji codes. But we missed the fact that the
emoji file names in emoji farm are based on emoji data's 'unified'
field and the value of this field has changed. Consequently the image
file names must also have been changed. We used `emoji_code` while
converting the span tags to img tags while processing notifications.
But since now `emoji_code` refers to non-qualified code while image
file names are based on qualified code, we need to rename images
to correctly do the conversion. This commit just fixes this.
2018-08-13 12:03:16 -07:00
Steve Howell 413a0174f4 Extract a zephyr.py library.
Right now it only has one function, but the function
we removed never really belonged in actions.py, and
now we have better test coverage on actions.py, which
is an important module to get to 100%.
2018-08-11 14:51:26 -07:00
Max Nussenbaum 570b9515c6 billing: Add location for errors on billing page.
This adds a location for error messages to the billing page,
using the standard error styling.
2018-08-10 22:48:10 -07:00
Vishnu Ks 0c84260faa billing: Enforce full coverage in zilencer/tests. 2018-08-10 13:52:35 -07:00
Vishnu Ks 9946202148 coverage: Enforce full coverage for zilencer/lib/stripe.py. 2018-08-10 13:52:35 -07:00
Tim Abbott aaedec1fdb tools: Move a few i18n scripts to tools/i18n.
These are likely to only be run by Zulip maintainers.
2018-08-09 15:46:27 -07:00
Tim Abbott ba626dcad6 tools: Move optimize-svg to tools/setup/. 2018-08-09 15:39:57 -07:00
Tim Abbott c2e5cc99a4 tools: Move generate-custom-icon-webfont to a subdirectory. 2018-08-09 15:39:57 -07:00
Tim Abbott 2bb04fc720 tools: Remove send-github-payloads.
This ancient tool predates our practice of collecting test fixtures
for third-party integrations, which is a better general system for the
problem this solved.
2018-08-09 15:39:57 -07:00
Tim Abbott 74dc3e9a3e tools: Remove unused munge-postgres-logs.
This hasn't been used in several years.
2018-08-09 15:39:57 -07:00
Tim Abbott bd945333a3 tools: Remove replace-tarball-shebang.
This was a temporary helper tool for part of
tools/build-release-tarball during our Python 3 migration, which
finished some time ago.
2018-08-09 15:27:09 -07:00
Tim Abbott 78539e7819 tools: Remove unused deployment-lock-ctl.
This interacted with a deployment locking system that is no longer
part of the project.
2018-08-09 15:25:15 -07:00
Rhea Parekh 26300110ca import: Fix rendered_content in imported messages.
After the messages have been imported, set the rendered_content of the
messages instead of leaving its value to be 'None'.

This is important to ensure that:
(1) Performance for users is good after completing the import.
(2) The database's full-text indexes have all of the imported messages
(which only happens properly when Message rows have their
rendered_content field edited).

Fixes #9168.
2018-08-09 15:12:53 -07:00
Yago González f6219745de users: Get all API keys via wrapper method.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.

Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.

This commit is the first step towards allowing users have multiple API
keys.
2018-08-08 16:35:17 -07:00
Yago González a0d8cbc9cb test-backend: Fix typo. 2018-08-08 16:34:29 -07:00
Cynthia Lin ecbc58c586 settings: Fix color styling for nested inputs to work in night mode. 2018-08-08 11:29:48 -07:00
Rhea Parekh 30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh 1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00
Vishnu Ks 119d108da0 droplets: Update code and remove nvm settings from bashrc.
* Synced zulip repo to master and provisioned
* Synced python-zulip-api repo to master and provisioned.
* Removed unnecessary nvm config from bashrc
2018-08-07 13:38:18 -07:00
Tim Abbott 4300683bca lint: Fix mypy error in check_pyflakes annotations.
Introduced in 78a93b8d9e.
2018-08-07 09:59:14 -07:00
Tim Abbott e67fa36803 lint: Update config to split files more evenly. 2018-08-07 09:42:37 -07:00
Tim Abbott 78a93b8d9e lint: Refactor pyflakes to use more consistent style.
In particular, we no longer pass in the full `by_lang` object and
expect it to pull out the Python piece.
2018-08-07 09:42:37 -07:00
Tim Abbott 458c17868f zulint: Move pycodestyle runner code to library.
This way, only the Zulip project's specific configuration ends up
living in pep8.py.
2018-08-04 19:54:18 -07:00
Tim Abbott 3b53ddf779 zulint: Stop importing color codes from zulip_tools.py.
This is I think the final dependency that zulint/ has on the rest of
the project.
2018-08-04 19:53:53 -07:00
Tim Abbott c0beeebc3e zulint: Move printer.py module to tools/zulint.
At this point, tools/linter_lib is intended to be the directory of
Zulip-specific code.
2018-08-04 19:53:53 -07:00
Tim Abbott 55ba434ecb zulint: Extract LinterConfig class to zulint. 2018-08-04 19:53:53 -07:00
Tim Abbott f74bb8a9b2 zulint: Extract run_parallel and do_lint. 2018-08-04 19:53:53 -07:00
Tim Abbott acf8ec492d zulint: Extract common linter argument parsing logic.
The remaining code is Zulip-specific business logic.
2018-08-04 19:53:53 -07:00
Tim Abbott 1fcc2a6ea4 zulint: Move lister.py to tools/zulint.
This is preparatory refactoring work for being able to extract the
linter as an external project called "zulint".
2018-08-04 19:53:53 -07:00
Joshua Pan 5303281dfb tests: Fully cover hash_util.js. 2018-08-04 17:56:02 -07:00
Abhilash Verma fd9652a9d7 settings_emoji: Make strings more user-friendly. 2018-08-04 09:37:48 -07:00
Steve Howell c8898e1dc8 refactor: Move by_conversation_and_time_uri to hash_util.
This removes the 100% coverage on hash_util, but we are
pretty careful about not messing with this code.
2018-08-04 09:32:27 -07:00
Anders Kaseorg d79555e9be update-locked-requirements: Fix shellcheck warnings.
In tools/update-locked-requirements line 66:
compile_requirements requirements/prod.in $OUTPUT_BASE_DIR/prod.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 67:
compile_requirements requirements/dev.in $OUTPUT_BASE_DIR/dev.txt
                                         ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 68:
compile_requirements requirements/mypy.in $OUTPUT_BASE_DIR/mypy.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 69:
compile_requirements requirements/docs.in $OUTPUT_BASE_DIR/docs.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 70:
compile_requirements requirements/thumbor.in $OUTPUT_BASE_DIR/thumbor.txt py2
                                             ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg 3e7bf9be9c production-helper: Fix shellcheck warnings.
In tools/travis/production-helper line 24:
if ! apt-get dist-upgrade -y $APT_OPTIONS; then
                             ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/travis/production-helper line 26:
    apt-get dist-upgrade -y $APT_OPTIONS
                            ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e05a56aa8b test-migrations: Fix shellcheck warnings.
In tools/test-migrations line 18:
    echo "$new_auto_named_migrations" | sed 's/\[[x ]\] /  /'
    ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/test-migrations line 27:
    echo 'ERROR: Migrations are not consistent with models!  Fix with `./tools/renumber-migrations`.'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c4ed4bc021 test-install: Fix shellcheck warnings.
In tools/test-install/destroy-all line 31:
  | while read c
          ^-- SC2162: read without -r will mangle backslashes.

In tools/test-install/install line 57:
    installer_dir="$(readlink -f $INSTALLER)"
                                 ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/test-install/lxc-wait line 30:
for i in {1..60}; do
^-- SC2034: i appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e1d770d451 test-documentation: Fix shellcheck warnings.
In tools/test-documentation line 6:
    echo -e "\e[${color_code}m${message}\e[0m" >&2
             ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                        ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".

In tools/test-documentation line 41:
scrapy crawl_with_status documentation_crawler $loglevel
                                               ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg bb51503907 test-all-docker: Fix shellcheck warnings.
In tools/test-all-docker line 7:
source /home/zulip/.bash_profile
^-- SC1091: Not following: /home/zulip/.bash_profile: openBinaryFile: does not exist (No such file or directory)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c93f53b463 test-all: Fix shellcheck warnings.
In tools/test-all line 7:
TEMP=`getopt -o f --long force -- "$@"`
     ^-- SC2006: Use $(..) instead of legacy `..`.

In tools/test-all line 24:
    echo "Running $@"
                  ^-- SC2145: Argument mixes string and array. Use * or separate argument.

In tools/test-all line 26:
        printf "\n\e[31;1mFAILED\e[0m $@\n"
               ^-- SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
                ^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\\n".
                  ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                      ^-- SC2145: Argument mixes string and array. Use * or separate argument.
                                        ^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\\n".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e9656564c2 start-dockers: Fix shellcheck warnings.
In tools/start-dockers line 7:
source /home/zulip/.bash_profile
^-- SC1091: Not following: /home/zulip/.bash_profile: openBinaryFile: does not exist (No such file or directory)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg f4c4e857ff generate-test-credentials: Fix shellcheck warnings.
In tools/setup/generate-test-credentials line 10:
password=$(./manage.py print_initial_password "$email" | fgrep "$email" | awk '{ print $2 }')
                                                         ^-- SC2197: fgrep is non-standard and deprecated. Use grep -F instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg af64bff9d4 reset-to-pull-request: Fix shellcheck warnings.
In tools/reset-to-pull-request line 25:
    git fetch "$remote" +"pull/$request_id/head":"$target_ref"
                                                ^-- SC2140: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c669626666 provision: Fix shellcheck warnings.
In tools/provision line 13:
FAIL="\033[91m"
      ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 14:
WARNING="\033[93m"
         ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 15:
ENDC="\033[0m"
      ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 19:
PARENT_PATH=$( cd "$(dirname "${BASH_SOURCE}")" ; pwd -P )
                              ^-- SC2128: Expanding an array without an index only gives the first element.

In tools/provision line 32:
if [ $failed = 1 ]; then
     ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/provision line 49:
    echo 'or just close this shell and start a new one (with Vagrant, `vagrant ssh`).'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg d7c7e8dd7c optimize-svg: Fix shellcheck warnings.
In tools/optimize-svg line 3:
if [ `node_modules/.bin/svgo -f static/images/integrations/logos | grep -o '\.[0-9]% = ' | wc -l` -ge 1 ]
     ^-- SC2046: Quote this to prevent word splitting.
     ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg cd451491ff find-unused-css: Fix shellcheck warnings.
In tools/find-unused-css line 5:
    if [ $(git grep "$n" | grep -v '^static/styles/zulip.css' | wc -l) -eq 0 ]; then
         ^-- SC2046: Quote this to prevent word splitting.
                           ^-- SC2126: Consider using grep -c instead of grep|wc -l.

In tools/find-unused-css line 6:
        echo $n
             ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg b920d19358 fetch-rebase-pull-request: Fix shellcheck warnings.
In tools/fetch-rebase-pull-request line 15:
git checkout -B "review-${request_id}" $remote/master
                                       ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg e7e25fb70b do-destroy-rebuild-test-database: Fix shellcheck warnings.
In tools/do-destroy-rebuild-test-database line 6:
"`dirname "$0"`/../tools/setup/generate-fixtures" --force
 ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Rhea Parekh ee37866687 import: Add gitter import file in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Rhea Parekh b8e1e8b31d import: Add slack import files in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Anders Kaseorg de5e3d3fbd tools/review: Don’t pretend to emulate shell=True either.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Anders Kaseorg 9a99d76296 tools: Remove pointless uses of shell=True.
Which, not coincidentally, is all of them.  Don’t use shell=True.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:06 -07:00
Steve Howell 95d136ca5e Extract search_util.js module.
We probably should have done this a while ago, even
though these functions are pretty tiny.  The goal here
is to make it easier to have more consistent search
semantics.

Our first use case is subs.js.  In this case we
are able to decouple a bit of generic string
matching from the subs-specific code.
2018-07-30 11:25:32 -07:00
Greg Price 4ba8015002 tools/reset...: Add experimental option to record the PR as a ref.
I often find myself looking manually through the reflog of `master` to
find a commit I previously reset to with tools/reset-to-pull-request .
Sometimes I want to see a previous version of a PR I'm reviewing a
revised version of; sometimes to look at two related PRs together.

So, here's a feature to automate that by saving each PR branch in its
own ref, with a name like `refs/remotes/pr/1234` -- or `pr/1234`, as
you'd normally refer to it.

To enable this, set the new config option:
  $ git config zulip.prPseudoRemote pr
(Or you can pick another name.)

The reason I hesitate to just make this the behavior for everyone
immediately is that the resulting `pr/1234` refs will naturally
accumulate and may clutter up the view -- and because with the
`refs/remotes/` style of name I've chosen, it requires a bit of
Git plumbing to clean them up.  (Use `git update-ref -d`.)
I'll play with it and iterate; comments welcome from other willing
early adopters.
2018-07-27 23:47:32 -07:00
Harshit Bansal cf5b2b4815 emoji: Change emoticon mapping for `:)`, `(:` and `:(`.
See discussion on CZO:
https://chat.zulip.org/#narrow/stream/101-design/subject/emoji.20picker/near/617811
2018-07-26 11:17:03 -07:00
Anders Kaseorg 19f37fb3d4 tools/test-queue-worker-reload: Avoid shelling out for touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg d3f8f74ac4 tools/test-locked-requirements: Avoid shelling out for cp, mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg dfed7da911 tools/lib/provision.py: Avoid shelling out for mkdir, touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Aditya Bansal 0042694e24 stylelint: Start using stylelint to lint our css.
Tweaked by tabbott to document it.

Fixes #8894.
2018-07-25 15:03:35 -07:00
Priyank Patel 8a012b9e7f Revert "webpack: Combine archive-style bundle into archive."
This reverts commit d1d9d63841, and fixes
the fonts in webapp being overwritten by bootstarap due to webpack bundling.
2018-07-24 16:00:26 -07:00
Greg Price 79dfe6e0ae push-to-pull-request: Fix shell usage in an error path.
Errors go to stderr, not stdout.

While we're here, match the message itself to the others in
this script.
2018-07-23 23:19:27 -07:00
Greg Price 26d0904436 push-to-pull-request: Give a nice error if `jq` missing. 2018-07-23 23:19:27 -07:00
Priyank Patel d1d9d63841 webpack: Combine archive-style bundle into archive. 2018-07-23 15:19:47 -07:00
Priyank Patel f8398238a9 webpack: Migrate archive bundle to webpack.
Tested by visiting localhost:9991/archive/streams/1/topics/denmark3.
2018-07-23 15:19:44 -07:00
Harshit Bansal c0b0fb7cce emoji: Move `EMOTICON_CONVERSIONS` mapping to build_emoji infra.
This commit closes a long pending issue which involved moving the
`EMOTICON_CONVERSION` mapping to build_emoji infrastructure so
that there is only one source of truth. This was pending from the
time when this feature was implemented.
2018-07-23 12:35:08 -07:00
Joshua Pan 565fd75661 tests: Test closing alert word status msg. 2018-07-23 14:53:49 -04:00
Tim Abbott 2918cc92f3 search: Duplicate HTML template content for search area.
This is preparation for being able to work on the search pills feature
without making any user-facing changes until we're ready to enable it.
2018-07-23 11:29:10 -07:00
Rhea Parekh 3a6921e54b gitter import: Add gitter data conversion script. 2018-07-23 08:36:30 -07:00
Harshit Bansal bf70955c0c emoji: Update `emoji-datasource` packages.
This commit updates the `emoji-datasource` packages to version 4.0.4.
This update brings following changes to emoji infra:

1: Fix for the bleeding sprite sheets.

2: The category of some emojis has been changed. Categorywise breakup of
   net gain or loss is as follows:
    Travel & Places: 58 (gain)
    Symbols: 47 (loss)
    Smileys & People: 52 (gain)
    Objects: 11 (loss)
    Food & Drink: 3 (gain)
    Animals and Nature: 46 (gain)
    Activities: 9 (loss)

3: There were some changes in the image farm of the package which were
   breaking our old emoji farm. I fixed them by modifying the remapped
   emoji map.

Fixes: #8235.
2018-07-23 08:04:58 -07:00
Harshit Bansal 6c28a60db0 emoji: Hardcode octopus emoji.
Google emojiset's octopus is really cute and whole Zulip community
loves it. So using a CSS hack, we hardcode octopus emoji to use image
from Google's emojiset only irrespective of the choosen emojiset.
2018-07-23 08:04:58 -07:00
Harshit Bansal 3f73826ffa emoji: Extract `get_emoji_code()`. 2018-07-23 08:04:58 -07:00
Harshit Bansal 69eaa2de67 emoji: Move `remapped_emojis` list to emoji_setup_utils.py. 2018-07-23 08:04:58 -07:00
Joshua Pan 015bc8b01e tests: Reach 100% coverage for keydown_util.
This commit pretty much just gets line coverage.
2018-07-23 08:42:11 -04:00
Joshua Pan 0596632457 lint: Exclude frontend_tests/node_tests from $().text rule. 2018-07-20 12:13:56 -07:00
Steve Howell b9c9dd9b5a lint: Turn off the CSS checker (officially).
Our CSS checker globs for .css files.  Since the
SCSS cutover, it has been a no-op, so there's no
sense launching it.  See #8894 for details on
future plans.
2018-07-20 10:03:28 -04:00
Anders Kaseorg 8698198452 tools/update-authors-json: Avoid shelling out for cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 6a8f0165ee tools/test-help-documentation: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg f01942e510 tools/test-emoji-name-scripts: Avoid shelling out for cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 07f5b81406 tools/minify-js: Avoid shelling out for mkdir, cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 2d12b5b3d9 tools/diagnose: Avoid shelling out for pwd.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Tim Abbott 811ac718d6 build_emoji: Document a quirk of generate_sprite_css_files. 2018-07-13 21:19:26 +05:30
Harshit Bansal ec1ce08523 emoji_setup_utils: Migrate to python3 type annotations. 2018-07-13 21:19:26 +05:30
Harshit Bansal f6fe5d6266 build_emoji: Migrate to python3 type annotations. 2018-07-13 21:19:25 +05:30
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Tim Abbott 7a86e83a6f build_emoji: Use with for writing to files.
This is a nice code cleanup.
2018-07-13 21:07:08 +05:30
Tim Abbott 85243c525d build_emoji: Pass around emoji_map data structure less.
It wasn't used in either of these functions.
2018-07-13 21:04:37 +05:30
Harshit Bansal b9f974ba61 tools: Add a tool to validate emoji name listing scripts.
This tools emoji name import/export tooling by verifying that no
information gets modified in a round-trip conversion.
2018-07-13 20:53:25 +05:30
Harshit Bansal 628e868d1e tools: Add `export_emoji_names_to_csv`.
This tool is used for exporting `emoji_names.py` to a csv file.
2018-07-13 20:53:08 +05:30
Harshit Bansal 05f85eb94d emoji: Add emoji_names.py file.
Credits to @rishig, Alice Lai, and @rntharu for naming all the emoji.

Names are inspired by iamcal, gemoji, and unicode names, sources like
emojipedia and iemoji, google search results for articles about emoji,
and emoji usage on twitter.
2018-07-13 20:53:07 +05:30
Harshit Bansal ae17a15b29 tools: Add generate_emoji_names_table tool.
This tool generates a listing of the emojis as per the current cache
in use.
2018-07-13 20:53:07 +05:30
Harshit Bansal fef5b43e22 minor: Fix directory name label in 'import_emoji_names_from_csv'. 2018-07-13 20:53:02 +05:30
Max Nussenbaum 8fd900626d billing: Style billing page.
This adds styles to /billing.
2018-07-13 18:30:38 +05:30
Tim Abbott 3cc93fd721 setup_venv: Fix missing libssl-dev dependency.
We were already correctly including libssl-dev in Zulip's dependencies
in development environment provisioning, but (at least now) it's
needed to build certain Python packages like pycurl when building a
Zulip virtualenv in production.  I haven't investigated why we didn't
need this on Ubuntu, but one possible reason would be that some other
library in our dependencies list happens to depend on it on Ubuntu.

We fix this by moving the dependency over to the shared
VENV_DEPENDENCIES list.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Harshit Bansal 8ada7cfe5b tools: Add import_emoji_names_from_csv tool. 2018-07-12 18:05:02 +05:30
Greg Price 36cf898589 mypy: Add a place to put type stubs.
This will allow us to begin to add our own stubs for external
libraries.  Writing stubs can be surprisingly little work to do, and
can have high leverage in keeping our type annotations high-quality.
2018-07-12 14:10:09 +05:30
Yago González a3d42d9901 test-api: Pass non-admin client to the test helpers. 2018-07-11 23:54:00 +05:30
Cynthia Lin 7d9281fa6c simplebar: Add simplebar dependency for help page scrolling. 2018-07-11 20:04:55 +05:30
Puneeth Chaganti 3dace27ed6 tools: Only files starting with same migration number conflict. 2018-07-10 21:09:34 +05:30
Shayan Toqraee 0757d022f5 messages: Add support for right-to-left messages.
This implements right-to-left message automatic detection support in
the compose box as well as the message feed.  Full unit tests and
support in the message-editing UI are for future work (as are
potentially more fancy things like supporting things like
right-to-left multi-word names for users/streams/etc.).

Fixes #3123.
2018-07-10 10:47:56 +05:30
Rishi Gupta 16334a1ba7 billing: Update billing system. 2018-07-09 14:33:07 +05:30
Aditya Bansal b6ba162769 generate-fixtures: Extract zulip_test_template creation as a function.
This will be helpful in the upcoming changes which will make use
of this extracted function to re-create zulip_test_template after
migrating zulip_test db so that we have latest schema in tests.
2018-07-09 09:11:48 +05:30
Armaan Ahluwalia ba6543b850 webpack: Fix missing exposed modules left out of transition.
This commit fixes some modules that were erroneously left out while
transitioning app.js to webpack. This commit exposes them using
expose-loader or setting them directly to window.
2018-07-07 08:00:34 +02:00
Shubham Dhama ced6f967bc voting_widget: Increase node test coverage to 100%. 2018-07-06 11:30:12 -04:00
Tim Abbott 0bd139af0c webpack: Fix missing expose for toMarkdown. 2018-07-05 13:24:19 +02:00