Commit Graph

2874 Commits

Author SHA1 Message Date
Tim Abbott 60eae53327 push-to-pull-request: Provide better output for invalid PR.
Previously, we were very confusingly showing the error message for
permissions when the issue was a not-found error.
2018-06-01 16:00:27 -07:00
Tim Abbott ad4617c953 push_to_pull_request: Fix handling of .git.
This was throwing the "you can't access the repo" error for this
reason.
2018-06-01 16:00:27 -07:00
Tim Abbott 95f1f1d363 mypy: Remove logic for linecoverage report. 2018-06-01 11:14:32 -07:00
Vishnu Ks 69f8b7e8df circleci: Add a base image for Ubuntu Bionic. 2018-05-31 18:51:32 -07:00
Tim Abbott 5b5d1f3e87 update-prod-static: Rewrite to use the run() scripting tool.
This provides nice `set -x` style logging of which commands are
running when.
2018-05-31 16:45:51 -07:00
Sampriti Panda e95f972e24 migrate: Add do_batch_update method for running batch updates.
* Refactor pgroonga_0002 migration to use new method.
2018-05-31 13:00:34 -07:00
Tim Abbott a2c9517f8d test_server: Don't continuously recompile handlebars templates.
This changes run-dev.py to ensure that we have in fact compiled
handlebars templates before running webpack, which is the right model.

Future work will likely include running the handlebars compiler from
webpack, and thus eliminating this extra process.
2018-05-30 20:16:06 -07:00
Tim Abbott 5a35507a7a success-http-headers: Add X-Frame-Options. 2018-05-30 09:41:50 -07:00
Tim Abbott 0fabff6dda setup_venv: Clean up VENV_DEPENDENCIES using parsed lsb_release.
This is mostly a cleanup, but it should also save 50ms in the runtime
of create-production-venv.
2018-05-29 10:57:36 -07:00
Raymond Akornor 5d39a0f0fc scripts: Replace calls to lsb release with our own parsing.
This improves the performance of these operations, by saving a ~50ms
Python process startup.  While not a major performance improvement, it
seems worth it, given how often these commands get run.

Fixes #9571.
2018-05-29 10:57:36 -07:00
Robert Hönig afa806a0ce linter: Add rule to favor Botserver over botserver or bot server. 2018-05-29 07:27:36 -07:00
Tim Abbott 2f571f9a45 webpack: Move font and pygments CSS to common.css.
First, it's silly that these weren't in common.css in the first place,
since that meant these were a bunch of duplicated code, but
additionally, that meant that these weren't available on the
`/activity` page (or other pages that don't include the portico styles).

Fixes #9561.
2018-05-27 23:19:40 -07:00
Yago González b4084d30d2 emails: Move user role logic for followup_day1 to notifications.py. 2018-05-26 07:29:58 -07:00
Yago González 5b0b9feb7d i18n: Atomize translation tags.
Makes the i18n strings in this file much easier to translate by splitting
them into smaller chunks (which avoids having a lot of code in the tagged
strings), and adds a string that was missing as well.
2018-05-26 07:06:43 -07:00
Tim Abbott 20aa3233b0 lint: Improve the check for realm->organization to support identifiers. 2018-05-25 12:03:28 -07:00
Tim Abbott 07b4da7659 Revert "node tests: Change timezone for node tests."
This reverts commit 311ddefd32.

This was causing tests to fail in some but not other environments.
2018-05-25 08:52:13 -07:00
Vishnu Ks abade460a9 tools: Add tool for listing outdated pip packages. 2018-05-24 15:16:02 -07:00
Aditya Bansal f9aa012486 linter: Fix issue with check-templates output.
We fix the issue of check-templates spitting out diff between
expected and found indentation of a file before mentioning the
error message and the file name. Basically stuff was being in the
wrong order despite the fact that in code stuff was happening in the
correct order ie, first print the error message along with the filename
and then the actual diff between expected and found file indentation.

Fixes: #9533.
2018-05-24 12:42:32 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 1b57e568ff Add "zform" functionality to the web client.
A "zform" knows how to render data that follows our
schema for widget messages with form elements like
buttons and choices.

This code won't be triggered until a subsequent
server-side commit takes widget_content from
API callers such as the trivial chat bot and
creates submessages for us.
2018-05-24 09:30:22 -07:00
Steve Howell 435719c65b Add a schema checking concept to the web app.
This starts the concept of a schema checker, similar to
zerver/lib/validator.py on the server.  We can use this
to validate incoming data.  Our server should filter most
of our incoming data, but it's useful to have client-side
checking to defend against things like upgrade
regressions (i.e. what if we change the name of the field
on the server side without updating all client uses).
2018-05-24 09:30:22 -07:00
Steve Howell d018995409 node tests: Make the failure message more obvious.
I mistakenly pushed a PR when my tests failed.  I ran with
the coverage option, so I saw this brightly colored summary
report that distracted me from the failure message.

This adds a couple newlines and some all caps.
2018-05-24 09:30:22 -07:00
Sivakar Sithamparanathan 311ddefd32 node tests: Change timezone for node tests.
The timezone environment variable was set to UTC initially. It was
changed to something other than UTC so that any local vs UTC
conversion issues will manifest in the tests.

Fixes: #5105.
2018-05-23 13:29:19 -07:00
Aditya Bansal 8cfb437677 provision: Don't run create_realm_internal_bots management command.
We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.

Fixes: #9467.
2018-05-23 13:34:52 +05:30
Aditya Bansal a27c9a08dd generate-fixtures: Export DJANGO_SETTINGS_MODULE=zproject.test_settings.
This is required because the --settings=zproject.test_settings param
doesn't work with migrate or the dumpdata management commands. Thus
untill now if one ran just this tool ended up with test database not
properly setup. We never noticed this because test-backend ran this
tool again (after exporting DJANGO_SETTINGS_MODULE) thus making the
tool work this time.
2018-05-23 11:32:40 +05:30
Sampriti Panda 51c71fd3fd provision: Add libfontconfig1 to bionic apt dependencies. 2018-05-22 22:37:04 -07:00
Shubham Dhama ec4e5e73ab org settings: Make time limit input elements inline to dropdown.
(To make the capitalization pass we added `N` to ignored phrases.)
2018-05-22 19:32:44 -07:00
Greg Price 5e89c33e7f tools: Add push-to-pull-request tool.
I've often done this by hand -- basically typed out the last line,
with the variables found from looking at the PR page in a browser.
Seems nicer for both us maintainers and the contributor, in particular
because the PR gets marked as merged, instead of closed.  But it's a
bit of a pain, and I do it maybe half the time or less; plus it's kind
of a subtle GitHub feature, and as a result I think other maintainers
of Zulip repos do this approximately never.

I've always figured this couldn't be hard to automate; today I decided
to take the 45 minutes to look up how, write out the script, QA it,
write up a nice usage message and some comments, and commit it. :)
2018-05-22 18:33:51 -07:00
Armaan Ahluwalia 1525e92058 logging/errors/webpack: Improve error stack traces with blueslip.
This commit improves the output that blueslip produces while
showing error stack traces on the front-end. This is done by
using a library called error-stack-parser to format the stack
traces.

This commit also edits the webpack config to use a different
devtool setting since the previous one did not support sourcemaps
within stack traces. It also removes a plugin that was obviated
by this change.
2018-05-22 14:36:41 -07:00
Eeshan Garg e0ef831993 webhooks: Migrate to UnexpectedWebhookEventType.
This improves test coverage for a lot of our webhooks that relied
on ad-hoc methods to handle unexpected event types.

Note that I have deliberately skipped github_legacy, it isn't
advertised and is officially deprecated.

Also, I have refrained from making further changes to Trello, I
believe further improvements to test coverage should be covered
in separate per-webhook commits/PRs.
2018-05-22 08:30:19 -07:00
Eeshan Garg 3ed20589f2 webhooks: Add generic exception for unexpected webhook events.
UnexpectedWebhookEventType is a generic exception that we may
now raise when we encounter a webhook event that is new or one
that we simply aren't aware of.
2018-05-22 08:30:19 -07:00
Michael J. Sullivan c81b276c1c mypy: Add basic support for invoking dmypy to run-mypy. 2018-05-21 22:41:08 -07:00
Aditya Bansal e41d32a15a reminders: Add various design improvements for the UI/reminder msg. 2018-05-21 09:07:37 -07:00
Tim Abbott c6fda61227 block_internet: Make compatible with httpretty library.
If httpretty is enabled, we can let it handle HTTP requests.
2018-05-20 23:31:42 -07:00
Tim Abbott a170bd70af provision: Don't block installing with Ubuntu bionic.
Now that we have tsearch_extras packages uploaded, this mostly works.

There's a few issues being debugged in #9460; they should be fixed
soon, and regardless, merging this will make it easier to develop.
2018-05-20 21:21:11 -07:00
Tim Abbott 06ed55e45a webpack: Add ALLOWED_HOSTS list for zulipdev.com and friends.
This makes it possible to again use the *.zulipdev.com domains in the
development environment.

Ideally, we'd also read REALM_HOSTS to make this more flexible.
2018-05-20 18:12:28 -07:00
Max Nussenbaum 0ff2051982 portico: Add tour section to homepage.
This adds a tour of Zulip to the bottom of the homepage.

In order to get the carousel nave, we use Bootstrap 2 from a CDN on
this page; this isn't ideal in the medium term, but upgrading
Bootstrap across the project is too much work for now.
2018-05-20 15:04:23 -07:00
Tim Abbott 4f8e09d5af provision: Fix buggy management of apt_dependencies_hash.
Apparently, we were incorrectly appending each new hash onto the end
of the file, basically resulting in every run of provision being
treated as a miss for this cache.

Fixing this saves about 4s (over 1/3) of the no-op provision time.
2018-05-20 14:22:32 -07:00
Tim Abbott 235002a549 provision: Don't install lsb-release if already installed.
This early bootstrapping step should be rearely required, and it saves
about 0.5s in our no-op provision time.
2018-05-20 14:22:32 -07:00
Shubham Padia 0824308a7a tools: Allow optional arguments after file arguments in test_backend.
Fixes #9233.
Uses nargs='*' instead of nargs='argparse.REMAINDER'.
nargs='argparse.REMAINDER' gathers remaining terms as arguments
even if it is an option e.g --coverage, while '*' gathers all the
command-line arguments until the next option is encountered.
2018-05-18 17:08:40 +05:30
Tim Abbott 0dcf873370 third: Rename thirdparty-fonts.css for better clarity.
Now it's clear that this is just legacy fontawesome styles.
2018-05-17 04:58:30 -07:00
Aman Ghebreweldi b05a702285 third: Use font awesome css file from NPM package. 2018-05-17 04:58:30 -07:00
Steve Howell b48f052b0c Add basic widgets framework (JS side). 2018-05-16 15:13:33 -07:00
Steve Howell 1f0ffdc415 Add widget.py to enable widgets on the server side.
The only slash command implemented in this initial
version is an extremely crippled version of a
"/stats" slash command that reports that you are
running 1 server.
2018-05-16 15:13:33 -07:00
Joshua Pan a67ba617d6 minor: Fix typo in tools/lint. 2018-05-16 13:19:28 -04:00
Tim Abbott 00861b910d test_presence: Add tests for seconds_usage_between.
This isn't widely used, but is a nice way to verify the logic further.
2018-05-15 18:37:49 -07:00
Tim Abbott e39a3e07d7 coverage: Mark zerver/lib/debug.py as not needing coverage. 2018-05-15 17:19:14 -07:00
Tim Abbott fd24eac8bb coverage: Exclude zerver/lib/parallel.py from coverage. 2018-05-15 17:19:14 -07:00
Tim Abbott c46149efcb migrate: Remove obsolete act_on_message_ranges.
This has a cool structure, but it's written against the long-dead
South API, and we can always pull it out of the Git history if we want
to use this approach in the future.
2018-05-15 17:19:14 -07:00
Tim Abbott 3e04aa99fa fonts: Use source-sans-pro CSS from upstream. 2018-05-15 16:22:44 -07:00
Greg Price 6dbd90835b mypy: Move remaining exclusion list to config file.
This leaves the wrapper script with very little left to do!

The main thing left is finding scripts by searching for shebang lines;
mypy itself would happily do the search for importable Python files.
2018-05-15 18:14:33 -04:00
Tim Abbott badfec2b49 coverage: Don't include settings files in test coverage. 2018-05-15 13:55:01 -07:00
Tim Abbott 191e6dc13a create_user: Remove user_profile_id option.
This hasn't been used in years, had no test coverage, and doesn't have
a clear use case.
2018-05-15 13:51:56 -07:00
RobbieClarken f81b936727 zerver/tests: Require 100% test coverage of zerver/lib/upload.py.
The last line here is impossible to test.

Fixes #4489.
2018-05-15 10:52:20 -07:00
Aman Ghebreweldi 83ee8211a8 third: Get sorttable package from npm.
We didn't have any local changes, so there's no need to have it
vendored in the repository.
2018-05-15 09:25:48 -07:00
Steve Howell 4f0b3fc1b6 test-js-with-node: Exit more cleanly for errors.
We don't need a Python stack trace when the node
piece fails, so just give a non-distracting error
message.
2018-05-15 08:24:44 -07:00
Joshua Pan df84e1d7eb tests: Reach 100% coverage for zerver/liv/management.py.
Cleaned up add_user_list_args(). The "help" and
"all_users_help" have all default values. As noted in
an earlier commit, "all_users_help" is always passed in,
so we can get rid of "all_users_arg". We keep the default
for "all_users_help" so we don't have to change variable order
in function definition.
2018-05-14 10:46:21 -07:00
Darshan Markandaiah 607cab2a53 tests: Add 100% test coverage to zerver/apps.py.
Tweaked by tabbott to rename to test_cache.py and remove the
sender_name argument (Since it was kinda confusing).
2018-05-14 10:36:10 -07:00
Vishnu Ks b73603a97c droplets: Do shutdown command along with clear history.
Otherwise shutdown command would be still present in the
history.
2018-05-14 08:12:43 -07:00
Vishnu Ks 861001f7b3 droplets: Update snapshot id.
Upgraded zulip and python-zulip-api repos.
Installed flask in python-zulip-api.
2018-05-14 08:12:43 -07:00
Tim Abbott f5ea661c1f tools: Upgrade pycodestyle.
We have to add a few rules that we're ignoring for now.
2018-05-14 06:15:13 -07:00
Aditya Bansal 64678b459c linter: Add rule to lint against use of typing.Text and promote str.
This commit is the last in a series of commits which migrated our
entire codebase to use str instead of typing.Text.

Fixes: #9203.
2018-05-13 17:17:32 -07:00
Aditya Bansal addfde7374 tools: Change use of typing.Text to str. 2018-05-13 17:17:32 -07:00
Joshua Pan 073ecaac66 provision: Give concrete NFS error message on older OSX versions. 2018-05-13 08:47:19 -07:00
Tim Abbott ff2157c787 coverage: Exclude api_test_helpers.py from backend test coverage.
These files are tested by a different test suite, and in practice have
100% coverage; no need to double-measure it.
2018-05-09 20:49:13 -07:00
Tim Abbott 7e2841b358 bulk_create: Mark bulk_create_streams as nocoverage.
It's only used in `populate_db`, so it gets tested that way.
2018-05-09 20:49:12 -07:00
Steve Howell 27b7461e0a Revert "js: Implement DynamicText class."
This reverts commit 6e7305f784.

We never ending up using this class.
2018-05-08 12:13:55 -07:00
Vishnu Ks 49b3cb9da5 droplets: Remove the step to activate virtualenv manually.
I didn't had to activate this manually during the
last few droplet upgrades.
2018-05-07 08:42:18 -07:00
Vishnu Ks c684333051 droplets: Fix the command for clearing bash history.
history -c only clears the history from memory and
would be populated back during the next SSH session.
2018-05-07 08:42:18 -07:00
Vishnu Ks 18314e57f8 droplets: Add origin remote to python-zulip-api as well. 2018-05-07 08:42:18 -07:00
Vishnu Ks 13f6cbeefd droplets: Change droplet region from SFO1 to NYC3.
Since the region of base.zulipdev.org is
NYC3 we had to add SFO1 as an additional
region each time a snapshot of base droplet
is created. This is required as droplets
can be created in SFO1 only if there is
an image present in that region. Adding
of droplet image to SFO1 takes a lot of
time as well as cost 2X as we are storing
2 images. It's better to just create new
droplets in NYC3 instead. Alternatively we
can create a new base droplet in SFO1 if
we want all the droplets to be created in
SFO1.
2018-05-07 08:42:18 -07:00
Vishnu Ks d6c5635550 droplets: Update snapshot id. 2018-05-07 08:42:18 -07:00
Greg Price 9f3052b0ef mypy: Move remaining options on type-check semantics to config file.
This puts all of this config in one place, and also needs a lot fewer
lines to describe it; which, combined, makes it a lot clearer what our
normal config actually is.  (I'd been looking at this script for a few
minutes without realizing that we have `--disallow-untyped-defs` *on*
by default, not off.)

Experimenting with different values is still easy; just comment the
line in the config.
2018-05-06 19:33:55 -07:00
Greg Price 6b1e76c1b1 mypy: Move cache-dir to config file. 2018-05-06 19:33:55 -07:00
Greg Price 724e849e2b mypy: Move follow-imports to config file.
It's cleaner to have this configuration in a nice declarative
config file than embedded in a script.
2018-05-06 19:33:55 -07:00
Greg Price ff178bb27a mypy: Drop now-redundant `-i` option.
This used to be a synonym for `--incremental`.  Since mypy 0.590,
incremental mode is the default, and the flag is ignored; so we
can happily drop it.
2018-05-06 19:33:55 -07:00
Tim Abbott 41841221ee scripts: Remove obsolete zesty configuration.
Zesty already reached end-of-life, so we'll never support it.

And in one place, we add support for bionic.
2018-05-05 11:41:57 -07:00
Tim Abbott d178c53a10 provision: Add official support for provisioning directly on Debian. 2018-05-05 11:00:47 -07:00
Tim Abbott 8ea8bfe285 puppet: Add basic configuration for Ubuntu bionic. 2018-05-05 10:49:09 -07:00
Tim Abbott b193330474 provision: Add tsearch_extras package for stretch. 2018-05-05 10:49:09 -07:00
Tim Abbott 78f3cff151 provision: Fix unescaped reference to vagrant.
Previously, this would actually try to run `vagrant ssh`, rather than
printing it :(.
2018-05-05 10:48:37 -07:00
Tim Abbott 9d6233a457 capitalization: Add an exclude rule for zulip_org_id. 2018-05-03 22:37:24 -07:00
Armaan Ahluwalia cca10beb78 css/webpack: Moved archive styles to webpack bundle.
This commit moves the stylesheets under the archive bundle in
the Django pipeline to being compiled by webpack instead. It
also removes a remaining call to a portico stylesheet that no
longer exists.
2018-05-03 19:15:49 -07:00
Armaan Ahluwalia 54bf2a6231 css/webpack: Transition landing-page.css to webpack.
This commit transitions landing-page.css from the Django pipeline
to being compiled by webpack as landing-page.scss under the
'landing-page' and 'integration' bundles.
2018-05-03 19:15:23 -07:00
Armaan Ahluwalia 1c016e990d css/webpack: Transition common.css to webpack.
This commit transitions common.css from the Django pipeline
to being compiled by webpack under the common bundle.
2018-05-03 19:15:23 -07:00
Tim Abbott bd110ccb3c test-migrations: Add django_two_factor migrations to exclude list.
We don't control the names for these migrations.
2018-05-03 12:23:25 -07:00
Priyank Patel 1e1b72f6c8 webpack: Transition stats.css to use webpack.
The stats.scss file is added to activity bundle.
2018-05-03 08:08:23 -07:00
Priyank Patel c780bc33ba webpack: Transition activity.css to use webpack.
The activity.scss file is added to activity bundle.
2018-05-03 07:53:24 -07:00
Armaan Ahluwalia 5f7b47e20c css: Transition 'app.css' to SCSS.
This commit transitions all styles in app.css in the Django pipeline
to being compiled by webpack in an app-styles bundle, and renames the
various files to now be processed as SCSS.

To implement this transition, we move the old CSS file refernces in
settings.py and replace them with a bundle declared in
`webpack.assets.json` and includedn in the index.html template

Tweaked by tabbott to keep the list of files in `app.css` in
`webpack.assets.json`, and to preserve the ordering from the old
`settings.py`.
2018-05-02 17:13:16 -07:00
Armaan Ahluwalia 64dadae697 webpack: Add css-hot-loader to remove flash on unstyled content.
This commit removes the flash on unstyled content while in dev
mode that was caused by the use of style-loader. Instead it
enables mini-css-extract-plugin in dev in combination with
css-hot-loader which enables HMR for development.

This is because mini-css-extract-plugin does not currently support
HMR out of the box. It also adds a SourceMapDevtoolPlugin to enable
sourcemaps with css since mini-css breaks sourcemaps when used in
combination with the cheap-module-evel-source-map setting.

Related issues:
https://github.com/webpack-contrib/mini-css-extract-plugin/issues/34
https://github.com/webpack-contrib/mini-css-extract-plugin/issues/29
2018-05-02 16:55:58 -07:00
Aditya Bansal a62efd55df linter: Make duplicate html tag id detection work with archives.
We modify check-templates to check for duplicate id's in archive
templates and app templates separately. This means we are allowing
app and archive templates to potentially use same id's. This is
needed because we intend to re use some js from the main app and
having same id's help achieve that.
Note: We haven't up until this point actually added archive
templates. This commit is more of a preparatory commit for merging
the basic archive infra.
2018-05-02 15:23:33 -07:00
Armaan Ahluwalia 9f80418d12 webpack: Add support for png file imports in file-loader.
Adds support for importing png files using file-loader in webpack.
Changes the name of the output directory to be files instead of
fonts for better readability.
2018-05-02 09:45:04 -07:00
Armaan Ahluwalia fce6882eb9 webpack: Move the styles consumed by 5xx.html to webpack.
This commit removes the need for portico.css to be generated
by the Django pipeline and makes the error page use the css
file compiled by webpack instead.
2018-05-02 09:45:01 -07:00
Tim Abbott a4ff917789 emails: Fix spelling of "Unknown IP" and tag for translation. 2018-04-30 12:04:39 -07:00
Priyank Patel 50b13219a3 webpack: Combine both js and css into one portico bundle.
Combines, both portico js and css into one bundle. This for now solve
the issue of an empty js bundle being generated by webpack for the
portico-styles stylesheet.
2018-04-30 10:23:39 -07:00
Tim Abbott 7d6bb3dcb4 settings: Remove obsolete default_desktop_notifications setting.
This actually hasn't been hooked up to do anything in years.

While we're at it, we remove the entire "Zulip Labs" settings page.
2018-04-28 13:46:07 -07:00
Steve Howell 605a90ce36 node tests: Reach 100% coverage for scroll_util.js. 2018-04-28 11:15:14 -07:00
Steve Howell 65d8eb3189 buddy list: Extract user_search.js.
This was a bit more than moving code.  I extracted the
following things:

        $widget (and three helper methods)
        $input
        text()
        empty()
        expand_column
        close_widget
        activity.clear_highlight

There was a minor bug before this commit, where we were inconsistent
about trimming spaces.  The introduction of text() and empty() should
prevent bugs where users type the space bar into search.
2018-04-28 11:15:14 -07:00
Armaan Ahluwalia fb0a421b8c webpack: Silence most webpack output for tests.
This commit adds a --quiet argument to tools/webpack which removes
the verbose output from webpack and replaces it with showing only
errors. It also makes tools/run-dev --tests use this argument while
running webpack for testing.

Tweaked by tabbott to clean up the code a bit.
2018-04-28 09:32:10 -07:00
Tim Abbott 699c4381f2 test-api: Add a PROVISION_VERSION --force option.
This brings this in line with the Casper and other tests.
2018-04-27 17:21:12 -07:00