Commit Graph

3733 Commits

Author SHA1 Message Date
Aman Agrawal 1324e7798c ci: Directly run production install script. 2020-04-28 11:11:00 -07:00
Aman Agrawal b5e31c04c9 ci: Rename production install script. 2020-04-28 11:11:00 -07:00
Aman Agrawal 70f7d04bf1 ci: Rename production builder script file. 2020-04-28 11:11:00 -07:00
Tim Abbott 1094589f79 test-all: Print a colorful notice on startup discouraging use.
The test-all tool is useful if you want to be really sure everything
passes, but it's almost always better to run a smaller subsuite when
doing active development, and just run test-all (or CI) at the end.

Our docs already covered this issue well, but this presents the idea
in a place one is definitely looking at when running test-all.
2020-04-27 16:35:29 -07:00
Rohitt Vashishtha 087b54aa45 markdown: Keep only 'text' for a no-op pygments rendering.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.

For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
2020-04-27 10:40:29 -07:00
Abhishek-Balaji 2ff1527be8 models: Switch from NullBooleanField to BooleanField.
In Django 2.1, the preferred way to express a nullable BooleanField
changed from NullBooleanField to passing null=True to BooleanField.

This updates our codebase to use the preferred API.  Tweaked by
tabbott to update the linter rules.

The migration is a noop for Django accounting only.

Part of #11341.
2020-04-26 22:13:28 -07:00
Aman Agrawal cf14cc211f ci: Separate prod tarball extraction and install process.
This allows to run scripts between extraction and install
process.

It will be used to restore npm caches for production install jobs.

We extract the tarball in the working directory so that yarn.lock and
package.json are available to restore cache.

(And also so the path is deterministic).
2020-04-25 16:10:41 -07:00
Aman Agrawal 0aa18e88a5 ci: Stop cloning repo as it is not required for prod install.
Move prod installer and test files to persistent dir so that
they can be utilized.
2020-04-25 16:07:55 -07:00
Tim Abbott 5e24d26f45 webpack: Remove useless print statement.
In I think every place we run this, you'll already be able to see
`tools/webpack` in the `set -x` style output anyway.
2020-04-25 15:39:22 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Tim Abbott c880886cc3 update-prod-static: Reduce verbosity of logging.
Previously, update-prod-static would produce thousands of lines of
basically useless output in `var/log/update-prod-static.log`.
2020-04-25 15:39:03 -07:00
Steve Howell b29c44ed94 list_render test: Test replace_list_data.
This gets us to 100% line coverage for
list_render, so we remove its exemption.
2020-04-24 18:24:12 -04:00
Steve Howell 951514dd7d check-templates: Be stricter about singleton tags.
We now forbid tags of the form `<foo ... />` in most
places, and we also forbid it even for several void
tags.

We make exceptions for tags that are already formatted
in two different ways in our codebase.  This is mostly
svg tags, plus these common cases:

    - br
    - hr
    - img
    - input

It would be nice to lock down a convention for these,
even though the HTML specification is unopinionated
on these.  We'll probably want to stay flexible for
svg tags, since they are sometimes copy/pasted from
other sources (although it's probably rare enough for
them that we can tolerate just doing minor edits as
needed).
2020-04-24 20:25:55 +00:00
Steve Howell 28f2a6950e check_templates: Parse '<foo/>' tags better.
If folks put something like '<br/>' in the HTML,
we would think the tag's name was "br/" instead
of "br".  I think we were assuming most folks
would write either "<br>" or <br />".

ASIDE:

We should probably have a consistent
preference among these styles:

    * <br>
    * <br/>
    * <br />

I prefer the first.
2020-04-24 20:25:55 +00:00
Steve Howell ca7c79c93e check-templates: Show filename for tokenize errors. 2020-04-24 20:25:54 +00:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Aman Agrawal 0e544ec40b ci: Add job to run production script to tests production install.
Only restore npm cache, venv cache not required.
Restart rabbitmq-server for production install.
2020-04-24 12:40:31 -07:00
Aman Agrawal 7c99555107 ci: Copy https headers to persistent dir for comparison. 2020-04-24 12:40:31 -07:00
Aman Agrawal b7c50034b8 ci: Remove unnecessary setup-production steps.
This steps won't be used since production install will run in
a separate job.
2020-04-24 12:40:26 -07:00
Aman Agrawal 1a096a9a10 ci: Move built production tarball to /tmp/ directory.
This directory will persist across build jobs.
2020-04-24 12:40:20 -07:00
Pranav 93b09d44d6 ci: Add test to check for untracked files.
Runs a test at the end of tools/ci/backend to check if any untracked
files have been created during this ci tests. Exits with error code 1
if untracked files are found, otherwise exits successfully.

Fixes #14691.
2020-04-24 12:35:47 -07:00
Steve Howell e47e1cd648 droplet settings: Fix hostname-related settings.
We recently changed our droplet setup such that their
host names no longer include zulipdev.org.  This caused
a few things to break.

The particular symptom that this commit fixes is that
we were trying to server static assets from
showell:9991 instead of showell.zulipdev.org:9991,
which meant that you couldn't use the app locally.
(The server would start, but the site's pretty unusable
without static assets.)

Now we rely 100% on `dev_settings.py` to set
`EXTERNAL_HOST` for any droplet users who don't set
that var in their own environment.  That allows us to
remove some essentially duplicate code in `run-dev.py`.

We also set `IS_DEV_DROPLET` explicitly, so that other
code doesn't have to make inferences or duplicate
logic to detemine whether we're a droplet or not.

And then in `settings.py` we use `IS_DEV_DROPLET` to
know that we can use a prod-like method of calculating
`STATIC_URL`, instead of hard coding `localhost`.

We may want to iterate on this further--this was
sort of a quick fix to get droplets functional again.
It's possible we can re-configure droplets to have
folks get reasonable `EXTERNAL_HOST` settings in their
bash profiles, or something like that, although that
may have its own tradeoffs.
2020-04-24 12:33:27 -07:00
Steve Howell cc1c517264 node tests: Report coverage url correctly.
This small patch accounts for our new
droplets dropping "zulipdev.org" from
the host name.
2020-04-24 10:50:08 -04:00
Rohitt Vashishtha fe5a1eeaeb settings: Extract settings_list_widget.js.
We still have a dependency on settings_org in settings_list_widget that
we should try to remove.
2020-04-22 17:57:16 -07:00
Puneeth Chaganti d5f5a99b07 tools: Verify that all integrations have bot avatars. 2020-04-22 17:45:30 -07:00
Puneeth Chaganti 18ae06404b tools: Use pre-generated avatar images for documentation bots.
Avatar images for bots used by the tool to generate integration
documentation screenshots are pre-generated and committed to the repository.
The `generate-integration-docs-screenshot` tool now uses these images,
instead of trying to create these avatar images on the fly.

Also, deleted the unused `create_png_from_svg` function.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti e97c39c587 tools: Extract png generation code to separate module. 2020-04-22 17:45:30 -07:00
Anders Kaseorg bfb314ff3a lint: Remove exclusions for Python 2 style type comments.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 16:16:08 -07:00
arpit551 aabad1c0bb test: run test-locked-requirements only for Bionic.
Currently our compiled requirements files will differ for different
python versions. So we will run test-locked-requirements only for Bionic.
2020-04-22 16:10:59 -07:00
Anders Kaseorg 6983491a0f install-{sgrep,shellcheck}: Check that we detect the version correctly.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 15:05:17 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Aman Agrawal 0f4b1076ad scripts: Remove Xenial and Stretch support from installation scripts.
Note that we leave support for them in `setup-apt-repo` and puppet,
since we're still supporting systems using Xenial for non-appserver
puppet rules.
2020-04-22 10:00:38 -07:00
Aman Agrawal 8944e0ad53 provision: Drop Xenial and stretch support. 2020-04-22 09:58:39 -07:00
Steve Howell 29ac0f36a9 rebuild-test-database: Clean up helper functions.
We now have helpers for the two places where
we create databases.

There was already one helper in place, and
I gave it a more concrete name, to match
its actual database name in postgres.
2020-04-22 09:24:42 -07:00
Steve Howell 8c2f8c8bcc db tools: Refactor generate-fixtures/rebuild-test-database.
So `generate-fixtures` only ever did 9 lines of code (really
3 lines of actual code) in its normal mode of operation.

But it was cluttered with lots of stuff that really only
happened when you called it with the `-force` option, which
was only invoked by `rebuild-test-database`.

Now we inline most of the code into `rebuild-test-database`.

And now `generate-fixtures` is simple (and doesn't support
a `-force` flag.
2020-04-22 09:24:42 -07:00
Steve Howell 239474124e test-backend: Remove generate_fixtures option.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).

The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.

The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.

And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.

This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.

The other caller wasn't using `use_force`.
2020-04-22 09:24:42 -07:00
Steve Howell 54151bb548 db tools: Rename do-destroy-*database.
The new tools now have more concise, more parallel names:

    - rebuild-dev-database
    - rebuild-test-database

The actual implementations are still pretty different:

rebuild-dev-database:
    mostly delegates to 5 management scripts

rebuild-test-database:
    is a very thin wrapper for generate-fixtures

We'll try to clean that up a bit soon.
2020-04-22 09:24:42 -07:00
Tim Abbott b5f2ba5566 tools: Move test-emoji-name-scripts into subdirectory. 2020-04-21 16:19:45 -07:00
Steve Howell 837369d339 tools: Remove js-dep-visualizer.
This tool was part of a very ad hoc investigation
during 2017 into our JS dep dependencies.

It's very out of date, and it has a non-trivial
maintenance cost, as these type of tools seem
to come up in every code sweep.
2020-04-21 16:18:29 -07:00
Anders Kaseorg 9620611ec9 check-openapi: Restore functionality after OpenAPI definitions moved.
Commit 35577a1f66 (#9406) moved the
OpenAPI definitions to zerver/openapi, and this script was not
updated, so it has been validating nothing for two years.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 16:11:56 -07:00
Anders Kaseorg 7bbbc46aad check-openapi: Remove check_duplicate_operation_ids.
This is handled by swagger-parser now.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 16:11:56 -07:00
arpit551 56a5beddc0 ci: Fixed errors related to running services in ci production build.
Restart postgres service if provision is called in production test suite.
This is required because terminate-psql-sessions script (used
in tools/ci/setup-production) throws error if postgres service is not running.

Restart rabbitmq service if provision is called in production test suite.
This is done to start the node as Circle CI don't start services on installation.

Removed memcached restart as flush-memcached script (which is furthur
used in tools/ci/production) throws UNKNOWN READ FAILURE if memcached is restarted
in development.
2020-04-21 14:46:40 -07:00
arpit551 e6edf469ee ci: Remove the need of using TRAVIS in env.
Since now we want to use production suites on Circle CI so there
is no need to set TRAVIS in env while running scripts.

CIRCLECI is set default in the enviroment of Circle CI builds
so we can use it directly.

Also Travis CI had rabbitmq-server installed so we had to add workaround
in install script to avoid the error. That workaround is removed.
2020-04-21 14:46:40 -07:00
arpit551 4946405e3a ci: Change production setup in ci according to bionic and Circle CI.
Used postgres 10 inplace of postgres 9.5 as it is used in Bionic.

Upgraded nginx version in success-http-headers which is in CircleCI
Bionic enviroment.
2020-04-21 14:46:40 -07:00
arpit551 dee8de3b88 ci: Replaced travis with test suite while setting production setup on ci. 2020-04-21 14:46:40 -07:00
arpit551 80af201ef9 ci: Removed hacky code required for travis CI.
Removed hacky code required for travis CI in production-helper
and setup production.
2020-04-21 14:46:40 -07:00
Steve Howell 900fe02471 tools: Remove replacer.
This seems to be a complicated wrapper around `sed`.
It's not used anywhere or documented anywhere.  I
assume it was used in some code sweep or something.

See cec45b0ae5
2020-04-21 14:12:57 -07:00