Commit Graph

3750 Commits

Author SHA1 Message Date
Aman Agrawal 5ab62a3514 sgrep/semgrep: Updgrade and rename sgrep to semgrep.
With its new upgrade, sgrep has been renamed to semgrep.

Rename sgrep.yml to semgrep.yml
2020-05-01 11:01:14 -07:00
Steve Howell ea52bc987d run-dev: Clean up argument parsing code.
We remove the import-tools code that was plunked
right into the middle of our command line
arguments.

Then we add a local var called `DESCRIPTION` to
fix some ugly code formatting, and we stop with the
unnecessary `r` prefix to the multi-line string.
2020-05-01 10:40:34 -07:00
Anders Kaseorg a552c2e5f9 auth: Use the clipboard instead of zulip:// for desktop auth flow.
This does not rely on the desktop app being able to register for the
zulip:// scheme (which is problematic with, for example, the AppImage
format).

It also is a better interface for managing changes to the system,
since the implementation exists almost entirely in the server/webapp
project.

This provides a smoother user experience, where the user doesn't need
to do the paste step, when combined with
https://github.com/zulip/zulip-desktop/pull/943.

Fixes #13613.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-04-30 16:45:00 -07:00
Aman Agrawal 5fa1dbf5b3 test_fixtures: Add DB function to write database digest.
This will give help up write new digest only if the db rebuild
succeeds. We were relying on the caller to
be successful in building db, this was hacky and unreliable.

We write new db digest once the caller succeeds, this ensures
that we write new digest after every successful attempt.

This fixes the anomality we were facing that Databases were rebuild
on the 2nd provision attempt with no changes to files or migrations.
This was happening because we didn't write a new digest for db
after the first provision (The case of DB didn't exist).

During the 1st provision, we check the template_status() of
Database both Dev and Test, but database_exists() of Databases
obviously returned false, and we rebuild the database,
but forgot to write_new_digest and hence the anomaly in the
second provision explained above.
2020-04-30 10:46:53 -07:00
Steve Howell adc0ed4206 provision: Avoid shelling out to clean caches.
Yes, it's slightly janky to create an
argparse.Namespace object like this, but it
saves us from shelling out to a script whose
only real value-add is parsing a single
`threshold_days` argument.

This saves about 130ms for a no-op provision.
2020-04-30 17:19:13 +00:00
Tim Abbott 34cd0d3530 provision: Avoid including all of dev-secrets.conf in digests.
This allows us to edit unrelated secrets without forcing a
reinitialization of RabbitMQ.
2020-04-29 22:53:49 -07:00
Steve Howell aff58ed9d9 db tools: Simplify args for postgres-init-dev-db.
We now just have two modes for setting up a dev/test
database.  This makes it easy to see these things
side-by-side, when you're trying to understand how
the two different databases get built:

    # dev:
        USERNAME=zulip
        DBNAME=zulip
        STATUS_FILE_NAME=migration_status_dev

    # test:
        USERNAME=zulip_test
        DBNAME=zulip_test
        STATUS_FILE_NAME=migration_status_test

And then we make it more explicit the things that
are common between dev and test (which are
important things to understand when troubleshooting
provision-related glitches):

    SEARCH_PATH=zulip,public
    PASSWORD=$("$(dirname "$0")/../../scripts/get-django-setting" LOCAL_DATABASE_PASSWORD)
    DBNAME_BASE=${DBNAME}_base

We lose some "generality" here, but passing in arbitrary
combinations of username/dbname/status_file to the script
would cause chaos for our digest checks, and all the different
template/base databases could cause confusion too.
2020-04-29 12:30:49 -07:00
Aman Agrawal 3442182d7b ci: Use absolute path to zulip folder.
Strangely focal resolves `~/zulip` as `/root/zulip`, this makes
sure we are in the correct folder.
2020-04-29 10:54:53 -07:00
Steve Howell cf3abcedfd provision: Run configure_rabbitmq on clean checkouts.
We now just automatically run configure_rabbitmq any
time there's a clean checkout, or if an important
file has changed.

This allows us to bypass the step to import
SimpleQueueClient in certain places and reduce
some log spam.

See https://chat.zulip.org/#narrow/stream/3-backend/topic/new.20base.20dev.20droplet/near/864672
2020-04-29 13:18:54 -04:00
Steve Howell cf78cb0d6e templates: Use `<hr>` and `<br>` consistently.
We now prevent these variations:

    * <hr/>
    * <hr />
    * <br/>
    * <br />

We could enforce similar consistency for other void
tags, if we wished, but these two are particularly
prevalent.
2020-04-28 17:05:48 -07:00
Puneeth Chaganti 80e432b466 tools: Move screenshot captured message to Python. 2020-04-28 13:00:09 -07:00
Puneeth Chaganti 88c297dffe tools: Allow specifying more options when capturing screenshot.
The name and directory into which the screenshot should be captured can be
specified now, apart from specifying the fixture file to use.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 4e5c30e7fd tools: Pass image path as an argument to the screenshot tool.
Instead of figuring out the image path from the integration name in the
puppeteer script, we do it in the `generate-integration-docs-screenshot`
script and pass it as an argument to `message-screenshot.js`.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 39bea656f7 generate-integration-docs-screenshot: Remove call to sys.exit.
Preparatory refactor to allow calling the function in a loop, to generate
screenshots for all integrations.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti ac7d8ce4b3 generate-integration-docs-screenshot: Notify on webhook's stream.
Use the stream specified by `stream_name` on the WebhookIntegration, instead
of using a common stream to send messages.
2020-04-28 13:00:09 -07:00
Tim Abbott 28cb0aa81a docs: Use consistent spelling of CircleCI. 2020-04-28 11:26:58 -07:00
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Aman Agrawal 1324e7798c ci: Directly run production install script. 2020-04-28 11:11:00 -07:00
Aman Agrawal b5e31c04c9 ci: Rename production install script. 2020-04-28 11:11:00 -07:00
Aman Agrawal 70f7d04bf1 ci: Rename production builder script file. 2020-04-28 11:11:00 -07:00
Tim Abbott 1094589f79 test-all: Print a colorful notice on startup discouraging use.
The test-all tool is useful if you want to be really sure everything
passes, but it's almost always better to run a smaller subsuite when
doing active development, and just run test-all (or CI) at the end.

Our docs already covered this issue well, but this presents the idea
in a place one is definitely looking at when running test-all.
2020-04-27 16:35:29 -07:00
Rohitt Vashishtha 087b54aa45 markdown: Keep only 'text' for a no-op pygments rendering.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.

For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
2020-04-27 10:40:29 -07:00
Abhishek-Balaji 2ff1527be8 models: Switch from NullBooleanField to BooleanField.
In Django 2.1, the preferred way to express a nullable BooleanField
changed from NullBooleanField to passing null=True to BooleanField.

This updates our codebase to use the preferred API.  Tweaked by
tabbott to update the linter rules.

The migration is a noop for Django accounting only.

Part of #11341.
2020-04-26 22:13:28 -07:00
Aman Agrawal cf14cc211f ci: Separate prod tarball extraction and install process.
This allows to run scripts between extraction and install
process.

It will be used to restore npm caches for production install jobs.

We extract the tarball in the working directory so that yarn.lock and
package.json are available to restore cache.

(And also so the path is deterministic).
2020-04-25 16:10:41 -07:00
Aman Agrawal 0aa18e88a5 ci: Stop cloning repo as it is not required for prod install.
Move prod installer and test files to persistent dir so that
they can be utilized.
2020-04-25 16:07:55 -07:00
Tim Abbott 5e24d26f45 webpack: Remove useless print statement.
In I think every place we run this, you'll already be able to see
`tools/webpack` in the `set -x` style output anyway.
2020-04-25 15:39:22 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Tim Abbott c880886cc3 update-prod-static: Reduce verbosity of logging.
Previously, update-prod-static would produce thousands of lines of
basically useless output in `var/log/update-prod-static.log`.
2020-04-25 15:39:03 -07:00
Steve Howell b29c44ed94 list_render test: Test replace_list_data.
This gets us to 100% line coverage for
list_render, so we remove its exemption.
2020-04-24 18:24:12 -04:00
Steve Howell 951514dd7d check-templates: Be stricter about singleton tags.
We now forbid tags of the form `<foo ... />` in most
places, and we also forbid it even for several void
tags.

We make exceptions for tags that are already formatted
in two different ways in our codebase.  This is mostly
svg tags, plus these common cases:

    - br
    - hr
    - img
    - input

It would be nice to lock down a convention for these,
even though the HTML specification is unopinionated
on these.  We'll probably want to stay flexible for
svg tags, since they are sometimes copy/pasted from
other sources (although it's probably rare enough for
them that we can tolerate just doing minor edits as
needed).
2020-04-24 20:25:55 +00:00
Steve Howell 28f2a6950e check_templates: Parse '<foo/>' tags better.
If folks put something like '<br/>' in the HTML,
we would think the tag's name was "br/" instead
of "br".  I think we were assuming most folks
would write either "<br>" or <br />".

ASIDE:

We should probably have a consistent
preference among these styles:

    * <br>
    * <br/>
    * <br />

I prefer the first.
2020-04-24 20:25:55 +00:00
Steve Howell ca7c79c93e check-templates: Show filename for tokenize errors. 2020-04-24 20:25:54 +00:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Aman Agrawal 0e544ec40b ci: Add job to run production script to tests production install.
Only restore npm cache, venv cache not required.
Restart rabbitmq-server for production install.
2020-04-24 12:40:31 -07:00
Aman Agrawal 7c99555107 ci: Copy https headers to persistent dir for comparison. 2020-04-24 12:40:31 -07:00
Aman Agrawal b7c50034b8 ci: Remove unnecessary setup-production steps.
This steps won't be used since production install will run in
a separate job.
2020-04-24 12:40:26 -07:00
Aman Agrawal 1a096a9a10 ci: Move built production tarball to /tmp/ directory.
This directory will persist across build jobs.
2020-04-24 12:40:20 -07:00
Pranav 93b09d44d6 ci: Add test to check for untracked files.
Runs a test at the end of tools/ci/backend to check if any untracked
files have been created during this ci tests. Exits with error code 1
if untracked files are found, otherwise exits successfully.

Fixes #14691.
2020-04-24 12:35:47 -07:00
Steve Howell e47e1cd648 droplet settings: Fix hostname-related settings.
We recently changed our droplet setup such that their
host names no longer include zulipdev.org.  This caused
a few things to break.

The particular symptom that this commit fixes is that
we were trying to server static assets from
showell:9991 instead of showell.zulipdev.org:9991,
which meant that you couldn't use the app locally.
(The server would start, but the site's pretty unusable
without static assets.)

Now we rely 100% on `dev_settings.py` to set
`EXTERNAL_HOST` for any droplet users who don't set
that var in their own environment.  That allows us to
remove some essentially duplicate code in `run-dev.py`.

We also set `IS_DEV_DROPLET` explicitly, so that other
code doesn't have to make inferences or duplicate
logic to detemine whether we're a droplet or not.

And then in `settings.py` we use `IS_DEV_DROPLET` to
know that we can use a prod-like method of calculating
`STATIC_URL`, instead of hard coding `localhost`.

We may want to iterate on this further--this was
sort of a quick fix to get droplets functional again.
It's possible we can re-configure droplets to have
folks get reasonable `EXTERNAL_HOST` settings in their
bash profiles, or something like that, although that
may have its own tradeoffs.
2020-04-24 12:33:27 -07:00
Steve Howell cc1c517264 node tests: Report coverage url correctly.
This small patch accounts for our new
droplets dropping "zulipdev.org" from
the host name.
2020-04-24 10:50:08 -04:00
Rohitt Vashishtha fe5a1eeaeb settings: Extract settings_list_widget.js.
We still have a dependency on settings_org in settings_list_widget that
we should try to remove.
2020-04-22 17:57:16 -07:00
Puneeth Chaganti d5f5a99b07 tools: Verify that all integrations have bot avatars. 2020-04-22 17:45:30 -07:00
Puneeth Chaganti 18ae06404b tools: Use pre-generated avatar images for documentation bots.
Avatar images for bots used by the tool to generate integration
documentation screenshots are pre-generated and committed to the repository.
The `generate-integration-docs-screenshot` tool now uses these images,
instead of trying to create these avatar images on the fly.

Also, deleted the unused `create_png_from_svg` function.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti e97c39c587 tools: Extract png generation code to separate module. 2020-04-22 17:45:30 -07:00
Anders Kaseorg bfb314ff3a lint: Remove exclusions for Python 2 style type comments.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 16:16:08 -07:00
arpit551 aabad1c0bb test: run test-locked-requirements only for Bionic.
Currently our compiled requirements files will differ for different
python versions. So we will run test-locked-requirements only for Bionic.
2020-04-22 16:10:59 -07:00
Anders Kaseorg 6983491a0f install-{sgrep,shellcheck}: Check that we detect the version correctly.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 15:05:17 -07:00