Commit Graph

3860 Commits

Author SHA1 Message Date
Dinesh 3d97050e75 puppeteer: Improve --interactive option for test-js-with-puppeteer.
Previously, --interactive used to run tests from the start on a
repeated run triggered when tests failed and we decide to trigger
a re-run by pressing Enter key. Rerunning passed tests is of no interest.
It also used to run all tests in a loop even if all pass.

This commit fixes those both issues i.e it runs again from the
test that failed on pressing Enter and exits if all tests pass.
2020-06-17 15:24:14 -07:00
Tim Abbott 4fff858aa2 templates: Fix missing quoting of attributes in HTML templates.
This fixes a bundle of issues where we were missing "" around
attributes coming from variables.  In most cases, the variables were
integers or fixed constants from the Zulip codebase (E.g. the name of
an installed integration), but in at least one case it was
user-provided data that could potentially have security impact.
2020-06-16 23:35:39 -07:00
Sara Gulotta 1cb040647b markdown: Add support for spoilers.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.

We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.

Fixes #5802.

Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
2020-06-16 16:14:10 -07:00
Aman Agrawal 6950d8d769 provision: Don't run migrations on `zulip` db in CircleCI.
The automated tests running in CircleCI don't actually use the `zulip`
db, so we can skip running migrations on it in some CircleCI shards to
save time.

NOTE: This only effects build jobs that run provision, except the
`production-build` job where we skip building the dbs altogether.
Migrations still run on `focal-backend` build job to ensure
we are testing all our development setup code.
2020-06-16 15:37:32 +05:30
Anders Kaseorg fa2496c229 terminate-psql-sessions: Rely on the caller to set PGHOST, PGUSER.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Alex Vandiver 7fe1780088 provision: Install `crudini` for zulip.conf shell scripting.
A few tools already use this, and it is part of server installation;
install it in development environments as well.
2020-06-15 17:58:39 -07:00
Anders Kaseorg cf6981eef0 lint: Remove other rules about percent formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg 769e198ab0 lint: Prohibit percent formatting on literal format strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 3461db7ef5 python: Convert percent formatting to "".format in certain files.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.

Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1ed2d9b4a0 logging: Use logging.exception and exc_info for unexpected exceptions.
logging.exception() and logging.debug(exc_info=True),
etc. automatically include a traceback.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 674158b817 migrations: Escape more pedantically in pgroonga.0003_v2_api_upgrade.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 21:50:37 -07:00
Anders Kaseorg 0cc897d08d migrations: Escape more pedantically in pgroonga.0001_enable.
The psycopg2.SQL API unfortunately doesn’t work with
django.db.migrations.RunSQL, so we need to take a detour into
PL/pgSQL for EXECUTE and format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 21:50:37 -07:00
Anders Kaseorg 80b991fc44 production-helper: Clarify log output on check_rabbitmq_consumers failure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 16:49:32 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Steve Howell 9d9342c2a5 dead code: Remove integration_bot_widget code.
This was work in progress from summer 2017
that never got completed.

See 534aa0b105
2020-06-13 15:35:39 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Dinesh 7229088b6c tools: Fix message-screenshot.js failing to take screenshot.
To take screenshots, we log in as Desdemonia who doesn't seem
to have permissions to the streams where we capture screenshots.
This commit fixes it by logging in as Iago.

Also added a waitForNavigation until DOMContentLoaded to
avoid flakes that could occur because of us going to another URL
before Iago gets logged in.
2020-06-13 07:20:32 -04:00
Anders Kaseorg 59e3c1f5e4 install-semgrep: Upgrade semgrep to 0.10.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 18:23:17 -07:00
Anders Kaseorg 48e55875fe lint: Run isort as a linter.
With a change to increase the line length to 100 characters by
tabbott.

Fixes #2665.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:47:07 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg ca4357fd64 python: Use standard NoReturn (Python ≥ 3.6).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 12:56:52 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
Anders Kaseorg d3e8af4ad2 python: Replace silly uses of filter().
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 23:31:59 -07:00
Anders Kaseorg 3decbd1e23 lint: Remove settings exemption for possibly undefined star imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:29:50 -07:00
Anders Kaseorg 24d320f245 dev_settings: Move prod_settings_template import to configured_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:20:42 -07:00
Anders Kaseorg c45962785c settings: Group {default,prod,dev}_settings as configured_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:20:42 -07:00
arpit551 662f4902f8 ci: Setup production job for Focal.
Run production suites on Ubuntu Focal.
Added separate success-http-headers files for Focal and Bionic.
Also excluded them from whitespace rules in lint.

memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.
We already use a workaround for this while applying puppet configurations in
99e71f3786 but for docker builds we used
do memcached hack since we can not use systemd in docker containers.
2020-06-08 21:59:57 -07:00
Anders Kaseorg 9ca5a65296 build-release-tarball: Add umask override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 21:57:05 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 95be7dcbab portico: Remove mystery hunt landing page.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
2020-06-08 18:10:45 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Sharif Naas b0a0ae215f js: Extract message_edit_history.js. 2020-06-07 13:57:28 -07:00
Anders Kaseorg cf60493f8e install-shellcheck: Upgrade ShellCheck to 0.7.1.
ShellCheck binary releases have moved to GitHub:

https://github.com/koalaman/shellcheck/issues/1871

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:06:57 -07:00
Anders Kaseorg d49e880c09 lint: Remove custom rules duplicating eslint and pycodestyle.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 17:43:30 -07:00
Anders Kaseorg 5ad33c9d62 install-semgrep: Upgrade semgrep to 0.9.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:35:11 -07:00
arpit551 569731da50 provision: Removed hack for running production suites in ci.
Since now, we use different jobs for getting release-tarball
and running production suites so we don't require this hack
2020-06-04 14:26:23 -07:00
arpit551 291bce93d0 provision: Rename --production-test-suite option in provision.
Since we use this option in our docker-zulip project also
so rather than using it as a test suite option we made it
more specific i.e. --build-release-tarball-only.
2020-06-04 14:26:23 -07:00
Dinesh d429b7145e tests: Make puppeteer tests run in full suites and CircleCI. 2020-06-03 17:39:35 -07:00
Anders Kaseorg dd4d8968da custom_check: Use unconditional type imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 789e612c70 requirements: Update Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 5ab1ddbad9 test_zulint_custom_rules: Fix open mock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
Puneeth Chaganti d2dadde381 tools: Use the correct path for bot avatar to generate screenshots. 2020-06-01 18:08:23 -07:00