Commit Graph

4077 Commits

Author SHA1 Message Date
Anders Kaseorg 123d527c9e check-issue-labels: Fix configparser import.
Commit ccf2792c1c (#7296) migrated this
incorrectly.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-24 19:26:02 -07:00
Alex Vandiver e0b725be63 test-install: Wait for network in the lxc container.
Ubuntu 20.04 "focal" comes up to runlevel 5 several seconds before it
is able to successfully resolve hosts, causing `prepare-base` to fail
while fetching from the apt repositories.

Add an additional check to verify that outbound networking is running
before returning from `lxc-wait`.
2020-06-24 16:07:20 -07:00
Alex Vandiver 8499b2c0dc test-install: Add support for focal. 2020-06-24 13:03:13 -07:00
Alex Vandiver 8719fba3c4 test-install: Stop installing postgres-10.
As in the previous commit, we can no longer pre-install the wrong
version of postgres.  Unfortunately, this leaves it out of the base
image and thus makes testing installs longer.
2020-06-24 13:03:13 -07:00
arpit551 e19aef569a ci: Update Circle CI images.
Removed postgresql and python 2 packages when creating docker images
for Circle CI. Updated the images.
2020-06-24 11:19:47 -07:00
Tim Abbott b7b173d2ae realm: Fix type-checking for message_retention_days.
The best way to handle this is to have the potentially string-typed
value be a separate variable.
2020-06-24 11:01:34 -07:00
Anders Kaseorg d6655689f5 validator: Remove unused type_structure system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 48719cb8c4 message_send: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg 3ffed617a2 mypy: Type simple generators as Iterator, not Iterable.
A generator that yields values without receiving or returning them is
an Iterator.  Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Felix a389c7390d calls: Add Big Blue Button as a Video Call Provider.
Big Blue Button needs an API secret so communication to creating a
room has to be done server side.

Fixes #14763.
2020-06-22 16:19:07 -07:00
Priyank Patel 89619767cd test_server: Double the timeout amount to avoid flakes.
These timeout flakes, when running puppeteer test, have been fairly
frequent in CI, so increase the timeout.
2020-06-22 12:30:32 -07:00
Aman Agrawal a052d24231 recent_topics: Support arrow key navigation.
Add arrow key navigation support for recent topics.

Simple jquery is used to allow navigation for filter buttons,
a grid system is used for navigation inside table.
2020-06-20 22:50:47 -07:00
Steve Howell 22ead64d54 tests: Extract USING_TYPE_STRUCTURE. 2020-06-20 05:22:35 -04:00
Anders Kaseorg 3916ea23a9 python: Combine some split import groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:54:11 -07:00
Clara Dantas e87df351f7 renumber-migrations: Skip migration 0261.
Because of a security release that required a migration, there
are two migrations numbered 0261. To avoid breaking existing
installs renumbering the migrations, we skipped migration 0261
when running tools/renumber-migrations.
2020-06-18 15:42:57 -07:00
Dinesh 0445311430 auth: Make apple log in and sign up buttons consistent with others. 2020-06-18 13:06:10 -07:00
Tim Abbott ba1f2dfdfa pointer: Merge pointer.js into message_scroll.js.
Ths remaining code here is naturally message scrolling logic, and fits
well into the existing file for handling that.
2020-06-18 12:55:59 -07:00
Dinesh 3d97050e75 puppeteer: Improve --interactive option for test-js-with-puppeteer.
Previously, --interactive used to run tests from the start on a
repeated run triggered when tests failed and we decide to trigger
a re-run by pressing Enter key. Rerunning passed tests is of no interest.
It also used to run all tests in a loop even if all pass.

This commit fixes those both issues i.e it runs again from the
test that failed on pressing Enter and exits if all tests pass.
2020-06-17 15:24:14 -07:00
Tim Abbott 4fff858aa2 templates: Fix missing quoting of attributes in HTML templates.
This fixes a bundle of issues where we were missing "" around
attributes coming from variables.  In most cases, the variables were
integers or fixed constants from the Zulip codebase (E.g. the name of
an installed integration), but in at least one case it was
user-provided data that could potentially have security impact.
2020-06-16 23:35:39 -07:00
Sara Gulotta 1cb040647b markdown: Add support for spoilers.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.

We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.

Fixes #5802.

Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
2020-06-16 16:14:10 -07:00
Aman Agrawal 6950d8d769 provision: Don't run migrations on `zulip` db in CircleCI.
The automated tests running in CircleCI don't actually use the `zulip`
db, so we can skip running migrations on it in some CircleCI shards to
save time.

NOTE: This only effects build jobs that run provision, except the
`production-build` job where we skip building the dbs altogether.
Migrations still run on `focal-backend` build job to ensure
we are testing all our development setup code.
2020-06-16 15:37:32 +05:30
Anders Kaseorg fa2496c229 terminate-psql-sessions: Rely on the caller to set PGHOST, PGUSER.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Alex Vandiver 7fe1780088 provision: Install `crudini` for zulip.conf shell scripting.
A few tools already use this, and it is part of server installation;
install it in development environments as well.
2020-06-15 17:58:39 -07:00
Anders Kaseorg cf6981eef0 lint: Remove other rules about percent formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg 769e198ab0 lint: Prohibit percent formatting on literal format strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 3461db7ef5 python: Convert percent formatting to "".format in certain files.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.

Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1ed2d9b4a0 logging: Use logging.exception and exc_info for unexpected exceptions.
logging.exception() and logging.debug(exc_info=True),
etc. automatically include a traceback.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 674158b817 migrations: Escape more pedantically in pgroonga.0003_v2_api_upgrade.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 21:50:37 -07:00
Anders Kaseorg 0cc897d08d migrations: Escape more pedantically in pgroonga.0001_enable.
The psycopg2.SQL API unfortunately doesn’t work with
django.db.migrations.RunSQL, so we need to take a detour into
PL/pgSQL for EXECUTE and format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 21:50:37 -07:00
Anders Kaseorg 80b991fc44 production-helper: Clarify log output on check_rabbitmq_consumers failure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 16:49:32 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Steve Howell 9d9342c2a5 dead code: Remove integration_bot_widget code.
This was work in progress from summer 2017
that never got completed.

See 534aa0b105
2020-06-13 15:35:39 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Dinesh 7229088b6c tools: Fix message-screenshot.js failing to take screenshot.
To take screenshots, we log in as Desdemonia who doesn't seem
to have permissions to the streams where we capture screenshots.
This commit fixes it by logging in as Iago.

Also added a waitForNavigation until DOMContentLoaded to
avoid flakes that could occur because of us going to another URL
before Iago gets logged in.
2020-06-13 07:20:32 -04:00
Anders Kaseorg 59e3c1f5e4 install-semgrep: Upgrade semgrep to 0.10.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 18:23:17 -07:00
Anders Kaseorg 48e55875fe lint: Run isort as a linter.
With a change to increase the line length to 100 characters by
tabbott.

Fixes #2665.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:47:07 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg ca4357fd64 python: Use standard NoReturn (Python ≥ 3.6).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 12:56:52 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
Anders Kaseorg d3e8af4ad2 python: Replace silly uses of filter().
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 23:31:59 -07:00
Anders Kaseorg 3decbd1e23 lint: Remove settings exemption for possibly undefined star imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:29:50 -07:00
Anders Kaseorg 24d320f245 dev_settings: Move prod_settings_template import to configured_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:20:42 -07:00
Anders Kaseorg c45962785c settings: Group {default,prod,dev}_settings as configured_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:20:42 -07:00
arpit551 662f4902f8 ci: Setup production job for Focal.
Run production suites on Ubuntu Focal.
Added separate success-http-headers files for Focal and Bionic.
Also excluded them from whitespace rules in lint.

memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.
We already use a workaround for this while applying puppet configurations in
99e71f3786 but for docker builds we used
do memcached hack since we can not use systemd in docker containers.
2020-06-08 21:59:57 -07:00
Anders Kaseorg 9ca5a65296 build-release-tarball: Add umask override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 21:57:05 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 95be7dcbab portico: Remove mystery hunt landing page.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
2020-06-08 18:10:45 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Sharif Naas b0a0ae215f js: Extract message_edit_history.js. 2020-06-07 13:57:28 -07:00
Anders Kaseorg cf60493f8e install-shellcheck: Upgrade ShellCheck to 0.7.1.
ShellCheck binary releases have moved to GitHub:

https://github.com/koalaman/shellcheck/issues/1871

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:06:57 -07:00
Anders Kaseorg d49e880c09 lint: Remove custom rules duplicating eslint and pycodestyle.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 17:43:30 -07:00
Anders Kaseorg 5ad33c9d62 install-semgrep: Upgrade semgrep to 0.9.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:35:11 -07:00
arpit551 569731da50 provision: Removed hack for running production suites in ci.
Since now, we use different jobs for getting release-tarball
and running production suites so we don't require this hack
2020-06-04 14:26:23 -07:00
arpit551 291bce93d0 provision: Rename --production-test-suite option in provision.
Since we use this option in our docker-zulip project also
so rather than using it as a test suite option we made it
more specific i.e. --build-release-tarball-only.
2020-06-04 14:26:23 -07:00
Dinesh d429b7145e tests: Make puppeteer tests run in full suites and CircleCI. 2020-06-03 17:39:35 -07:00
Anders Kaseorg dd4d8968da custom_check: Use unconditional type imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 789e612c70 requirements: Update Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 5ab1ddbad9 test_zulint_custom_rules: Fix open mock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
Puneeth Chaganti d2dadde381 tools: Use the correct path for bot avatar to generate screenshots. 2020-06-01 18:08:23 -07:00
Puneeth Chaganti 788203778d tools: Use the correct logo path to generate bot avatar.
Integrations can be configured with specific logo paths, which weren't
correctly being used for generating the bot avatars.
2020-06-01 18:08:23 -07:00
Puneeth Chaganti 29f981c836 tools: Allow choosing one integration from the screenshot config. 2020-06-01 18:08:22 -07:00
Puneeth Chaganti 752fe58f4f tools: Allow specifying & capturing screenshots for all integrations. 2020-06-01 18:08:22 -07:00
Puneeth Chaganti 04b2653364 tools: Allow generating screenshot without fixture file.
All the data is passed as URL parameters, and no fixture file is used.
2020-06-01 18:03:27 -07:00
Puneeth Chaganti ce51c6ade9 tools: Generate screenshots for integrations with txt fixtures. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti b533420d3b tools: Allow specifying extra query params in screenshot config. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 56c31a8647 tools: Allow specifying the bot name for notifications. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 55444c33a0 tools: Allow passing payload as a query param. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti d4c6195bca tools: Allow specifying use of basic auth in ScreenshotConfig. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 5ffc51382d tools: Allow specifying custom headers in ScreenshotConfig. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti d0f8744d74 provision: Switch to fonts-freefont-ttf from fonts-liberation.
Some UTF-8 characters (★ for example) are not displayed correctly, with
fonts-liberation. Puppeteer recommends[1] installing fonts-freefont-ttf in
their docs on running Puppeteer in docker.

Provisioning forward is sufficient. There's no need to remove the
new font and replace it with the old font, I think.

[1]: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#running-puppeteer-in-docker
2020-06-01 18:03:25 -07:00
Steve Howell 43e5b2d28b right sidebar: Remove "GROUP PMs" section.
We remove the "GROUP PMs" section that used
to be in the lower right sidebar.

Most of this is straightforward code removal.

A couple quick notes:

    - The message fetching code now just
      calls `huddle_data.process_loaded_messages`,
      which we still need for search suggestions.
      We removed `activity.process_loaded_messages`.

    - The `huddle_data.process_loaded_messages`
      function no longer needs to return `need_resize`.

    - In `resize.js` we now just calculate
      `res.buddy_list_wrapper_max_height` directly
      from `usable_height`.
2020-05-27 17:57:50 -07:00
Aman Agrawal c9bff71410 ci: Don't remove .coverage file & set empty suffix.
Fixes #14962
* codecov needs `.coverage` file in the pwd to upload coverage
results.
* concurrency='multiprocessing' forces `.coverage` file to have a
data_suffix, we explicity set it to "" for the file to have no suffix.
Filed issue as https://github.com/nedbat/coveragepy/issues/989
2020-05-27 17:54:54 -07:00
Anders Kaseorg 30ab261f91 styles: Undo calc(x + y) → calc(x - -y) workaround.
The bug this was working around does not affect our current toolchain,
as confirmed by grepping through the minified output.

(Also, this linter rule only matched calc(x + y) with two arguments
and we were already using calc($far_left_gutter_size + $left_col_size
+ 4px).)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-27 16:29:25 -07:00
sahil839 1db0775e6e tests: Use do_change_user_role in lint rules and test docs.
This commit changes test docs and lint rules to use do_change_user_role
instead of do_change_is_admin and do_change_is_guest.
2020-05-27 15:38:25 -07:00
Tim Abbott d4dfeb57fd lint: Add i18n linter rule for invalid i18n.t tags.
After seeing yet another contributor accidentally try to add i18n tags
that don't work using this pattern, it's time for a lint rule.
2020-05-27 14:09:56 -07:00
Anders Kaseorg e950344078 tsconfig: Enable resolveJsonModule.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 23:33:40 -07:00
Anders Kaseorg 8bcdf4ca97 python: Convert TypedDict declarations to Python 3.6 style.
A subset of the diff generated by pyupgrade --py36-plus
--keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Rohitt Vashishtha fa9431c0a4 markdown: Extract rendered_markdown.js to update dynamic elements.
Previously, we handled this code only in message_list_view.js.
Now we support rendering stream descriptions and some dynamic
elements can be rendered in them, so we extract this new module
and use it in both the places.
2020-05-21 12:32:52 -07:00
Rohitt Vashishtha 61f6a6e9bb linter: Allow using javascript instead of JavaScript for calling examples.
This allows us to call js code examples as follows:

`{generate_code_example(javascript)|/users:post|example()}`
2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 6dd4030e67 js-api: Add hooks to run tests and render docs of JS API examples.
This commit adds python code to call javascript_examples.js in its
two supported modes. tools/test-api asserts that the example output
is as expected, whereas the API markdown extension is used to render
these examples in the docs.
2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 1091615b17 edit-bot: Show dropdown list widget for selecting owner.
This commit removes user_dropdown in favor of the common
dropdown_list_widget for our bot edit forms.
2020-05-20 09:51:38 -07:00
Aman Agrawal 0504c61bfd semgrep: Use pattern-where-python operator to filter patterns.
See https://github.com/returntocorp/semgrep/blob/experimental/docs/config/advanced.md#pattern-where-python for usage.

This helps us minimize duplication of similar patterns.
2020-05-20 09:37:26 -07:00
Vishnu KS 9442835386 droplets: Make sure the username is lowercase.
Otherwise, if someone tries to create a droplet with a mixed case username,
the existing DNS records (which are always lowercase) would not be deleted.

Also, existing droplets would not be destroyed(--recreate) if there is a mismatch
in case. Unlike DNS records, droplets are created with case sensitive
hostnames.

Plus it's neater to have lowercase hostnames.
2020-05-20 09:32:11 -07:00
Dinesh c915e2c6cf puppeteer_tests: Use casper's webpack setup.
As puppeteer tests are similar to casper i.e both being frontend
screen scraping tests, we need to use the same webpack setup as
that of casper's instead of `build_for_most_tests` which is used
mostly for backend related tests.

This sets env variable `CASPER_TESTS` to `1` which does all the
work to make it use the same setup of casper. We would want to
rename that to `PUPPETEER_TESTS` and make changes to do the
same as casper webpack setup when we completely replace casper
with puppeteer.
2020-05-19 15:53:59 -07:00
Aman Agrawal 1a8798d9c5 ci: Disable casper tests in ci due to flaky behaviour. 2020-05-13 14:02:06 -07:00
Rohitt Vashishtha e2b0a4cba1 list-widget: Rename settings_list_widget => dropdown_list_widget.
We want to use this widget outside of the settings panels as well.
2020-05-13 10:08:51 -07:00
orientor 9170931da3 openapi: Add test for validating examples.
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.

Part of #14100.
2020-05-12 23:03:06 -07:00
Rohan Prashant 44a56a66d0 tests-backend: Bring stream_recipient.py to 100 percent coverage.
Fixes part of #7089.
2020-05-12 16:43:01 -07:00
Vishnu KS 8fb1f2af58 billing: Support downgrading plan from /billing page. 2020-05-11 17:20:54 -07:00
Vishnu KS f1b1bf5a0d billing: Add support for Zulip Standard free trial. 2020-05-11 17:20:54 -07:00
Steve Howell 9f137c3a05 bots: Extract user_dropdown widget.
We use this new widget in bot settings panels
(personal and org).  It lets you re-assign a
bot to a new human user.

Ideally we can improve this code to use
our existing list widgets to make it more
performant for realms with lots of users.
2020-05-11 16:16:58 -07:00
Puneeth Chaganti 322e84ffeb integrations: Add ERROR prefix to missing bot avatar message. 2020-05-11 14:49:11 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Anders Kaseorg caf64585e4 lint: Disallow old-style type comments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Vishnu KS 060e54e814 droplets: Add instruction to create a new base image. 2020-05-07 17:09:05 -07:00
Vishnu Ks 2bd0e59462 droplets: Disable SSH password authentication.
We cannot disable this in base droplet as DigitalOcean overrides
the file and enable password authentication during droplet creation.
2020-05-07 17:09:05 -07:00
Vishnu KS 4796206629 droplets: Use bash script instead of cloud-config as user data.
We use cloud-config for setting up the SSH keys and executing
some commands. When cloud-config sets the SSH key it doesn't override
the existing keys. So we need to set the SSH keys manually using a command
instead. This means we no longer require cloud config. We can instead
pass a bash script as the user data instead of cloud-config.

I also included a command to set the SSH key of the root.
2020-05-07 17:09:05 -07:00
Vishnu KS c07fb47aab droplets: Set template_id to the id of the new image on 18.04.
size_slug represents the plan the droplet should be created on.
Since the new base droplet is created on the cheaper but more
feature rich new plan we have to update the slug_size as well
to take advantage of the cheaper plan.
2020-05-07 17:09:05 -07:00
Vishnu KS 87ff8ada6d droplets: Move git clean to setup_repo string. 2020-05-07 17:09:05 -07:00
Vishnu KS 6079c19304 droplets: Set the hostname correctly using cloud-init. 2020-05-07 17:09:05 -07:00
Anders Kaseorg 7f52329256 check-openapi: Improve error handling.
Previously a YAML syntax error resulted in an
UnhandledPromiseRejectionWarning and a successful exit code; now it
gives a clear message and a failing exit code.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-06 16:28:55 -07:00
Aman Agrawal b25f2ed5b3 populate_db: Add option to specify max no of topics to create.
This will help us create more topics per stream than we currently do
by default.
2020-05-04 09:41:31 -07:00
Anders Kaseorg b501d04f6a semgrep: Lint against common SQL injection patterns.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:35:30 -07:00
Tim Abbott d0ffba96d9 webpack: Use cacheLoader for handlebars files.
This seems to save about 1s for run-dev.py startup time.
2020-05-03 11:03:04 -07:00
Tim Abbott af5ba9bf36 webpack: Use cacheLoader for JS files.
This seems to save over 50% of the webpack startup time in run-dev.py.
2020-05-03 11:03:04 -07:00
Anders Kaseorg baab041acd lint: Upgrade semgrep from 0.5.0-dev.2 to 0.5.0-dev.4.
The major PROVISION_VERSION bump would not be needed, but it was
missing in commit 5ab62a3514 (#14834),
so I’m doing it here.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:19:40 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Anders Kaseorg 82f629091a lint: Check docs/THIRDPARTY for format errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:10:22 -07:00
Tim Abbott 4f3976b917 process_fts_updates: Clean up logging output.
This saves a couple lines of spammy output in the run-dev.py startup
experience, and will be better output in production as well.
2020-05-01 11:51:20 -07:00
Steve Howell ce55ef6e4f tests: Use streamlined server for api/casper tests.
We now use the `--streamlined` options for `run-dev.py`
when we use `test_server_running` for `test-api` and
`test-js-with-casper` (and its experimental
replacement, `test-js-with-puppeteer`).

This means we don't slow anything down with
processes like thumbor, process_fts_updates, etc.,
which aren't meaningfully exercised by these tests.

We may eventually want some tests to meaningfully
exercise those processes, and when that day comes,
we will need to add an extra argument to
`test_server_running`, probably, but until then,
we just always set `--streamlined` in that codepath.

There is actually a tool called `./tools/test-run-dev`
that we run in CI, and it will use the full mode.
It just doesn't verify much stuff--it mostly polls
the server without testing specific features.

This seems to save about 1s of the startup time on a system I use
(~10.6s -> ~9.7s).
2020-05-01 11:37:09 -07:00
Steve Howell d9f8ec1fe7 run-dev: Add streamlined option.
For basic testing (either manual or automated), we
generally only need the server and tornado running.

Obviously, it's nice to test the complete system,
but if you're on a slow PC, the overhead can be
annoying.

Note that we don't launch any of these processes
in `--streamlined` mode:

    process_queue
    process_fts_updates
    deliver_scheduled_messages
    thumbor

And then by not launching process_queue, we avoid
several child processes.

Basic functionality like sending messages will
still work here.

The streamlined mode may be helpful in debugging
our generally slow server startup time.  Obviously,
some of the problem with startup is the auxiliary
processes here, but removing them as a variable
could help us focus on getting the core stuff fast.

Note that we still have the webpack watcher running
in streamlined mode.

For the particular case of thumbor, note that we
modify the proxy server to explicitly print and
return an error if we get a `/thumbor/*` request.
2020-05-01 11:36:43 -07:00
Steve Howell 28a2b90b04 run-dev: Extract server_processes().
We clean up the code related to launching
processes here.

We extract:

    server_processes

We also extract these helper for webpack
stuff:

    do_one_time_webpack_compile
    start_webpack_watcher

And then we move the code to actually launch
them lexically within the file (so as not to
be obscured by various function definitions).
2020-05-01 11:36:43 -07:00
Steve Howell 9cdc9cbca6 run-dev: Display ports more nicely.
Here is the new output for displaying ports:

    Zulip services will listen on ports:
       9991: web proxy
       9992: Django
       9993: Tornado
       9994: webpack
       9995: Thumbor

    Note to Vagrant users: Only the proxy port (9991) is exposed.

I tone down the yellow for the Vagrant warning, and I show
the web proxy in cyan to emphasize it.

I also extracted the code into a function, and I don't call
that function until after `app.listen()`.  (The users probably
won't notice much difference in the timing of this message, but
the message won't show if the `listen` step fails for some
reason, which I think is what we want here.)
2020-05-01 11:36:43 -07:00
Aman Agrawal 9089fd5b08 semgrep: Remove unrequired file exclude.
This file doen't contain any matching pattern which violates the
rule.
2020-05-01 11:08:47 -07:00
Aman Agrawal af0c267cd4 semgrep: Move migrations import check lint rule to semgrep.
We change how a few imports in migrations are done to be easier to
lint and more consitsent with our typical import style.
2020-05-01 11:07:53 -07:00
Aman Agrawal 8e29c88beb semgrep: Add rule to enforce no use of stream.objects.filter. 2020-05-01 11:01:14 -07:00
Aman Agrawal 06c831f146 semgrep: Reformat and correct doc link. 2020-05-01 11:01:14 -07:00
Aman Agrawal 5ab62a3514 sgrep/semgrep: Updgrade and rename sgrep to semgrep.
With its new upgrade, sgrep has been renamed to semgrep.

Rename sgrep.yml to semgrep.yml
2020-05-01 11:01:14 -07:00
Steve Howell ea52bc987d run-dev: Clean up argument parsing code.
We remove the import-tools code that was plunked
right into the middle of our command line
arguments.

Then we add a local var called `DESCRIPTION` to
fix some ugly code formatting, and we stop with the
unnecessary `r` prefix to the multi-line string.
2020-05-01 10:40:34 -07:00
Anders Kaseorg a552c2e5f9 auth: Use the clipboard instead of zulip:// for desktop auth flow.
This does not rely on the desktop app being able to register for the
zulip:// scheme (which is problematic with, for example, the AppImage
format).

It also is a better interface for managing changes to the system,
since the implementation exists almost entirely in the server/webapp
project.

This provides a smoother user experience, where the user doesn't need
to do the paste step, when combined with
https://github.com/zulip/zulip-desktop/pull/943.

Fixes #13613.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-04-30 16:45:00 -07:00
Aman Agrawal 5fa1dbf5b3 test_fixtures: Add DB function to write database digest.
This will give help up write new digest only if the db rebuild
succeeds. We were relying on the caller to
be successful in building db, this was hacky and unreliable.

We write new db digest once the caller succeeds, this ensures
that we write new digest after every successful attempt.

This fixes the anomality we were facing that Databases were rebuild
on the 2nd provision attempt with no changes to files or migrations.
This was happening because we didn't write a new digest for db
after the first provision (The case of DB didn't exist).

During the 1st provision, we check the template_status() of
Database both Dev and Test, but database_exists() of Databases
obviously returned false, and we rebuild the database,
but forgot to write_new_digest and hence the anomaly in the
second provision explained above.
2020-04-30 10:46:53 -07:00
Steve Howell adc0ed4206 provision: Avoid shelling out to clean caches.
Yes, it's slightly janky to create an
argparse.Namespace object like this, but it
saves us from shelling out to a script whose
only real value-add is parsing a single
`threshold_days` argument.

This saves about 130ms for a no-op provision.
2020-04-30 17:19:13 +00:00
Tim Abbott 34cd0d3530 provision: Avoid including all of dev-secrets.conf in digests.
This allows us to edit unrelated secrets without forcing a
reinitialization of RabbitMQ.
2020-04-29 22:53:49 -07:00
Steve Howell aff58ed9d9 db tools: Simplify args for postgres-init-dev-db.
We now just have two modes for setting up a dev/test
database.  This makes it easy to see these things
side-by-side, when you're trying to understand how
the two different databases get built:

    # dev:
        USERNAME=zulip
        DBNAME=zulip
        STATUS_FILE_NAME=migration_status_dev

    # test:
        USERNAME=zulip_test
        DBNAME=zulip_test
        STATUS_FILE_NAME=migration_status_test

And then we make it more explicit the things that
are common between dev and test (which are
important things to understand when troubleshooting
provision-related glitches):

    SEARCH_PATH=zulip,public
    PASSWORD=$("$(dirname "$0")/../../scripts/get-django-setting" LOCAL_DATABASE_PASSWORD)
    DBNAME_BASE=${DBNAME}_base

We lose some "generality" here, but passing in arbitrary
combinations of username/dbname/status_file to the script
would cause chaos for our digest checks, and all the different
template/base databases could cause confusion too.
2020-04-29 12:30:49 -07:00
Aman Agrawal 3442182d7b ci: Use absolute path to zulip folder.
Strangely focal resolves `~/zulip` as `/root/zulip`, this makes
sure we are in the correct folder.
2020-04-29 10:54:53 -07:00
Steve Howell cf3abcedfd provision: Run configure_rabbitmq on clean checkouts.
We now just automatically run configure_rabbitmq any
time there's a clean checkout, or if an important
file has changed.

This allows us to bypass the step to import
SimpleQueueClient in certain places and reduce
some log spam.

See https://chat.zulip.org/#narrow/stream/3-backend/topic/new.20base.20dev.20droplet/near/864672
2020-04-29 13:18:54 -04:00
Steve Howell cf78cb0d6e templates: Use `<hr>` and `<br>` consistently.
We now prevent these variations:

    * <hr/>
    * <hr />
    * <br/>
    * <br />

We could enforce similar consistency for other void
tags, if we wished, but these two are particularly
prevalent.
2020-04-28 17:05:48 -07:00
Puneeth Chaganti 80e432b466 tools: Move screenshot captured message to Python. 2020-04-28 13:00:09 -07:00
Puneeth Chaganti 88c297dffe tools: Allow specifying more options when capturing screenshot.
The name and directory into which the screenshot should be captured can be
specified now, apart from specifying the fixture file to use.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 4e5c30e7fd tools: Pass image path as an argument to the screenshot tool.
Instead of figuring out the image path from the integration name in the
puppeteer script, we do it in the `generate-integration-docs-screenshot`
script and pass it as an argument to `message-screenshot.js`.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 39bea656f7 generate-integration-docs-screenshot: Remove call to sys.exit.
Preparatory refactor to allow calling the function in a loop, to generate
screenshots for all integrations.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti ac7d8ce4b3 generate-integration-docs-screenshot: Notify on webhook's stream.
Use the stream specified by `stream_name` on the WebhookIntegration, instead
of using a common stream to send messages.
2020-04-28 13:00:09 -07:00
Tim Abbott 28cb0aa81a docs: Use consistent spelling of CircleCI. 2020-04-28 11:26:58 -07:00
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Aman Agrawal 1324e7798c ci: Directly run production install script. 2020-04-28 11:11:00 -07:00
Aman Agrawal b5e31c04c9 ci: Rename production install script. 2020-04-28 11:11:00 -07:00
Aman Agrawal 70f7d04bf1 ci: Rename production builder script file. 2020-04-28 11:11:00 -07:00
Tim Abbott 1094589f79 test-all: Print a colorful notice on startup discouraging use.
The test-all tool is useful if you want to be really sure everything
passes, but it's almost always better to run a smaller subsuite when
doing active development, and just run test-all (or CI) at the end.

Our docs already covered this issue well, but this presents the idea
in a place one is definitely looking at when running test-all.
2020-04-27 16:35:29 -07:00
Rohitt Vashishtha 087b54aa45 markdown: Keep only 'text' for a no-op pygments rendering.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.

For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
2020-04-27 10:40:29 -07:00
Abhishek-Balaji 2ff1527be8 models: Switch from NullBooleanField to BooleanField.
In Django 2.1, the preferred way to express a nullable BooleanField
changed from NullBooleanField to passing null=True to BooleanField.

This updates our codebase to use the preferred API.  Tweaked by
tabbott to update the linter rules.

The migration is a noop for Django accounting only.

Part of #11341.
2020-04-26 22:13:28 -07:00
Aman Agrawal cf14cc211f ci: Separate prod tarball extraction and install process.
This allows to run scripts between extraction and install
process.

It will be used to restore npm caches for production install jobs.

We extract the tarball in the working directory so that yarn.lock and
package.json are available to restore cache.

(And also so the path is deterministic).
2020-04-25 16:10:41 -07:00
Aman Agrawal 0aa18e88a5 ci: Stop cloning repo as it is not required for prod install.
Move prod installer and test files to persistent dir so that
they can be utilized.
2020-04-25 16:07:55 -07:00
Tim Abbott 5e24d26f45 webpack: Remove useless print statement.
In I think every place we run this, you'll already be able to see
`tools/webpack` in the `set -x` style output anyway.
2020-04-25 15:39:22 -07:00
Tim Abbott 5187d5032c update-prod-static: Remove unused authors-not-required.
This argument hasn't done anything since we moved constructing the
/team page to a cron job.
2020-04-25 15:39:03 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Tim Abbott c880886cc3 update-prod-static: Reduce verbosity of logging.
Previously, update-prod-static would produce thousands of lines of
basically useless output in `var/log/update-prod-static.log`.
2020-04-25 15:39:03 -07:00
Steve Howell b29c44ed94 list_render test: Test replace_list_data.
This gets us to 100% line coverage for
list_render, so we remove its exemption.
2020-04-24 18:24:12 -04:00
Steve Howell 951514dd7d check-templates: Be stricter about singleton tags.
We now forbid tags of the form `<foo ... />` in most
places, and we also forbid it even for several void
tags.

We make exceptions for tags that are already formatted
in two different ways in our codebase.  This is mostly
svg tags, plus these common cases:

    - br
    - hr
    - img
    - input

It would be nice to lock down a convention for these,
even though the HTML specification is unopinionated
on these.  We'll probably want to stay flexible for
svg tags, since they are sometimes copy/pasted from
other sources (although it's probably rare enough for
them that we can tolerate just doing minor edits as
needed).
2020-04-24 20:25:55 +00:00
Steve Howell 28f2a6950e check_templates: Parse '<foo/>' tags better.
If folks put something like '<br/>' in the HTML,
we would think the tag's name was "br/" instead
of "br".  I think we were assuming most folks
would write either "<br>" or <br />".

ASIDE:

We should probably have a consistent
preference among these styles:

    * <br>
    * <br/>
    * <br />

I prefer the first.
2020-04-24 20:25:55 +00:00
Steve Howell ca7c79c93e check-templates: Show filename for tokenize errors. 2020-04-24 20:25:54 +00:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Aman Agrawal 0e544ec40b ci: Add job to run production script to tests production install.
Only restore npm cache, venv cache not required.
Restart rabbitmq-server for production install.
2020-04-24 12:40:31 -07:00
Aman Agrawal 7c99555107 ci: Copy https headers to persistent dir for comparison. 2020-04-24 12:40:31 -07:00
Aman Agrawal b7c50034b8 ci: Remove unnecessary setup-production steps.
This steps won't be used since production install will run in
a separate job.
2020-04-24 12:40:26 -07:00
Aman Agrawal 1a096a9a10 ci: Move built production tarball to /tmp/ directory.
This directory will persist across build jobs.
2020-04-24 12:40:20 -07:00
Pranav 93b09d44d6 ci: Add test to check for untracked files.
Runs a test at the end of tools/ci/backend to check if any untracked
files have been created during this ci tests. Exits with error code 1
if untracked files are found, otherwise exits successfully.

Fixes #14691.
2020-04-24 12:35:47 -07:00
Steve Howell e47e1cd648 droplet settings: Fix hostname-related settings.
We recently changed our droplet setup such that their
host names no longer include zulipdev.org.  This caused
a few things to break.

The particular symptom that this commit fixes is that
we were trying to server static assets from
showell:9991 instead of showell.zulipdev.org:9991,
which meant that you couldn't use the app locally.
(The server would start, but the site's pretty unusable
without static assets.)

Now we rely 100% on `dev_settings.py` to set
`EXTERNAL_HOST` for any droplet users who don't set
that var in their own environment.  That allows us to
remove some essentially duplicate code in `run-dev.py`.

We also set `IS_DEV_DROPLET` explicitly, so that other
code doesn't have to make inferences or duplicate
logic to detemine whether we're a droplet or not.

And then in `settings.py` we use `IS_DEV_DROPLET` to
know that we can use a prod-like method of calculating
`STATIC_URL`, instead of hard coding `localhost`.

We may want to iterate on this further--this was
sort of a quick fix to get droplets functional again.
It's possible we can re-configure droplets to have
folks get reasonable `EXTERNAL_HOST` settings in their
bash profiles, or something like that, although that
may have its own tradeoffs.
2020-04-24 12:33:27 -07:00
Steve Howell cc1c517264 node tests: Report coverage url correctly.
This small patch accounts for our new
droplets dropping "zulipdev.org" from
the host name.
2020-04-24 10:50:08 -04:00
Rohitt Vashishtha fe5a1eeaeb settings: Extract settings_list_widget.js.
We still have a dependency on settings_org in settings_list_widget that
we should try to remove.
2020-04-22 17:57:16 -07:00
Puneeth Chaganti d5f5a99b07 tools: Verify that all integrations have bot avatars. 2020-04-22 17:45:30 -07:00
Puneeth Chaganti 18ae06404b tools: Use pre-generated avatar images for documentation bots.
Avatar images for bots used by the tool to generate integration
documentation screenshots are pre-generated and committed to the repository.
The `generate-integration-docs-screenshot` tool now uses these images,
instead of trying to create these avatar images on the fly.

Also, deleted the unused `create_png_from_svg` function.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti e97c39c587 tools: Extract png generation code to separate module. 2020-04-22 17:45:30 -07:00
Anders Kaseorg bfb314ff3a lint: Remove exclusions for Python 2 style type comments.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 16:16:08 -07:00
arpit551 aabad1c0bb test: run test-locked-requirements only for Bionic.
Currently our compiled requirements files will differ for different
python versions. So we will run test-locked-requirements only for Bionic.
2020-04-22 16:10:59 -07:00
Anders Kaseorg 6983491a0f install-{sgrep,shellcheck}: Check that we detect the version correctly.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 15:05:17 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Aman Agrawal 0f4b1076ad scripts: Remove Xenial and Stretch support from installation scripts.
Note that we leave support for them in `setup-apt-repo` and puppet,
since we're still supporting systems using Xenial for non-appserver
puppet rules.
2020-04-22 10:00:38 -07:00
Aman Agrawal 8944e0ad53 provision: Drop Xenial and stretch support. 2020-04-22 09:58:39 -07:00
Steve Howell 29ac0f36a9 rebuild-test-database: Clean up helper functions.
We now have helpers for the two places where
we create databases.

There was already one helper in place, and
I gave it a more concrete name, to match
its actual database name in postgres.
2020-04-22 09:24:42 -07:00
Steve Howell 8c2f8c8bcc db tools: Refactor generate-fixtures/rebuild-test-database.
So `generate-fixtures` only ever did 9 lines of code (really
3 lines of actual code) in its normal mode of operation.

But it was cluttered with lots of stuff that really only
happened when you called it with the `-force` option, which
was only invoked by `rebuild-test-database`.

Now we inline most of the code into `rebuild-test-database`.

And now `generate-fixtures` is simple (and doesn't support
a `-force` flag.
2020-04-22 09:24:42 -07:00
Steve Howell 239474124e test-backend: Remove generate_fixtures option.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).

The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.

The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.

And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.

This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.

The other caller wasn't using `use_force`.
2020-04-22 09:24:42 -07:00
Steve Howell 54151bb548 db tools: Rename do-destroy-*database.
The new tools now have more concise, more parallel names:

    - rebuild-dev-database
    - rebuild-test-database

The actual implementations are still pretty different:

rebuild-dev-database:
    mostly delegates to 5 management scripts

rebuild-test-database:
    is a very thin wrapper for generate-fixtures

We'll try to clean that up a bit soon.
2020-04-22 09:24:42 -07:00
Tim Abbott b5f2ba5566 tools: Move test-emoji-name-scripts into subdirectory. 2020-04-21 16:19:45 -07:00
Steve Howell 837369d339 tools: Remove js-dep-visualizer.
This tool was part of a very ad hoc investigation
during 2017 into our JS dep dependencies.

It's very out of date, and it has a non-trivial
maintenance cost, as these type of tools seem
to come up in every code sweep.
2020-04-21 16:18:29 -07:00
Anders Kaseorg 9620611ec9 check-openapi: Restore functionality after OpenAPI definitions moved.
Commit 35577a1f66 (#9406) moved the
OpenAPI definitions to zerver/openapi, and this script was not
updated, so it has been validating nothing for two years.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 16:11:56 -07:00
Anders Kaseorg 7bbbc46aad check-openapi: Remove check_duplicate_operation_ids.
This is handled by swagger-parser now.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 16:11:56 -07:00
arpit551 56a5beddc0 ci: Fixed errors related to running services in ci production build.
Restart postgres service if provision is called in production test suite.
This is required because terminate-psql-sessions script (used
in tools/ci/setup-production) throws error if postgres service is not running.

Restart rabbitmq service if provision is called in production test suite.
This is done to start the node as Circle CI don't start services on installation.

Removed memcached restart as flush-memcached script (which is furthur
used in tools/ci/production) throws UNKNOWN READ FAILURE if memcached is restarted
in development.
2020-04-21 14:46:40 -07:00
arpit551 e6edf469ee ci: Remove the need of using TRAVIS in env.
Since now we want to use production suites on Circle CI so there
is no need to set TRAVIS in env while running scripts.

CIRCLECI is set default in the enviroment of Circle CI builds
so we can use it directly.

Also Travis CI had rabbitmq-server installed so we had to add workaround
in install script to avoid the error. That workaround is removed.
2020-04-21 14:46:40 -07:00
arpit551 4946405e3a ci: Change production setup in ci according to bionic and Circle CI.
Used postgres 10 inplace of postgres 9.5 as it is used in Bionic.

Upgraded nginx version in success-http-headers which is in CircleCI
Bionic enviroment.
2020-04-21 14:46:40 -07:00
arpit551 dee8de3b88 ci: Replaced travis with test suite while setting production setup on ci. 2020-04-21 14:46:40 -07:00
arpit551 80af201ef9 ci: Removed hacky code required for travis CI.
Removed hacky code required for travis CI in production-helper
and setup production.
2020-04-21 14:46:40 -07:00
Steve Howell 900fe02471 tools: Remove replacer.
This seems to be a complicated wrapper around `sed`.
It's not used anywhere or documented anywhere.  I
assume it was used in some code sweep or something.

See cec45b0ae5
2020-04-21 14:12:57 -07:00
Steve Howell 735f098ff5 tools: Remove get-handlebars-vars.
I built this in 2013 to help me quickly build
some sample data for our very early node tests,
I believe.

Ever since then it's just been changed for various
code sweeps.  Also, if we wanted to resurrect this
idea for some reason, we now have the template
parser that was written since then.
2020-04-21 14:12:57 -07:00
Steve Howell 3dd1a9211c tools: Remove html-grep.
I built this in August 2016, but I haven't used it
since, and all the more recent commits are just for
things like mypy sweeps.
2020-04-21 14:12:57 -07:00
Steve Howell 5bf43b77df tools: Remove django-template-graph.
This tool was last modified in 2016, has no
documentation, and requires `dot` to run, so
I think we can remove it to reduce some clutter
in the tools directory.
2020-04-21 14:12:57 -07:00
Steve Howell 2488438d90 tools: Convert fix-unused-css to look at zulip.scss.
This tool is hackish and incomplete, but it's still
worth looking at if somebody wants to hunt down
obsolete CSS.  There are probably better ways to
tackle this problem, so we should eventually just
remove this tool, but it's pretty low maintenance.

Current output looks like this:

    $ ./tools/find-unused-css
    actions_hovered
    actions_link
    bookend_tr
    btn-skip
    company-name
    flatpickr-months
    label_for_text
    loading_more_messages_indicator_box
    loading_more_messages_indicator_box_container
    logoimage
    messages-collapse
    messages-expand
    numInputWrapper
    page_loading_indicator_box
    page_loading_indicator_box_container
    skinny-user-gravatar
    sp-input
    summary_colorblock
    summary_row
    summary_row_private_message
    tutorial-done-button
2020-04-21 14:12:57 -07:00
Vishnu KS 87fdef2472 team: Don't remove duplicate commits inside the retry loop.
Else for each retry the duplicate commits would be removed
again and again from the contributor's zulip/zulip commits.
This is a bug in the original commit
6bed6ccdcf that added the
functionality to remove duplicate commits.
2020-04-21 13:41:10 -07:00
Puneeth Chaganti 7a5033852d provision: Add dependencies for Chromium used by latest puppeteer.
a51902a870 upgraded the puppeteer version to
3.0.0, which upgraded the Chromium version being used[1]. This version of
Chromium requires a couple of extra libraries to be installed.

[1]: https://github.com/puppeteer/puppeteer/releases/tag/v3.0.0
2020-04-20 23:25:45 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell 2bd836bc4c provision: Promote pygments import to module scope.
There's no real reason to do the lazy import any
more, as we use this unconditionally inside `main`
(indirectly), and `provision_inner` runs after we
have set up the venv.
2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Steve Howell ca49f38619 provision: Extract helpers for paths to hash.
I make these all functions for consistency,
and in particular I want to continue to avoid
`glob.glob` calls until we are actually
computing hashes.

This is mostly a prep to allow us to do
hashing in two separate places:

    - check hashes
    - update hashes

We would only update hashes **after** running
processes anew.

For `provision_inner` I considered using a
class to put the three path-related helpers
into a mini namespace, but it felt too heavy.

It wouldn't be completely implausible here
to extract something like a JSON config
file that has a list of globs for each
process that we do path-hashing for, but I
want to clean up other stuff first.
2020-04-20 15:06:47 -07:00
Steve Howell 4822f8d7d6 refactor: Add Database.template_status.
This is mostly a pure code move from
template_database_status().
2020-04-20 15:06:47 -07:00
Steve Howell 0ea4f727d4 refactor: Change params to template_database_status(). 2020-04-20 15:06:47 -07:00
Steve Howell 5c5d85cf19 test databases: Add Database.run_db_migrations().
We can reduce some code duplication by having this
on the class.
2020-04-20 15:06:47 -07:00
Puneeth Chaganti e55d967f6e generate_zulip_bots_static_files: Lazy import cairosvg.
Importing cairosvg in fails in production, because `libgtk-3-dev` is not
available. This commit moves the import for `cairosvg` into the function
where it is used. This code will soon be moved into a separate script that
will not be run on production.
2020-04-20 09:14:08 -07:00
Anders Kaseorg 438a545477 gitlint-rules: Fix bogus type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 13:36:58 -07:00
Aman Agrawal b13beb5ea8 circleci: Remove Xenial build as we are deprecating Xenial.
Docs updated to reflect the change.
2020-04-19 11:38:49 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Anders Kaseorg d3c55c166e requirements: Upgrade mypy from 0.761 to 0.770.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Anders Kaseorg 21c2a4cc42 update-locked-requirements: Generate prod locks from dev locks.
This guarantees that we don’t accidentally upgrade one without the
other, which could happen for example due to different third-party
version constraints between the two.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Steve Howell 7eb6d32d59 provision: Let build_emoji build its own cache.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.

And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.

There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient.  The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.

(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)

ASIDE:

Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":

    node_modules
    yarn.lock
    zulip-emoji-cache
    zulip-npm-cache
    zulip-py3-venv
    zulip-thumbor-venv
    zulip-venv-cache
    zulip-yarn

I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?

    ./var/webpack-cache
    ./var/mypy-cache

In `docs/subsystems/emoji.md` we say this:

```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```

I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.

And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
2020-04-17 09:53:26 -07:00
Steve Howell bf3decfd0c build_emoji: Inline get_success_stamp.
So, `source_emoji_dump` is not the greatest variable
name, but at least we now define it relative to its
parent instead of the `.success-stamp` file.  (And
then `success_stamp` is just another join.)
2020-04-17 09:45:59 -07:00
Steve Howell aae28e41d0 provision: Extract create_var_directories().
This fixes the pattern of
SIX_DIR_PATH_CONSTANTS_IN_ALL_CAPS
just to create a few subdirectories.
2020-04-17 09:45:59 -07:00
Steve Howell bc62796276 provision: Check for compiled templates.
If the directory `templates/zerver/emails/compiled/`
is missing, then we need to run `inline_email_css`
again.

This can happen if somebody gets overzealous about
cleaning untracked files.
2020-04-17 09:45:59 -07:00
Steve Howell ec416a315f provision: Extract need_to_run_inline_email_css.
This extraction also prevents an unnecessary
call to `glob.glob` when `is_force` is `True`.
2020-04-17 09:45:59 -07:00
Steve Howell 66234850ba provision: Extract need_to_run_build_pygments_data.
This is more encapsulated and more efficient.

In the cases where `is_force` is `True` or
`pygments_data.json` is missing, we now avoid
the unnecessary step of importing `pygments`, at
least up front.

(Of course, we probably import that once we generate
the artifacts.)
2020-04-17 09:45:59 -07:00
Steve Howell 796351ed9e provision: Detect missing language_name_map.json.
If somebody is having issues with provision, it's
plausible they'll do something like `git clean -fX`
to clean up old artifacts of earlier provision runs,
as part of debugging things.

We defend against this by detecting the most obvious
symptom as cheaply as possible.
2020-04-17 09:45:59 -07:00
Steve Howell 765d40b34c provision: Extract need_to_run_compilemessages.
This is slightly more than a code move, as we
now skip some unnecessary file I/O when
`is_force` is `True`.
2020-04-17 09:45:59 -07:00
Steve Howell 067196c508 provision: Simplify `is_force` codepaths.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.

If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.

Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.

For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.

We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
2020-04-17 09:45:59 -07:00
Steve Howell f98843d197 provision: Add build_emoji prefix to message.
This is similar to what we do in generate_secrets,
and it makes the output from a no-op provision
a bit more consistent.
2020-04-17 09:45:59 -07:00
Steve Howell 478b50dde3 provision: Improve message for configure-rabbitmq.
We change the message for skipping RabbitMQ
configuration to match nearby messages:

    No need to run `tools/setup/build_pygments_data`.
    No need to run `scripts/setup/inline_email_css.py`.
    No need to run `scripts/setup/configure-rabbitmq.
    No need to regenerate the dev DB.
    No need to regenerate the test DB.
    No need to run `manage.py compilemessages`.
2020-04-17 09:45:59 -07:00
Puneeth Chaganti 6b3358b14f generate-integration-docs-screenshot: Allow sending custom headers.
This commit makes it convenient to specify additional headers to send in the
integration webhook request from the command line.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 84e411c543 generate-integration-docs-screenshot: Fix crash on ignored fixtures.
When using a fixture that is correctly handled by our webhooks, but don't
cause a notification message to be sent, the tool shouldn't crash.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 9248b16e86 message-screenshot: Create image directory if it doesn't exist. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti cee240fb74 generate-integration-docs-screenshot: Raise ArgumentTypeError.
When an invalid fixture is passed as an argument to the tool, raise an
`ArgumentTypeError`, instead of a `ValueError`.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti ae754887eb generate-integration-docs-screenshot: Message to run dev server.
If the dev server is not running, when this tool is run, we print a clear
error message to start the dev server first, and then run this tool.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 874e473fc4 generate-integration-docs-screenshot: Refactor code into functions. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti 4719226d0f generate-integration-docs-screenshot: Ensure puppeteer is installed. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti dd6966b9af test-js-with-puppeteer: Extract code to prepare for puppeteer run. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti 572e188b36 tools: Extract code to find js test files to test_script. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti 26e199035d test_script: Use Python3 style type hints. 2020-04-17 09:41:55 -07:00
Tim Abbott 561ded5e59 production: Fix generating bot static files in production.
For upgrade-zulip-from-git to work, we need to be able to run
update-prod-static on production systems, which means provision code
like this cairosvg logic needs to be there for now.
2020-04-17 09:25:48 -07:00
Puneeth Chaganti 464cbcd9c1 tools: Capture incoming bot message screenshot using puppeteer. 2020-04-16 19:25:13 -07:00
Puneeth Chaganti 4d2ce607c9 tools: Add script to trigger webhook notification using fixtures.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.

Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.

Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
2020-04-16 19:25:13 -07:00
Puneeth Chaganti 4c142b778a generate_zulip_bots_static_files: Move imports outside the function. 2020-04-16 19:25:13 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
wowol fcf4c9639e emails: Add tests for sending custom emails.
This requires configuring the linter to allow using "subject" in these
files.
2020-04-14 10:50:43 -07:00
Tim Abbott 4f23f13c55 fetch-contributor-data: Fix running in production.
The previous logic attempted to check for a venv, rather than putting
itself into the virtualenv automatically.
2020-04-12 12:14:07 -07:00
Aman Agrawal cf4349d1c1 tools: Add script to start database services for WSL.
It needs to be run everytime WSL is restarted.
2020-04-11 12:05:33 -07:00
Steve Howell b8552e9524 test-js-with-node: Show correct coverage link in color.
We figure out the dev host using the same logic as
dev_settings.py, so that we don't use wrong things
like 127.0.0.1 for droplet users.

And we display the link in cyan.
2020-04-11 07:05:04 -04:00
Steve Howell f28a1a4c6c test-js-with-node: Remove stray comments.
When I introduced EXEMPT_FILES, I forgot
to delete these comments, and they became
totally nonsensical after sorting the list
of files.
2020-04-11 07:05:04 -04:00
wowol 71dfb85607 emails: Allow usage of word subject in send_email.py. 2020-04-10 15:53:35 -07:00
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Anders Kaseorg d8fce9417b run-dev: Automatically set EXTERNAL_HOST for droplet dev servers.
As of commit 99242138a7 (#14530), this
is required when visiting a droplet dev server remotely.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 16:50:13 -04:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 00:48:02 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Rohitt Vashishtha 3f6541b306 bugdown: Add 'none' as alias for no syntax highlighting in codeblocks.
This is a precursor to #14404.
2020-04-09 16:02:02 -07:00