Commit Graph

3797 Commits

Author SHA1 Message Date
Anders Kaseorg 30ab261f91 styles: Undo calc(x + y) → calc(x - -y) workaround.
The bug this was working around does not affect our current toolchain,
as confirmed by grepping through the minified output.

(Also, this linter rule only matched calc(x + y) with two arguments
and we were already using calc($far_left_gutter_size + $left_col_size
+ 4px).)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-27 16:29:25 -07:00
sahil839 1db0775e6e tests: Use do_change_user_role in lint rules and test docs.
This commit changes test docs and lint rules to use do_change_user_role
instead of do_change_is_admin and do_change_is_guest.
2020-05-27 15:38:25 -07:00
Tim Abbott d4dfeb57fd lint: Add i18n linter rule for invalid i18n.t tags.
After seeing yet another contributor accidentally try to add i18n tags
that don't work using this pattern, it's time for a lint rule.
2020-05-27 14:09:56 -07:00
Anders Kaseorg e950344078 tsconfig: Enable resolveJsonModule.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 23:33:40 -07:00
Anders Kaseorg 8bcdf4ca97 python: Convert TypedDict declarations to Python 3.6 style.
A subset of the diff generated by pyupgrade --py36-plus
--keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Rohitt Vashishtha fa9431c0a4 markdown: Extract rendered_markdown.js to update dynamic elements.
Previously, we handled this code only in message_list_view.js.
Now we support rendering stream descriptions and some dynamic
elements can be rendered in them, so we extract this new module
and use it in both the places.
2020-05-21 12:32:52 -07:00
Rohitt Vashishtha 61f6a6e9bb linter: Allow using javascript instead of JavaScript for calling examples.
This allows us to call js code examples as follows:

`{generate_code_example(javascript)|/users:post|example()}`
2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 6dd4030e67 js-api: Add hooks to run tests and render docs of JS API examples.
This commit adds python code to call javascript_examples.js in its
two supported modes. tools/test-api asserts that the example output
is as expected, whereas the API markdown extension is used to render
these examples in the docs.
2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 1091615b17 edit-bot: Show dropdown list widget for selecting owner.
This commit removes user_dropdown in favor of the common
dropdown_list_widget for our bot edit forms.
2020-05-20 09:51:38 -07:00
Aman Agrawal 0504c61bfd semgrep: Use pattern-where-python operator to filter patterns.
See https://github.com/returntocorp/semgrep/blob/experimental/docs/config/advanced.md#pattern-where-python for usage.

This helps us minimize duplication of similar patterns.
2020-05-20 09:37:26 -07:00
Vishnu KS 9442835386 droplets: Make sure the username is lowercase.
Otherwise, if someone tries to create a droplet with a mixed case username,
the existing DNS records (which are always lowercase) would not be deleted.

Also, existing droplets would not be destroyed(--recreate) if there is a mismatch
in case. Unlike DNS records, droplets are created with case sensitive
hostnames.

Plus it's neater to have lowercase hostnames.
2020-05-20 09:32:11 -07:00
Dinesh c915e2c6cf puppeteer_tests: Use casper's webpack setup.
As puppeteer tests are similar to casper i.e both being frontend
screen scraping tests, we need to use the same webpack setup as
that of casper's instead of `build_for_most_tests` which is used
mostly for backend related tests.

This sets env variable `CASPER_TESTS` to `1` which does all the
work to make it use the same setup of casper. We would want to
rename that to `PUPPETEER_TESTS` and make changes to do the
same as casper webpack setup when we completely replace casper
with puppeteer.
2020-05-19 15:53:59 -07:00
Aman Agrawal 1a8798d9c5 ci: Disable casper tests in ci due to flaky behaviour. 2020-05-13 14:02:06 -07:00
Rohitt Vashishtha e2b0a4cba1 list-widget: Rename settings_list_widget => dropdown_list_widget.
We want to use this widget outside of the settings panels as well.
2020-05-13 10:08:51 -07:00
orientor 9170931da3 openapi: Add test for validating examples.
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.

Part of #14100.
2020-05-12 23:03:06 -07:00
Rohan Prashant 44a56a66d0 tests-backend: Bring stream_recipient.py to 100 percent coverage.
Fixes part of #7089.
2020-05-12 16:43:01 -07:00
Vishnu KS 8fb1f2af58 billing: Support downgrading plan from /billing page. 2020-05-11 17:20:54 -07:00
Vishnu KS f1b1bf5a0d billing: Add support for Zulip Standard free trial. 2020-05-11 17:20:54 -07:00
Steve Howell 9f137c3a05 bots: Extract user_dropdown widget.
We use this new widget in bot settings panels
(personal and org).  It lets you re-assign a
bot to a new human user.

Ideally we can improve this code to use
our existing list widgets to make it more
performant for realms with lots of users.
2020-05-11 16:16:58 -07:00
Puneeth Chaganti 322e84ffeb integrations: Add ERROR prefix to missing bot avatar message. 2020-05-11 14:49:11 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Anders Kaseorg caf64585e4 lint: Disallow old-style type comments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Vishnu KS 060e54e814 droplets: Add instruction to create a new base image. 2020-05-07 17:09:05 -07:00
Vishnu Ks 2bd0e59462 droplets: Disable SSH password authentication.
We cannot disable this in base droplet as DigitalOcean overrides
the file and enable password authentication during droplet creation.
2020-05-07 17:09:05 -07:00
Vishnu KS 4796206629 droplets: Use bash script instead of cloud-config as user data.
We use cloud-config for setting up the SSH keys and executing
some commands. When cloud-config sets the SSH key it doesn't override
the existing keys. So we need to set the SSH keys manually using a command
instead. This means we no longer require cloud config. We can instead
pass a bash script as the user data instead of cloud-config.

I also included a command to set the SSH key of the root.
2020-05-07 17:09:05 -07:00
Vishnu KS c07fb47aab droplets: Set template_id to the id of the new image on 18.04.
size_slug represents the plan the droplet should be created on.
Since the new base droplet is created on the cheaper but more
feature rich new plan we have to update the slug_size as well
to take advantage of the cheaper plan.
2020-05-07 17:09:05 -07:00
Vishnu KS 87ff8ada6d droplets: Move git clean to setup_repo string. 2020-05-07 17:09:05 -07:00
Vishnu KS 6079c19304 droplets: Set the hostname correctly using cloud-init. 2020-05-07 17:09:05 -07:00
Anders Kaseorg 7f52329256 check-openapi: Improve error handling.
Previously a YAML syntax error resulted in an
UnhandledPromiseRejectionWarning and a successful exit code; now it
gives a clear message and a failing exit code.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-06 16:28:55 -07:00
Aman Agrawal b25f2ed5b3 populate_db: Add option to specify max no of topics to create.
This will help us create more topics per stream than we currently do
by default.
2020-05-04 09:41:31 -07:00
Anders Kaseorg b501d04f6a semgrep: Lint against common SQL injection patterns.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:35:30 -07:00
Tim Abbott d0ffba96d9 webpack: Use cacheLoader for handlebars files.
This seems to save about 1s for run-dev.py startup time.
2020-05-03 11:03:04 -07:00
Tim Abbott af5ba9bf36 webpack: Use cacheLoader for JS files.
This seems to save over 50% of the webpack startup time in run-dev.py.
2020-05-03 11:03:04 -07:00
Anders Kaseorg baab041acd lint: Upgrade semgrep from 0.5.0-dev.2 to 0.5.0-dev.4.
The major PROVISION_VERSION bump would not be needed, but it was
missing in commit 5ab62a3514 (#14834),
so I’m doing it here.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:19:40 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Anders Kaseorg 82f629091a lint: Check docs/THIRDPARTY for format errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:10:22 -07:00
Tim Abbott 4f3976b917 process_fts_updates: Clean up logging output.
This saves a couple lines of spammy output in the run-dev.py startup
experience, and will be better output in production as well.
2020-05-01 11:51:20 -07:00
Steve Howell ce55ef6e4f tests: Use streamlined server for api/casper tests.
We now use the `--streamlined` options for `run-dev.py`
when we use `test_server_running` for `test-api` and
`test-js-with-casper` (and its experimental
replacement, `test-js-with-puppeteer`).

This means we don't slow anything down with
processes like thumbor, process_fts_updates, etc.,
which aren't meaningfully exercised by these tests.

We may eventually want some tests to meaningfully
exercise those processes, and when that day comes,
we will need to add an extra argument to
`test_server_running`, probably, but until then,
we just always set `--streamlined` in that codepath.

There is actually a tool called `./tools/test-run-dev`
that we run in CI, and it will use the full mode.
It just doesn't verify much stuff--it mostly polls
the server without testing specific features.

This seems to save about 1s of the startup time on a system I use
(~10.6s -> ~9.7s).
2020-05-01 11:37:09 -07:00
Steve Howell d9f8ec1fe7 run-dev: Add streamlined option.
For basic testing (either manual or automated), we
generally only need the server and tornado running.

Obviously, it's nice to test the complete system,
but if you're on a slow PC, the overhead can be
annoying.

Note that we don't launch any of these processes
in `--streamlined` mode:

    process_queue
    process_fts_updates
    deliver_scheduled_messages
    thumbor

And then by not launching process_queue, we avoid
several child processes.

Basic functionality like sending messages will
still work here.

The streamlined mode may be helpful in debugging
our generally slow server startup time.  Obviously,
some of the problem with startup is the auxiliary
processes here, but removing them as a variable
could help us focus on getting the core stuff fast.

Note that we still have the webpack watcher running
in streamlined mode.

For the particular case of thumbor, note that we
modify the proxy server to explicitly print and
return an error if we get a `/thumbor/*` request.
2020-05-01 11:36:43 -07:00
Steve Howell 28a2b90b04 run-dev: Extract server_processes().
We clean up the code related to launching
processes here.

We extract:

    server_processes

We also extract these helper for webpack
stuff:

    do_one_time_webpack_compile
    start_webpack_watcher

And then we move the code to actually launch
them lexically within the file (so as not to
be obscured by various function definitions).
2020-05-01 11:36:43 -07:00
Steve Howell 9cdc9cbca6 run-dev: Display ports more nicely.
Here is the new output for displaying ports:

    Zulip services will listen on ports:
       9991: web proxy
       9992: Django
       9993: Tornado
       9994: webpack
       9995: Thumbor

    Note to Vagrant users: Only the proxy port (9991) is exposed.

I tone down the yellow for the Vagrant warning, and I show
the web proxy in cyan to emphasize it.

I also extracted the code into a function, and I don't call
that function until after `app.listen()`.  (The users probably
won't notice much difference in the timing of this message, but
the message won't show if the `listen` step fails for some
reason, which I think is what we want here.)
2020-05-01 11:36:43 -07:00
Aman Agrawal 9089fd5b08 semgrep: Remove unrequired file exclude.
This file doen't contain any matching pattern which violates the
rule.
2020-05-01 11:08:47 -07:00
Aman Agrawal af0c267cd4 semgrep: Move migrations import check lint rule to semgrep.
We change how a few imports in migrations are done to be easier to
lint and more consitsent with our typical import style.
2020-05-01 11:07:53 -07:00
Aman Agrawal 8e29c88beb semgrep: Add rule to enforce no use of stream.objects.filter. 2020-05-01 11:01:14 -07:00
Aman Agrawal 06c831f146 semgrep: Reformat and correct doc link. 2020-05-01 11:01:14 -07:00
Aman Agrawal 5ab62a3514 sgrep/semgrep: Updgrade and rename sgrep to semgrep.
With its new upgrade, sgrep has been renamed to semgrep.

Rename sgrep.yml to semgrep.yml
2020-05-01 11:01:14 -07:00
Steve Howell ea52bc987d run-dev: Clean up argument parsing code.
We remove the import-tools code that was plunked
right into the middle of our command line
arguments.

Then we add a local var called `DESCRIPTION` to
fix some ugly code formatting, and we stop with the
unnecessary `r` prefix to the multi-line string.
2020-05-01 10:40:34 -07:00
Anders Kaseorg a552c2e5f9 auth: Use the clipboard instead of zulip:// for desktop auth flow.
This does not rely on the desktop app being able to register for the
zulip:// scheme (which is problematic with, for example, the AppImage
format).

It also is a better interface for managing changes to the system,
since the implementation exists almost entirely in the server/webapp
project.

This provides a smoother user experience, where the user doesn't need
to do the paste step, when combined with
https://github.com/zulip/zulip-desktop/pull/943.

Fixes #13613.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-04-30 16:45:00 -07:00