Commit Graph

39367 Commits

Author SHA1 Message Date
Alex Vandiver fd20e54c79 webhooks: Never log JsonableError to webook loggers.
These represent known errors in what the user submitted.  This is
slightly complicated by UnsupportedWebhookEventType being an instance
of JsonableError.
2020-09-22 15:11:48 -07:00
Alex Vandiver 7001004ec0 webhooks: Do not predicate on the "payload" key.
If we are to log to the webhook logger, do so no matter which
arguments are passed.
2020-09-22 15:11:48 -07:00
Alex Vandiver 1a763696f7 webhooks: Only enable webhook logging if it is a webhook.
allow_webhook_access may be true if the request allows webhook
requests, regardless of if it only used for a webhook integration.

Only actually log to the verbose webhook logger if it is explicitly a
webhook endpoint, as judged by `webhook_client_name`.  This prevents
requests for `POST /api/v1/messages` from being logged to the webhook
logger if they mistakenly contain a `payload` argument.
2020-09-22 15:11:48 -07:00
Alex Vandiver 77d1a4a5c0 webhooks: Simplify logic around is_webhook_access.
We clearly allow webhook access if we are setting the
webhook_client_name.  This removes the need for the `or`s later.
2020-09-22 15:11:48 -07:00
Alex Vandiver d24869e484 webhooks: Rename is_webhook to allow_webhook_access.
This argument does not define if an endpoint "is a webhook"; it is set
for "/api/v1/messages", which is not really a webhook, but allows
access from webhooks.
2020-09-22 15:11:48 -07:00
Alex Vandiver b515c2bcbf webhooks: Add a missing format for the realm. 2020-09-22 15:11:48 -07:00
Aman Agrawal 1b5b82e712 RealmFilterPattern: Mark converted content as AtomicString.
If multiple filters match the same string, we run into an infinite
loop of converting string into urls. To fix it, we mark the matched
string as atomic after first conversion.
2020-09-22 15:10:38 -07:00
Anders Kaseorg 0228acf0f5 rest: Add rest_path shortcut for path with rest_dispatch.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:51:00 -07:00
Anders Kaseorg 5297e4a30a urls: Use unqualified imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:51:00 -07:00
Anders Kaseorg e70f2ae58d rest: Specify rest_dispatch handlers by function, not by string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Anders Kaseorg faf600e9f5 urls: Remove unused URL names and shorten others.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Mateusz Mandera f078509105 saml: Extract logic for determining if subdomain is allowed by an attr.
This logic has some trickiness due to the need to handle root subdomain
aliases, so this is worth extracting into a helper for use in other
backends.
2020-09-21 22:58:59 -07:00
Mateusz Mandera a0dea84edb saml: Add defensive code against org_membership attr not being a list. 2020-09-21 22:58:59 -07:00
Alex Vandiver db8daf4175 linkifiers: Allow tildes in target URLs. 2020-09-21 21:04:02 -07:00
Alex Vandiver 03c6a0f182 markdown: Skip other common file extensions in linking, sort. 2020-09-21 21:03:29 -07:00
Alex Vandiver 4361ce1246 markdown: Use tlds package to keep updated list of TLDs.
Also remove a useage of "blacklist."
2020-09-21 21:03:29 -07:00
Alex Vandiver 29459ea61f release: Group and indent changes for major releases. 2020-09-21 21:03:29 -07:00
Aman Agrawal 37e77c77e2 message_fetch: Better handle unauthorized requests.
We raise MissingAuthenticationError now, which adds
`www_authenticate=session` header to the error response. This
stops modern web-browsers from displaying a login form everytime
a 401 response it sent to the client.
2020-09-21 16:07:43 -07:00
Aman Agrawal a8350ebd63 events: Disable events or presence for web-public guest. 2020-09-21 16:07:40 -07:00
Tim Abbott 7fa8bafe81 lint: Fix type of initial 0 in queue monitoring. 2020-09-21 15:47:30 -07:00
Priyansh Garg 6684247147 compose: Refactor autosize_textarea to work while editing messages.
Previously, compose_ui.autosize_textarea didn't work while editing
messages in many cases (uploading files, typeaheads, keydown handling,
etc.).

Refactored the autosize_textarea function in compose_ui to work
while editing messages too and added appropriate argument for the
introduced function parameter at all occurences of the function
use.

Also, updated the corresponding test cases.
2020-09-21 15:43:17 -07:00
Priyansh Garg abe876a4a4 compose: Fix auto-resize issue on uploading files in preview mode.
On uploading a few files from markdown_preview mode of compose box and
then switching back to edit mode, the compose box doesn't get resized.
It even doesn't allow to scroll through the content.

Fixed this by switching back to the edit mode everytime user uploads
some file in markdown_preview mode as there's no use of staying in
markdown_preview mode anyways after uploading a file as the preview
doesn't get updated.

Also, updated the corresponding test cases.

Fixes: #16296.
2020-09-21 15:40:12 -07:00
Alex Vandiver e5f62d083e tornado: Merge the TORNADO_SERVER and TORNADO_PORTS configs.
Having both of these is confusing; TORNADO_SERVER is used only when
there is one TORNADO_PORT.  Its primary use is actually to be _unset_,
and signal that in-process handling is to be done.

Rename to USING_TORNADO, to parallel the existing USING_RABBITMQ, and
switch the places that used it for its contents to using
TORNADO_PORTS.
2020-09-21 15:36:16 -07:00
Alex Vandiver 4b3121db0b certbot: Explicitly apt-get update before installing certbot.
There is no guarantee that the apt data is up-to-date, unless we
explicitly update.

Fixes: zulip/docker-zulip#275
2020-09-21 15:26:28 -07:00
Mateusz Mandera e2dcdc2758 queue: Increase allowed expected_time_to_clear_backlog for embed_links.
It's okay for this queue to be a bit slow, and the default limits are
kind of too low for it.
2020-09-21 15:24:04 -07:00
Mateusz Mandera cd9b194d88 queue: Eliminate useless "burst" concept in monitoring.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
2020-09-21 15:24:04 -07:00
Mateusz Mandera 810514dd9d queue: Update stats file every 30 seconds.
This system can't update stats while the queue is idle, without using
threads for this, but at least we ensure to update the file after
consuming an event if more than MAX_SECONDS_BEFORE_UPDATE_STATS passed
since the last update, regardless of the number of iterations done so
far.
2020-09-21 15:24:02 -07:00
Mateusz Mandera 40c4511a9c queue: Fix misspelled consume_iteration_counter variable. 2020-09-21 15:22:58 -07:00
Mateusz Mandera 2365a53496 queue: Fix a race condition in monitoring after queue stops being idle.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
2020-09-21 15:22:56 -07:00
Amitsinghyadav a72e9476ee api_docs: Add fragment references for all parameters.
This makes it convenient to link to a specific parameter accepted by
Zulip API endpoint.
2020-09-21 12:18:10 -07:00
Sumanth V Rao 2b9f7916c5 api: Document markdown changes for code blocks and bump API_FEATURE_LEVEL. 2020-09-18 17:15:47 -07:00
Sumanth V Rao c563cdba61 markdown: Add data-code-lang attribute for locally echoed messages.
This mimics the backend logic for adding the data-attribute -
to know what Pygments language was used to highlight the code
block - in locally echoed messages.

New test added checks our logic for canonicalizing pygments alias
(for both frontend and backend).

Other fixtures and tests amended.
2020-09-18 17:12:26 -07:00
Sumanth V Rao 4a2791e2a3 tools/build_pygments_data: Map pygments language aliases to lexer name.
We need this information in the frontend to:
    * Display the 'view in playground' option for locally echoed messages.
    * When we add a UI settings for realm admins to configure their
      playground choices, we'll need to use these canonicalized aliases
      for displaying the option.

Hence, this tweaks the tool which generates pygments_data.json to contain
the data we need.

Bumping major PROVISION_VERSION since folks need to provision in both
directions.

Tests amended.
2020-09-18 17:12:26 -07:00
Sumanth V Rao 564b199fe6 markdown: Escape lang when echoing back custom non-pygments languages.
In ae58ed5a7 we decided to echo back the text, when no Pygments lexer
matching that language was found. When we do so, we must take care to
HTML escape the lang before wrapping it in a data-code-language attribute.

Tweaked by tabbott to make clear the escaping is defensive.
2020-09-18 17:12:11 -07:00
Ryan Rehman d6fce41800 stream edit: Update stream subscription info text.
This changes the success text of the `subscriber_list_add`
form to display the subscribed and already subscribed users
on success. We also display the user profile as a popover.

Previously we would only display the email ids of the already
subscribed users.

Formatting tweaked by tabbott.
2020-09-18 16:53:01 -07:00
Tim Abbott a4e499a543 docs: Adjust spelling s/back end/backend/. 2020-09-18 16:39:28 -07:00
Alex Vandiver 2a12fedcf1 tornado: Remove explicit tornado_processes setting; compute it.
We can compute the intended number of processes from the sharding
configuration.  In doing so, also validate that all of the ports are
contiguous.

This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding.  Having behaviour which
changes merely based on if an empty section exists is surprising.

This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.

After this commit, configuring sharding can be done by adding to
`zulip.conf`:

```
[tornado_sharding]
9800 =              # default
9801 = other_realm
```

Followed by running `./scripts/refresh-sharding-and-restart`.
2020-09-18 15:13:40 -07:00
Alex Vandiver ece0aaa6cc tests: Remove use_db flag from test_server_running.
It was only ever true, since 2563f93d46.
2020-09-18 15:13:40 -07:00
Alex Vandiver 1928696829 tests: Remove three references to Casper in the comments. 2020-09-18 15:13:40 -07:00
Alex Vandiver 922940e223 tornado: Be more explicit about tornado configuration in testing.
Passing this through the environment needlessly complicates the logic.
2020-09-18 15:13:40 -07:00
Alex Vandiver f638518722 tornado: Move default production port to 9800.
In development and test, we keep the Tornado port at 9993 and 9983,
respectively; this allows tests to run while a dev instance is
running.

In production, moving to port 9800 consistently removes an odd edge
case, when just one worker is on an entirely different port than if
two workers are used.
2020-09-18 15:13:40 -07:00
Alex Vandiver 5c806fbd52 tornado: Remove a misleading comment and reformat.
tornado.web.Application does not share any inheritance with Django at
all; it has a similar router interface, but tornado.web.Application is
not an instance of Django anything.

Refold the long lines that follow it.
2020-09-18 15:13:40 -07:00
Alex Vandiver 4354386e69 tornado: Remove an unused port argument.
This was added in ec065e92ee for the WebSocket codepath, which was
subsequently removed in ea6934c26d.
2020-09-18 15:13:40 -07:00
Alex Vandiver de1db2c838 sentry: Provide more metadata in queue processors.
This allows aggregation by queue, makes the event data more readily
accessible, and clears out the breadcrumbs upon every batch that is
serviced.
2020-09-18 15:13:08 -07:00
Tim Abbott 9c0d6becc5 docs: Add comment links to i18n documentation. 2020-09-18 11:44:04 -07:00
Tim Abbott a3285d4568 docs: Update changelog since 3.0 release.
Changes through 456b0dd258 are recorded here.
2020-09-18 11:36:43 -07:00
neiljp (Neil Pilgrim) ec60b1e757 typeahead: Add `/settings` slash command. 2020-09-18 11:36:12 -07:00
Aman Agrawal ef19e4870d css: Don't use `Monaco` font for <code>.
bootstrap sets <code> to use `Monaco` font by default. We don't
want to use this font since some characters are not clearly
readable like `()` appearing as `0`.

Hence, we use Menlo font by default if available.

Since `Monaco` font is only installed in macOS by default, this
mostly affected mac users.
2020-09-18 11:35:13 -07:00
Alex Vandiver cbede2d3fc circleci: Put the branch name into the topic on failure. 2020-09-18 10:37:56 -07:00
Alex Vandiver 173ee2177b docs: Mention pulling release notes back over. 2020-09-17 17:16:06 -07:00