Commit Graph

39152 Commits

Author SHA1 Message Date
Alex Vandiver 4361ce1246 markdown: Use tlds package to keep updated list of TLDs.
Also remove a useage of "blacklist."
2020-09-21 21:03:29 -07:00
Alex Vandiver 29459ea61f release: Group and indent changes for major releases. 2020-09-21 21:03:29 -07:00
Aman Agrawal 37e77c77e2 message_fetch: Better handle unauthorized requests.
We raise MissingAuthenticationError now, which adds
`www_authenticate=session` header to the error response. This
stops modern web-browsers from displaying a login form everytime
a 401 response it sent to the client.
2020-09-21 16:07:43 -07:00
Aman Agrawal a8350ebd63 events: Disable events or presence for web-public guest. 2020-09-21 16:07:40 -07:00
Tim Abbott 7fa8bafe81 lint: Fix type of initial 0 in queue monitoring. 2020-09-21 15:47:30 -07:00
Priyansh Garg 6684247147 compose: Refactor autosize_textarea to work while editing messages.
Previously, compose_ui.autosize_textarea didn't work while editing
messages in many cases (uploading files, typeaheads, keydown handling,
etc.).

Refactored the autosize_textarea function in compose_ui to work
while editing messages too and added appropriate argument for the
introduced function parameter at all occurences of the function
use.

Also, updated the corresponding test cases.
2020-09-21 15:43:17 -07:00
Priyansh Garg abe876a4a4 compose: Fix auto-resize issue on uploading files in preview mode.
On uploading a few files from markdown_preview mode of compose box and
then switching back to edit mode, the compose box doesn't get resized.
It even doesn't allow to scroll through the content.

Fixed this by switching back to the edit mode everytime user uploads
some file in markdown_preview mode as there's no use of staying in
markdown_preview mode anyways after uploading a file as the preview
doesn't get updated.

Also, updated the corresponding test cases.

Fixes: #16296.
2020-09-21 15:40:12 -07:00
Alex Vandiver e5f62d083e tornado: Merge the TORNADO_SERVER and TORNADO_PORTS configs.
Having both of these is confusing; TORNADO_SERVER is used only when
there is one TORNADO_PORT.  Its primary use is actually to be _unset_,
and signal that in-process handling is to be done.

Rename to USING_TORNADO, to parallel the existing USING_RABBITMQ, and
switch the places that used it for its contents to using
TORNADO_PORTS.
2020-09-21 15:36:16 -07:00
Alex Vandiver 4b3121db0b certbot: Explicitly apt-get update before installing certbot.
There is no guarantee that the apt data is up-to-date, unless we
explicitly update.

Fixes: zulip/docker-zulip#275
2020-09-21 15:26:28 -07:00
Mateusz Mandera e2dcdc2758 queue: Increase allowed expected_time_to_clear_backlog for embed_links.
It's okay for this queue to be a bit slow, and the default limits are
kind of too low for it.
2020-09-21 15:24:04 -07:00
Mateusz Mandera cd9b194d88 queue: Eliminate useless "burst" concept in monitoring.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
2020-09-21 15:24:04 -07:00
Mateusz Mandera 810514dd9d queue: Update stats file every 30 seconds.
This system can't update stats while the queue is idle, without using
threads for this, but at least we ensure to update the file after
consuming an event if more than MAX_SECONDS_BEFORE_UPDATE_STATS passed
since the last update, regardless of the number of iterations done so
far.
2020-09-21 15:24:02 -07:00
Mateusz Mandera 40c4511a9c queue: Fix misspelled consume_iteration_counter variable. 2020-09-21 15:22:58 -07:00
Mateusz Mandera 2365a53496 queue: Fix a race condition in monitoring after queue stops being idle.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
2020-09-21 15:22:56 -07:00
Amitsinghyadav a72e9476ee api_docs: Add fragment references for all parameters.
This makes it convenient to link to a specific parameter accepted by
Zulip API endpoint.
2020-09-21 12:18:10 -07:00
Sumanth V Rao 2b9f7916c5 api: Document markdown changes for code blocks and bump API_FEATURE_LEVEL. 2020-09-18 17:15:47 -07:00
Sumanth V Rao c563cdba61 markdown: Add data-code-lang attribute for locally echoed messages.
This mimics the backend logic for adding the data-attribute -
to know what Pygments language was used to highlight the code
block - in locally echoed messages.

New test added checks our logic for canonicalizing pygments alias
(for both frontend and backend).

Other fixtures and tests amended.
2020-09-18 17:12:26 -07:00
Sumanth V Rao 4a2791e2a3 tools/build_pygments_data: Map pygments language aliases to lexer name.
We need this information in the frontend to:
    * Display the 'view in playground' option for locally echoed messages.
    * When we add a UI settings for realm admins to configure their
      playground choices, we'll need to use these canonicalized aliases
      for displaying the option.

Hence, this tweaks the tool which generates pygments_data.json to contain
the data we need.

Bumping major PROVISION_VERSION since folks need to provision in both
directions.

Tests amended.
2020-09-18 17:12:26 -07:00
Sumanth V Rao 564b199fe6 markdown: Escape lang when echoing back custom non-pygments languages.
In ae58ed5a7 we decided to echo back the text, when no Pygments lexer
matching that language was found. When we do so, we must take care to
HTML escape the lang before wrapping it in a data-code-language attribute.

Tweaked by tabbott to make clear the escaping is defensive.
2020-09-18 17:12:11 -07:00
Ryan Rehman d6fce41800 stream edit: Update stream subscription info text.
This changes the success text of the `subscriber_list_add`
form to display the subscribed and already subscribed users
on success. We also display the user profile as a popover.

Previously we would only display the email ids of the already
subscribed users.

Formatting tweaked by tabbott.
2020-09-18 16:53:01 -07:00
Tim Abbott a4e499a543 docs: Adjust spelling s/back end/backend/. 2020-09-18 16:39:28 -07:00
Alex Vandiver 2a12fedcf1 tornado: Remove explicit tornado_processes setting; compute it.
We can compute the intended number of processes from the sharding
configuration.  In doing so, also validate that all of the ports are
contiguous.

This removes a discrepancy between `scripts/lib/sharding.py` and other
parts of the codebase about if merely having a `[tornado_sharding]`
section is sufficient to enable sharding.  Having behaviour which
changes merely based on if an empty section exists is surprising.

This does require that a (presumably empty) `9800` configuration line
exist, but making that default explicit is useful.

After this commit, configuring sharding can be done by adding to
`zulip.conf`:

```
[tornado_sharding]
9800 =              # default
9801 = other_realm
```

Followed by running `./scripts/refresh-sharding-and-restart`.
2020-09-18 15:13:40 -07:00
Alex Vandiver ece0aaa6cc tests: Remove use_db flag from test_server_running.
It was only ever true, since 2563f93d46.
2020-09-18 15:13:40 -07:00
Alex Vandiver 1928696829 tests: Remove three references to Casper in the comments. 2020-09-18 15:13:40 -07:00
Alex Vandiver 922940e223 tornado: Be more explicit about tornado configuration in testing.
Passing this through the environment needlessly complicates the logic.
2020-09-18 15:13:40 -07:00
Alex Vandiver f638518722 tornado: Move default production port to 9800.
In development and test, we keep the Tornado port at 9993 and 9983,
respectively; this allows tests to run while a dev instance is
running.

In production, moving to port 9800 consistently removes an odd edge
case, when just one worker is on an entirely different port than if
two workers are used.
2020-09-18 15:13:40 -07:00
Alex Vandiver 5c806fbd52 tornado: Remove a misleading comment and reformat.
tornado.web.Application does not share any inheritance with Django at
all; it has a similar router interface, but tornado.web.Application is
not an instance of Django anything.

Refold the long lines that follow it.
2020-09-18 15:13:40 -07:00
Alex Vandiver 4354386e69 tornado: Remove an unused port argument.
This was added in ec065e92ee for the WebSocket codepath, which was
subsequently removed in ea6934c26d.
2020-09-18 15:13:40 -07:00
Alex Vandiver de1db2c838 sentry: Provide more metadata in queue processors.
This allows aggregation by queue, makes the event data more readily
accessible, and clears out the breadcrumbs upon every batch that is
serviced.
2020-09-18 15:13:08 -07:00
Tim Abbott 9c0d6becc5 docs: Add comment links to i18n documentation. 2020-09-18 11:44:04 -07:00
Tim Abbott a3285d4568 docs: Update changelog since 3.0 release.
Changes through 456b0dd258 are recorded here.
2020-09-18 11:36:43 -07:00
neiljp (Neil Pilgrim) ec60b1e757 typeahead: Add `/settings` slash command. 2020-09-18 11:36:12 -07:00
Aman Agrawal ef19e4870d css: Don't use `Monaco` font for <code>.
bootstrap sets <code> to use `Monaco` font by default. We don't
want to use this font since some characters are not clearly
readable like `()` appearing as `0`.

Hence, we use Menlo font by default if available.

Since `Monaco` font is only installed in macOS by default, this
mostly affected mac users.
2020-09-18 11:35:13 -07:00
Alex Vandiver cbede2d3fc circleci: Put the branch name into the topic on failure. 2020-09-18 10:37:56 -07:00
Alex Vandiver 173ee2177b docs: Mention pulling release notes back over. 2020-09-17 17:16:06 -07:00
Tim Abbott 456b0dd258 docs: Update changelog for 3.1/3.2 releases. 2020-09-17 16:54:08 -07:00
Alex Vandiver 5a9b6feb18 tornado: Retry POST requests from Django to Tornado.
While urllib3 retries all connection errors, it only retries a subset
of read errors, since not all requests are safe to retry if they are
not idempotent, and the far side may have already processed them once.
By default, the only methods that are urllib3 retries read errors on
are GET, TRACE, DELETE, OPTIONS, HEAD, and PUT.  However, all of the
requests into Tornado from Django are POST requests, which limits the
effectiveness of bb754e0902.

POST requests to `/api/v1/events/internal` are safe to retry; at worst,
they will result in another event queue, which is low cost and will be
GC'd in short order.

POST requests to `/notify_tornado` are _not_ safe to retry, but this
codepath is only used if USING_RABBITMQ is False, which only occurs
during testing.

Enable retries for read errors during all POSTs to Tornado, to better
handle Tornado restarts without 500's.
2020-09-17 16:50:36 -07:00
Dinesh 1dc44703f1 auth: Remove redundant call to pad_method_dict().
This was called in both if and else with the same argument.
I believe there's no reason for it to exist twice and having
it just once would be a bit cleaner.
2020-09-17 13:41:41 -07:00
Anders Kaseorg 144296f9c0 release-checklist: Restore and clarify -dev version/tag for master.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-17 13:04:12 -07:00
Anders Kaseorg 22eefdd3cd zulip-icons: Set writeFiles to false.
webfonts-loader now defaults writeFiles to true, which makes spurious
copies of zulip-icons.{css,eot,svg,ttf,woff,woff2} for no reason.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-17 08:06:23 -04:00
sankalp 737d796e1c devlogin: Fix handling of fragments in the URL.
When a fragment (i.e. section starting with `#`) is present in the URL
when landing the development login page, dev-login.js used to append
it to the appends it to the `formaction` attribute of all the input
tag present in the `dev_login.html`.

This made sense before 139cb8026f, which
adjusted the structure of how `next` was passed.

To fix this, we adjust the JavaScript login to set the `next` hidden
input instead.

fixes #16215
2020-09-16 14:31:00 -07:00
Aman Agrawal 2befc007cf landing-page: Don't transition all li elements.
This looks weird and shouldn't be present in the first place.
2020-09-16 10:29:23 -07:00
Alex Vandiver 9d028aeb1b release: Update release process. 2020-09-16 10:28:53 -07:00
Aman Agrawal d9814e5e1d recent_topics: Test deleting messages not stored locally.
This commits adds test coverage for
5142bdd317.
2020-09-15 17:03:27 -07:00
Aman Agrawal 93a41e95a1 message_util: Extract get_topics_for_message_ids. 2020-09-15 17:03:27 -07:00
Anders Kaseorg b7874ac82e install-node: Upgrade Node.js to 12.18.4 and Yarn to 1.22.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg f9f9d6aed4 zulip-icons: Disable ligature generation.
This makes our icon font smaller.  We only reference its glyphs by
Unicode private-use codepoints and not by ligatures.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg b00a0bfe53 js: Use simplebar ES module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg 7c755f1394 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg a3d26d701e styles: Rename .scss files back to .css.
css-loader@4 broke @import statements referencing files with
extensions other than .css, unless those @import statements are
compiled away by another loader.  Upstream is more interested in
arguing that such @import statements are semantically incorrect than
applying the one line fix.

https://github.com/webpack-contrib/css-loader/issues/1164

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00