Commit Graph

38666 Commits

Author SHA1 Message Date
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 123790a72d webhooks: Fix invalid JSON in Hello World test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 5043f6039b webhooks: Fix invalid JSON in Freshdesk test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg ec6856bc3b webhooks: Fix invalid JSON in Dialogflow test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 114b0a2982 message_edit: Output a list, not a reversed iterator.
A list_reverseiterator object is not JSON serializable, and orjson
enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 03d2540899 export: Post-process authentication_methods BitHandler field to list.
A BitHandler object is not JSON serializable, and orjson enforces
this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 2cf2547b27 export: Add missing datetime fields for post-processing.
datetime objects are not ordinarily JSON serializable.  While both
ujson and orjson have special cases to serialize datetime objects,
they do it in different ways.  So we want to fix the post-processing
code to do its job.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Dinesh 62d3107ac9 puppeteer: Migrate mention test from casper.
Also add helper functions needed.
`select_item_via_typeahead` has been ported from casper
and is exactly same in puppeteer to as I couldn't find
any better way for that purpose.
2020-08-11 10:43:29 -07:00
Dinesh feb2556253 puppeteer: Migrate admin test from casper.
Also adds a required helper function to common.js.
2020-08-11 10:41:44 -07:00
Dinesh a1934797ea puppeteer: Migrate navigation test from casper.
Also adds a helper function that is used in later
tests too.
2020-08-11 10:41:44 -07:00
Dinesh 184b2b055d tests: Remove casper tests that were added in puppeteer. 2020-08-11 10:41:43 -07:00
Dinesh 3b3ed0c2d5 puppeteer: Replace `hidden: false` with `visible: true`.
I had a misconception with hidden and visible options
and thought `hidden: false` was same as `visible: true`
and other way too.

But `hidden: false` or `visible: false` does nothing
more than checking if the selector exists.

Also, to mention, `visible: false`'s were fixed in
33e19fa7d1
2020-08-11 10:41:36 -07:00
Alex Vandiver 28c627452f sentry: Ignore DisallowedHost messages.
This is a misconfiguration of the client, not the server.
2020-08-11 10:38:14 -07:00
Alex Vandiver f00ff1ef62 middleware: Make HostDomain into a process_request, not process_response.
It is more suited for `process_request`, since it should stop
execution of the request if the domain is invalid.  This code was
likely added as a process_response (in ea39fb2556) because there was
already a process_response at the time (added 7e786d5426, and no
longer necessary since dce6b4a40f).

It quiets an unnecessary warning when logging in at a non-existent
realm.

This stops performing unnecessary work when we are going to throw it
away and return a 404.  The edge case to this is if the request
_creates_ a realm, and is made using the URL of the new realm; this
change would prevent the request before it occurs. While this does
arise in tests, the tests do not reflect reality -- real requests to
/accounts/register/ are made via POST to the same (default) realm,
redirected there from `confirm-preregistrationuser`.  The tests are
adjusted to reflect real behavior.

Tweaked by tabbott to add a block comment in HostDomainMiddleware.
2020-08-11 10:37:55 -07:00
Alex Vandiver 816f91ae27 auth: Remove redirect on invalid realm during password reset.
This redirect was never effective -- because of the
HostDomainMiddleware, all requests to invalid domains have their
actual results thrown away, and replaced by an "Invalid realm" 404.
These lines are nonetheless _covered_ by coverage, because they do
run; the redirect is simply ineffective.  This can be seen by the test
that was added with them, in c8edbae21c, actually testing the contents
for the invalid realm wording, not the "find your accounts" wording.
2020-08-11 10:27:29 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg 64b2212c23 docs: Correct “setup” as a verb to “set up”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg fb2e56e3c9 docs: Fix capitalization of keyboard keys.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg c155403884 docs: Fix various capitalization errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:52 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Alex Vandiver e3a093b73f sentry: Add two more languages seen. 2020-08-11 10:15:02 -07:00
Alex Vandiver 34250d76a2 sentry: Don't drop (and server error) on languages we don't know.
It is better to show the stacktrace without highlighting than to drop
it entirely.
2020-08-11 10:15:02 -07:00
Alex Vandiver 569de55b82 sentry: Rename mapping variable to be more explicitly-named. 2020-08-11 10:15:02 -07:00
Alex Vandiver 712affd4df sentry: Document default topic. 2020-08-11 10:15:02 -07:00
Mateusz Mandera a7039c815e queue_processors: Fix UnboundLocalError in QueueProcessingWorker.
consume_time_seconds wasn't properly defined at the beginning, so when
a BaseException that isn't a subclass of Exception is thrown, the
finally: block could be entered with it still undefined.
2020-08-11 10:09:42 -07:00
Vishnu KS 4dc83a139c counts: Create 7day_actives::day counstat. 2020-08-10 17:22:19 -07:00
Anders Kaseorg 6698fedd1e docs: Change Zoom app type to user-managed.
Based on the Zoom documentation, a user-managed app seems more
appropriate for our use.

https://marketplace.zoom.us/docs/guides/build#account-level-user-managed-apps

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-10 17:00:41 -07:00
Alex Vandiver 8fc6d740cd tornado: Set a half-second default timeout on requests from django.
By defaults, `requests` has no timeout on requests, which can lead to
waiting indefinitely.  Add a half-second timeout on these; this is
applied _inside_ each retry, not overall -- that is, with retries any
of these functions may take a total of 1.5s.
2020-08-10 16:55:56 -07:00
Alex Vandiver 89cd31e07c tornado: Switch trust_env = False out for no_proxy.
Use the `no_proxy` proxy, which explicitly disables proxy usage for
particular hosts.  This is a slightly cleaner solution than ignoring
all of the environment, as removing proxies is specifically what we
are attempting to accomplish.
2020-08-10 16:55:56 -07:00
Alex Vandiver 5bea344fde tornado: Factor out common error-handling pattern into an HTTP adapter.
The change in #2764 provided a better error message on one of the
three calls into Tornado, but left the other two with the old error
message.  `raise_for_status` was used on two out of three.

Use a custom HTTPAdapter to apply this pattern to all requests from
Django to Tornado.
2020-08-10 16:55:56 -07:00
Alex Vandiver bb754e0902 tornado: Add a retry with backoff to django-to-tornado requests.
This better hides errors from users during the moments when Tornado is
being restarted.
2020-08-10 16:55:56 -07:00
Alex Vandiver e87aecf61c tornado: Make requests_client a singleton, not a global. 2020-08-10 16:55:56 -07:00
Alex Vandiver 6918556648 tornado: Swap to early-abort, for fewer long hanging blocks. 2020-08-10 16:55:56 -07:00
Alex Vandiver 39368cad3a tornado: Extract functions called from django into one module.
This makes clearer the separation of concerns.
2020-08-10 16:55:56 -07:00
Sahil Batra 33e19fa7d1
puppeteer: Replace 'visible: false' with 'hidden: true'.
We need to replace 'visible: false' with 'hidden: true' to wait
for elements to get hidden. Using 'visible: false' just checks
whether the selector exists or not and does not check whether
the element is hidden or not.
2020-08-10 16:48:53 -07:00
Tim Abbott dedb28055a stats: Fix a few strings not tagged for translation. 2020-08-10 16:26:37 -07:00
Anders Kaseorg f869701d15 Revert "vagrant: Add installation support for MacOS 10.15.6."
This reverts commit 6e0d4b9502.  It was
merged with no review and has various issues (see #16086).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-10 12:10:15 -07:00
Anders Kaseorg 18dcb86de5 Revert "provision: Fix missing <sasl/sasl.h> headers during provision."
This reverts commit cc9dc07b07.  It was
merged with no review and has various issues (see #16086).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-10 12:09:37 -07:00
Aman cc9dc07b07 provision: Fix missing <sasl/sasl.h> headers during provision.
This fixes the error of missing sasl.h when compiling python-ldap
from source during provision.
2020-08-09 23:21:13 -07:00
Aman 6e0d4b9502 vagrant: Add installation support for MacOS 10.15.6.
There are file sharing issues with the mac version 10.15.6 and
vagrant. Using parallels as a provider for vagrant fixes the issue.
2020-08-09 23:21:13 -07:00
Tim Abbott d63cf04a20 docs: Clarify how manage.py backup --output works. 2020-08-09 17:44:32 -07:00
Tim Abbott e12225ace7 realm_audit_log: Fix malformed RealmAuditLog entries.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.

This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.

Fixes #16066.
2020-08-09 16:51:29 -07:00
Alex Vandiver 83645a3115 sentry: Ignore SystemExit and similar exceptions.
There are three exceptions in Python3 which are descended from
BaseException, but not Exception: GeneratorExit, KeyboardInterrupt,
and SystemExit.  None of these are suitable to be sent to Sentry.
For example, SystemExit is raised by `sys.exit`; in that sense, it is
never "uncaught" because we chose to cause it explicitly.

Use the suggested form[1] for ignoring specific classes of exceptions.

[1] https://github.com/getsentry/sentry-python/issues/149#issuecomment-434448781
2020-08-09 16:49:12 -07:00
Shanu b5b7bc9a7c
loading: Add aria-hidden="true" attribute to loading spinners.
Fixes #16082.
2020-08-09 16:47:56 -07:00
Steve Howell 8ba55b090a node fixtures: Add id field to realm_filters tuple. 2020-08-08 11:30:46 -04:00
Steve Howell 6f97e9dfa9 mypy: Use object, not Any, in event_schema.
This requires a few redundant runtime isinstance
checks, but the extra assertions arguably make
the code more readable, and isinstance checks
are extremely negligible.
2020-08-08 11:30:46 -04:00
Anders Kaseorg 3582183fba setup_venv: Install libyaml-dev.
This will let PyYAML link against LibYAML when PyYAML is next
installed.  Due to virtualenv-clone, that won’t happen until the next
Python package removal anyway, so we don’t bother bumping
PROVISION_VERSION.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 20:58:07 -07:00
Tim Abbott ba34e2a61c help: Mark Zoom integration as stable.
We've had plenty of happy production usage at this point.
2020-08-07 16:45:30 -07:00