Commit Graph

124 Commits

Author SHA1 Message Date
Anders Kaseorg 21dc34cc52 open graph: HTML-escape og:description, twitter:description.
The entire idea of doing this operation with unchecked string
replacement in a middleware class is in my opinion extremely
ill-conceived, but this fixes the most pressing problem with it
generating invalid HTML.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:53:59 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Tim Abbott 983e24a7f5 auth: Use HTTP status 404 for invalid realms.
Apparently, our invalid realm error page had HTTP status 200, which
could be confusing and in particular broken our mobile app's error
handling for this case.
2019-03-14 13:50:09 -07:00
Tim Abbott de6f724bc5 middleware: Avoid doing work for statsd when not enabled.
This saves about 8% of the runtime of our total response middleware,
or equivalently close to 2% of the total Tornado response time.  Which
is pretty significant given that we're not sure anyone is using statsd
in production.

It's also useful outside Tornado, but the effect is particularly
significant because of how important Tornado performance is.
2019-02-27 17:53:15 -08:00
Tim Abbott c955b20131 middleware: Don't repreatedly regenerate open graph functions.
This avoids parsing these functions on every request, which was
adding roughly 350us to our per-request response times.

The overall impact was more than 10% of basic Tornado response
runtime.
2019-02-27 17:53:13 -08:00
Rishi Gupta 028874bab3 open graph: Remove extraneous spaces from descriptions.
Our html collects extra spaces in a couple of places. The most prominent is
paragraphs that look like the following in the .md file:
* some text
  continued

The html will have two spaces before "continued".
2019-02-11 12:05:19 -08:00
Rishi Gupta d3125f59e1 open graph: Omit .code-section navigation from open graph. 2019-02-11 12:05:19 -08:00
Rishi Gupta e1f02dc6f2 open graph: Include multiple paragraphs in description tags. 2019-02-11 12:05:19 -08:00
Anders Kaseorg f0ecb93515 zerver core: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:41:24 -08:00
Wyatt Hoodes 8eac361fb5 docs: Refactor BS work with use of cache_with_key.
Refactor the potentially expensive work done by Beautiful Soup into a
function that is called by the alter_content function, so that we can
cache the result.  Saves a significant portion of the runtime of
loading of all of our /help/ and /api/ documentation pages (e.g. 12ms
for /api).

Fixes #11088.

Tweaked by tabbott to use the URL path as the cache key, clean up
argument structure, and use a clearer name for the function.
2019-01-28 15:21:52 -08:00
Tim Abbott 9c3f38a564 docs: Automatically construct OpenAPI metadata for help center.
This is somewhat hacky, in that in order to do what we're doing, we
need to parse the HTML of the rendered page to extract the first
paragraph to include in the open graph description field.  But
BeautifulSoup does a good job of it.

This carries a nontrivial performance penalty for loading these pages,
but overall /help/ is a low-traffic site compared to the main app, so
it doesn't matter much.

(As a sidenote, it wouldn't be a bad idea to cache this stuff).

There's lots of things we can improve in this, largely through editing
the articles, but we can deal with that over time.

Thanks to Rishi for writing all the tests.
2018-12-19 10:18:20 -08:00
Tim Abbott ae6fc0a471 sessions: Resync session middleware from Django upstream.
Until we resolve https://github.com/zulip/zulip/issues/10832, we will
need to maintain our own forked copy of Django's SessionMiddleware.
We apparently let this get out of date.

This fixes a few subtle bugs involving the user logout experience that
were throwing occasional exceptions (e.g. the UpdateError fix you can
see).
2018-11-14 15:16:12 -08:00
Tim Abbott 10ac671cd4 middleware: Fix logging of query counts in websockets requests.
Apparently, we weren't resetting the query counters inside the
websockets codebase, resulting in broken log results like this:

SOCKET  403   2ms (db: 1ms/2q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   5ms (db: 2ms/3q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 3ms/4q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 3ms/5q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 4ms/6q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 5ms/7q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 5ms/8q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   3ms (db: 6ms/9q) /socket/auth [transport=websocket] (unknown via ?)

The correct fix for this is to call reset_queries at the start of each
endpoint within the websockets system.  As it turns out, we're already
calling record_request_start_data there, and in fact should be calling
`reset_queries` in all code paths that use that function (the other
code paths, in zerver/middleware.py, do it manually with
connection.connection.queries = []).

So we can clean up the code in a way that reduces risk for similar
future issues and fix this logging bug with this simple refactor.
2018-10-31 16:22:17 -07:00
Tim Abbott e4813e462b tornado: Rename async_request_{restart,stop} to mention timer.
Previously, these timer accounting functions could be easily mistaken
for referring to starting/stopping the request.  By adding timer to
the name, we make the code easier for the casual observer to read and
understand.
2018-10-16 15:39:10 -07:00
Vishnu Ks d2e4417a72 urls: Separate endpoint for signup and new realm email confirm.
This is preparation for the next commit.
2018-08-26 22:53:57 -07:00
Aditya Bansal 993d50f5ab zerver: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
neiljp (Neil Pilgrim) 2ed6da77c7 mypy: Rewrite some middleware annotations to use ViewFuncT. 2018-03-17 23:25:05 +00:00
Greg Price 53c57cf002 errors: Include request info on error mails for JSON routes too.
When our code raises an exception and Django converts it to a 500
response (in django.core.handlers.exception.handle_uncaught_exception),
it attaches the request to the log record, and we use this in our
AdminNotifyHandler to include data like the user and the URL path
in the error email sent to admins.

On this line, when our code raises an exception but we've decided (in
`TagRequests`) to format any errors as JSON errors, we suppress the
exception so we have to generate the log record ourselves.  Attach the
request here, just like Django does when we let it do the job.

This still isn't an awesome solution, in that there are lots of other
places where we call `logging.error` or `logging.exception` while
inside a request; this just covers one of them.  This is one of the
most common, though, so it's a start.
2018-03-01 15:12:32 -08:00
Callum Fraser aa9567ce37 mypy: Use Python 3 type syntax in zerver/middleware.py. 2017-12-11 18:43:24 -08:00
rht a1cc720860 zerver: Use Python 3 syntax for typing.
Tweaked by tabbott to fix some minor whitespace errors.
2017-11-28 16:49:36 -08:00
Greg Price b6cc21b438 debug: Add facility to dump tracemalloc snapshots.
Originally this used signals, namely SIGRTMIN.  But in prod, the
signal handler never fired; I debugged fruitlessly for a while, and
suspect uwsgi was foiling it in a mysterious way (which is kind of
the only way uwsgi does anything.)

So, we listen on a socket.  Bit more code, and a bit trickier to
invoke, but it works.

This was developed for the investigation of memory-bloating on
chat.zulip.org that led to a331b4f64 "Optimize query_all_subs_by_stream()".

For usage instructions, see docstring.
2017-11-28 15:52:07 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
Tim Abbott 10ab9410c9 python: Sort imports in easy files in zerver/. 2017-11-15 15:50:28 -08:00
derAnfaenger 3ac09b3e9b queue processors: Add coverage for SlowQueryWorker. 2017-11-09 15:20:40 -08:00
rht 5ee40bf718 Remove usage of six.moves.binary_type. 2017-11-09 10:00:00 -08:00
Felix Yan aea33fc738 Fix a comment typo in zerver/middleware.py. 2017-10-30 10:36:35 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
Greg Price 093bae4bc5 subdomains: Fix some implicit uses of "" for the root subdomain.
These are just instances that jumped out at me while working on the
subdomains code, mostly while grepping for get_subdomain call sites.
I haven't attempted a comprehensive search, and there are likely
still others left.
2017-10-26 10:29:17 -07:00
Tim Abbott 1ab2ca5986 subdomains: Extract zerver.lib.subdomains library.
These never really belonged with the rest of zerver.lib.utils.py, and
having a separate library makes it easier to enforce full test
coverage.
2017-10-18 22:27:48 -07:00
Alena Volkova 5515a075ec urls: Move the report endpoints to be API-style routes. 2017-10-17 22:05:56 -07:00
Tim Abbott 46485322eb middleware: Remove logic for redirecting to zulipdev.com domains.
We originally wrote this because when testing subdomains, you wanted
to be sure you were actually testing subdomains.  Now that subdomains
is the default, doesn't seem to actually be a good reason why we
should need this.
2017-10-05 23:21:02 -07:00
Tim Abbott 40c59f2878 middleware: Fix losing sub-URL when pushing to zulipdev.com.
Previously, this would always send one to homepage, making visiting
the /help/ documentation in the development environment using the
localhost URL unpleasant.

While this fixes the proximal bug, it's not clear to me that we need
this redirect logic at all, so I'm going to try removing it soon.
2017-10-05 16:36:34 -07:00
Tim Abbott 1d72629dc4 subdomains: Hardcode REALMS_HAVE_SUBDOMAINS=True. 2017-10-02 16:42:43 -07:00
rht 2949d1c1e8 zerver: Remove the rest of absolute_import. 2017-09-27 10:02:39 -07:00
neiljp (Neil Pilgrim) bb83742906 mypy: Correct 2 type annotations in zerver/middleware.py. 2017-08-15 17:50:18 -07:00
neiljp (Neil Pilgrim) e772e89fe2 mypy: Add None return path for RateLimitMiddleware.process_exception(). 2017-08-03 11:03:14 -07:00
Umair Khan 9e33917d25 rate_limiter: Upgrade max_api_calls to generic API. 2017-08-02 18:01:39 -07:00
Greg Price 192ec7c0f6 middleware: Use a proper error code on CSRF failure.
This allows us to reliably parse the error in code, rather than
attempt to parse the error text.  Because the error text gets
translated into the user's language, this error-handling path
wasn't functioning at all for users using Zulip in any of the
seven non-English languages for which we had a translation for
this string.

Together with 709c3b50f which fixed a similar issue in a
different error-handling path, this fixes #5598.
2017-07-25 14:02:12 -07:00
Greg Price 5de7c4f2af JsonErrorHandler: Take advantage of the new JsonableError structured data. 2017-07-24 16:41:22 -07:00
Greg Price 9faa44af60 JsonableError: Optionally carry error codes and structured data.
This provides the main infrastructure for fixing #5598.  From here,
it's a matter of on the one hand upgrading exception handlers -- the
many except-blocks in the codebase that look for JsonableError -- to
look beyond the string `msg` and pass on the machine-readable full
error information to their various downstream recipients, and on the
other hand adjusting places where we raise errors to take advantage
of this mechanism to give the errors structured details.

In an ideal future, I think all exception handlers that look (or
should look) for a JsonableError would use its contents in structured
form, never mentioning `msg`; but the majority of error sites might
continue to just instantiate JsonableError with a string message.  The
latter is the simplest thing to do, and probably most error types will
never have code looking for them specifically.

Because the new API refactors the `to_json_error_msg` method which was
designed for subclasses to override, update the 4 subclasses that did
so to take full advantage of the new API instead.
2017-07-24 16:41:22 -07:00
Greg Price 6dfb46dc08 JsonableError: Rename `status_code` and rely more on its default.
With #5598 there will soon be an application-level error code
optionally associated with a `JsonableError`, so rename this
field to make clear that it specifically refers to an
HTTP status code.

Also take this opportunity to eliminate most of the places
that refer to it, which only do so to repeat the default value.
2017-07-24 16:41:22 -07:00
Greg Price a5597e91a1 exceptions: Move zerver/exceptions.py under zerver/lib/.
Seems like a more appropriate place for it.  Preparation for
moving a bit more into that file.
2017-07-24 16:41:22 -07:00
Greg Price ff5013c619 JsonableError: Add types, and eliminate duck-typing.
In order to benefit from the modern conveniences of type-checking,
add concrete, non-Any types to the interface for JsonableError.

Relatedly, there's no need at this point to duck-type things at
the places where we receive a JsonableError and try to use it.
Simplify those by using straightforward standard typing.
2017-07-24 16:41:22 -07:00
Umair Khan 95fc16d90d Django 1.11: MIDDLEWARE_CLASSES setting is deprecated.
Django provides MiddlewareMixin to upgrade old-style middlewares. See
https://docs.djangoproject.com/en/1.11/topics/http/middleware/#upgrading-middleware
2017-06-13 15:04:04 -07:00
Tim Abbott 8e978df957 mypy: Fix missing import from recent mypy merge. 2017-05-25 16:22:19 -07:00
Ethan d2e72b0082 mypy: correct process response type. 2017-05-25 15:41:51 -07:00
Ethan b6e7e36c86 mypy: correct from int to str in rate limiting. 2017-05-25 15:41:49 -07:00
Tim Abbott 17f87cfc9e mypy: Fix missing Optional on process_exception. 2017-05-23 17:44:30 -07:00
Lukasz Prasol 5eaccc550a rate_limit: Make retry-after data machine-readable.
Fixes #4831.
2017-05-22 17:35:12 -07:00
hollywoodno 75d9630258 Add notifications on new logins to Zulip.
This adds helpful email notifications for users who just logged into a
Zulip server, as a security protection against accounts being hacked.

Text tweaked by tabbott.

Fixes #2182.
2017-03-25 16:50:52 -07:00