This removes strict_optional = True and warn_no_return = True (which
are already the default), and adds disallow_incomplete_defs = True and
warn_unused_configs = True, as well as several options that need to
remain False for now.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, these were interpreted relative to the current directory.
This was notably causing dozens of extra var/mypy-cache directories to
be created in different places when mypy was run from Emacs Flycheck.
Use the $MYPY_CONFIG_FILE_DIR variable added in mypy 0.800.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
Also enable warn_unused_ignores. I think the fact that there are so
few of these is good evidence that it’s not a significant burden for
people fixing type errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since essentially the first use of Tornado in Zulip, we've been
maintaining our Tornado+Django system, AsyncDjangoHandler, with
several hundred lines of Django code copied into it.
The goal for that code was simple: We wanted a way to use our Django
middleware (for code sharing reasons) inside a Tornado process (since
we wanted to use Tornado for our async events system).
As part of the Django 2.2.x upgrade, I looked at upgrading this
implementation to be based off modern Django, and it's definitely
possible to do that:
* Continue forking load_middleware to save response middleware.
* Continue manually running the Django response middleware.
* Continue working out a hack involving copying all of _get_response
to change a couple lines allowing us our Tornado code to not
actually return the Django HttpResponse so we can long-poll. The
previous hack of returning None stopped being viable with the Django 2.2
MiddlewareMixin.__call__ implementation.
But I decided to take this opportunity to look at trying to avoid
copying material Django code, and there is a way to do it:
* Replace RespondAsynchronously with a response.asynchronous attribute
on the HttpResponse; this allows Django to run its normal plumbing
happily in a way that should be stable over time, and then we
proceed to discard the response inside the Tornado `get()` method to
implement long-polling. (Better yet might be raising an
exception?). This lets us eliminate maintaining a patched copy of
_get_response.
* Removing the @asynchronous decorator, which didn't add anything now
that we only have one API endpoint backend (with two frontend call
points) that could call into this. Combined with the last bullet,
this lets us remove a significant hack from our
never_cache_responses function.
* Calling the normal Django `get_response` method from zulip_finish
after creating a duplicate request to process, rather than writing
totally custom code to do that. This lets us eliminate maintaining
a patched copy of Django's load_middleware.
* Adding detailed comments explaining how this is supposed to work,
what problems we encounter, and how we solve various problems, which
is critical to being able to modify this code in the future.
A key advantage of these changes is that the exact same code should
work on Django 1.11, Django 2.2, and Django 3.x, because we're no
longer copying large blocks of core Django code and thus should be
much less vulnerable to refactors.
There may be a modest performance downside, in that we now run both
request and response middleware twice when longpolling (once for the
request we discard). We may be able to avoid the expensive part of
it, Zulip's own request/response middleware, with a bit of additional
custom code to save work for requests where we're planning to discard
the response. Profiling will be important to understanding what's
worth doing here.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
zerver/openapi/python_examples.py:105: error: Argument 1 to "get_user_presence" of "Client" has incompatible type "str"; expected "Dict[str, Any]"
zerver/openapi/python_examples.py:563: error: Argument 1 to "add_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
zerver/openapi/python_examples.py:576: error: Argument 1 to "remove_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
zerver/worker/queue_processors.py:587: error: Argument "client" to "extract_query_without_mention" has incompatible type "EmbeddedBotHandler"; expected "ExternalBotHandler"
These were only missed because mypy daemon mode requires us to set
`follow_imports = skip` for the `zulip` package.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
tools/linter_lib/pyflakes.py:35: error: Argument 3 to "run_pyflakes" has incompatible type "List[Tuple[bytes, bytes]]"; expected "List[Tuple[str, str]]"
tools/linter_lib/custom_check.py:110: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:214: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:214: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:502: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:502: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:519: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:706: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:728: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:738: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:779: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:779: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
tools/linter_lib/custom_check.py:803: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
tools/linter_lib/custom_check.py:805: error: Unsupported operand types for + ("List[Rule]" and "List[Dict[str, Any]]")
tools/linter_lib/custom_check.py:819: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
These were missed the `zulint` package was missing PEP 561 type
annotation markers, and if it’d had them, mypy daemon mode would’ve
required us to set `follow_imports = skip` for it.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Anders Kaseorg
Mohit Gupta
Anders Kaseorg
Tim Abbott
Wyatt Hoodes
neiljp (Neil Pilgrim)