We had a bunch of ugly hacks to monkey patch things due to upstream
being temporarily unmaintained and not merging PRs. Now the project is
active again and the fixes have been merged and included in the latest
version - so we clean up all that code.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.
Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:
* urls: django.urls.resolvers.RegexURLPattern has been replaced by
django.urls.resolvers.URLPattern; affects OpenAPI code and related
features which re-parse Django's internals.
https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.
There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.
Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.
Fixes#10835.
responses is an module analogous to httpretty for mocking external
URLs, with a very similar interface (potentially cleaner in that it
makes use of context managers).
The most important (in the moment) problem with httpretty is that it
breaks the ability to use redis in parts of code where httpretty is
enabled. From more research, the module in general has tendency to
have various troublesome bugs with breaking URLs that it shouldn't be
affecting, caused by it working at the socket interface layer. While
those issues could be fixed, responses seems to be less buggy (based
on both third-party reports like ckan/ckan#4755 and our own experience
in removing workarounds for bugs in httpretty) and is more actively
maintained.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
For a long time, we've been only doing the zxcvbn password strength
checks on the browser, which is helpful, but means users could through
hackery (or a bug in the frontend validation code) manage to set a
too-weak password. We fix this by running our password strength
validation on the backend as well, using python-zxcvbn.
In theory, a bug in python-zxcvbn could result in it producing a
different opinion than the frontend version; if so, it'd be a pretty
bad bug in the library, and hopefully we'd hear about it from users,
report upstream, and get it fixed that way. Alternatively, we can
switch to shelling out to node like we do for KaTeX.
Fixes#6880.
Send the `csrftoken` and `sessionid` cookies with `SameSite=Lax`.
This adds a layer of defense against CSRF attacks and matches the new
default in Django 2.1:
https://docs.djangoproject.com/en/2.1/releases/2.1/#samesite-cookies
This can be reverted when we upgrade to Django ≥ 2.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It happens that commonmark, python-jose, and python-twitter don’t
actually use future on Python 3, and moto uses aws-xray-sdk in such a
way that it doesn’t use future, but this was a weird game to be
playing just to remove one dependency, and it caused CI failures after
new releases of future, so let’s just include it.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This avoids expensive `git clone` operations during provisioning and
installation, and will also allow us to use `pip-compile
--generate-hashes` for better security.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This makes no changes to the locked versions in *.txt, but it reduces
duplicate information and gives us sane workflows for
* upgrading packages: remove some or all lines from *.txt and re-run
`update-locked-requirements`;
* marking packages as intentionally held back: add a version bound
to *.in with an explanatory comment.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The reason that `pip-tools` running on Python 3 didn’t detect the
right requirements for `thumbor` on Python 2 is simply that some of
them are conditional on the Python version.
As for the requirements that had been manually added as a workaround:
`backports-abc` and `singledispatch` are now correctly detected, while
`backports.ssl-match-hostname` was vendored into `urllib3` some time
ago and `certifi` is no longer necessary.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Also move it to dev.in.
Other notes for posterity: this should have been installed with a
pinned commit hash, and could have been installed directly from the
upstream Git repository, even on Python 3.7, as long as Cython was
installed as well.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It’s unclear why pip-tools considers these packages unsafe, and
excluding them from being pinned has resulted in nondeterministic
output that makes our test suite unhappy.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
futures is no longer there to be removed. Be clear about why we’re
removing future (it was never a “pip-tools bug”), and leave evidence
behind to help indicate how long that will be needed.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
These are not the latest versions, but pip-tools 3.9.0 or 4.0.0 fails
to resolve dependencies from Git URLs:
pip._internal.exceptions.DistributionNotFound: No matching distribution found for zulip==0.6.1_git (from -r requirements/common.in (line 135))
while pip 19.2 breaks pip-tools 3.8.0:
TypeError: __init__() got an unexpected keyword argument 'find_links'
Fixes#10802.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
zerver/openapi/python_examples.py:105: error: Argument 1 to "get_user_presence" of "Client" has incompatible type "str"; expected "Dict[str, Any]"
zerver/openapi/python_examples.py:563: error: Argument 1 to "add_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
zerver/openapi/python_examples.py:576: error: Argument 1 to "remove_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
zerver/worker/queue_processors.py:587: error: Argument "client" to "extract_query_without_mention" has incompatible type "EmbeddedBotHandler"; expected "ExternalBotHandler"
These were only missed because mypy daemon mode requires us to set
`follow_imports = skip` for the `zulip` package.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The original seems to be unmaintained
(johnsensible/django-sendfile#65). Notably, this fixes a bug in the
filename parameter, which perviously showed the Python 3 repr of a
byte string (johnsensible/django-sendfile#49).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
tools/linter_lib/pyflakes.py:35: error: Argument 3 to "run_pyflakes" has incompatible type "List[Tuple[bytes, bytes]]"; expected "List[Tuple[str, str]]"
tools/linter_lib/custom_check.py:110: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:214: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:214: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:502: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:502: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:519: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:706: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:728: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:738: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:779: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
tools/linter_lib/custom_check.py:779: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
tools/linter_lib/custom_check.py:803: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
tools/linter_lib/custom_check.py:805: error: Unsupported operand types for + ("List[Rule]" and "List[Dict[str, Any]]")
tools/linter_lib/custom_check.py:819: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
These were missed the `zulint` package was missing PEP 561 type
annotation markers, and if it’d had them, mypy daemon mode would’ve
required us to set `follow_imports = skip` for it.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.
(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We already had this as an indirect dependency, but now it's going to be
needed to write a decorator for rate limiting authenticate() functions,
so it should be added to common.in.
Fixes#11209.
This requires changing how zadd is used in rate_limiter.py:
In redis-py >= 3.0 the pairs to ZADD need to be passed as a dictionary,
not as *args or **kwargs, as described at
https://pypi.org/project/redis/3.2.1/ in the section
"Upgrading from redis-py 2.X to 3.0".
The rate_limiter change has to be in one commit with the redis upgrade,
because the dict format is not supported before redis-py 3.0.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend. These are:
* GoogleMobileOauth2Backend, the ancient code path for mobile
authentication last used by the EOL original Zulip Android app.
* The `finish_google_oauth2` code path in zerver/views/auth.py, which
was the webapp (and modern mobile app) Google authentication code
path.
This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
If a url doesn't have a scheme, browsers would treat it as a relative
url and open something like: https://chat.zulip.org/google.com instead.
This PR fixes the issue on the backend; the frontend implementation
remains out of sync and the user sending the message wouldn't see
any linkification for urls without a scheme.
Fixes#12791.
* backports-abc: For old Python versions.
* backports.ssl-match-hostname: For old Python versions.
* docopt: Has never been used directly.
* gitdb: Has never been used directly.
* ndg-httpsclient: No longer used by requests ≥ 2.12.1.
* pycrypto: Has never been used directly.
* smmap: Has never been used directly.
* typing: For old Python versions.
* typing_extensions: For old Python versions.
PROVISION_VERSION is not bumped because these were already unused
since at least the last major bump.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Reverts c09962b and 697b4b2. The lines that break look like
stripe.Invoice.finalize_invoice(stripe_invoice).
Something like stripe_invoice.finalize_invoice() would work, but it's a big
change given how the tests in test_stripe.py currently work.
This is required for Python 3.7 support; line_profiler needs a release
with Cython rerun following the Python 3.7 release, and this achieves
that goal.
This updates recommonmark so that relative links to .md files will
correctly be converted to links to html pages by Sphinx/ReadTheDocs.
This was repoprted in https://github.com/rtfd/recommonmark/issues/89
This will allow us to in a future commit use relative links to .md
files so all of the hyperlinks work both on ReadTheDocs and on GitHub.
This commit leverages the ahocorasick algorithm to build a set of user_ids
that have their alert_words present in the message. It runs in linear time
of the order of length of the input message as opposed to number of
alert_words. This is after building a ahocorasick Automaton which runs
in O(number of alert_words in entire realm) which is usually cached.
This is a major upgrade, and requires some significant compatibility
work:
* Migrating the pattern-removal logic to use the Registry feature.
* Handling the removal of positional arguments in markdown extensions.
* Handling the removal of safe mode.
Now, if you pass an api_key, we'll initialize the public room
subscribers to be whatever they were at the time the import happened.
Also, document the situation on the caveats section.
Broaden the type of the AbstractEnum __reduce_ex__ parameter to object; this
matches the parameter type specified in the latest enum.pyi file in typeshed.
Fixes#10996.
This was generated by:
* `rm -f requirements/*.txt`
* `./tools/update-locked-requirements`
and passes tests. The effect is to unpin ourselves from old versions
of a bunch of recursive dependencies of our third-party dependencies.
This optimizes the import performance of Zulip by using a fork of
pkgresources. The fork just has a single commit, which replaces
libthumbor's use of pkgresources (which is really slow to import) with
a direct query to the version number.
This fixes part of #9953.
The autenticate function now follows the signature of
Django 2.0 https://github.com/django-auth-ldap/
django-auth-ldap/commit/27a8052b26f1d3a43cdbcdfc8e7dc0322580adae
Also AUTH_LDAP_CACHE_GROUPS is depricated in favor of
AUTH_LDAP_CACHE_TIMEOUT.
This package is important in order to avoid scary-looking errors
whenever we upgrade the dependencies in thumbor.txt (where
virtualenv-clone isn't installed in the venv, and then gets installed
by the code we just added a TODO comment to.
Bunch of bug fixes.
* Allow non-ascii chars in namedtuple fields (regression
introduced fixing ticket #211).
* Fixed adaptation of arrays of arrays of nulls (ticket #325).
* Fixed building on Solaris 11 and derivatives such as
SmartOS and illumos (ticket #677).
* Maybe fixed building on MSYS2 (as reported in ticket #658).
* Allow string subclasses in connection and other places (ticket #679).
* Don’t raise an exception closing an unused named cursor (ticket #716).
* Wheel package compiled against PostgreSQL 10.4 libpq and OpenSSL 1.0.2o.
Some minor changes.
* Clean up handling for description file - pull in content
from this file into setup()
* Allows the nightly build to fail
* Add Trove version classifiers to make it explicit what
is supported
* Add python_requires to help pip
* Drop support for EOL Python 2.6 and 3.3
We add this dependancy to thumbor for no use other than making an
import possible in one of the upcoming commits. Basically we wanted to
import LOCAL_UPLOADS_DIR from zproject.prod_settings or
zproject.dev_settings and prod_settings_template.py imports
django-auth-ldap (which depends on python-ldap and django).
This seems counterproductive, but it makes it possible for us to save
significant thumbor server startup time that would have been consumed
in `get-django-setting`, and once thumbor supports Python 3, we'll
probably be merging the virtualenvs anyway (in which case this change
would become a no-op).
Mateusz Mandera
rht
Anders Kaseorg
Tim Abbott
Rafid Aslam
David Rosa
Vishnu Ks
Aman
Wyatt Hoodes
neiljp (Neil Pilgrim)
Harshit Bansal
Rohitt Vashishtha
Eeshan Garg
Priyank Patel
Rishi Gupta
Puneeth Chaganti
Mayank Madan
Samuel Searles-Bryant
Bennet Sunder
Shoumorup
Pragati Agrawal
Aaditya Arora
Yago González
Rhea Parekh
Anders Kaseorg
Aditya Bansal