The list_to_streams() method now uses create_streams_if_needed() to
do its heavy lifting during the autocreate=True case.
This commit gets us to 100% coverage on the streams view. (The
recently created action.create_streams_if_needed() was easy
to test in isolation, and it has 100% coverage as well, so we are
not cheating here.)
Fixes: #1005.
When we push a device token, we want to clean out any other user's
tokens on the device, but not the current user's. We were wiping
away our own token, if it existed, before creating it again. This
was probably never a user-facing problem; it just made for dead code
and a little unnecessary DB churn. By excluding the current user
from the delete() call, we exercise the update path in our tests now,
so we have 100% coverage.
This commit extracts compose_views() from update_subscriptions_backend(),
and it implements the correct behavior for forcing transactions to roll
back, which is to raise an exception.
There were really three steps in this commit:
- Extract buggy code to compose_views().
- Add tests on compose_views().
- Fix bugs exposed by the new tests by converting errors to exceptions.
These annotations aren't perfect because the sqlalchemy stubs in
typeshed are broken (e.g. a `Select` doesn't have the ability to do
`.where()`, but we've at least used some typevars to make it easy to
address that when the sqlalchemy stubs are less broken).
This adds support for using PGroonga to back the Zulip full-text
search feature. Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.
Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.
PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.
Fixes#615.
[docs and tests tweaked by tabbott]
Apparently, we had incorrectly concluded that our highlight_string
search result highlighting offsets coming from tsearch_extras were
measured in bytes, whereas in fact it is measured in characters.
Also changes all links to /integrations to be relative rather than
absolute, so that users will primarily access the /integrations page
for their subdomain.
This sends an event when a new avatar is uploaded that refreshes the
avatar for all browser clients without the need to reload the browser.
Fixes: #1359.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm
This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
Most directly useful for the migration to zulipchat.com.
Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.
Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
Adds a new field default language in the zerver_realm model.
This realm level default language will be used as default language
for newly created users. Realm level default language can be
changed from the administration page.
Fixes#1372.
The previous export tool would only work properly for small realms,
and was missing a number of important features:
* Export of avatars and uploads from S3
* Export of presence data, activity data, etc.
* Faithful export/import of timestamps
* Parallel export of messages
* Not OOM killing for large realms
The new tool runs as a pair of documented management commands, and
solves all of those problems.
Also we add a new management command for exporting the data of an
individual user.
Often, users will copy email addresses with a name (rather than pure
email addresses) into the Zulip "invite users" UI. Previously, that
would throw an error.
This change also adds a get_invitee_emails_set function for parsing
emails content and a test suite for this new feature.
Fixes: #1419.
While logging through GitHub if the realm of the user doesn't
exist then we are redirected to registration page but the action
points to the complete url of the GitHub oAuth overflow.
The MitUser model caused a constant series of little problems for
users with mit.edu email addresses trying to sign up for different
Zulip servers.
The new implementation just uses conditionals on the realm object when
selecting the confirmation template to use.
In python 3, subprocess uses bytes for input and output if
universal_newlines=False (the default). It uses str for input and
output if universal_newlines=True.
Since we mostly deal with strings, add universal_newlines=True to
subprocess.check_output.
This allows the frontend to fetch data on the subscribers list (etc.)
for streams where the user has never been subscribed, making it
possible to implement UI showing details like subscribe counts on the
subscriptions page.
This is likely a performance regression for very large teams with
large numbers of streams; we'll want to do some testing to determine
the impact (and thus whether we should make this feature only fully
enabled for larger realms).
The muting logic in approximate_unread_count() was confusing
stream/subject and only using the first of many stream/subject
pairs, so it was rarely excluding rows from the count, and when
it did exclude rows, they were the wrong rows.
This fixes part of #1300, but we may want to keep the issue open.
There were a bunch of authorization and well-formedness checks in
zerver.lib.actions.do_update_message that I moved to
zerver.views.messages.update_message_backend.
Reason: by convention, functions in actions.py complete their actions;
error checking should be done outside the file when possible.
Fixes: #1150.
This is controlled through the admin tab and a new field in the Realms table.
Notes:
* The admin tab setting takes a value in minutes, whereas the backend stores it
in seconds.
* This setting is unused when allow_message_editing is false.
* There is some generosity in how the limit is enforced. For instance, if the
user sees the hovering edit button, we ensure they have at least 5 seconds to
click it, and if the user gets to the message edit form, we ensure they have
at least 10 seconds to make the edit, by relaxing the limit.
* This commit also includes a countdown timer in the message edit form.
Resolves#903.
Taiga's webhook integration would give output events in a random
order which caused test failures on python 3 (seems like python
3 is more prone to non-deterministic failures). Fix that by
sorting the outputs obtained from events before concatenating them.
Use ujson.dumps to render raw messages sent by the PagerDuty
integration instead of using pprint.pformat. pprint.pformat
gives different results on python 2 and 3.
Bitbucket changed the format of their API. The old format is still
useful for BitBucket enterprise, but for the main cloud verison of
Bitbucket, we need a new BitBucket integration supporting the new API.
This is controlled through the admin tab and a new field in the Realms
table. This mirrors the behavior of the old hardcoded setting
feature_flags.disable_message_editing. Partially resolves#903.
This fixes some tracebacks I got while testing the Zulip htpasswd SSO
functionality.
I think that this stopped working as a result of the Jinja2 migration.
We would like to know which kind of authentication backends the server
supports.
This is information you can get from /login, but not in a way easily
parseable by API apps (e.g. the Zulip mobile apps).
This prototype from Dropbox Hack Week turned out to be too inefficient
to be used for realms with any significant amount of history, so we're
removing it.
It will be replaced by https://github.com/zulip/zulip/pull/673.
This reverts commit e985b57259.
This commit will break production when we next do a release, because
we haven't done a migration to create Attachment objects for
previously uploaded files.
For a long time, rest_dispatch has had this hack where we have to
create a copy of it in each views file using it, in order to directly
access the globals list in that file. This removes that hack, instead
making rest_dispatch just use Django's import_string to access the
target method to use.
[tweaked and reorganized from acrefoot's original branch in various
ways by tabbott]
generate_random_token used to return a value of type six.binary_type
and its return type was annotated as `str`. This commit fixes that
by making it return a value of type `six.text_type` and updating
the annotation accordingly.
Also fix clashing annnotations.
This changes the type annotations for the cache keys in Zulip to be
consistently text_type, and updates the annotations for values that
are used as cache keys across the codebase.
Previously, uploaded files were served:
* With S3UploadBackend, via get_uploaded_file (redirects to S3)
* With LocalUploadBackend in production, via nginx directly
* With LocalUploadBackend in development, via Django's static file server
This changes that last case to use get_uploaded_file in development,
which is a key step towards being able to do proper access control
authorization.
Does not affect production.
Also, fixed a a small type annotation in users.py because email must
be a string because emails don't support UTF-8 at this time (according
a comment in gravatar_hash in avatar.py).
Previously, api_fetch_api_key would not give clear error messages if
password auth was disabled or the user's realm had been deactivated;
additionally, the account disabled error stopped triggering when we
moved the active account check into the auth decorators.
The security model for deactivated users (and users in deactivated
realms) being unable to access the service is intended to work via two
mechanisms:
* All active user sessions are deleted, and all login code paths
(where a user could get a new session) check whether the user (or
realm) is inactive before authorizing the request, preventing the
user from accessing the website and AJAX endpoints.
* All API code paths (which don't require a session) check whether the
user (and realm) are active.
However, this security model was not implemented correctly. In
particular, the check for whether a user has an active account in the
login process was done inside the login form's validators, which meant
that authentication mechanisms that did not use the login form
(e.g. Google and REMOTE_USER auth) could succeed in granting a session
even with an inactive account. The Zulip homepage would still fail to
load because the code for / includes an API call to Tornado authorized
by the user's token that would fail, but this mechanism could allow an
inactive user to access realm data or users to access data in a
deactivated realm.
This fixes the issue by adding explicit checks for inactive users and
inactive realms in all authentication backends (even those that were
already protected by the login form validator).
Mirror dummy users are already inactive, so we can remove the explicit
code around mirror dummy users.
The following commits add a complete set of tests for Zulip's inactive
user and realm security model.
This results in a substantial performance improvement for all of
Zulip's backend templates.
Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.
Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`
Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
instead of `RequestContext` object. There are two reasons to do this:
1. `RequestContext` is not compatible with Jinja2
2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
includes a custom backend and a template renderer, compressors for js
and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.
Fixes#620.
To avoid the potential for introducing regressions here, we carefully
pass a default to REQ or not based on how the existing webhook's
parsing code worked. In the longer term, we'll want to make the
behavior consistent.
This integration relies on the Teamcity "tcWebHooks" plugin which is
available at
https://netwolfuk.wordpress.com/category/teamcity/tcplugins/tcwebhooks/
It posts build fail and success notifications to a stream specified in
the webhook URL.
It uses the name of the build configuration as the topic.
For personal builds, it tries to map the Teamcity username to a Zulip
username, and sends a private message to that person.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.
As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.
When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.
We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.
Fixes#441.
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.
This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
The previous implementation didn't work because HomepageForm rejected
the email as not having a domain. Additionally, the logic in
accounts_register didn't work with Google auth because that code path
doesn't pass through accounts_home. Since whether there's a unique
open realm for the server is effectively a configuration property, we
can fix the bug and make the logic clearer by moving it into the
"figure out the user's realm" function.
The browser registers for events via loading the home view, not this
interface, and this functionality is available via the API-format
register route anyway.
This makes it possible to use DevAuthBackend when doing
performance/scalability testing on Zulip with many thousands of users.
It's unlikely that anyone testing this backend will find it valuable
to have more than 100 login buttons on the same page, and if they do,
they can always just change this limit.
Thanks to @dbiollo for the suggestion!
Previously we only did this when new human users were created via the
login process, which meant the management command to create a user did
not add the user to default streams (for example) and any future code
that might want to register a new Zulip user (such as the LDAP
integration) would need to import views/__init__.py in order to
properly set this up.
In b59b5cac35, we upgraded our Google
Oauth code to support new python-requests, but because Ubuntu precise
still has old python-requests, this broke the codepath for older
systems.
requests 1.0 changed response.json attribute to response.json()
instancemethod. The code wasn't updated to match that change,
causing a TypeError when attempting to use the Google OAuth
Authenticator backend.
This is fixed simply by using response.json() instead of response.json.
Steve Howell
Rishi Gupta
Tim Abbott
Tomasz Kolek
Kouhei Sutou
Taranjeet Singh
Tim Abbott
Brock Whittaker
Umair Khan
Rag Sagar
Christie Koehler
Eklavya Sharma
Ashish Kumar
Kartik Maji
Steve Howell
Vishnu Ks
acrefoot
rahuldeve
dhanus
kunall17
John Hergenroeder
Babak
medullaskyline
Hyunchel Kim
Evan Palmer
Daw-Ran Liou
Dalek-Sec
Max
Reid Barton
David Adamec
Oren Leaffer
Pei-Wei Wu
Ashley Dunn
Aristeidis Fkiaras
Preston Hansen
Antek Grzanka
Vladislav Manchev
David Payne
Anindya Chakravarti
Varshit
Ryan Moore
Reid Barton
Josh Mandel
Chris Chapman
Allie Jones
Darren Worrall
Jason Michalski
Yuvi Panda
Tim Abbott