Commit Graph

11015 Commits

Author SHA1 Message Date
Mateusz Mandera 6e4dcc714e auth: Fix bug in backend configuration checks in start_social_login.
We shouldn't check if *every* backend in the backends list is
configured, but only the one the user is attempting to log in with.
2020-03-01 22:30:38 -08:00
Steve Howell 4fba227898 backend tests: Test include_all_active for streams.
We were only checking error handling before, not
the happy path.  The structure of the code
made it so that we effectively tested most of the
logic for this use case (since all the other flags
are sort of just filters on top of this), but
obviously we want explicit coverage here.  Also,
we weren't testing the is-admin-but-not-api-super-user
error checking until this commit.
2020-03-01 07:49:38 -05:00
Mateusz Mandera 2d55c7e0d5 home: Don't assume user agent header is set for insecure_desktop_app.
The header may not be set - this leads to CI failures on 2.1.x branch,
but in any case is a real bug.
2020-02-28 16:51:23 -08:00
Chris Bobbe 23ba2b63c5 push_notifications: In dev, make APNs or GCM config suffice. 2020-02-28 16:49:35 -08:00
Steve Howell 504ec9d489 typing: Remove recipient-related complexity.
For historical reasons we were creating Recipient
objects at some point in the typing-notifications
codepath.  Now we just work with UserProfiles.
This removes some queries, as indicated by
the change to `len(queries)` in a couple of the
tests.

The one subtle thing that changes here is huddles.
If user 10 sends a typing notification that they
are talking to users 20 and 30, there might not
actually be a huddle for users 10/20/30, but
we were actually creating huddles on the fly!
There is no need to create huddles just for
typing notifications, since we don't even
share huddle ids with our clients.  The clients
just infer the huddles.

Some of the code that gets killed off here as
somewhat "collateral damage" is some
defensive code related to formerly supporting streams
in typing indicators.  The support for streams
was killed off almost as soon as we released
the feature, and the codepath is pretty clearly
user-centric at this point.
2020-02-28 12:46:20 -08:00
Steve Howell f224f215c1 refactor: Simplify handling of emails for typing endpoint.
Instead of duplicating code for the email case, just
convert emails to user_ids and then run the same code.
2020-02-28 12:39:36 -08:00
Steve Howell bed6d5a789 typing: Inline check_typing_notification.
I actually like this pattern:

    def check_send_typing_notification(...):
        typing_notification = check_typing_notification(...)
        do_send_typing_notification(...)

It can help divide responsibilities nicely and make it easy
to write detailed unit tests against each of the two helpers.

Unfortunately, the good things didn't really happen here, and
instead we got the worst aspects of the pattern:

    - The responsibilities for validation leaked into
      the second function.

    - Both functions were doing sane things individually
      that became not-so-sane in the big picture (namely,
      we ended up making Recipient objects for no reason,
      but if you read each of the helpers, it was just one
      step that seemed reasonable).

    - Passing around dictionaries for results can be annoying.

Also, the pattern made a lot more sense when the validation
for typing was a lot more complicated.  My prior commit makes
it so that we only ever deal with a list of user_ids.

Anyway, now I'm inlining it. :)

Subsequent commits will clean up the more substantive issue
here, which is that we are building Recipients for no reason.
2020-02-28 12:39:36 -08:00
Steve Howell b26f2dcd4b typing: Deprecate emails in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

Unfortunately, we still have some older versions
of mobile that still send emails.

In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.

One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion).  It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.

We also clean up the test structure for the
typing tests:

    TypingHappyPathTest.test_start_to_another_user
    TypingHappyPathTest.test_start_to_multiple_recipients
    TypingHappyPathTest.test_start_to_self
    TypingHappyPathTest.test_start_to_single_recipient
    TypingHappyPathTest.test_stop_to_another_user
    TypingHappyPathTest.test_stop_to_self

    TypingValidateOperatorTest.test_invalid_parameter
    TypingValidateOperatorTest.test_missing_parameter

    TypingValidateUsersTest.test_argument_to_is_not_valid_json
    TypingValidateUsersTest.test_bogus_user_id
    TypingValidateUsersTest.test_empty_array
    TypingValidateUsersTest.test_missing_recipient

    TypingValidationHelpersTest.test_recipient_for_user_ids
    TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id

    TypingLegacyMobileSupportTest.test_legacy_email_interface
2020-02-28 12:39:36 -08:00
Tim Abbott d79a7a8c35 panels: Show a banner for users with legacy desktop apps.
Users who are using ZulipDesktop or haven't managed to auto-update to
ZulipElectron should be strongly encouraged to upgrade.

We'll likely want to move to something even stricter that blocks
loading the app at all, but this is a good start.
2020-02-28 01:54:46 -08:00
Mateusz Mandera 7db3d4560f do_delete_messages: Archive the messages in bulk.
The test added in this commit shows 37 queries - compared to 181 without
the change to the function. That seems very much worth it.
2020-02-27 23:12:32 -08:00
Mateusz Mandera b4186fb680 do_delete_messages: Remove unused message_ids list. 2020-02-27 23:12:32 -08:00
Wyatt Hoodes 6ed944c761 test_runner: Update database ids to be human readable.
Before the Django 2.x upgrade, the DatabaseCreation
argument took an integer value.  To deal with running
mulitple test instances, we created a random start
range that could count up 100 workers until the next
random id.  Arbitrarily limiting the number of workers
to 100.

Post upgrade, we can now use string values. Enabling
the database + worker numbers to be more readable, as
well as removing the cap on the worker count.
2020-02-27 23:01:29 -08:00
Mateusz Mandera efb3065158 social_auth: Take user to find_account if invalid subdomain is given.
This allows to also clean up some code that's not really useful.
2020-02-27 17:27:55 -08:00
Tim Abbott 2fb967b735 do_update_message: Remove sender field from update_message events.
This field wasn't accessed by any clients and was a less robust
version of the user_id field.  Any client hoping to be interested in
who did message edits should be able to handle working with user IDs
rather than email addresses.
2020-02-26 16:16:01 -08:00
Tim Abbott 588bcb37cf do_update_message: Avoid using a direct query to fetch a Stream.
We have a helper designed for the purpose, and it fixes potentially
misbehavior where the previous code did not do `.select_related()`.
2020-02-26 16:14:34 -08:00
Tim Abbott 49ca7cf717 topic: Add recipient_id to fields for message edit saves.
This is preparation for supporting moving messages between streams in
some cases.

It doesn't actually have any functional effect, since flush_message
clears the message unconditionally anyway.
2020-02-26 16:12:07 -08:00
Steve Howell b75fb579e3 typing tests: Test unwanted Huddle side effect.
This test shows that we are, among other
things, creating Huddle records every
time somebody starts typing a message
to multiple people.
2020-02-25 16:17:47 -08:00
Steve Howell 600fcd6c52 typing tests: Add query_count checks.
We should not need so many queries here,
although a couple of the queries are just
standard things that apply to all requests.

I will reduce the number of queries in a
later commit.
2020-02-25 16:17:47 -08:00
Steve Howell 995353fb28 message validation: Clean up extract_private_recipients.
This is mostly refactoring, but we also prevent a new
type of value error (list of non-int-or-string).  The
new test code helps enforce that.

Cleanup includes:

    - Use early-exit for email case.
    - Rename helpers to get_validate_*.
    - Avoid clumsy rebuilding of lists in helpers.
    - Avoid the confusing `recipient` name (which
      can be confused with the model by the same
      name).
    - Just delegate duplicate-id/email-removal to
      the helpers.

The cleaner structure allows us to elminate a couple
mypy workarounds.
2020-02-25 16:17:47 -08:00
Vishnu KS 303cd9bb9e actions: Make do_change_plan_type support changing plan to SELF_HOSTED.
Credits to @xpac1985 for reporting, debugging and proposing fix to the
issue. The proposed fix was modified slightly by @hackerkid to set the
correct value for max_invites and upload_quota_gb. Tests added by
@hackerkid.

Fixes #13974
2020-02-25 16:14:45 -08:00
Anders Kaseorg 1cdab5ae61 emoji: Resolve emoji sprite sheets and stylesheets through Webpack.
This gives them cache-compatible URLs, and also avoids some extra
copies of the sprite sheet images.

Comments on the Octopus emoji added by tabbott.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:43:46 -08:00
Tim Abbott 27edc18330 test_classes: Use realistic web and mobile User-Agent strings.
This fixes a confusing aspect of how our automated tests worked
previously, where we'd almost all HTTP requests in the unlikely
configuration with no User-Agent string specified.

We need to adjust query counts in a few tests that now are a bit
cheaper because they now can take advantage of a Client object created
in server_initialization.py in `process_client`.
2020-02-24 23:19:43 -08:00
Tim Abbott 27b267026e test_classes: Rename set_http_host to set_http_headers.
This supports the goal of setting other headers like User-Agent in the
future.
2020-02-24 23:19:43 -08:00
Tim Abbott d80175d29e server_initialization: Create Client objects for mobile/desktop.
This replaces the "API" client, which isn't used by any real clients,
with the "ZulipMobile" and "ZulipElectron" client strings, which are.
2020-02-24 23:19:43 -08:00
Steve Howell 29dd7d2267 minor: Fix comment in send_message_backend.
The `send_message_backend` function no longer
calls `recipient_for_emails`.
2020-02-24 15:32:29 -08:00
Mateusz Mandera 2343f80d72 auth: Remove redundant get_subdomain(request) call in remote_user_sso.
subdomain = get_subdomain(request) is already called earlier in the
function.
2020-02-24 12:39:48 -08:00
Mateusz Mandera f2903e9c70 auth: Refactor - convert remote_username to email in remote_user_sso.
So far the conversion was in a very random place -
register_remote_user(). All other codepaths that use
login_or_register_remote_user() call it with the user's email address.
Making remote_user_sso convert remote_username to the email address
before calling login_or_register_remote_user makes this usage consistent
across the board.
2020-02-24 12:39:48 -08:00
Mateusz Mandera 98ae2fb940 auth: Remove redundant realm argument to finish_desktop_flow.
finish_desktop_flow is called with the assumption that the request
successfully proved control over the user_profile and generates a
special link to log into the user_profile account. There's no reason to
pass the realm param, as user_profile.realm can be assumed.
2020-02-24 12:39:48 -08:00
Dinesh 1308544a70 auth: Remove `if` blocks to redirect to config error page.
In `auth.py` there are three `if` blocks for different backends
to redirect to config error page with similar code. It is better
handled with common code using `get_attr()` function on
constructed setting names.
2020-02-24 12:19:45 -08:00
Dinesh 144304c798 auth: Move `ConfigErrorTest` from `test_docs` to `test_auth_backends`.
There was some duplicated code to test config error pages for
different auths which could be handled with less duplicated code
by adding those functions to `SocialAuthBase`.
Also moving the other tests makes it easier to access tests related
to a backend auth when they are in the same file.
2020-02-24 12:19:45 -08:00
harshavardhanpb cac4feb263 openapi: Move openapi.py into zerver/openapi.py.
Fixes #14006
2020-02-24 12:21:26 -05:00
Steve Howell ed859617e4 minor: Add test for extract_stream_indicator. 2020-02-24 07:40:31 -05:00
Mateusz Mandera ac041956d9 test_cache: Test caching of None values is handled correctly. 2020-02-21 09:05:46 -08:00
Mateusz Mandera a9794ec001 cache: Delete unused function cache(). 2020-02-21 09:05:46 -08:00
Mateusz Mandera bf0f1274fa saml: Make the bad idp param KeyError log message more verbose.
Original idea was that KeyError was only going to happen there in case
of user passing bad input params to the endpoint, so logging a generic
message seemed sufficient. But this can also happen in case of
misconfiguration, so it's worth logging more info as it may help in
debugging the configuration.
2020-02-20 14:49:27 -08:00
vsvipul 020a263a67 auth: Create a new page hop for desktop auth.
Create a new page for desktop auth flow, in which
users can select one from going to the app or
continue the flow in the browser.

Co-authored-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-02-20 11:59:55 -08:00
Mateusz Mandera c78d0712f7 tests: For ldap tests, give each ldap user a unique password.
To avoid some hidden bugs in tests caused by every ldap user having the
same password, we give each user a different password, generated based
on their uids (to avoid some ugly hard-coding in a bunch of places).
2020-02-19 14:46:29 -08:00
Vishnu KS 51f5701879 export: Canonicalize the email of cross realm bot to default value.
Fixes #13496
2020-02-19 14:44:50 -08:00
Vishnu KS 0d4bf86130 management: Make backup command work when DB is not in localhost.
This is useful preparatory work for supporting the backup management
command inside docker-zulip.
2020-02-19 12:41:05 -08:00
Vishnu KS e1a7716578 emails: Translate from_name of account security emails. 2020-02-18 17:45:33 -08:00
Tim Abbott 0075c6cd56 do_update_message: Clean up timestamp code.
By moving this logic to the topic of the functon, we make the code a
lot more readable.
2020-02-18 16:38:34 -08:00
Ryan Rehman 1bda3babbe tests: Minor refactoring of test_messages.
This avoids using `.save()` directly for editing stream properties,
and also uses the API in _send_and_verify_message to avoid confusing
logic around which user is doing what request.

Fixes part of #13823
2020-02-18 11:07:41 -08:00
Mateusz Mandera 6a0b68bc7f models: Delete get_stream_recipient function and its uses.
With recipient being now a Stream field, there's no more use for
this helper function.
2020-02-18 10:49:14 -08:00
Mateusz Mandera 0d6f78b381 models: Delete get_personal_recipient function and its uses.
With recipient being now a UserProfile field, there's no more use for
this helper function.
2020-02-18 10:49:14 -08:00
Mateusz Mandera 920d22524b import: Use re_map_foreign_keys on the realm column of UserPresence.
We forgot to make this adjustment in the recent denormalization of realm
into UserPresence. It's needed for imports to work correctly.
2020-02-18 10:45:38 -08:00
akashaviator 6a36edef9e api: Document PATCH ../users/{user_id} endpoint.
This adds update_user to python_examples.py in zerver/openapi.
This also adds update-user.md to templates/zerver/api and adds
update-user to rest-endpoints.md (templates/zerver/help/include).
2020-02-15 23:08:13 -08:00
akashaviator dc6a5e3ca2 api: Document DELETE ../users/{user_id} endpoint.
This adds deactivate_user to python_examples.py in zerver/openapi.
This also adds delete-user.md to templates/zerver/api and adds
delete-user to rest-endpoints.md (templates/zerver/help/include).
2020-02-15 23:08:13 -08:00
akashaviator 08efb00321 api: Document GET ../users/{user_id} endpoint.
This adds get_single_user to python_examples.py in zerver/openapi.
This also adds get-single-user.md to templates/zerver/api and adds
get-single-user to rest-endpoints.md (templates/zerver/help/include).
2020-02-15 23:08:13 -08:00
akashaviator 7bc470c699 openapi: Add OpenAPI format data for /users/{user_id} endpoint.
This adds the OpenAPI format data for /users/{user_id} endpoint
and also removes 'users/{user_id}' from 'pending_endpoints' in
zerver/tests/test_openapi.py .
2020-02-15 23:08:10 -08:00
Anders Kaseorg b2ec8e157b has_request_variables: Remove query_params dict.
‘req_var in request.GET’ was previously believed to be slow from
profiling results.  However, the real explanation for those profiling
results is that WSGIRequest.GET is a lazy cached property, so there’s
no reason to avoid it if we’re accessing request.GET anyway.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-15 11:37:18 -08:00
Chris Heald 18e3982acd integrations: Add AlertManager webhook. 2020-02-14 17:43:15 -08:00
Mateusz Mandera cbdfef28a8 retention: Update to account for the zulipinternal realm.
In https://github.com/zulip/zulip/pull/12823 some changes to the realms
structure have been made, so now both in production and development
cross-realm bots live in the realm with string_id "zulipinternal".
There was a TODO in retention code to eliminate a conditional in a query
that became redundant with this change, and also the zulipinternal realm
should be omitted from the archiving process in archive_messages().
2020-02-14 17:15:26 -08:00
Tim Abbott 229090a3a5 middleware: Avoid running APPEND_SLASH logic in Tornado.
Profiling suggests this saves about 600us in the runtime of every GET
/events request attempting to resolve URLs to determine whether we
need to do the APPEND_SLASH behavior.

It's possible that we end up doing the same URL resolution work later
and we're just moving around some runtime, but I think even if we do,
Django probably doesn't do any fancy caching that would mean doing
this query twice doesn't just do twice the work.

In any case, we probably want to extend this behavior to our whole API
because the APPEND_SLASH redirect behavior is essentially a bug there.
That is a more involved refactor, however.
2020-02-14 16:15:57 -08:00
Tim Abbott 10e7e15088 user_agent: Compile the regular expression.
We use this single regular expression for processing essentially every
request, so it's definitely worth hinting to Python that we're going
to do so by compiling it.  Saves about 40us per request.
2020-02-14 10:26:37 -08:00
Tim Abbott 800312c976 has_request_variables: Fix slow extraction of parameters.
A sloppy implementation of the main has_request_variables wrapper
function meant that it did two very inefficient things:

* To combine together the GET and POST parameters, it would make a
  copy of the request.GET QueryDict object, which combined with the
  fact that these objects are slow to access, consumed about 90us per
  argument.
* Doing this in a loop (one time per argument), rather than once,
  which resulted in us doing this 11 times for a `GET /events` query.

Fixing this to just make a dictionary and combine things with some
small loops saved about 1 millisecond from the total runtime of GET
/events (for comparison, the total actual work of that view function
is about 700ms).

We need to fix at least one test that used a bad mock HttpRequest
object that didn't have a .GET property.
2020-02-14 09:45:26 -08:00
Tim Abbott 4fbcbeeea7 settings: Disable django.request logging at WARNING log level.
The comment explains this issue, but effectively, the upgrade to
Django 2.x means that Django's built-in django.request logger was
writing to our errors logs WARNING-level data for every 404 and 400
error.  We don't consider user errors to be a problem worth
highlighting in that log file.
2020-02-13 23:50:53 -08:00
Anders Kaseorg 8e356368f7 markdown: Fix HTML escaping of &.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-13 17:50:59 -08:00
rht 41e3db81be dependencies: Upgrade to Django 2.2.10.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.

Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:

* urls: django.urls.resolvers.RegexURLPattern has been replaced by
  django.urls.resolvers.URLPattern; affects OpenAPI code and related
  features which re-parse Django's internals.
  https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
  ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
  https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
  https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
  the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.

There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.

Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.

Fixes #10835.
2020-02-13 16:27:26 -08:00
Tim Abbott 1ea2f188ce tornado: Rewrite Django integration to duplicate less code.
Since essentially the first use of Tornado in Zulip, we've been
maintaining our Tornado+Django system, AsyncDjangoHandler, with
several hundred lines of Django code copied into it.

The goal for that code was simple: We wanted a way to use our Django
middleware (for code sharing reasons) inside a Tornado process (since
we wanted to use Tornado for our async events system).

As part of the Django 2.2.x upgrade, I looked at upgrading this
implementation to be based off modern Django, and it's definitely
possible to do that:
* Continue forking load_middleware to save response middleware.
* Continue manually running the Django response middleware.
* Continue working out a hack involving copying all of _get_response
  to change a couple lines allowing us our Tornado code to not
  actually return the Django HttpResponse so we can long-poll.  The
  previous hack of returning None stopped being viable with the Django 2.2
  MiddlewareMixin.__call__ implementation.

But I decided to take this opportunity to look at trying to avoid
copying material Django code, and there is a way to do it:

* Replace RespondAsynchronously with a response.asynchronous attribute
  on the HttpResponse; this allows Django to run its normal plumbing
  happily in a way that should be stable over time, and then we
  proceed to discard the response inside the Tornado `get()` method to
  implement long-polling.  (Better yet might be raising an
  exception?).  This lets us eliminate maintaining a patched copy of
  _get_response.

* Removing the @asynchronous decorator, which didn't add anything now
  that we only have one API endpoint backend (with two frontend call
  points) that could call into this.  Combined with the last bullet,
  this lets us remove a significant hack from our
  never_cache_responses function.

* Calling the normal Django `get_response` method from zulip_finish
  after creating a duplicate request to process, rather than writing
  totally custom code to do that.  This lets us eliminate maintaining
  a patched copy of Django's load_middleware.

* Adding detailed comments explaining how this is supposed to work,
  what problems we encounter, and how we solve various problems, which
  is critical to being able to modify this code in the future.

A key advantage of these changes is that the exact same code should
work on Django 1.11, Django 2.2, and Django 3.x, because we're no
longer copying large blocks of core Django code and thus should be
much less vulnerable to refactors.

There may be a modest performance downside, in that we now run both
request and response middleware twice when longpolling (once for the
request we discard).  We may be able to avoid the expensive part of
it, Zulip's own request/response middleware, with a bit of additional
custom code to save work for requests where we're planning to discard
the response.  Profiling will be important to understanding what's
worth doing here.
2020-02-13 16:13:11 -08:00
Chris Heald a91358e186 webhooks: Fix hellosign webhook.
Hellosign now posts their callback as form/multipart, which Django only
permits to be read once. Attempts to access request.body after the
initial read throw "django.http.request.RawPostDataException: You
cannot access body after reading from request's data stream".

Fixes #13847.
2020-02-12 22:36:11 -08:00
Mateusz Mandera 27b15a9722 install: Don't create internal realm in the installation process. 2020-02-12 12:00:10 -08:00
Mateusz Mandera bde495db87 registration: Add support for mobile and desktop flows.
This makes it possible to create a Zulip account from the mobile or
desktop apps and have the end result be that the user is logged in on
their mobile device.

We may need small changes in the desktop and/or mobile apps to support
this.

Closes #10859.
2020-02-12 11:22:16 -08:00
Mateusz Mandera fe33966642 sessions: Implement the concept of expirable session variables.
This can be useful in the future for various things, and right now it'll
specifically be used in the signup mobile/desktop flows.
2020-02-12 11:09:55 -08:00
Hashir Sarwar eb23c6fa6c test_fixtures: Clean up interface for `template_database_status()`.
1) Created a new class `DatabaseType` and access its objects inside
`template_database_status()` instead of sending five arguments with
default values.

2) Made `check_files` and `setting_name` local variables instead of
function parameters since they had same value(None) for every call.

Fixes #13845.
2020-02-12 11:07:10 -08:00
Tim Abbott 96b0ec705d email_notifications: Fix missing translation tags on sender. 2020-02-12 10:54:34 -08:00
Anders Kaseorg e257253e64 emoji_codes: Replace JS module with JSON module.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.

Update the backend to use emoji_codes.json too instead of the three
separate JSON files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Tim Abbott cb2c96f736 test_templates: Remove shallow template rendering code.
This code was very useful when first implemented to help catch errors
where our backend templates didn't render, but has been superceded by
the success of our URL coverage testing (which ensures every URL
supported by Zulip's urls.py is accessed by our tests, with a few
exceptions) and other tests covering all of the emails Zulip sends.

It has a significant maintenance cost because it's a bit hacky and
involves generating fake context, so it makes sense to remove these.
Any future coverage issues with templates should be addressed with a
direct test that just accessing the relevant URL or sends the relevant
email.
2020-02-11 18:00:15 -08:00
Mateusz Mandera 2475adbf8a messages_for_topic: Use stream.recipient_id for more efficient query. 2020-02-11 17:39:43 -08:00
Chris Heald bddb370750 tests: Reorder python version logic to be more clear. 2020-02-11 17:34:56 -08:00
Chris Heald 3236483d0e tests: Fix type reflection for Python 3.7.
In python 3.5-3.6, generic types had an __origin__ attribute which
indicated which generic they originated from; the code was reflecting on
that value to check types against the openapi spec. In python3.7, this
changed, and there's no longer an immediately simple way to get this
information in all cases. __origin__ appears to be the implementing
class now, returning `list` or `collections.abc.Iterator` rather than
`typing.List` and `typing.Iterator`. This adds a sloppy-but-effective
mechanism for inferring if a type maps to the List/Dict/Iterator/Mapping
types and gets the test suite passing again.
2020-02-11 17:34:56 -08:00
Dinesh 4304d5f8db auth: Add support for GitLab authentication.
With some tweaks by tabbott to the documentation and comments.

Fixes #13694.
2020-02-11 13:54:17 -08:00
Steve Howell 900f98c0c5 presence: Use realm_id for UserPresence queries.
We now use realm_id for querying UserPresence
instead of building a big WHERE clause from the
list of user_ids.

This commit may be a bit hard to measure, since
we still get the list of user_ids for the PushToken
query in the same method.
2020-02-11 13:11:58 -08:00
Steve Howell d68052b68d presence: Add realm/timestamp index to UserPresence.
It adds this index:

    "zerver_userpresence_realm_id_timestamp_25f410da_idx" btree (realm_id, "timestamp")

We expect this index to provide a major performance improvement when
fetching presence data for the whole realm from the database on
servers like zulipchat.com hosting several realms.
2020-02-11 13:11:28 -08:00
Tim Abbott fcac3a4342 recipients: Rename extract_recipients to extract_private_recipients.
Recent changes mean this function is now only used for private
messages.
2020-02-11 12:28:14 -08:00
Steve Howell 1b6578cafd messages: Fix bug with commas in stream names.
We now validate streams with a separate
function from PM recipients.

It's confusing enough all the ways you can
encode a stream or encode the PM recipients,
but trying to do it all in one function was
hard to reason about and led to at least one
bug.

In particular, there was a bug where streams
with commas in them would get split.  Now
we just don't ever split on commas inside
of `extract_stream_indicator`.

Fixes #13836
2020-02-11 12:20:54 -08:00
Steve Howell 96132fe0e9 extract_recipients: Enforce str as incoming type.
After removing internal_send_message() in a recent
commit, we now have only two callers for
extract_recipients, and they are both related
to our REQ mechanism that always passes strings
to converters.  (If there are default values,
REQ does not call the converters.)

We therefore make two changes:

    - use the more strict annotation of "str"
      for the `s` parameter

    - don't bother with the isinstance check
2020-02-11 12:20:54 -08:00
Steve Howell 8c3eaeb872 Remove obsolete internal_send_messages().
We have been phasing this out for a couple years,
and I fixed the last stragglers over the last
couple days.
2020-02-11 12:20:54 -08:00
Steve Howell 2e8dec233e slow queries: Use internal_send_stream_message().
Note that while the test mocks the actual message
send, we now have a `get_stream` call in the queue
worker, so we have to set up a real stream for
testing (or we could have mocked that as well, but
it didn't seem necessary).  The setup queries add
to the amount of queries reported by the test,
plus the `get_stream` call.  I just made the
query count a digits regex, which is a little bit
lame, but I don't think it's worth risking test
flakes for this.
2020-02-11 12:20:54 -08:00
Steve Howell e37d660d19 error_notify: Use internal_send_stream_message(). 2020-02-11 12:20:53 -08:00
Steve Howell c4e3cfebb0 presence: Add realm_id to UserPresence.
This index is intended to optimize the performance of the very
frequently run query of "what is the presence status of all users in a
realm?".

Main changes:
    - add realm_id to UserPresence
    - add index for realm_id
    - backfill realm_id for old rows
    - change all writes to UserPresence to include
      realm_id

The index is of this form:

    "zerver_userpresence_realm_id_5c4ef5a9" btree (realm_id)

We will create an index on (realm_id, timestamp) in a
future commit, but I think it's a bit faster if you do
the backfill before the index.

There's also a minor tweak to the populate_db script.
2020-02-10 17:21:45 -08:00
Steve Howell 28a8ffbc4c email_mirror: Use internal_send_stream_message().
This is just a refactoring to the more modern API
for sending internal messages.

To make this work we now plumb the email_gateway
flag through `internal_send_stream_message` instead
of `internal_send_message`.

We also change `send_zulip` to have its callers
pass in a full UserProfile object (which one of
them already had).
2020-02-10 15:45:13 -08:00
Steve Howell 6922eef380 signups: Use internal_send_stream_message().
We prefer this to internal_send_message().

We are trying to deprecate `internal_send_message`,
which has extra moving parts related to
`extract_recipients` and `Addressee.legacy_build`.

There are two chunks of code that I touch here
that look pretty similar, but I'm not quite
sure they're worth de-duplicating, since they
use different topics and different message
content.
2020-02-10 15:45:13 -08:00
Steve Howell f1ac16973c tests: Create signups stream in RealmCreationTests. 2020-02-10 15:45:13 -08:00
Steve Howell b33552997e cross realm bots: Simplify notify_new_user.
Instead of having `notify_new_user` delegate
all the heavy lifting to `send_signup_message`,
we just rename `send_signup_message` to be
`notify_new_user` and remove the one-line
wrapper.

We remove a lot of obsolete complexity:

    - `internal` was no longer ever set to True
      by real code, so we kill it off as well
      as well as killing off the internal_blurb code
      and the now-obsolete test

    - the `sender` parameter was actually an
      email, not a UserProfile, but I think
      that got past mypy due to the caller
      passing in something from settings.py

    - we were only passing in NOTIFICATION_BOT
      for the sender, so we just hard code
      that now

    - we eliminate the verbose
      `admin_realm_signup_notifications_stream`
      parameter and just hard code it to
      "signups"

    - we weren't using the optional realm
      parameter

There's also a long ugly comment in
`get_recipient_info` related to this code
that I amended for now.
We should try to take action in a subsequent
commit.
2020-02-10 15:45:13 -08:00
Steve Howell 6e40db4b1f minor: Fix misleading comments.
These comments were naming the wrong function.
2020-02-10 15:45:13 -08:00
Hashir Sarwar dcbd3e486f stream_subscription: Remove unused TypedDict `SubInfo`. 2020-02-10 14:04:22 -08:00
Steve Howell 2ff41bf9e5 /json/users: Use field.realm for realm lookup.
This avoids an unnecessary join to UserProfile.

To verify this, you can do `print(queries)` in the
`test_get_custom_profile_fields_from_api` test.  It's
kinda noisy, so I excerpted them below...

Before:

    SELECT ...
    FROM "zerver_customprofilefieldvalue"
    INNER JOIN "zerver_userprofile" ON ("zerver_customprofilefieldvalue"."user_profile_id" = "zerver_userprofile"."id")
    INNER JOIN "zerver_customprofilefield" ON ("zerver_customprofilefieldvalue"."field_id" = "zerver_customprofilefield"."id")
    WHERE "zerver_userprofile"."realm_id" = 2

After:

    SELECT ...
    FROM "zerver_customprofilefieldvalue"
    INNER JOIN "zerver_customprofilefield" ON ("zerver_customprofilefieldvalue"."field_id" = "zerver_customprofilefield"."id")
    WHERE "zerver_customprofilefield"."realm_id" = 2'

I don't have any way to measure the two queries with
realistic data, but I would assume the second
query is significantly faster on most of our instances,
since CustomProfileField should be tiny.
2020-02-09 22:04:02 -08:00
Steve Howell 9303c386b8 tests: Count queries for /json/users.
I am trying to optimize a query in this endpoint.
I don't think I'll actually reduce the number of
queries, but I wanted to capture the query and
this was the easiest way to do it, so might as
well check in the code! :)
2020-02-09 22:04:02 -08:00
Steve Howell 01f180d042 minor: Remove unused line of code in get_raw_user_data().
The line removed here is a noop, as both sides of the
immediately following conditional reassign the
same variable.

This harmless cruft was the result of the recent commit
1ae5964ab8, which added
support for single-user GETs.
2020-02-09 22:04:02 -08:00
Tim Abbott 986706c7e5 tornado: Use common code for copying headers.
This fixes a bug where our asynchronous requests were only copying the
Content-Type header (i.e. the one case where we're noticed) from the
Django HttpResponse.  I'm not sure what the impact of this would be;
the rate-limiting headers rarely come up when breaking a long-polled
request.  But it seems clearly an improvement to do this in a
consistent fashion.

Only the headers piece is a change; in Tornado

    self.finish(x)

is equivalent to:

    self.write(x)
    self.finish()
2020-02-07 16:14:19 -08:00
Tim Abbott 224a73a3ec tornado: Extract a function for writing Tornado responses.
This increases the readability of what's happening in our core Tornado
handlers code, as well as making this logic reusable.
2020-02-07 16:13:49 -08:00
Tim Abbott 5305e8af85 tornado: Extract convert_tornado_request_to_django_request. 2020-02-07 16:03:58 -08:00
Tim Abbott fc58ae117a handlers: Rename confusingly named response to result_dict.
This should somewhat increase the readability of zulip_finish.
2020-02-07 16:03:58 -08:00
Vishnu KS 4572be8c27 api: Rename subject_links to topic_links.
Fixes #13588
2020-02-07 14:35:22 -08:00
Tim Abbott 84edb5c516 test_fixtures: Fix buggy reuse of status_dir between databases.
Apparently, the arguments passed to template_database_status were
incorrect for the manual testing development database, in that we
didn't pass a status_dir when calling into that code from provision.

The result was that provisioning before running `test-backend` would
ignore changes to the list of check_files (etc.) made after rebasing,
and vice versa.

The cleanest fix is to compute status_dir from other values passed in;
I'm also going to open a follow-up issue for creating a better overall
interface here.
2020-02-07 13:33:08 -08:00
akashaviator 1ae5964ab8 api: Add an api endpoint for GET /users/{id}
This adds a new API endpoint for querying basic data on a single other
user in the organization, reusing the existing infrastructure (and
view function!) for getting data on all users in an organization.

Fixes #12277.
2020-02-07 10:36:31 -08:00
Tim Abbott e39840c705 users: Add read-only mode for access_user_by_id.
We've be using this in the upcoming GET /users/{id} method.
2020-02-07 10:36:31 -08:00
Tim Abbott aa9286a1f9 users: Move query into caller of get_custom_profile_field_values.
This will be useful for supporting a smaller query for a single user.
2020-02-07 10:36:31 -08:00
Tim Abbott 79e5dd1374 users: Rename get_raw_user_data user parameter to acting_user.
This is for improved clarity as we extend this function to take
multiple user objects.
2020-02-07 10:36:31 -08:00
Steve Howell 7e99e7feb2 presence: Extract get_legacy_user_info.
This code is a bit flatter and just preps the data
for a single user.  There is never any interaction
between the data for user A and user B, so we can
mostly avoid complicated nested data structures
and do most of the data-crunching on a per-user basis.

We also do an explicit sort of the data before
running it through groupby.  The explicit sort
simplifies how we calculate `most_recent_info`
and also avoids needing to add `dt` to an intermediate
data structure.

Finally, when it comes to the individual client data,
the code has relied on the assumption that there is
only one row per client, which I believe to be true,
but now the code is more explicit about that.
2020-02-06 17:16:22 -08:00
Steve Howell bf3baa14ac presence: Rename get_status_dict_by_user(). 2020-02-06 17:16:22 -08:00
Steve Howell 675f8514e8 presence: Rename get_status_dict().
We renamed this to get_presences_for_realm(),
and we have the caller pass in realm, not
user_profile.
2020-02-06 17:16:22 -08:00
Steve Howell 363e6bf239 presence: Move get_status_dicts_for_rows(). 2020-02-06 17:16:22 -08:00
Steve Howell 36fba1076f presence: Move get_status_dict_by_user. 2020-02-06 17:16:22 -08:00
Steve Howell 6f027d84a9 presence: Move get_status_dict_by_realm. 2020-02-06 17:16:22 -08:00
Steve Howell 703338dfa3 presence: Extract lib/presence.py.
This will make more sense when we pull some
code out of the model.
2020-02-06 17:16:22 -08:00
Steve Howell a5093be867 presence: Rename get_status_list.
The word "status" is vague, and this isn't
actually returning a list, so we now name it
get_presence_response.

I originally was gonna rename this to
get_presence_dict, but there's a function
called get_status_dict that returns a subset
of the response, so I think it's a bit more
clear that this is the bigger dict that
actually gets sent back.
2020-02-06 17:16:22 -08:00
Steve Howell 8a1fb2dcd6 presence: Calculate server_timestamp slightly earlier.
We want to err on the side of server_timestamp being
old, since we may eventually use this to make responses
just include incremental changes, and we don't want a
time window (however small) when we miss presence rows.
The clients will be able to deal with duplicate data
to the extent that the time windows are overlapping.

Also, extracting the other local var here
(for `presences`) will set up a subsequent commit
where we re-format the data for clients with
slim_presence=True.
2020-02-06 17:16:22 -08:00
Tim Abbott b5cc1a3afe home: Finish removing assumption that user_profile isn't None. 2020-02-05 16:16:33 -08:00
Tim Abbott be02604fe5 home: Refactor logic for several more settings.
The main functional change is there's now a code path for
`user_profile is None`.
2020-02-05 16:13:29 -08:00
Tim Abbott 5f8af84ad1 home: Refactor emojiset and billing logic for None UserProfile. 2020-02-05 16:09:03 -08:00
Tim Abbott 7c0a98754a home: Refactor logic for show_invites and show_add_streams. 2020-02-05 16:05:02 -08:00
Tim Abbott fc4400cfa7 home: Remove unused avatar_url context parameter. 2020-02-05 16:05:02 -08:00
Tim Abbott 7044aa8e5d home: Extract updates to last_reminder to a function. 2020-02-05 16:05:02 -08:00
Tim Abbott 85690ce649 home: Extract handling of narrow windows to a function. 2020-02-05 16:05:02 -08:00
Tim Abbott 42d529e749 home: Extract terms of service check to a function. 2020-02-05 16:05:01 -08:00
Tim Abbott 7032f49f8e exceptions: Move default json_unauthorized string to response.py.
This small refactor should make it easier to reuse this exception for
other situations as well.
2020-02-05 15:40:10 -08:00
Tim Abbott 2aab71e153 event_queue: Fix confusing event_queue.push interface.
In e3ad9baf1d, we introduced yet another
bug where we incorrectly shared event dictionaries between multiple
queues.

Fortunately, the logging that reports on "event was not in the queue"
issues worked and detected this on chat.zulip.org, but this is a clear
indication that the comments we have around this system were not
sufficient to produce correct behavior.

We fix this by changing event_queue.push, the code that mutates the
event dictionaries, to do the shallow copies itself.  The only
downside here is process_message_event, a relatively low-traffic code
path, does an extra per-queue dictionary copy.  Given that presence,
heartbeat, and message reading events are likely more traffic and
dealing with HTTP is likely much more expensive than a dictionary
copy, this probably doesn't matter performance-wise.

(And if profiling later finds it is, there are potential workarounds
like passing a skip_copy argument we can do).
2020-02-05 12:40:01 -08:00
Anders Kaseorg 8e5a45267d test_classes: Use a valid (but reserved as fictional) phone number.
django-phonenumber-field 2.4.0 adds tighter phone number validation
that rejects +12223334444 for having an invalid area code.  This was
reverted in 4.0.0, but django-two-factor-auth still requires <3.99.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg 4d49a20430 requirements: Upgrade django-sendfile2 from 0.4.3 to 0.5.1.
The module was renamed from sendfile to django_sendfile.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Mateusz Mandera 30d02c2e2c test_fixtures: app_label should be a positional arg in call_command.
We were incorrectly passing it as a kwarg, which would cause an
exception on Django 2.
2020-02-04 12:46:53 -08:00
Mateusz Mandera 5de832283b test_auth_backends: Remove logger mocking that will fail on Django 2.2.
On Django 2.2 there is no longer a logger object in that module, so it's
best to remove this mocking as it's not essential to the tests.
2020-02-04 12:46:53 -08:00
Mateusz Mandera 0e7c97378e is_safe_url: Use allowed_hosts instead of depreciated host argument.
Judging by comparing django 1.11 with django 2.2 code of this function,
this shouldn't change any behavior.
2020-02-04 12:46:53 -08:00
Steve Howell e3ad9baf1d presence: Add process_presence_event.
This lets us conditionally remove the email
field from a presence event if the client
has registered with the slim_presence flag.
2020-02-04 12:30:36 -08:00
Steve Howell 9847d4d9a3 refactor: Use user_id in get_status_dict_by_user.
This avoids a needless user lookup in apply_event.
2020-02-04 12:30:36 -08:00
Steve Howell a672a00677 presence: Add user_id to presence event.
In a later commit, we will eliminate email for
clients who have set slim_presence as their
preference.
2020-02-04 12:30:36 -08:00
Steve Howell bf9144ff69 presence: Add slim_presence flag.
This flag affects page_params and the
payload you get back from POSTs to this
url:

    users/me/presence

The flag does not yet affect the
presence events that get sent to a
client.
2020-02-04 12:30:34 -08:00
Vishnu KS df5345705c import: Support importing team icon from slack. 2020-02-03 14:09:05 -08:00
Vishnu Ks 5dfd4ea38d export: Remove unused parameter from _get_exported_s3_record. 2020-02-03 14:09:05 -08:00
Vishnu Ks 5a59bf329e import: Skip setting user_profile_id metadata only if unavailable. 2020-02-03 14:09:05 -08:00
Vishnu Ks 2ea53a347a import: Support importing realm icon and logo.
Fixes #11216
2020-02-03 14:09:05 -08:00
Vishnu Ks af3a37b58b upload: Refactor out realm_avatar_and_logo_path function. 2020-02-03 14:09:05 -08:00
Tim Abbott df110e8ff9 test_fixtures: Note populate_db depends on server_initialization.py.
This should ensure that folks rebasing past this commit from an older
database model get their database rebuilt in the way that will
match the test_subs.py query count of 40.
2020-02-03 10:38:04 -08:00
Ryan Rehman 3dc7d60ffe muting: Record DateTime when a Topic is muted.
This includes the necessary migration to add
the date_muted field to the MutedTopic class
and populates it with a hard coded value.
2020-02-02 20:49:53 -08:00
Mateusz Mandera 6aacc4195e login: Make authentication_methods data available to JavaScript.
This is intended to simplify overriding these buttons' controls in the
desktop app to do the authentication in the user's default browser.
2020-02-02 20:22:49 -08:00
Mateusz Mandera b685d506ad auth: Make login_context available to TwoFactorLoginView. 2020-02-02 20:22:49 -08:00
Tim Abbott 4fba725803 test_auth_backends: Fix errors after rebasing.
Apparently, the rate-limiting PR had some import conflicts with our
recent authentication backend testing changes.
2020-02-02 20:22:32 -08:00
Hemanth V. Alluri 7858e5ee40 webhooks/ansibletower: Update for AWX 9.1.1.
Add a simple compatibility function for AWX 9.x.x. Before AWX 9.x.x
a "friendly_name" key was sent by default. Afterwards it was removed
from being a default key but we can still more or less determine if
the triggering event was a job from the REST-style URL.

Note: It is also technically possible to add the key back by defining
a custom notification template in AWX/Tower.

Resolves #13295.
2020-02-02 20:21:03 -08:00
Mateusz Mandera 7b34853328 rate_limiter: Rename authenticate domain to authenticate_by_username.
This prepares for adding authenticate_by_ip_address.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 7c78d8a966 rate_limiter: Limit the amount of password reset emails to one address.
This limits the possibility to use the password reset form to make us
spam an email address with password reset emails.
2020-02-02 19:15:13 -08:00
Mateusz Mandera bf89cf2b4b rate_limiter: Use ABC for defining the abstract class RateLimitedObject. 2020-02-02 19:15:13 -08:00
Mateusz Mandera cb71a6571e rate_limiter: Rename 'all' domain to 'api_by_user'. 2020-02-02 19:15:13 -08:00
Mateusz Mandera 06198af5b9 auth: Handle rate limiting in OurAuthenticationForm and user_settings.
These parts of the code should catch the RateLimited exception and
generate their own, apprioprate user-facing error message.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 5f94ea3d54 auth: Rate limit username+password authenticate() calls.
This applies rate limiting (through a decorator) of authenticate()
functions in the Email and LDAP backends - because those are the ones
where we check user's password.
The limiting is based on the username that the authentication is
attempted for - more than X attempts in Y minutes to a username is not
permitted.

If the limit is exceeded, RateLimited exception will be raised - this
can be either handled in a custom way by the code that calls
authenticate(), or it will be handled by RateLimitMiddleware and return
a json_error as the response.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 335b804510 exceptions: RateLimited shouldn't inherit from PermissionDenied.
We will want to raise RateLimited in authenticate() in rate limiting
code - Django's authenticate() mechanism catches PermissionDenied, which
we don't want for RateLimited. We want RateLimited to propagate to our
code that called the authenticate() function.
2020-02-02 19:15:00 -08:00
Mateusz Mandera a6a2d70320 rate_limiter: Handle multiple types of rate limiting in middleware.
As more types of rate limiting of requests are added, one request may
end up having various limits applied to it - and the middleware needs to
be able to handle that. We implement that through a set_response_headers
function, which sets the X-RateLimit-* headers in a sensible way based
on all the limits that were applied to the request.
2020-02-02 19:15:00 -08:00
Mateusz Mandera 677764d9ca auth: Pass request kwarg in authenticate() calls with username+password.
These authenticate() calls use either Email or LDAP backends, which will
be rate limited and will need access to the request object.
2020-02-02 19:15:00 -08:00
Mateusz Mandera 4cc5d2464c rate_limiter: Expand support for different domains. 2020-02-02 19:15:00 -08:00
Mateusz Mandera d5786ee67a auth: Ensure only one of mobile and desktop otps in validate_otp_params.
validate_otp_params needs to be moved to backends.py, because as of this
commit it'll be used both there and in views.auth - and import from
views.auth to backends.py causes circular import issue.
2020-02-02 19:14:40 -08:00