Commit Graph

2665 Commits

Author SHA1 Message Date
Tim Abbott 3657717ade views: Extract message_send.py for sending views. 2020-06-22 15:08:34 -07:00
Tim Abbott 55448ecf24 views: Extract message_flags.py for flags views. 2020-06-22 15:08:34 -07:00
Tim Abbott 4d7550d705 views: Extract message_edit.py for message editing views.
This is a pretty clean extraction of files that lets us shrink one of
our largest files.
2020-06-22 15:08:34 -07:00
MariaGkoulta b10f156250 settings: Add automatic theme detection feature.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.

This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.

Fixes: #14451.

Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
2020-06-21 01:09:01 -07:00
sahil839 63389b3bd3 invite: Add option to invite user as an organization owner.
We can now invite new users as realm owners. We restrict only
owners to invite new users as owners both for single invite
and multiuse invite link. Also, only owners can revoke or resend
owner invitations.
2020-06-20 23:04:59 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c0fa42dd8c complete_zoom_user: Validate state type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 64038163e7 update_user_backend: Validate profile_data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c584462b32 update_storage: Validate storage type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 717e349834 events_register_backend: Validate client_capabilities type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 475fc5910f update_user_custom_profile_data: Validate data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Steve Howell fff2a81536 mypy: Fix payload_url. 2020-06-20 17:09:15 -07:00
Steve Howell 82036741b9 mypy: Fix service_interface 2020-06-20 17:09:15 -07:00
Steve Howell 704c84502b mypy: Fix authentication_methods.
We know the key of the dictionary is a string
here, since we get it via json.
2020-06-20 17:09:15 -07:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell 2e9f689add mypy: Fix annotation for principals. 2020-06-19 21:23:23 -07:00
Tim Abbott 14c0a9a228 messages: Deduplicate non-search code path in messages_in_narrow.
The non-search code path here was simulating the response and escaping
logic from get_search_fields by duplicating what it would do with an
empty set of highlight locations.

We can produce much more readable code by just passing an empty list
of locations in this case.
2020-06-19 15:09:57 -07:00
Steve Howell 7c6ddf90ae mypy: Fix default_language annotation. 2020-06-19 12:35:47 -04:00
Hashir Sarwar 5200598a31 events: Don't send avatar URLs of long term idle users.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.

Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.

This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.

Fixes #15287.
2020-06-18 21:35:16 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Tim Abbott f8ea5f3769 pointer: Remove pointer from GET /users/me.
This cleans up a bit of mess in the Zulip API.
2020-06-18 12:55:59 -07:00
Mateusz Mandera 8d2d64c100 CVE-2020-14215: Fix validation in PreregistrationUser queries.
The most import change here is the one in maybe_send_to_registration
codepath, as the insufficient validation there could lead to fetching
an expired PreregistrationUser that was invited as an administrator
admin even years ago, leading to this registration ending up in the
new user being a realm administrator.

Combined with the buggy migration in
0198_preregistrationuser_invited_as.py, this led to users incorrectly
joining as organizations administrators by accident.  But even without
that bug, this issue could have allowed a user who was invited as an
administrator but then had that invitation expire and then joined via
social authentication incorrectly join as an organization administrator.

The second change is in ConfirmationEmailWorker, where this wasn't a
security problem, but if the server was stopped for long enough, with
some invites to send out email for in the queue, then after starting it
up again, the queue worker would send out emails for invites that
had already expired.
2020-06-16 23:35:39 -07:00
Clara Dantas ddbde66af5 realm: Remove Google Hangouts integration.
Google  has removed the Google Hangouts brand, thus we are removing
them as video chat provider option.
This commit removes Google Hangouts integration and make a migration
that sets all realms that are using Hangouts as their video chat
provider to the default, jitsi.

With changes by tabbott to improve the overall video call documentation.

Fixes: #15298.
2020-06-16 17:02:27 -07:00
Mateusz Mandera 2ac6a8f829 auth: Change the "continue in browser" link in desktop flow end page.
Fixes #14828.
Giving the /subdomain/<token>/ url there could feel buggy if the user
ended up using the token in the desktop app, and then tried clicking the
"continue in browser" link - which had the same token that would now be
expired. It's sufficient to simply link to /login/ instead.
2020-06-16 16:27:53 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Mateusz Mandera f467ab046e messages: Remove redundant "assert is not None" for mypy.
Apparently new mypy is able to infer correctly and doesn't need this
assert anymore.
2020-06-15 16:20:43 -07:00
Aman Agrawal ccc0c8706c delete_message: Support `bulk_message_deletion`.
Fixes #15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
2020-06-15 10:41:12 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Aman Agrawal cda7b2f539 deletion: Add support for bulk message deletion events.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.

Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).

Backend implementation of #15285.
2020-06-14 22:34:00 -07:00
Aman Agrawal 21ac1fb32c register: Refactor to pass client_capabilities directly.
We extract values from client_capabilities directly in
do_events_register where we decide how to process the
extracted variables.
2020-06-14 21:42:50 -07:00
Tim Abbott d97c891afe realm owners: Remove unnecessary duplicate strings. 2020-06-14 21:32:10 -07:00
sahil839 18429cfd29 realm: Allow only owners to set message retention policy for a realm.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.

Dropdown for changing retention policy is disabled in UI for admins also.
2020-06-14 21:23:53 -07:00
sahil839 87e72ac8e2 realm: Allow only owners to configure auth methods for a realm.
This commit adds the restriction on configuring auth methods for
admins. We now allow only owners to configure the auth methods
for realm.
2020-06-14 21:23:51 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Graham Bleaney 461d5b1a3e pysa: Introduce sanitizers, models, and inline marking safe.
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
2020-06-11 12:57:49 -07:00
Graham Bleaney 56345f98cb registration: Incorporate redirect URL building helper function.
This commit adds 'add_query_to_redirect_url' to one additional
function which had not yet been written when
'add_query_to_redirect_url' was introduced. This helper centralizes
URL manipulation for redirects, making it easier to add Pysa
sanitization in subsequent commits.
2020-06-11 12:57:49 -07:00
Kartik Srivastava 8c39ddfd28 api: Add GET /users/{user_id}/subscription/{stream_id} endpoint.
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.

Fixes #14966.
2020-06-10 17:59:14 -07:00
Steve Howell d5cc29755e typing: Accept only user_ids in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.

There may be a very tiny number of mobile
clients that still use the old emails API.

This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.

Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.

The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:

    validator=check_list(check_int)

This also allows us to remove a test hack
related to the API documentation.  (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
2020-06-10 17:50:32 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
sahil839 86b52ef7bf users: Owners can only be deactivated by other organization owners.
This commit adds restriction on deactivation of organization owners.
Only owners can deactivate other organization owners.
2020-06-10 17:33:02 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
sahil839 6d667dbe53 realm owner: Add ability to change realm owner status of user.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.

This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.

We need to apply that restriction both in the role change code path
and the deactivate code path.
2020-06-10 12:49:37 -07:00
Anders Kaseorg 5c9d56d2f7 get_safe_redirect_to: Remove incorrect allowed_hosts setting.
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:51:52 -07:00
Dinesh dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
Tim Abbott 2beaf2cab2 pointer: Remove have_initial_messages code.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history.  However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
2020-06-08 22:36:18 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Anders Kaseorg b1b34b6f6a dev_login: Move owners to the top.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 17:26:34 -07:00
sahil839 7de23b8b5c users: Remove short_name and client_id from get_profile_backend.
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.

* short_name hasn't been present anywhere else in the Zulip API for
  several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
  returned by the register_event_queue process.

The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
2020-06-08 17:01:08 -07:00
sahil839 1f5778bad7 users: Refactor get_profile_backend to be based on format_user_row.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.

To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.

This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
2020-06-08 16:57:44 -07:00
Anders Kaseorg 8e4f22c184 auth: Require algorithms setting for JWT auth.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning.  This is for protecting against
symmetric/asymmetric key confusion attacks.

This is a backwards-incompatible configuration change.

Fixes #15207.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 16:22:25 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 139cb8026f auth: Accept next as POST parameter in POST requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 11:07:32 -07:00
Anders Kaseorg 22178c169e request: Replace type argument hack with a different workaround.
This works in mypy 0.770 and is needed for mypy 0.780.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
sahil839 43bff1f67a streams: Extract removing_someone_else logic to a function.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.

This is a follow up to 2187c84.
2020-06-05 16:04:28 -07:00
Aman Agrawal a096f34cab move_topic_to_stream: Add option to disable breadcrumb messages.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.

User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
2020-06-05 12:28:51 -07:00
Anders Kaseorg 4d04fa3118 compose: Rewrite Zoom video call integration to use OAuth.
This reimplements our Zoom video call integration to use an OAuth
application.  In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.

Fixes #11672.

Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-06-03 16:39:12 -07:00
Anders Kaseorg 7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
sahil839 0b432e26b6 test_subs: Add tests for unsubscribing multiple users from stream.
This commit adds tests for unsusbcribing multiple users from a
stream and fixes the missing coverage issue introduced in 2187c84.
2020-06-02 16:27:22 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Mateusz Mandera 3e7fc17788 auth: Delegate RemoteUser SSO to browser when using the desktop app. 2020-06-02 13:00:17 -07:00
Tim Abbott fbcf6a7d72 streams: Add nocoverage to fix CI. 2020-06-01 16:12:13 -07:00
sahil839 2187c84ed9 stream: Add support for sending user_id to endpoint when subscribing users.
This commit modifies the backend to accept user ids when subscribing
users to streams.

It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
2020-06-01 15:59:04 -07:00
sahil839 9ef1c5b1a6 users: Add is_owner field to user objects returned by get endpoints.
This commit adds 'is_owner' field to the user object returned by
'/users', 'users/{user_id}', and '/users/me' endpoints.
2020-06-01 15:33:51 -07:00
sahil839 bd9e891e1c dev_login: Change dev env login page to show Desdemona as owner.
This commits changes the dev login page to show Desdemona as
owner.
2020-06-01 15:22:44 -07:00
Mateusz Mandera 4b10a07c3e auth: Fix social signup with the desktop app.
Only start_social_login had the code for handling the desktop
application, it was missing from start_social_signup.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 676305f6ab auth: Extract handle_desktop_flow wrapper. 2020-06-01 14:14:58 -07:00
sahil839 9fa60672e6 users: Modify update user API endpoint to accept role as parameter.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.

User role dropdown in user info modal is also modified to use
"dropdown_options_widget".

Modified by tabbott to document the API change.
2020-05-29 14:29:17 -07:00
clarammdantas f0c85c5796 registration: Show error if invalid key is posted on /accounts/register.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
2020-05-27 15:37:16 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
arunikaydav42 4680d504de invite: Fix invitations removed from list too soon.
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.

Fixes: #12281
2020-05-26 21:45:44 -07:00
Vishnu KS 8784539d53 free trial: Send users to /upgrade after realm creation. 2020-05-26 17:01:32 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
Vishnu KS 4e0d8953b4 models: Create ensure_not_on_limited_plan instance method. 2020-05-19 14:48:59 -07:00
Vishnu Ks 66b1ad7002 billing: User FREE_TRIAL_DAYS instead of FREE_TRIAL_MONTHS. 2020-05-16 14:52:01 -07:00
Puneeth Chaganti 734d651b45 home: Fix furthest read time when user has multiple clients.
The query to fetch the latest user activity was missing an
`.order_by('last_visit')`. This meant that the results were being
ordered by the `id`, which resulted in us getting `update_message_flags`
action performed on the client that the user installed last, instead of
being client agnostic and fetching the "global" last
`update_message_flags` action performed by the user.
2020-05-14 13:38:09 -07:00
Anders Kaseorg 4362cceffb portico: Add setting to put Google Analytics on selected portico pages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-11 23:22:50 -07:00
Vishnu KS f1b1bf5a0d billing: Add support for Zulip Standard free trial. 2020-05-11 17:20:54 -07:00
Pragati Agrawal 60a762704a org settings: Fix limited plan realm can change message_retention_days.
These changes should be included in bd9b74436c,
as it makes sure that Zulip limited plan realm won't be able to change the
`message_retention_days` setting.
2020-05-09 16:40:50 -07:00
Puneeth Chaganti ded3b00760 home: Use last update_message_flags activity instead of pointer.
The pointer doesn't get updated when a user is only reading messages in
narrowed views. But, we use the pointer position to determine the
furthest read time, which causes the bankruptcy banner to show up even
for users who have been actively reading and sending messages.

This commit switches to using the time of the last update_message_flags
activity by a user to determine the time of last activity.
2020-05-08 15:06:06 -07:00
Puneeth Chaganti 1adb88cf62 home: Extract code to get the furthest read time. 2020-05-08 15:01:08 -07:00
Aman Agrawal 898e717b9d register:api: Provide additional fields that can be useful to clients.
Fixes #14166

* Bumped the api version to feature level 4.
See https://github.com/zulip/zulip/issues/14166#issuecomment-598961462
for details about implementation and future steps.
2020-05-08 14:54:54 -07:00
Aman Agrawal 0c2b25cab3 home/page_params: Add suffix `_mib` to clarify the size units.
The docs say "megabytes" or "MB", but client software needs to
know the actual meaning, which is in mebibytes.
2020-05-08 14:54:54 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Tim Abbott a702894e0e middleware: Stop using X_REAL_IP.
The comment was wrong, in that REMOTE_ADDR is where the real external
IP was; X_REAL_IP was the loadbalancer's IP.
2020-05-08 11:40:54 -07:00
Tim Abbott 87ddd73dcb help: Add article for how to get support.
This page isn't polished properly and I'm not sure it's the best
decision tree here, but it's definitely better to have this page than
not, and we can always adjust forward.

Fixes #10033.
2020-05-04 09:38:36 -07:00
Hashir Sarwar 55a8e7dff2 settings: Offer hiding presence info from other users.
For privacy-minded folks who don't want to leak the
information of whether they're online, this adds an
option to disable sending presence updates to other
users.

The new settings lies in the "Other notification
settings" section of the "Notification settings"
page, under a "Presence" subheading.

Closes #14798.
2020-05-02 15:09:05 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
jagansivam28 bbf5a5efed invitation: Make Member to see invitations sent by him/her.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.

Fixes #14007.
2020-05-01 17:42:49 -07:00
Anders Kaseorg a552c2e5f9 auth: Use the clipboard instead of zulip:// for desktop auth flow.
This does not rely on the desktop app being able to register for the
zulip:// scheme (which is problematic with, for example, the AppImage
format).

It also is a better interface for managing changes to the system,
since the implementation exists almost entirely in the server/webapp
project.

This provides a smoother user experience, where the user doesn't need
to do the paste step, when combined with
https://github.com/zulip/zulip-desktop/pull/943.

Fixes #13613.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-04-30 16:45:00 -07:00
Wyatt Hoodes 82e7ad8e25 data exports: Handle pending and failed exports.
Prior to this change, there were reports of 500s in
production due to `export.extra_data` being a
Nonetype.  This was reproducible using the s3
backend in development when a row was created in
the `RealmAuditLog` table, but the export failed in
the `DeferredWorker`.  This left an entry lying
about that was never updated with an `extra_data`
field.

To fix this, we catch any exceptions in the
`DeferredWorker`, and then update `extra_data` to
encode the failure.  We also fix the fact that we
never updated the export UI table with pending exports.

These changes also negated the use for the somewhat
hacky `clear_success_banner` logic.
2020-04-30 13:00:59 -07:00
Hashir Sarwar 3a110bd647 actions: Remove redundant 'url_embed_preview_enabled' check.
We add URLs to the `links_for_embed set`, only when
the `url_embed_preview_enabled` flag is turned on.
So, it is sufficient to check if `links_for_embed`
is not empty.
2020-04-28 22:41:00 -07:00
Mateusz Mandera f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
akashaviator 28d58c848d api: Guess emoji_type when adding a reaction.
This changes add_reaction in zerver.views.reactions to allow
calling POST ../messages/{message_id}/reactions api endpoint with
emoji_name only, even in the case of a custom emoji.
2020-04-28 20:49:39 -07:00
Puneeth Chaganti f280b64caa integrations: Remove hasattr check for stream_name on Integrations.
`stream_name` is always set on `Integration`. If it is not explicitly
specified, it is set to the name of the integration.
2020-04-28 13:00:09 -07:00
Tim Abbott 976e554799 Revert "home: Don't use pointer to compute furthest_read_time."
This reverts commit 8f32db81a1.

This change unfortunately requires an index that we don't have, and
thus is incredibly expensive.  We'll need to do a thoughtful reworking
before we can integrate it again.
2020-04-27 22:29:39 -07:00
Puneeth Chaganti 8f32db81a1 home: Don't use pointer to compute furthest_read_time.
When a user is reading messages only in stream or topic narrows, the pointer
can be left far behind. Using this to compute the furthest_read_time causes
the banckruptcy banner to be shown even when a user has been actively
reading messages. This commit switches to using the sent time on the last
message that the user has read to compute the furthest read time.
2020-04-26 22:41:52 -07:00
Tim Abbott d8def0de55 pointer: Remove useless last_pointer_updater field.
This hasn't done anything in at least 5 years.
2020-04-26 22:39:24 -07:00
Tim Abbott 731cd79436 messages: Remove first_unread pointer hack.
This hack was important when only the mobile apps (and not the webapp)
were using the unread_msgs data structure and the first_unread
infrastructure.  Now that the webapp is using those things, there
aren't leaked ancient unread messages that aren't accessible on the
webapp, so any few users still in this situation can get out of it by
just reading the problematic messages.
2020-04-26 22:37:41 -07:00
Dinesh 2735860f01 browser_support: Drop support for Internet Explorer.
Internet Explorer does not support `position: sticky` which improves
floating recipient bar behavior during scrolling which is one of the
issues blocking PR #9910.
IE also does not support some features that modern browsers support
hence may not super well.
This commit adds an error page that'll be displayed when a user logs
in from Internet Explorer. Also, a test is added.
2020-04-23 17:50:28 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Kartik Srivastava b29ccdf51c
api: Refactor get_members_backend to return a single bot's data.
This makes `get_members_backend` in zerver/views/users.py to
return a single bot's data too.
2020-04-21 13:45:58 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
Mateusz Mandera 4018dcb8e7 upload: Include filename at the end of temporary access URLs. 2020-04-20 10:25:48 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Udit107710 db30cf470c refactor: Making email_mirror independent of actions.
Moved truncate_body, truncate_content and truncate_topic
to message.py.
2020-04-18 16:58:29 -07:00
Divyanshu Agrawal 8a64ed6e20 alert words: Limit alert word length to 100.
We do limit the length to 100 in the frontend, but no such check
exists in the backend.

Check that a new alert word has a maximum length of 100 in the
alert_words endpoint.
2020-04-17 11:32:16 -07:00
Tim Abbott 0ccc0f02ce upload: Support requesting a temporary unauthenticated URL.
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).

The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.

Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-04-17 09:08:10 -07:00
Tim Abbott d30c8d4a66 terms: Fix email address visibility displayed when accepting ToS.
Apparently, we never tested this unusual flow (primarily used after
Slack import) with an organization with EMAIL_ADDRESS_VISIBILITY_ADMINS.
2020-04-16 11:44:51 -07:00
Kartik Srivastava a060691389 default streams: Change add api to use stream_id.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.

Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
2020-04-15 15:47:04 -07:00
Puneeth Chaganti 67822a0544 messages: Fix 500 on queries with empty string as search operand.
The error is PGroonga specific since `pgroonga_query_extract_keywords` does
not handle empty string inputs correctly. This commit prevents search
narrows from having empty operands.

Closes #14405
2020-04-12 11:04:09 -07:00
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Rohitt Vashishtha f9caf522f0 markdown: Allow setting a default language for code blocks.
This adds a new realm setting: default_code_block_language.

This PR also adds a new widget to specify a language, which
behaves somewhat differently from other widgets of the same
kind; instead of exposing methods to the whole module, we
just create a single IIFE that handles all the interactions
with the DOM for the widget.

We also move the code for remapping languages to format_code
function since we want to preserve the original language to
decide if we override it using default_code_clock_language.

Fixes #14404.
2020-04-09 16:02:02 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job.
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
2020-04-08 12:52:31 -07:00
Mateusz Mandera 9911c6a0f0 rate_limiter: Put secs_to_freedom as message when raising RateLimited.
That's the value that matters to the code that catches the exception,
and this change allows simplifying the plumbing somewhat, and gets rid
of the get_rate_limit_result_from_request function.
2020-04-08 10:29:18 -07:00
Tim Abbott 843345dfee message_edit: Add backend for moving a topic to another stream.
This commit reuses the existing infrastructure for moving a topic
within a stream to add support for moving topics from one stream to
another.

Split from the original full-feature commit so that we can merge just
the backend, which is finished, at this time.

This is a large part of #6427.

The feature is incomplete, in that we don't have real-time update of
the frontend to handle the event, documentation, etc., but this commit
is a good mergable checkpoint that we can do further work on top of.
We also still ideally would have a test_events test for the backend,
but I'm willing to leave that for follow-up work.

This appears to have switched to tabbott as the author during commit
squashing sometime ago, but this commit is certainly:

Co-Authored-By: Wbert Adrián Castro Vera <wbertc@gmail.com>
2020-04-07 14:19:19 -07:00
Mateusz Mandera 8686bbf637 auth: Show a user-facing page when wrong token given to /subdomain/.
This used to show a blank page. Considering that the links remain valid
only for 15 seconds it's important to show something more informative to
the user.
2020-04-05 12:29:09 -07:00
Tim Abbott e27cafd785 settings: Don't use nice REQ syntax for language_codes.
This breaks provisioning because running this as import time would
require language_name_map.json to be generated by `manage.py
compilemessages` before we can run any management commands :(.

We could potentially fix this in the future by changing the generate
language files to be things we commit to the project.
2020-04-03 16:00:47 -07:00
sahil839 b66a67272f models: Reuse constants for common realm permissions policies.
This is a prep commit for making use of same choices for
create_stream_policy and invite_to_stream_policy as both fields
have same set of choices.

This will be useful as we add other fields using these same types.
2020-04-03 15:17:51 -07:00
sahil839 386215614c models: Replace WAITING_PERIOD with FULL_MEMBERS in choice variables.
This commit replaces the WAITING _PERIOD with FULL_MEMBERS from
create_stream_policy and invite_to_stream_policy choices to
achieve consistency and making the variables more descriptive.
2020-04-03 15:14:13 -07:00
Tim Abbott a745e533fe settings: Use cleaner validators for display settings.
This simplifies the update_display_settings endpoint to use REQ for
validation, rather than custom if/else statements.

The test changes just take advantage of the now more consistent
syntax.
2020-04-03 15:09:14 -07:00
Tim Abbott 17723b1617 version: Move minimum desktop version configuration to version.py.
This makes it relatively easy for a system administrator to
temporarily override these values after a desktop app security
release that they want to ensure all of their users take.

We're not putting this in settings, since we don't want to encourage
accidental long-term overrides of these important-to-security values.
2020-04-01 13:23:08 -07:00
Steve Howell 2fe21de733 dev login: Sort users. 2020-03-30 10:16:27 -04:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Tim Abbott 737a36a9f8 portico: Use a backend variable to determine desktop version.
This makes it possible to change this parameter without rebuilding all
the server's static assets.
2020-03-27 01:37:56 -07:00
Anders Kaseorg 56aadf6503 tests: Fix CI apocalypse perpetrated by previous commit.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:25:25 -07:00
Steve Howell 697b00dd6e default streams: Change remove api to use stream_id.
This is a full-stack change:

    - server
    - JS code
    - templates

It's all pretty simple--just use stream_id instead
of stream_name.

I am 99% sure we don't document this API nor use it
in mobile, so it should be a safe change.
2020-03-25 17:11:25 -07:00
Graham Bleaney 2fe9d85a5f redirects: Refactor redirect code to use central helper function.
This commit introduces two new functions in 'url_encoding.py' which
centralize two common patterns for constructing redirect URLs. It
also migrates the files using those patterns to use the new
functions.
2020-03-25 16:39:17 -07:00
Graham Bleaney 3e602a9bd4 integrations: Harden development views against path traversal.
This commit hardens the code in the developer integrations views by
ensuring that file paths can only be constructed with valid
integration names.
2020-03-25 16:39:17 -07:00
Tim Abbott d9bb6d0081 compatibility: Add more strict desktop app blocking.
This allows us to block use of the desktop app with insecure versions
(we simply fail to load the Zulip webapp at all, instead rendering an
error page).

For now we block only versions that are known to be both insecure and
not auto-updating, but we can easily adjust these parameters in the
future.
2020-03-24 20:33:11 -07:00
Tim Abbott 85c9ffd91c message: Validate propagate_mode parameters.
This improves the error handling for invalid values of the
propagate_mode parameter to our message editing endpoints.
Previously, invalid values would just work like change_one rather than
doing nothing.
2020-03-24 12:36:45 -07:00
Dinesh 5888d7c0f5 auth: Change how config error URLs are configured.
URLs for config errors were configured seperately for each error
which is better handled by having error name as argument in URL.
A new view `config_error_view` is added containing context for
each error that returns `config_error` page with the relevant
context.
Also fixed tests and some views in `auth.py` to be consistent with
changes.
2020-03-22 17:15:18 -07:00
Steve Howell a041d9e4aa minor: Clean up lstrip() for help article titles.
Saying `foo.lstrip('# ')` does more than just remove
a '# ' prefix.  It removes any combination of '#' and
spaces.

We now make the intention slightly more clear.

We would strip these as you'd expect:

    # foo
    ## foo
    ### foo

but for this we now only strip the first "#":

    # # # # # foo
2020-03-22 11:32:29 -07:00
Steve Howell edf1b1e5e8 minor: Fix buggy lstrip() call in integrations dev panel.
Thanks to @minusworld for catching this--see #14264, which
points out that lstrip() doesn't do what your intuition
might tell you it does.

Now we properly remove the "HTTP_" prefix.

It's not clear to me why we need these prefixes for Django
purposes in the fixtures, but I didn't want to go down
the rabbit hole of fixing those.

To test:

    got to http://YOUR-DEV_SERVER/devtools/integrations/
    select "bitbucket3" for the integration.
    select "diagnostics_ping.json" for the fixture.
    see "X_EVENT_KEY" in "Custom HTTP Headers"

Fixes #14264
2020-03-22 11:32:29 -07:00
Steve Howell d71111f3dc presence api: Use email to look up presence.
We don't want to use delivery_email to look up
presence on email-restricted realms.
2020-03-19 14:32:18 -07:00
Mateusz Mandera f5e95c4fc1 requirements: Bump python-social-auth version.
We had a bunch of ugly hacks to monkey patch things due to upstream
being temporarily unmaintained and not merging PRs. Now the project is
active again and the fixes have been merged and included in the latest
version - so we clean up all that code.
2020-03-18 12:14:31 -07:00
Tim Abbott b064559652 zephyr: Add strict assertion about username format.
This ensures that even if it were possible to create an MIT Kerberos
account with a malicious username and/or hack webathena to pretend
that's the case, one couldn't do anything malicious.

This security improvement only impacts a single installation of Zulip
where Zephyr mirroring is in use that has already had the fix applied,
so there's no reason to do a security notice for it.

Found by Graham Bleaney using pysa.
2020-03-17 05:37:25 -07:00
Mateusz Mandera 2000608a9e report_error: Fix inaccurate docstring.
do_report_error isn't actually below.
2020-03-09 13:54:58 -07:00
Mateusz Mandera 0255ca9b6a middleware: Log user.id/realm.string_id instead of _email. 2020-03-09 13:54:58 -07:00
akashaviator 9c63976da5 api: Refactor get_members_backend in zerver/views/users.py.
This refactors get_members_backend to return user data of a single
user in the form of a dictionary (earlier being a list with a single
dictionary).

This also refactors it to return the data with an appropriate key
(inside a dictionary), "user" or "members", according to the type of
data being returned.

Tweaked by tabbott to use somewhat less opaque code and simple OpenAPI
descriptions.
2020-03-08 18:43:30 -07:00
Tim Abbott 2c75b39078 templates: Delete show_debug feature.
As far as I know, this hasn't been used in at least 5 years, and I'm
not sure there's a real use case for it with the current app.
2020-03-08 18:34:59 -07:00
Tim Abbott 53cc00c21c messages: Ban the sender property when not mirroring.
The "sender" property in `send_message_backend` is meant to only do
something when doing Zephyr mirroring (or similar).  We should help
clients behave correctly by banning this property in requests that are
not specifically requesting mirroring behavior.

This commit requires changes to a number of tests that incorrectly
passed this parameter or didn't use the right setup for mirroring.
2020-03-08 14:09:32 -07:00
Tim Abbott 9230213bde settings: Add EMAIL_ADDRESS_VISIBILITY_NOBODY.
This extends our email address visibility settings to deny access to
user email addresses even to organization administrators.

At the moment, they can of course change the setting (which leaves an
audit trail), but in the future only organization owners will be able
to change that setting.

While we're at this, we rewrite the settings_data.js test to cover all
the cases in a more consistent way.

Fixes #14111.
2020-03-06 16:34:08 -08:00
Steve Howell f2b8eef21a refactor: Avoid hacky use of ValidationError.code.
We were using `code` to pass around messages.

The `code` field is designed to be a code, not
a human-readable message.

It's possible that we don't actually need two
flavors of messages for these type of validations,
but I didn't want to change that yet.

We **definitely** don't need to put two types of
message in the exception, so I fix that.  Instead,
I just have the caller ask what level of detail
it needs.

I added a non-verbose message for the case of
system bots.

I removed the non-translated version of the message
for deactivated accounts, which didn't have test
coverage and is slightly more prone to leaking
email info that we don't want to leak.
2020-03-06 11:53:22 -08:00
Steve Howell 62fb3ad801 refactor: Move validate_email_not_already_in_realm.
We move this to email_validation.py.
2020-03-06 11:53:22 -08:00
Steve Howell ad85e286de user settings: Inline call to validate_email.
We are trying to elminate the version of
`validate_email` that lives in `actions.py`.

Inlining it barely increases the code size, and
it removes some noise related the three-item
tuple that `check_incoming_email` returns.
2020-03-06 11:53:22 -08:00
Steve Howell 4f5b07a7e6 refactor: Extract zerver/lib/email_validation.py. 2020-03-06 11:53:22 -08:00
Steve Howell 57f1aa722c refactor: Rename validate_email_for_realm.
Now called:

    validate_email_not_already_in_realm

We have a separate validation function that
makes sure that the email fits into a realm's
domain scheme, and we want to avoid naming
confusion here.
2020-03-06 11:53:22 -08:00
Mateusz Mandera 6e4dcc714e auth: Fix bug in backend configuration checks in start_social_login.
We shouldn't check if *every* backend in the backends list is
configured, but only the one the user is attempting to log in with.
2020-03-01 22:30:38 -08:00
Mateusz Mandera 2d55c7e0d5 home: Don't assume user agent header is set for insecure_desktop_app.
The header may not be set - this leads to CI failures on 2.1.x branch,
but in any case is a real bug.
2020-02-28 16:51:23 -08:00
Tim Abbott d79a7a8c35 panels: Show a banner for users with legacy desktop apps.
Users who are using ZulipDesktop or haven't managed to auto-update to
ZulipElectron should be strongly encouraged to upgrade.

We'll likely want to move to something even stricter that blocks
loading the app at all, but this is a good start.
2020-02-28 01:54:46 -08:00
Mateusz Mandera efb3065158 social_auth: Take user to find_account if invalid subdomain is given.
This allows to also clean up some code that's not really useful.
2020-02-27 17:27:55 -08:00
Anders Kaseorg 1cdab5ae61 emoji: Resolve emoji sprite sheets and stylesheets through Webpack.
This gives them cache-compatible URLs, and also avoids some extra
copies of the sprite sheet images.

Comments on the Octopus emoji added by tabbott.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:43:46 -08:00
Steve Howell 29dd7d2267 minor: Fix comment in send_message_backend.
The `send_message_backend` function no longer
calls `recipient_for_emails`.
2020-02-24 15:32:29 -08:00
Mateusz Mandera 2343f80d72 auth: Remove redundant get_subdomain(request) call in remote_user_sso.
subdomain = get_subdomain(request) is already called earlier in the
function.
2020-02-24 12:39:48 -08:00
Mateusz Mandera f2903e9c70 auth: Refactor - convert remote_username to email in remote_user_sso.
So far the conversion was in a very random place -
register_remote_user(). All other codepaths that use
login_or_register_remote_user() call it with the user's email address.
Making remote_user_sso convert remote_username to the email address
before calling login_or_register_remote_user makes this usage consistent
across the board.
2020-02-24 12:39:48 -08:00
Mateusz Mandera 98ae2fb940 auth: Remove redundant realm argument to finish_desktop_flow.
finish_desktop_flow is called with the assumption that the request
successfully proved control over the user_profile and generates a
special link to log into the user_profile account. There's no reason to
pass the realm param, as user_profile.realm can be assumed.
2020-02-24 12:39:48 -08:00
Dinesh 1308544a70 auth: Remove `if` blocks to redirect to config error page.
In `auth.py` there are three `if` blocks for different backends
to redirect to config error page with similar code. It is better
handled with common code using `get_attr()` function on
constructed setting names.
2020-02-24 12:19:45 -08:00
vsvipul 020a263a67 auth: Create a new page hop for desktop auth.
Create a new page for desktop auth flow, in which
users can select one from going to the app or
continue the flow in the browser.

Co-authored-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-02-20 11:59:55 -08:00
Vishnu KS e1a7716578 emails: Translate from_name of account security emails. 2020-02-18 17:45:33 -08:00
Mateusz Mandera 6a0b68bc7f models: Delete get_stream_recipient function and its uses.
With recipient being now a Stream field, there's no more use for
this helper function.
2020-02-18 10:49:14 -08:00
rht 41e3db81be dependencies: Upgrade to Django 2.2.10.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.

Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:

* urls: django.urls.resolvers.RegexURLPattern has been replaced by
  django.urls.resolvers.URLPattern; affects OpenAPI code and related
  features which re-parse Django's internals.
  https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
  ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
  https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
  https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
  the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.

There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.

Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.

Fixes #10835.
2020-02-13 16:27:26 -08:00
Mateusz Mandera bde495db87 registration: Add support for mobile and desktop flows.
This makes it possible to create a Zulip account from the mobile or
desktop apps and have the end result be that the user is logged in on
their mobile device.

We may need small changes in the desktop and/or mobile apps to support
this.

Closes #10859.
2020-02-12 11:22:16 -08:00
Mateusz Mandera 2475adbf8a messages_for_topic: Use stream.recipient_id for more efficient query. 2020-02-11 17:39:43 -08:00
Dinesh 4304d5f8db auth: Add support for GitLab authentication.
With some tweaks by tabbott to the documentation and comments.

Fixes #13694.
2020-02-11 13:54:17 -08:00
Tim Abbott fcac3a4342 recipients: Rename extract_recipients to extract_private_recipients.
Recent changes mean this function is now only used for private
messages.
2020-02-11 12:28:14 -08:00
Steve Howell 1b6578cafd messages: Fix bug with commas in stream names.
We now validate streams with a separate
function from PM recipients.

It's confusing enough all the ways you can
encode a stream or encode the PM recipients,
but trying to do it all in one function was
hard to reason about and led to at least one
bug.

In particular, there was a bug where streams
with commas in them would get split.  Now
we just don't ever split on commas inside
of `extract_stream_indicator`.

Fixes #13836
2020-02-11 12:20:54 -08:00
akashaviator 1ae5964ab8 api: Add an api endpoint for GET /users/{id}
This adds a new API endpoint for querying basic data on a single other
user in the organization, reusing the existing infrastructure (and
view function!) for getting data on all users in an organization.

Fixes #12277.
2020-02-07 10:36:31 -08:00
Tim Abbott 79e5dd1374 users: Rename get_raw_user_data user parameter to acting_user.
This is for improved clarity as we extend this function to take
multiple user objects.
2020-02-07 10:36:31 -08:00
Steve Howell bf3baa14ac presence: Rename get_status_dict_by_user(). 2020-02-06 17:16:22 -08:00
Steve Howell 36fba1076f presence: Move get_status_dict_by_user. 2020-02-06 17:16:22 -08:00
Steve Howell 703338dfa3 presence: Extract lib/presence.py.
This will make more sense when we pull some
code out of the model.
2020-02-06 17:16:22 -08:00
Steve Howell a5093be867 presence: Rename get_status_list.
The word "status" is vague, and this isn't
actually returning a list, so we now name it
get_presence_response.

I originally was gonna rename this to
get_presence_dict, but there's a function
called get_status_dict that returns a subset
of the response, so I think it's a bit more
clear that this is the bigger dict that
actually gets sent back.
2020-02-06 17:16:22 -08:00
Steve Howell 8a1fb2dcd6 presence: Calculate server_timestamp slightly earlier.
We want to err on the side of server_timestamp being
old, since we may eventually use this to make responses
just include incremental changes, and we don't want a
time window (however small) when we miss presence rows.
The clients will be able to deal with duplicate data
to the extent that the time windows are overlapping.

Also, extracting the other local var here
(for `presences`) will set up a subsequent commit
where we re-format the data for clients with
slim_presence=True.
2020-02-06 17:16:22 -08:00
Tim Abbott b5cc1a3afe home: Finish removing assumption that user_profile isn't None. 2020-02-05 16:16:33 -08:00
Tim Abbott be02604fe5 home: Refactor logic for several more settings.
The main functional change is there's now a code path for
`user_profile is None`.
2020-02-05 16:13:29 -08:00
Tim Abbott 5f8af84ad1 home: Refactor emojiset and billing logic for None UserProfile. 2020-02-05 16:09:03 -08:00
Tim Abbott fc4400cfa7 home: Remove unused avatar_url context parameter. 2020-02-05 16:05:02 -08:00
Tim Abbott 7044aa8e5d home: Extract updates to last_reminder to a function. 2020-02-05 16:05:02 -08:00
Tim Abbott 85690ce649 home: Extract handling of narrow windows to a function. 2020-02-05 16:05:02 -08:00
Tim Abbott 42d529e749 home: Extract terms of service check to a function. 2020-02-05 16:05:01 -08:00
Anders Kaseorg 4d49a20430 requirements: Upgrade django-sendfile2 from 0.4.3 to 0.5.1.
The module was renamed from sendfile to django_sendfile.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Mateusz Mandera 0e7c97378e is_safe_url: Use allowed_hosts instead of depreciated host argument.
Judging by comparing django 1.11 with django 2.2 code of this function,
this shouldn't change any behavior.
2020-02-04 12:46:53 -08:00
Steve Howell 9847d4d9a3 refactor: Use user_id in get_status_dict_by_user.
This avoids a needless user lookup in apply_event.
2020-02-04 12:30:36 -08:00
Steve Howell bf9144ff69 presence: Add slim_presence flag.
This flag affects page_params and the
payload you get back from POSTs to this
url:

    users/me/presence

The flag does not yet affect the
presence events that get sent to a
client.
2020-02-04 12:30:34 -08:00
Ryan Rehman 3dc7d60ffe muting: Record DateTime when a Topic is muted.
This includes the necessary migration to add
the date_muted field to the MutedTopic class
and populates it with a hard coded value.
2020-02-02 20:49:53 -08:00
Mateusz Mandera b685d506ad auth: Make login_context available to TwoFactorLoginView. 2020-02-02 20:22:49 -08:00
Mateusz Mandera 06198af5b9 auth: Handle rate limiting in OurAuthenticationForm and user_settings.
These parts of the code should catch the RateLimited exception and
generate their own, apprioprate user-facing error message.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 677764d9ca auth: Pass request kwarg in authenticate() calls with username+password.
These authenticate() calls use either Email or LDAP backends, which will
be rate limited and will need access to the request object.
2020-02-02 19:15:00 -08:00