Commit Graph

2772 Commits

Author SHA1 Message Date
sahil839 4e8f5b5b31 streams: Change access_stream_for_delete_or_update to also return sub.
We modify access_stream_for_delete_or_update function to return
Subscription object also along with stream. This change will be
helpful in avoiding an extra query to get subscription object in
code for updating subscription role.
2020-10-08 17:07:30 -07:00
Mateusz Mandera 6e83bcc0d5 custom_profile_fields: Don't allow leading/trailing whitespaces.
Allowing such whitespaces can lead to hard to debug issues e.g. with
ldap sync.
2020-10-02 14:58:06 -07:00
Tim Abbott 351d73ac5a home: Pass realm to build_page_params_for_home_page_load.
This is preparation for this needing to be a separate parameter from
the user.
2020-10-01 15:00:36 -07:00
sahil839 78b98d8067 realm: Add wildcard_mention_policy setting.
We add a new wildcard_mention_policy setting to handle wildcard
mentions in large streams, with a wide range of policies available to
organizations.

We set the default to the safe option for preventing accidental spam:
only stream administrators being able to use wildcard mentions in
large streams.
2020-10-01 12:18:03 -07:00
Steve Howell c199571112 mypy: Add StreamDict.
This requires us to rework the view code a little
bit to explicitly assign fields.
2020-09-29 16:49:10 -07:00
Dinesh 232eb8b7cf auth: Render config error page on configuration error.
We previously used to to redirect to config error page with
a different URL. This commit renders config error in the same
URL where configuration error is encountered. This way when
conifguration error is fixed the user can refresh to continue
normally or go back to login page from the link provided to
choose any other backend auth.

Also moved those URLs to dev_urls.py so that they can be easily
accessed to work on styling etc.

In tests, removed some of the asserts checking status code to be 200
as the function `assert_in_success_response` does that check.
2020-09-25 16:16:17 -07:00
Vishnu KS a888e65ea4 i18n: Don't include email tags in translation strings. 2020-09-25 15:53:26 -07:00
Anders Kaseorg faf600e9f5 urls: Remove unused URL names and shorten others.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Aman Agrawal 37e77c77e2 message_fetch: Better handle unauthorized requests.
We raise MissingAuthenticationError now, which adds
`www_authenticate=session` header to the error response. This
stops modern web-browsers from displaying a login form everytime
a 401 response it sent to the client.
2020-09-21 16:07:43 -07:00
Aman Agrawal 2bc3924672 move_topic_to_stream: Allow moving to/between/from private streams.
Fixes #16284.

Most of the work for this was done when we implemented correct
behavior for guest users, since they treat public streams like private
streams anyway.

The general method involves moving the messages to the new stream with
special care of UserMessage.

We delete UserMessages for subs who are losing access to the message.
For private streams with protected history, we also create UserMessage
elements for users who are not present in the old stream, since that's
important for those users to access the moved messages.
2020-09-14 15:00:55 -07:00
Anders Kaseorg 463929f349 urls: Migrate re_path routes to path.
Django treats path("<name>") like re_path(r"(?P<name>[^/]+)") and
path("<path:name>") like re_path(r"(?P<name>.+)").

This is more readable and consistent than the mix of slightly
different regexes we had before, and fixes various bugs:

• The r'apps/(.*)$' regex was missing a start anchor ^, so it
  incorrectly matched all URLs that included apps/ as a substring
  anywhere.
• The r'accounts/login/(google)/$' regex was missing a start anchor ^,
  so it incorrectly matched all URLs that ended with
  accounts/login/google/.
• The type annotation of zerver.views.realm_export.delete_realm_export
  takes export_id as an int, but it was previously passed as a string.
• The type annotation of zerver.views.users.avatar takes medium as a
  bool, but it was previously passed as a string.
• The [0-9A-Za-z]+ pattern for uidb64 was missing the - and _
  characters that can validly be part of a base64url encoded
  string (although I think the id is actually a decimal integer here,
  in which case only 012345ADEIMNOQTUYcgjkwxyz are present in its
  base64url encoding).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 14:29:26 -07:00
Aman Agrawal ed8796f23f report: Allow anonymous queries to report (un)narrow times. 2020-09-11 16:57:27 -07:00
Anders Kaseorg b7b7475672 python: Use standard secrets module to generate random tokens.
There are three functional side effects:

• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.

• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.

• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.

(The base32 alphabet is A-Z 2-7.  We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-09 15:52:57 -07:00
Aman c114447e11 topic_history: Allow anonymous access in web-public streams.
For web-public streams, clients can access full topic history
without being authenticated. They only need to additionally
send "streams:web-public" narrow with their request like all
the other web-public queries.
2020-09-04 10:17:56 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bef46dab3c python: Prefer kwargs form of dict.update.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Aman Agrawal d9431a5e66 exceptions: Raise InvalidSubdomainError when realm is invalid.
When user requests for a realm that doesn't exists, we raise
a InvalidSubdomainError.

This reduces our effort at repeatedly ensuring realm is valid
in request in web-public queries.
2020-09-03 10:45:14 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Tim Abbott 992c6126a8 docs: Update many references to Casper. 2020-08-30 17:16:02 -07:00
Kartik Srivastava b9b9a79382 user_groups: Use 'path_only=True' in REQ for 'user_group_id' in views.
'user_groups' endpoints are currently under 'pending_endpoints' in
test_openapi.py (even after being documented except one), due to the
'user_group_id' and 'group_id' parameter name mismatch in the
url config and the view functions.

This commit includes 'path_only=True' for 'user_group_id' parameter in
views to avoid the failure of 'test_openapi_arguments', in
test_openapi.py, which excludes the path parameters. This is a prep
commit for renaming 'group_id' to 'user_group_id' in the documentation
and removing the 'user_groups' endpoints from 'pending_endpoints'.
2020-08-26 15:40:19 -07:00
Aman c3a8492697 topic: Rename get_topic_history_for_web_public_stream. 2020-08-25 17:01:12 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00
Tim Abbott 28b43b4edc message_fetch: Refactor get_base_query_for_search. 2020-08-12 16:33:59 -07:00
Tim Abbott 221cd496c6 NarrowBuilder: self.user_profile.realm -> self.realm. 2020-08-12 16:33:59 -07:00
Tim Abbott 35bd14574d message_fetch: Refactor public-streams query logic. 2020-08-12 16:33:59 -07:00
Tim Abbott d2e191b94e NarrowBuilder: Rename self.user_realm to self.realm.
The previous name will be very confusing in upcoming commits where the
UserProfile becomes Optional but Realm is not (E.g. for web-public
queries).
2020-08-12 16:33:59 -07:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Alex Vandiver 153f16ee6a links: Flatten the set into a list before serializing into the queue.
orjson does not transparently do this set-to-list translation, unlike
ujson.
2020-08-12 11:42:24 -07:00
Kartik Srivastava 63173d5554 api: Return 'user_id' in 'POST /users' response.
This adds 'user_id' to the simple success response for 'POST /users'
api endpoint, to make it convenient for API clients to get details
about users they just created.  Appropriate changes have been made in
the docs and test_users.py.

Fixes #16072.
2020-08-11 16:40:12 -07:00
Aman Agrawal 94839c9492 NarrowBuilder: Directly use denormalized 'recipient_id' of streams.
'recipient_id' was is now directly accessible and we can now use
it instead of doing an extra query to Recipient model.
2020-08-11 11:22:22 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 114b0a2982 message_edit: Output a list, not a reversed iterator.
A list_reverseiterator object is not JSON serializable, and orjson
enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Alex Vandiver 816f91ae27 auth: Remove redirect on invalid realm during password reset.
This redirect was never effective -- because of the
HostDomainMiddleware, all requests to invalid domains have their
actual results thrown away, and replaced by an "Invalid realm" 404.
These lines are nonetheless _covered_ by coverage, because they do
run; the redirect is simply ineffective.  This can be seen by the test
that was added with them, in c8edbae21c, actually testing the contents
for the invalid realm wording, not the "find your accounts" wording.
2020-08-11 10:27:29 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Anders Kaseorg 36bae2e86c message_fetch: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 80c0e60edc integrations: Decode webhook responses to str.
bytes is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 28143c0ee2 drafts: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Mateusz Mandera bb46b21123 find_account: Fix the email search query.
The search should be case-insensitive.
2020-08-02 12:37:20 -07:00
Clara Dantas a9af80d7a2 streams: Make /streams endpoint return also web-public streams.
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
2020-07-29 17:52:36 -07:00
Hemanth V. Alluri 21c9adbb3c drafts: Add an API endpoint for fetching drafts.
This endpoint will allow a user to fetch their drafts.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 630fd10be1 drafts: Add an API endpoint for deleting drafts.
This endpoint will allow a user to delete a single draft.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 2d307c760e drafts: Add an API endpoint for editing drafts.
This endpoint will allow a user to edit a single draft.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri a0f71b7458 drafts: Add an API endpoint for creating drafts.
This endpoint will allow a user to create drafts in bulk.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Dinesh 9583554d44 auth: Add check_config for apple auth.
Apple has some other obligatory settings other than key and secret.
To handle that this commit adds a function check_config() similar
to that of SAML.
2020-07-28 17:12:49 -07:00
Gittenburg 87d8a54b91 help: Stop collapsing sidebar sections.
Three reasons:

1. The sliding was disorienting.
2. The collapsing disallowed searching for other pages with Ctrl+F.
3. The collapsing mechanism wasn't accessible (not usable with the
   keyboard / no ARIA tags).

Tweaked by tabbott to center the left sidebar on the selected page.

Part of #15948.
2020-07-28 16:28:58 -07:00
majordwarf ad23c6e808 emoji: Switch to `user_profile.realm` for cross-realm reactions.
A temporary fix to enable cross-realm emoji reactions till `Message.realm`
is denormalized.

Fixes #15866.
2020-07-24 13:11:27 -07:00
arpit551 0d79b55b2e audit_log: Record RealmAuditLog in do_change_notification_settings.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:04:40 -07:00
arpit551 94d2de8b4a audit_log: Record RealmAuditLog while changing default streams.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:31 -07:00
arpit551 753713fde7 audit_log: Record RealmAuditLog in do_change_subscription_property.
Removed logging with log_subscription_property_change
and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:29 -07:00
arpit551 e6f54a3278 audit_log: Record RealmAuditLog in do_change_icon_source.
Removed logging with  log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 11:56:40 -07:00
arpit551 ccdf52fef6 audit_log: Log RealmAuditLog for realm notification properties.
Log RealmAuditLog for do_set_realm_notifications_stream and
do_set_realm_signup_notifications_stream function.
Added tests for the same.
2020-07-24 11:51:47 -07:00
arpit551 8f6a1c3f40 actions.py: Removed do_set_realm_message_deleting function.
Used do_set_realm_property function instead of
do_set_realm_message_deleting and removed it.
2020-07-24 11:51:47 -07:00
arpit551 422fa0ff7d audit_log: Log RealmAuditLog for do_set_realm_message_editing.
Log RealmAuditLog for do_set_realm_message_editing. Added tests for
same.
2020-07-24 11:51:46 -07:00
Vishnu KS 9e0ff58a6d team: Rename contrib to contributors in page_params. 2020-07-23 10:22:28 -07:00
Steve Howell 1fa6ae1e16 refactor: Extract build_page_params_for_home_page_load. 2020-07-22 17:15:03 -07:00
Steve Howell 27072289ce refactor: Extract zerver/lib/home.py.
The two functions extracted here are mostly
copied verbatim, but we use dataclasses
to marshal the values back.
2020-07-22 17:15:02 -07:00
Steve Howell 26e4d34e81 refactor: Move code up higher in home view.
The two pieces of code here don't need to be
intermingled with the bigger task of building
page_params.
2020-07-22 17:13:53 -07:00
Vishnu KS 67bacd6e31 billing: Don't allow guest users to upgrade. 2020-07-22 16:57:49 -07:00
Tim Abbott 9efec1f929 auth: Clean up DevAuthBackend error messages.
We had a user confused by these error messages, which suggested they
needed to enable something rather than using a development
environment.
2020-07-21 12:55:11 -07:00
Tim Abbott 6be3fca037 auth: Remove short_name from LDAP API.
As best I can tell, we fetched this field and then ignored it, so
unlike the last few commits, this is more a code cleanup than a
functional change.
2020-07-17 11:28:08 -07:00
Steve Howell c44500175d database: Remove short_name from UserProfile.
A few major themes here:

    - We remove short_name from UserProfile
      and add the appropriate migration.

    - We remove short_name from various
      cache-related lists of fields.

    - We allow import tools to continue to
      write short_name to their export files,
      and then we simply ignore the field
      at import time.

    - We change functions like do_create_user,
      create_user_profile, etc.

    - We keep short_name in the /json/bots
      API.  (It actually gets turned into
      an email.)

    - We don't modify our LDAP code much
      here.
2020-07-17 11:15:15 -07:00
Steve Howell c60f4236a9 api: Do not require short_name to create user.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.

An upcoming commit will eliminate it from the
database.
2020-07-17 11:15:15 -07:00
arpit551 a2428017e9 audit_log: Log RealmAuditLog in do_set_realm_authentication_methods.
Log RealmAuditLog in do_set_realm_authentication_methods and added
tests for it.
2020-07-06 17:38:20 -07:00
arpit551 ba92666dbd audit_log: Log RealmAuditLog in do_set_realm_property.
Log RealmAuditLog in do_set_realm_property and do_remove_realm_domain.
Tests for the changes are written in test_events because it will save
duplicate code for test_change_realm_property.
2020-07-06 17:38:19 -07:00
arpit551 6af337c07e audit_log: Log RealmAuditLog when stream is deactivated. 2020-07-06 17:32:11 -07:00
arpit551 87aaa84b42 audit_log: Log acting_user in do_change_user_role. 2020-07-06 17:32:11 -07:00
arpit551 01f12b9fc2 audit_log: Log acting_user in user creation and user activation. 2020-07-06 17:32:09 -07:00
arpit551 653928bdfe audit_log: Log acting_user in do_change_avatar_fields. 2020-07-06 17:24:18 -07:00
arpit551 2279fef316 audit_log: Log acting_user in do_change_logo_source. 2020-07-06 17:24:18 -07:00
Anders Kaseorg 2fb96e94f1 registration: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Steve Howell 50cbe1b19b minor: Rename var to max_message_id.
We are phasing out the concept of pointer.
2020-07-03 13:08:40 +00:00
Vishnu KS cbd73a6dc1 home: Use get_customer_by_realm for getting customer. 2020-07-01 16:45:38 -07:00
Vishnu KS 4c6350fa4b billing: Add option to request a sponsorship in /upgrade. 2020-07-01 16:45:38 -07:00
Vishnu KS a74cdf3123 18n: Cleanup strings to be translated in add_subscriptions_backend.
Also expand the test to check the details of the user who created the
stream.
2020-07-01 11:03:29 -07:00
Vishnu KS eb50ecf07a 18n: Cleanup strings to be translated in you_were_just_subscribed_message. 2020-07-01 11:03:29 -07:00
Anders Kaseorg f2e7076e2a decorator: Replace type: ignore with cast, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg dcbc8e66fb decorator: Remove authenticated_json_post_view.
It’s effectively a combination of require_post with
authenticated_json_view and has one use.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg d6ecb67d75 report: Split hostname from port more carefully.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Mohit Gupta bbec1f72a8 refactor: Rename bugdown to markdown in message_edit.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 15:03:20 -07:00
Mateusz Mandera 9392e8bec7 emails: Add a custom header specifying the organization when possible.
Closes #15135.
2020-06-28 13:33:02 -07:00
Steve Howell 69be97e365 pointer: Remove pointer from API and page_params.
There is still some miscellaneous cleanup that
has to happen for things like analytics queries
and dead code in node tests, but this should
remove the main use of pointers in the backend.

(We will also still need to drop the DB field.)
2020-06-27 16:44:38 -07:00
Mohit Gupta 3f5fc13491 refactor: Rename zerver.lib.bugdown to zerver.lib.markdown .
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
2020-06-26 17:08:37 -07:00
Vishnu KS cc0b3a08c9 i18n: Set the correct language for translation in add_subscriptions_backend. 2020-06-26 14:57:18 -07:00
Tim April 8e2a79095d mobile: Add support for alternative mobile URI.
Due to authentication restrictions, a deployment may need to direct
traffic for mobile applications to an alternate uri to take advantage
alternate authentication mechansism. By default the standard realm URI
will be usedm but if overridden in the settings file, an alternate uri
can be substituted.
2020-06-26 12:13:26 -07:00
Tim Abbott def6189d53 docs: Document local echo paramters for sending messages. 2020-06-25 14:44:16 -07:00
Steve Howell 2fb67b3f32 refactor: Extract add/remove_subscriptions_schema.
Now we are consistent about validating color/description.

Ideally we wouldn't need to validate the
`streams_raw` parameters multiple times per
request, but the outer function here changes
the error messages to explicitly reference
the "delete" and "add" request variables.

And for the situation where the user-supplied
parameters are correct, the performance penalty
for checking them twice is extremely negligible.

So it's probably fine for now to just make sure
we use the same validators in all the relevant
places.

There's probably some deeper refactor that we
can do to eliminate the whole `compose_views`
scheme.  And it's also not entirely clear to
me that we really need to support the update
endpoint.  But that's all out of the scope of
this commit.
2020-06-25 13:52:59 -07:00
Steve Howell f960df04e8 narrows: Validate negated field. 2020-06-25 13:52:59 -07:00
Steve Howell 80c057d91d REQ: Use check_dict_only in update_user_backend.
Update the REQ check for profile_data in
update_user_backend by tweaking `check_profile_data`
to use `check_dict_only`.

Here is the relevant URL:

    path('users/<int:user_id>', rest_dispatch,
         {'GET': 'zerver.views.users.get_members_backend',

It would be nice to unify the validator
for these two views, but they are different:

    update_user_backend
    update_user_custom_profile_data

It's not completely clear to me why update_user_backend
seems to support a superset of the functionality
of `update_user_custom_profile_data`, but it has
this code to allow you to remove custom profile fields:

    clean_profile_data = []
    for entry in profile_data:
        assert isinstance(entry["id"], int)
        if entry["value"] is None or not entry["value"]:
            field_id = entry["id"]
            check_remove_custom_profile_field_value(target, field_id)
        else:
            clean_profile_data.append({
                "id": entry["id"],
                "value": entry["value"],
            })

Whereas the other view is much simpler:

def update_user_custom_profile_data(
    <snip>
) -> HttpResponse:

    validate_user_custom_profile_data(user_profile.realm.id, data)
    do_update_user_custom_profile_data_if_changed(user_profile, data)
    # We need to call this explicitly otherwise constraints are not check
    return json_success()
2020-06-25 10:54:15 -07:00
Steve Howell 92a552d2ad REQ: Check value in update_user_custom_profile_data.
This tightens our checking of user-supplied data
for this endpoint:

    path('users/me/profile_data', rest_dispatch,
         {'PATCH': 'zerver.views.custom_profile_fields.update_user_custom_profile_data',
         ...

We now explicitly require the `value` field
to be present in the dicts being passed in
here, as part of `REQ`.  There is no reason
that our current clients would be sending
extra fields here, and we would just ignore
them anyway, so we also move to using
check_dict_only.

Here is some relevant webapp code (see settings_account.js):

    fields.push({id: field.id, value: user_ids});
    update_user_custom_profile_fields(fields, channel.patch);

    settings_ui.do_settings_change(method, "/json/users/me/profile_data",
                                   {data: JSON.stringify([field])}, spinner_element);

The webapp code sends fields one at a time
as one-element arrays, which is strange, but
that is out of the scope of this change.
2020-06-25 10:54:15 -07:00
orientor 8ab6182683 fetch_api_key: Return `email` in json_response.
`/api/v1/fetch_api_key`'s response had a key `email` with the user's
delivery email. But its JSON counterpart `/json/fetch_api_key`, which
has a completely different implementation, did not return `email` in
its success response.

So to avoid confusion, the non-API endpoint, `/json/fetch_api_key`
response has been made identical with it's `/api` counterpart by
adding the `email` key. Also it is safe to send as the calling user
will only see their own email.
2020-06-24 15:13:31 -07:00
Tim Abbott b7b173d2ae realm: Fix type-checking for message_retention_days.
The best way to handle this is to have the potentially string-typed
value be a separate variable.
2020-06-24 11:01:34 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 7b3f4755ff streams: Fix bug that allows 0 value for message_retention_days. 2020-06-24 10:38:58 -07:00
Hashir Sarwar 9b8521faee people: Enable `user_avatar_url_field_optional` in webapp.
In 5200598a31, we introduced a new
client capability that can be used to avoid unreasonable network
bandwidth consumed sending avatar URLs of long term idle users in
organizations with 10,000s members.

This commit enables this feature and adds support for it to the web
client.
2020-06-24 10:06:35 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 48719cb8c4 message_send: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg f6c73ed45a auth: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg feeb39a55f timezone: Strengthen get_timezone type.
pytz.BaseTzInfo is a subclass of datetime.tzinfo that adds the
localize and normalize methods.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 1d545efac2 custom_profile_fields: Replace a type: ignore with an assert.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg a3a39494e3 video_calls: Annotate a type: ignore with a specific bug URL.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 30ab0524b5 video_calls: Don’t request a scope for the Zoom OAuth session.
As requested by a Zoom reviewer for unclear reasons.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 18:48:35 -07:00
Felix a389c7390d calls: Add Big Blue Button as a Video Call Provider.
Big Blue Button needs an API secret so communication to creating a
room has to be done server side.

Fixes #14763.
2020-06-22 16:19:07 -07:00
Tim Abbott 866965f162 views: Rename messages.py to message_fetch.py.
This completes to effort to split zerver.views.messages into smaller
modules focused on the different core type of work.
2020-06-22 15:08:34 -07:00
Tim Abbott 3657717ade views: Extract message_send.py for sending views. 2020-06-22 15:08:34 -07:00
Tim Abbott 55448ecf24 views: Extract message_flags.py for flags views. 2020-06-22 15:08:34 -07:00
Tim Abbott 4d7550d705 views: Extract message_edit.py for message editing views.
This is a pretty clean extraction of files that lets us shrink one of
our largest files.
2020-06-22 15:08:34 -07:00
MariaGkoulta b10f156250 settings: Add automatic theme detection feature.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.

This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.

Fixes: #14451.

Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
2020-06-21 01:09:01 -07:00
sahil839 63389b3bd3 invite: Add option to invite user as an organization owner.
We can now invite new users as realm owners. We restrict only
owners to invite new users as owners both for single invite
and multiuse invite link. Also, only owners can revoke or resend
owner invitations.
2020-06-20 23:04:59 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c0fa42dd8c complete_zoom_user: Validate state type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 64038163e7 update_user_backend: Validate profile_data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c584462b32 update_storage: Validate storage type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 717e349834 events_register_backend: Validate client_capabilities type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 475fc5910f update_user_custom_profile_data: Validate data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Steve Howell fff2a81536 mypy: Fix payload_url. 2020-06-20 17:09:15 -07:00
Steve Howell 82036741b9 mypy: Fix service_interface 2020-06-20 17:09:15 -07:00
Steve Howell 704c84502b mypy: Fix authentication_methods.
We know the key of the dictionary is a string
here, since we get it via json.
2020-06-20 17:09:15 -07:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell 2e9f689add mypy: Fix annotation for principals. 2020-06-19 21:23:23 -07:00
Tim Abbott 14c0a9a228 messages: Deduplicate non-search code path in messages_in_narrow.
The non-search code path here was simulating the response and escaping
logic from get_search_fields by duplicating what it would do with an
empty set of highlight locations.

We can produce much more readable code by just passing an empty list
of locations in this case.
2020-06-19 15:09:57 -07:00
Steve Howell 7c6ddf90ae mypy: Fix default_language annotation. 2020-06-19 12:35:47 -04:00
Hashir Sarwar 5200598a31 events: Don't send avatar URLs of long term idle users.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.

Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.

This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.

Fixes #15287.
2020-06-18 21:35:16 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Tim Abbott f8ea5f3769 pointer: Remove pointer from GET /users/me.
This cleans up a bit of mess in the Zulip API.
2020-06-18 12:55:59 -07:00
Mateusz Mandera 8d2d64c100 CVE-2020-14215: Fix validation in PreregistrationUser queries.
The most import change here is the one in maybe_send_to_registration
codepath, as the insufficient validation there could lead to fetching
an expired PreregistrationUser that was invited as an administrator
admin even years ago, leading to this registration ending up in the
new user being a realm administrator.

Combined with the buggy migration in
0198_preregistrationuser_invited_as.py, this led to users incorrectly
joining as organizations administrators by accident.  But even without
that bug, this issue could have allowed a user who was invited as an
administrator but then had that invitation expire and then joined via
social authentication incorrectly join as an organization administrator.

The second change is in ConfirmationEmailWorker, where this wasn't a
security problem, but if the server was stopped for long enough, with
some invites to send out email for in the queue, then after starting it
up again, the queue worker would send out emails for invites that
had already expired.
2020-06-16 23:35:39 -07:00
Clara Dantas ddbde66af5 realm: Remove Google Hangouts integration.
Google  has removed the Google Hangouts brand, thus we are removing
them as video chat provider option.
This commit removes Google Hangouts integration and make a migration
that sets all realms that are using Hangouts as their video chat
provider to the default, jitsi.

With changes by tabbott to improve the overall video call documentation.

Fixes: #15298.
2020-06-16 17:02:27 -07:00
Mateusz Mandera 2ac6a8f829 auth: Change the "continue in browser" link in desktop flow end page.
Fixes #14828.
Giving the /subdomain/<token>/ url there could feel buggy if the user
ended up using the token in the desktop app, and then tried clicking the
"continue in browser" link - which had the same token that would now be
expired. It's sufficient to simply link to /login/ instead.
2020-06-16 16:27:53 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Mateusz Mandera f467ab046e messages: Remove redundant "assert is not None" for mypy.
Apparently new mypy is able to infer correctly and doesn't need this
assert anymore.
2020-06-15 16:20:43 -07:00
Aman Agrawal ccc0c8706c delete_message: Support `bulk_message_deletion`.
Fixes #15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
2020-06-15 10:41:12 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Aman Agrawal cda7b2f539 deletion: Add support for bulk message deletion events.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.

Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).

Backend implementation of #15285.
2020-06-14 22:34:00 -07:00
Aman Agrawal 21ac1fb32c register: Refactor to pass client_capabilities directly.
We extract values from client_capabilities directly in
do_events_register where we decide how to process the
extracted variables.
2020-06-14 21:42:50 -07:00
Tim Abbott d97c891afe realm owners: Remove unnecessary duplicate strings. 2020-06-14 21:32:10 -07:00
sahil839 18429cfd29 realm: Allow only owners to set message retention policy for a realm.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.

Dropdown for changing retention policy is disabled in UI for admins also.
2020-06-14 21:23:53 -07:00
sahil839 87e72ac8e2 realm: Allow only owners to configure auth methods for a realm.
This commit adds the restriction on configuring auth methods for
admins. We now allow only owners to configure the auth methods
for realm.
2020-06-14 21:23:51 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Graham Bleaney 461d5b1a3e pysa: Introduce sanitizers, models, and inline marking safe.
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
2020-06-11 12:57:49 -07:00
Graham Bleaney 56345f98cb registration: Incorporate redirect URL building helper function.
This commit adds 'add_query_to_redirect_url' to one additional
function which had not yet been written when
'add_query_to_redirect_url' was introduced. This helper centralizes
URL manipulation for redirects, making it easier to add Pysa
sanitization in subsequent commits.
2020-06-11 12:57:49 -07:00
Kartik Srivastava 8c39ddfd28 api: Add GET /users/{user_id}/subscription/{stream_id} endpoint.
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.

Fixes #14966.
2020-06-10 17:59:14 -07:00
Steve Howell d5cc29755e typing: Accept only user_ids in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.

There may be a very tiny number of mobile
clients that still use the old emails API.

This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.

Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.

The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:

    validator=check_list(check_int)

This also allows us to remove a test hack
related to the API documentation.  (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
2020-06-10 17:50:32 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
sahil839 86b52ef7bf users: Owners can only be deactivated by other organization owners.
This commit adds restriction on deactivation of organization owners.
Only owners can deactivate other organization owners.
2020-06-10 17:33:02 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
sahil839 6d667dbe53 realm owner: Add ability to change realm owner status of user.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.

This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.

We need to apply that restriction both in the role change code path
and the deactivate code path.
2020-06-10 12:49:37 -07:00
Anders Kaseorg 5c9d56d2f7 get_safe_redirect_to: Remove incorrect allowed_hosts setting.
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:51:52 -07:00
Dinesh dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
Tim Abbott 2beaf2cab2 pointer: Remove have_initial_messages code.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history.  However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
2020-06-08 22:36:18 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Anders Kaseorg b1b34b6f6a dev_login: Move owners to the top.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 17:26:34 -07:00
sahil839 7de23b8b5c users: Remove short_name and client_id from get_profile_backend.
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.

* short_name hasn't been present anywhere else in the Zulip API for
  several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
  returned by the register_event_queue process.

The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
2020-06-08 17:01:08 -07:00
sahil839 1f5778bad7 users: Refactor get_profile_backend to be based on format_user_row.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.

To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.

This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
2020-06-08 16:57:44 -07:00
Anders Kaseorg 8e4f22c184 auth: Require algorithms setting for JWT auth.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning.  This is for protecting against
symmetric/asymmetric key confusion attacks.

This is a backwards-incompatible configuration change.

Fixes #15207.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 16:22:25 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 139cb8026f auth: Accept next as POST parameter in POST requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 11:07:32 -07:00
Anders Kaseorg 22178c169e request: Replace type argument hack with a different workaround.
This works in mypy 0.770 and is needed for mypy 0.780.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
sahil839 43bff1f67a streams: Extract removing_someone_else logic to a function.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.

This is a follow up to 2187c84.
2020-06-05 16:04:28 -07:00
Aman Agrawal a096f34cab move_topic_to_stream: Add option to disable breadcrumb messages.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.

User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
2020-06-05 12:28:51 -07:00
Anders Kaseorg 4d04fa3118 compose: Rewrite Zoom video call integration to use OAuth.
This reimplements our Zoom video call integration to use an OAuth
application.  In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.

Fixes #11672.

Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-06-03 16:39:12 -07:00
Anders Kaseorg 7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
sahil839 0b432e26b6 test_subs: Add tests for unsubscribing multiple users from stream.
This commit adds tests for unsusbcribing multiple users from a
stream and fixes the missing coverage issue introduced in 2187c84.
2020-06-02 16:27:22 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Mateusz Mandera 3e7fc17788 auth: Delegate RemoteUser SSO to browser when using the desktop app. 2020-06-02 13:00:17 -07:00
Tim Abbott fbcf6a7d72 streams: Add nocoverage to fix CI. 2020-06-01 16:12:13 -07:00
sahil839 2187c84ed9 stream: Add support for sending user_id to endpoint when subscribing users.
This commit modifies the backend to accept user ids when subscribing
users to streams.

It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
2020-06-01 15:59:04 -07:00
sahil839 9ef1c5b1a6 users: Add is_owner field to user objects returned by get endpoints.
This commit adds 'is_owner' field to the user object returned by
'/users', 'users/{user_id}', and '/users/me' endpoints.
2020-06-01 15:33:51 -07:00
sahil839 bd9e891e1c dev_login: Change dev env login page to show Desdemona as owner.
This commits changes the dev login page to show Desdemona as
owner.
2020-06-01 15:22:44 -07:00
Mateusz Mandera 4b10a07c3e auth: Fix social signup with the desktop app.
Only start_social_login had the code for handling the desktop
application, it was missing from start_social_signup.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 676305f6ab auth: Extract handle_desktop_flow wrapper. 2020-06-01 14:14:58 -07:00
sahil839 9fa60672e6 users: Modify update user API endpoint to accept role as parameter.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.

User role dropdown in user info modal is also modified to use
"dropdown_options_widget".

Modified by tabbott to document the API change.
2020-05-29 14:29:17 -07:00
clarammdantas f0c85c5796 registration: Show error if invalid key is posted on /accounts/register.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
2020-05-27 15:37:16 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
arunikaydav42 4680d504de invite: Fix invitations removed from list too soon.
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.

Fixes: #12281
2020-05-26 21:45:44 -07:00
Vishnu KS 8784539d53 free trial: Send users to /upgrade after realm creation. 2020-05-26 17:01:32 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
Vishnu KS 4e0d8953b4 models: Create ensure_not_on_limited_plan instance method. 2020-05-19 14:48:59 -07:00
Vishnu Ks 66b1ad7002 billing: User FREE_TRIAL_DAYS instead of FREE_TRIAL_MONTHS. 2020-05-16 14:52:01 -07:00
Puneeth Chaganti 734d651b45 home: Fix furthest read time when user has multiple clients.
The query to fetch the latest user activity was missing an
`.order_by('last_visit')`. This meant that the results were being
ordered by the `id`, which resulted in us getting `update_message_flags`
action performed on the client that the user installed last, instead of
being client agnostic and fetching the "global" last
`update_message_flags` action performed by the user.
2020-05-14 13:38:09 -07:00
Anders Kaseorg 4362cceffb portico: Add setting to put Google Analytics on selected portico pages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-11 23:22:50 -07:00
Vishnu KS f1b1bf5a0d billing: Add support for Zulip Standard free trial. 2020-05-11 17:20:54 -07:00
Pragati Agrawal 60a762704a org settings: Fix limited plan realm can change message_retention_days.
These changes should be included in bd9b74436c,
as it makes sure that Zulip limited plan realm won't be able to change the
`message_retention_days` setting.
2020-05-09 16:40:50 -07:00
Puneeth Chaganti ded3b00760 home: Use last update_message_flags activity instead of pointer.
The pointer doesn't get updated when a user is only reading messages in
narrowed views. But, we use the pointer position to determine the
furthest read time, which causes the bankruptcy banner to show up even
for users who have been actively reading and sending messages.

This commit switches to using the time of the last update_message_flags
activity by a user to determine the time of last activity.
2020-05-08 15:06:06 -07:00
Puneeth Chaganti 1adb88cf62 home: Extract code to get the furthest read time. 2020-05-08 15:01:08 -07:00
Aman Agrawal 898e717b9d register:api: Provide additional fields that can be useful to clients.
Fixes #14166

* Bumped the api version to feature level 4.
See https://github.com/zulip/zulip/issues/14166#issuecomment-598961462
for details about implementation and future steps.
2020-05-08 14:54:54 -07:00
Aman Agrawal 0c2b25cab3 home/page_params: Add suffix `_mib` to clarify the size units.
The docs say "megabytes" or "MB", but client software needs to
know the actual meaning, which is in mebibytes.
2020-05-08 14:54:54 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Tim Abbott a702894e0e middleware: Stop using X_REAL_IP.
The comment was wrong, in that REMOTE_ADDR is where the real external
IP was; X_REAL_IP was the loadbalancer's IP.
2020-05-08 11:40:54 -07:00