Commit Graph

3488 Commits

Author SHA1 Message Date
Tim Abbott 85222b790d edit_history: Refactor to use FormattedEditHistoryEvent type.
We fix the mutation of caller and other bad patterns, as well as
adding explicit typing to make the code readable.

We also update the OpenAPI documentation for previously
undocumented `prev_strem` field in the `/get-message-history`
endpoint for API validation testing.

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Mateusz Mandera 3c5ea830a9 create_preregistration_user: Add additional hardening assertion.
TestMaybeSendToRegistration needs tweaking here, because it wasn't
setting the subdomain for the dummy request, so
maybe_send_to_registration was actually running with realm=None, which
is not right for these tests.

Also, test_sso_only_when_preregistration_user_exists was creating
PreregistrationUser without setting the realm, which was also incorrect.
2022-02-25 14:02:24 -08:00
Mateusz Mandera 7b795b6338 registration: Change create_preregistration_user to take realm as arg.
create_preregistration_user is a footgun, because it takes the realm
from the request. The calling code is supposed to validate that
registration for the realm is allowed
first, but can sometimes do that on "realm" taken from something else
than the request - and later on calls create_preregistration_user, thus
leading to prereg user creation on unvalidated request.realm.

It's safer, and makes more sense, for this function to take the intended
realm as argument, instead of taking the entire request. It follows that
the same should be done for prepare_activation_url.
2022-02-25 14:02:24 -08:00
Mateusz Mandera 88917019f0 CVE-2022-21706: Prevent use of multiuse invites to join other orgs.
The codepaths for joining an organization via a multi-use invitation
(accounts_home_from_multiuse_invite and maybe_send_to_registration)
weren't validating whether
the organization the invite was generated for matches the organization
the user attempts to join - potentially allowing an attacker with access
to organization A to generate a multi-use invite and use it to join
organization B within the same deployment, that they shouldn't have
access to.
2022-02-25 14:02:24 -08:00
Sahil Batra 392b17da5f invite: Add backend support for "Never expires" option.
The database value for expiry_date is None for the invite
that will never expire and the clients send -1 as value
in the API similar to the message retention setting.

Also, when passing invite_expire_in_days as an argument
in various functions, invite_expire_in_days is passed as
-1 for "Never expires" option since invite_expire_in_days
is an optional argument in some functions and thus we cannot
pass "None" value.
2022-02-24 16:32:19 -08:00
Lauryn Menard 8e1f62af89 api: Include realm_web_public_access_enabled in get-server-settings.
Adds `realm_web_public_access_enabled` as a realm-specific server
setting potentially returned by the `/get-server-settings` endpoint
so that clients that support browsing of web-public stream content
without an account can generate a login page that supports that
type of access.
2022-02-22 10:11:11 -08:00
Sahil Batra 630b37f2d8 users: Refactor get_user_by_email.
This commit refactors get_user_by_email function
to use access_user_by_email which is similar to
already existing access_user_by_id and thus using
get_user_data function added recently.

We also remove the unnecessary check for email as
email will always be passed to this endpoint.

Preparatory commit for #10970.
2022-02-21 06:47:54 -08:00
Sahil Batra 5471584fa7 users: Add get_user_data function.
This commit adds get_user_data which is called by
get_members_backend to compute the client_gravatar
value and then return the data of a single user or
all accessible users.

This function will also be used by get_user_by_email
in further commtis.
2022-02-21 06:47:54 -08:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Alya Abbott e93320c40d portico: Add a self-hosting page.
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
2022-02-17 12:43:13 -08:00
Alex Vandiver a40b3e1118 realm_emoji: Stop swallowing all exceptions from upload_emoji_image.
Putting all of the logic in a `finally` block is equivalent to a bare
`except` block, which silently consumes all exceptions.

Move only the most-necessary parts into the except; this lets
`BadImageError` exceptions from `zerver/lib/upload.py` to escape,
allowing better the generic "Image file upload failed" to be replaced
with a more specific message.

It also allows unexpected exceptions, as the previous commit resolved,
to escape and 500.  This lets them be detected and resolved, rather
than give users a silently bad experience.
2022-02-17 12:19:47 -08:00
Anders Kaseorg 2a5adf1453 auth: Remove unused kwargs for saml_sp_metadata.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-16 13:01:34 -08:00
Anders Kaseorg ce4261cbfb auth: Fix incorrect kwargs forwarding for logout_then_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-16 13:01:34 -08:00
Anders Kaseorg 12da3ac0ad pgroonga: Re-enable PGroonga in development.
This partially reverts commit fdabf0b357
(#21104).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-13 19:11:49 -08:00
Tim Abbott fdabf0b357 pgroonga: Disable PGroonga in development to fix CI.
A recent Postgres upstream release appears to have broken PGroonga.
While we wait for https://github.com/pgroonga/pgroonga/issues/203 to
be resolved, disable PGroonga in our automated tests so that Zulip
CI passes.
2022-02-11 11:35:43 -08:00
Anders Kaseorg 29330c180a sqlalchemy_utils: Make get_sqlalchemy_connection a context manager.
Although our NonClosingPool prevents the SQLAlchemy connection from
closing the underlying Django connection, we still want to properly
dispose of the associated SQLAlchemy structures.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg 8e5ae4e829 message_fetch: Use Row._mapping.
Fixes these warnings with SQLALCHEMY_WARN_20=1:

RemovedIn20Warning: Using non-integer/slice indices on Row is
deprecated and will be removed in version 2.0; please use
row._mapping[<key>], or the mappings() accessor on the Result
object. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

RemovedIn20Warning: Using the 'in' operator to test for string or
column keys, or integer indexes, in a :class:`.Row` object is
deprecated and will be removed in a future release. Use the
`Row._fields` or `Row._mapping` attribute, i.e. 'key in
row._fields' (Background on SQLAlchemy 2.0 at:
https://sqlalche.me/e/b8d9)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg fbb8f54d80 message_fetch: Use modern select() calling style.
Fixes this warning with SQLALCHEMY_WARN_20=1:

RemovedIn20Warning: The legacy calling style of select() is deprecated
and will be removed in SQLAlchemy 2.0.  Please use the new calling
style described at select(). (Background on SQLAlchemy 2.0 at:
https://sqlalche.me/e/b8d9)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg 8769c0c485 message_fetch: Replace Select.column() with Select.add_columns().
Fixes “SADeprecationWarning: The Select.column() method is deprecated
and will be removed in a future release.  Please use
Select.add_columns() (deprecated since: 1.4)”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg d5a5b0b5ac message_fetch: Replace SELECT coercion with .subquery().
Fixes “SADeprecationWarning: Implicit coercion of SELECT and textual
SELECT constructs into FROM clauses is deprecated; please call
.subquery() on any Core select or ORM Query object in order to produce
a subquery object.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
madrix01 a1e71e8639 topic: Return JsonableError for race condition in topic mute.
To avoid an uncaught IntegrityError causing a 500 HTTP response in a
race between two processes trying to mute a topic, we catch the
integrity error and raise the error exception with status 400 we'd
have gotten if the second request had been a bit later.

Fixes #21011.
2022-02-09 14:35:12 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Tim Abbott de411be576 home: Simplify and comment terms of service notice implementation. 2022-02-04 15:48:38 -08:00
Aman Agrawal 37d7d6ccd4 tos: Adjust migration tos to look same as our new standard tos. 2022-02-04 15:48:38 -08:00
Lauryn Menard 3be622ffa7 backend: Add request as parameter to json_success.
Adds request as a parameter to json_success as a refactor towards
making `ignored_parameters_unsupported` functionality available
for all API endpoints.

Also, removes any data parameters that are an empty dict or
a dict with the generic success response values.
2022-02-04 15:16:56 -08:00
Lauryn Menard dbddbee5a1 backend: Change return value of `compose_views`.
As a preparatory step to refactoring json_success to accept
request as a parameter, update helper function `compose_views`
in `views.streams.py` to return the response data and call
json_success from view functions that utilize `compose_views`.

Also, updates related test in `zerver.tests.test_subs.py`.
2022-02-04 15:16:55 -08:00
Lauryn Menard a4b347bb5f backend: Move json_success to callers vs helper functions.
As a preparatory step to refactoring json_success to accept
request as a parameter, change interface of helper functions:
`handle_deferred_message` in `views.message_send.py` and
`mute_topic` and `unmute_topic` in `views.muting.py`, so
that they return None or data for json_success.

Instead call json_sucess in the caller function, which already
has the HttpRequest as a parameter.
2022-02-04 15:12:22 -08:00
Anders Kaseorg 90e202cd38 docs: Consistently hyphenate “web-public”.
In English, compound adjectives should essentially always be
hyphenated.  This makes them easier to parse, especially for users who
might not recognize that the words “web public” go together as a
phrase.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-28 17:45:45 -08:00
jai2201 c14b3a8844 settings: Don't mention username in fetch API key error message.
There isn't a username prompt in the form for it, so the only
possibility for triggering this error is a wrong password.

Fixes #20924.
2022-01-28 14:10:15 -08:00
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Eeshan Garg 0d99809fd3 streams: Add notifications for permission policy changes.
The change to curl_param_value_generators.py warrants a brief
explanation. Stream permission changes now generate a notification
message. Our curl example test for removing a reaction comes after
the two tests for updating the stream permission changes, thus the
hardcoded message ID in that test needs to be incremented by 2 to
account for the two notification messages that now come before it.

This is a part of #20289.
2022-01-21 13:59:34 -08:00
Eeshan Garg fab1b7f5d5 actions: Refactor functions for stream permission changes.
do_make_stream_web_public and do_change_stream_invite_only seem
to contain very similar logic that could just live inside the
do_change_stream_permission function that handles all permission
changes in one place.
2022-01-21 13:59:34 -08:00
Alex Vandiver 915c2b2fd9 muting: Fix a race in topic unmuting.
Rather than check if the topic exists and then try to delete it, just
try to delete it, and catch the lack of matching rows.
2022-01-18 14:15:06 -08:00
Anders Kaseorg 87b4e9259f documentation: Replace deprecated request.is_ajax() method.
This was deprecated in Django 3.1 for being jQuery-specific, and
removed in Django 4.0.  Replicate the jQuery-specific check.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 15:40:22 -08:00
Anders Kaseorg 78e54a0d7a python: Replace deprecated jinja2.utils.Markup with markupsafe.Markup.
Fixes “DeprecationWarning: 'jinja2.Markup' is deprecated and will be
removed in Jinja 3.1. Import 'markupsafe.Markup' instead.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:22:48 -08:00
Mateusz Mandera ff688c3a8d actions: Give do_get_user_invites a more specific name.
The added docstrings elaborates on why the new name is more appropriate.
2022-01-12 13:53:34 -08:00
Alex Vandiver 3574637fbf string_validation: Factor out stream name validation.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-01-11 15:17:53 -08:00
Eeshan Garg c30458e174 streams: Add notifications for posting policy changes.
An explanatory note on the changes in zulip.yaml and
curl_param_value_generators is warranted here. In our automated
tests for our curl examples, the test for the API endpoint that
changes the posting permissions of a stream comes before our
existing curl test for adding message reactions.

Since there is an extra notification message due to the change in
posting permissions, the message IDs used in tests that come after
need to be incremented by 1.

This is a part of #20289.
2022-01-10 18:29:04 -08:00
Eeshan Garg 80f30f187e streams: Add notifications for description changes.
This is a part of #20289.
2022-01-07 16:13:11 -08:00
Eeshan Garg 1b303e7b2f signup: Stop prompting self-hosted customers for marketing emails.
Fixes #20595.
2022-01-06 13:22:50 -08:00
Steve Howell c4bd4496dd peformance: Cache user mentions for multiple PMs.
It's slightly annoying to plumb Optional[MentionBackend]
down the stack, but it's a one-time change.

I tried to make the cache code relatively unobtrusive
for the single-message use case.

We should be able to eliminate redundant stream queries
using similar techniques.

I considered caching at the level of rendering the message
itself, but this involves nearly as much plumbing, and
you have to account for the fact that several users on
your realm may have distinct default languages (French,
Spanish, Russian, etc.), so you would not eliminate as
many query hops. Also, if multiple streams were involved,
users would get slightly different messages based on
their prior subscriptions.
2021-12-30 11:28:15 -08:00
Steve Howell fd925e6045 streams: Add id to user mentions for stream notifications. 2021-12-30 11:23:25 -08:00
Steve Howell 01ebb2c85f refactor: Pass realm to bulk_remove_subscriptions.
We made a very similar change to bulk_add_subscriptions
earlier in the year.
2021-12-28 12:15:02 -08:00
Alex Vandiver e43373cc1f video_calls: Drop VIDEO_ZOOM_TESTING_ configurations.
These are no longer needed.
2021-12-13 15:17:34 -08:00
Tim Abbott ee77c6365a portico: Use /help/ style pages for displaying policies.
This replaces the TERMS_OF_SERVICE and PRIVACY_POLICY settings with
just a POLICIES_DIRECTORY setting, in order to support settings (like
Zulip Cloud) where there's more policies than just those two.

With minor changes by Eeshan Garg.
2021-12-10 17:56:12 -08:00
Tim Abbott 95854d9d94 terms: Rename and tweak FIRST_TIME_TERMS_OF_SERVICE_TEMPLATE.
We do s/TOS/TERMS_OF_SERVICE/ on the name, and while we're at it,
remove the assumed zerver/ namespace for the template, which isn't
correct -- Zulip Cloud related content should be in the corporate/
directory.
2021-12-10 17:56:12 -08:00
Tim Abbott 1c180a9f57 documentation: Avoid potential unused variable code path.
These variables can be unset if the `os.path.exists` check fails.

That should be rare, since we've previously checked the files do
exist before getting here.
2021-12-09 17:51:52 -08:00
Tim Abbott 4cb189fc63 settings: Rename TOS_VERSION to TERMS_OF_SERVICE_VERSION.
The previous version was appropriate in a setting where it was only
used for Zulip Cloud, but it's definitely clearer to spell it out.
2021-12-09 17:51:16 -08:00
Eeshan Garg 3714a30e63 stream notifications: Add helper for silent user mention syntax.
In many of our stream notification messages, we make use of the
same silent user mention syntax, the template for which was always
hardcoded. This commit adds a helper function that all relevant
callers can call to get the right syntax when mentioning users.

Thanks to Tim Abbott for this suggestion!
2021-12-07 14:53:50 -08:00
Eeshan Garg d2901892e2 streams: Add notifications for message retention policy updates.
This is a part of #20289.
2021-12-07 14:53:50 -08:00
Tim Abbott 92d717b8a5 accounts: Improve code documentation for confirmation endpoint. 2021-12-02 09:50:41 -08:00
Mateusz Mandera 53a44a17e0 check_prereg_key: Raise exception in case of validation failure.
The code is cleaner when this raises an exception that can be translated
into an error response by the caller.
2021-12-02 09:50:41 -08:00
Mateusz Mandera 39ce424fd5 confirmation: Delete rendundant confirmation_link_expired_error page.
This template provides nothing that link_expired.html does not and is
redundant.
2021-12-02 09:50:41 -08:00
Mateusz Mandera 4a030f616f auth: Improve Confirmation link error handling in social auth codepath.
render_confirmation_key_error provides better error responses than
simply rendering the "link expired" template.
2021-12-02 09:50:41 -08:00
Mateusz Mandera 7c5daac8b6 get_object_from_key: Accept multiple allowed Confirmation types.
This allows making check_prereg_key significantly cleaner.
2021-12-02 09:50:41 -08:00
Mateusz Mandera f40b39ba0e registration: Make check_prereg_key return PreregistrationUser. 2021-12-02 09:50:41 -08:00
Mateusz Mandera a014ef75a3 CVE-2021-43791: Validate confirmation keys in /accounts/register/ codepath.
A confirmation link takes a user to the check_prereg_key_and_redirect
endpoint, before getting redirected to POST to /accounts/register/. The
problem was that validation was happening in the check_prereg_key_and_redirect
part and not in /accounts/register/ - meaning that one could submit an
expired confirmation key and be able to register.

We fix this by moving validation into /accouts/register/.
2021-12-01 23:14:04 +00:00
Mateusz Mandera a1cd660147 confirmation: Use error status codes for confirmation link error pages. 2021-12-01 23:14:04 +00:00
Lauryn Menard 7713b371a5 api: Migrate `/update-subscription-settings` response value.
Migrates the `/update-subscription-settings` api endpoint to the
`ignored_parameters_unsupported` model, which is also currently used
by `/update-settings` and `update-realm-user-settings-defaults`.

This change is a step towards preparing for an eventual migration to
have all endpoints return an `ignored_parameters_unsupported` block.

Previously the `/update-subscription-settings` endpoint returned a
copy of the data object sent in the request.

Fixes #15307.
2021-11-26 22:25:53 -08:00
strifel 11211bd9c4 integrations: Switch BigBlueButton over to SHA256.
This commit switches the BigBlueButton integration
to use SHA256 instead of SHA1 as BigBlueButton supports
it and scalelite does now, too.

Fixes #19966.
2021-11-25 07:58:27 -08:00
Sahil Batra ad99b4fac9 streams: Allow changing stream to be web-public based on creation setting.
We allow a user to make an existing stream web-public only if user is
allowed to create web-public streams.
2021-11-23 10:48:20 -08:00
Aman Agrawal 2a922409aa realm: Add option to enable/disable spectator login.
We restrict access of messages from web public streams if
anonymous login is disabled via `enable_spectator_access`.

Display of `Anonymous login` button is now controlled by
the value of `enable_spectator_access`.

Admins can toggle `enable_spectator_access` via org settings in UI.
2021-11-18 12:55:12 -08:00
Mateusz Mandera 0abbb87155 auth: Include user_id in the params returned at the end of mobile flow.
The user id is a very useful piece of information that the mobile
client should have access to - instead of only getting the email. This
makes it much simpler to impleent clients that might be robust to
changes in email address.
2021-11-16 12:12:59 -08:00
Alex Vandiver a2f89f21e3 email_change: Show deactivated page if realm is now deactivated.
We also catch if the user is now deactivated.
2021-11-14 18:55:40 -08:00
YashRE42 778b7c0bdb links: Replace all references to Django docs to link to /3.2/ version.
Previously, our codebase contained links to various versions of the
Django docs, eg https://docs.djangoproject.com/en/1.8/ref/
request-response/#django.http.HttpRequest and https://
docs.djangoproject.com/en/2.2/ref/settings/#std:setting-SERVER_EMAIL
opening a link to a doc with an outdated Django version would show a
warning "This document is for an insecure version of Django that is no
longer supported. Please upgrade to a newer release!".

Most of these links are inside comments.

Following the replacement of these links in our docs, this commit uses
a search with the regex "docs.djangoproject.com/en/([0-9].[0-9]*)/"
and replaces all matches with "docs.djangoproject.com/en/3.2/".

All the new links in this commit have been generated by the above
replace and each link has then been manually checked to ensure that
(1) the page still exists and has not been moved to a new location
(and it has been found that no page has been moved like this), (2)
that the anchor that we're linking to has not been changed (and it has
been found that no anchor has been changed like this).

One comment where we mentioned a Django version in text before linking
to a page for that version has also been changed, the comment
mentioned the specific version when a change happened, and the history
is no longer relevant to us.
2021-11-08 18:06:16 -08:00
Alex Vandiver c8badbd858 reset_password: Show user-facing page on rate-limit. 2021-11-04 20:34:39 -07:00
Tim Abbott 1cad29fc3a settings: Add rate limiting for email address changes.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-11-04 20:34:39 -07:00
Tim Abbott 01e2a495fc rate_limit: Fix missing IP rate limiting on confirmation.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-11-04 20:34:39 -07:00
Alex Vandiver 0cfb156545 rate_limit: Merge two IP rate limits domains that send emails.
Both `create_realm_by_ip` and `find_account_by_ip` send emails to
arbitrary email addresses, and as such can be used to spam users.
Lump their IP rate limits into the same bucket; most legitimate users
will likely not be using both of these endpoints at similar times.

The rate is set at 5 in 30 minutes, the more quickly-restrictive of
the two previous rates.
2021-11-04 20:34:39 -07:00
Aman Agrawal 0879e5e0e2 home: Delete prefers_web_public_view key after user is logged in.
Since `prefers_web_public_view` key in session is only
relevant to users without an account, this key should no longer
be present in the user's session object.

Fixes #19907
2021-11-03 16:52:51 -07:00
Aman Agrawal 3e689ebae9 users: Allow spectators to view user avatars.
If realm is web_public, spectators can now view avatar of other
users.

There is a special exception we had to introduce in rest model to
allow `/avatar` type of urls for `anonymous` access, because they
don't have the /api/v1 prefix.

Fixes #19838.
2021-11-02 11:26:19 -07:00
Lauryn Menard 73710e1cf0 user_settings: Add option to disable escape key navigation to default view.
Add `escape_navigates_to_default_view` as a bool setting in
UserBaseSettings model and implement it as a checkbox that toggles
the hotkey implementation of escape to the default view in the
advanced user display settings.

With /help/ documentation edits from Alya Abbott.

Fixes #20043.
2021-10-29 18:15:30 -07:00
Eeshan Garg b325a4f1be realm: Rename plan type constants to be more descriptive.
It is confusing to have the plan type constants not be namespaced
by the thing they represent. We already have a namespacing
convention in place for constants, so we should use it for
Realm.plan_type as well.
2021-10-19 12:20:39 -07:00
Gaurav Pandey 1c1a1e2cad api: Remove encoding of string in update_user endpoint.
* Remove unnecessary json_validator for full_name parameter.
* Update frontend to pass the right parameter.
* Update documentation and note the change.

Fixes #18409.
2021-10-18 19:03:43 -07:00
Sahil Batra 6be20a8c6d users: Allow admins to get subscription status in private stream.
We pass allow_realm_admin as True to access_stream_by_id for
`GET users/{user_id}/subscriptions{stream_id}` endpoint
because we want to allow non-subscribed admins to get
subscription status in private streams.

Fixes #19077.
2021-10-15 10:58:56 -07:00
Anders Kaseorg f42e191776 url_encoding: Use proper parsing for query string appending.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Sahil Batra e47662aa19 user_groups: Rename get_memberships_of_users.
This is a prep commit for new permissions model in
which a user group would be able to have a subgroup.

This commit renames get_memberships_of_users to
get_direct_memberships_of_users to specify that
the function is used only to fetch the direct
memberships and not memberships of subgroups of
the direct group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:47 -07:00
Sahil Batra 82af2d3db5 user_groups: Rename get_user_group_members to get_user_group_direct_members.
This is a prep commit for new permissions model in which a user group would
be able to have a subgroup.

This commit renames get_user_group_members to get_user_group_direct_members
to specify that the function is used only to fetch direct members of group
and excludes the subgroup's members.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:26 -07:00
Dinesh e2df0d171f user_settings: Add send_read_receipts setting.
This will be useful to let users enable/disable
sharing read receipts once we add that feature.

Note: Added "I've" to IGNORED_PHRASES in
tools/lib/capitalization.py to avoid capitalization
errors for the label text of this setting.
2021-10-07 17:46:05 -07:00
Dinesh 9cea88c1a5 typing_notifications: Make privacy settings functional.
Changes the view code to skip sending typing events
when these settings are disabled. Returns a json success
response with a msg.
2021-10-07 17:39:22 -07:00
Dinesh 6d2b8f5ca9 user_settings: Add settings to configure sending typing notifications.
Note: These are not functional in enabling/disabling sending of
typing notifications with this commit.

Refactored the privacy settings update to keep the code less
duplicated along with making the addition of new settings easier.
2021-10-07 17:39:21 -07:00
Eeshan Garg 29b354346b update_realm: Allow demo orgs to be converted to regular orgs.
This commit adds support to the `PATCH /realm` endpoint for
converting a demo organization to a regular organization.

This is a part of #19523.
2021-10-07 14:43:03 -07:00
Aman Agrawal 3b29c61409 auth: Add login buttons for web public access.
For users who are not logged in and for those who don't have
'prefers_web_public_view' set in session, we redirect them
to the default login page where they can choose to login
as spectator or authenticated user.
2021-10-05 16:30:40 -07:00
Sahil Batra 5f950e3efd models: Add create_web_public_stream_policy setting.
This commit adds create_web_public_stream_policy
field to Realm table which controls the roles that
can create web-public streams and by default its
value is set to POLICY_OWNERS_ONLY.
2021-10-05 09:28:28 -07:00
Ganesh Pawar fa928d5cd1 streams: Split setting for stream creation policy.
Users wanted a feature where they could specify
which users can create public streams and which users can
create private streams.

This splits stream creation code into two parts,
public and private stream creation.

Fixes #17009.
2021-10-01 10:26:42 -07:00
sahil839 9dd69c17ee settings: Add moderators and members option in delete_own_message_policy.
This commit adds moderators, full members and members options to
delete_own_message_policy in backend.
2021-09-30 14:59:31 -07:00
sahil839 909a3cde76 realm: Replace allow_message_deleting with delete_own_message_policy.
This commit replaces 'allow_message_deleting' boolean setting
with an integer setting 'delete_own_message_policy'. We have a
separate dropdown now for deciding which user-roles can delete
messages sent by themselves and the time-limit setting droddown
is different.

This new setting has two options - everyone and admins only. Other
options including moderators will be added further.

We also remove the "Never" option from the original time-limit
dropdown, as admins are always allowed to delete message. This
never option resembled the case of only admins being allowed to
delete but this state is now resembled by setting the dropdown
to "admins only" and we also disable the time-limit dropdown in
this case as admins are allowed to delete irrespective of limit.

Note, this setting is only for deleting messages sent by the
deleting user themselves, and only admins are allowed to delete
messages sent by others as before.
2021-09-30 14:59:31 -07:00
sahil839 b13bfa09c5 message: Make zero invalid value for message_content_delete_limit_seconds.
We make zero invalid value for message_content_delete_limit_seconds and
for handling the case of "Allow to delete message any time", the API-level
value of message_content_delete_limit_seconds is "anytime" and "None"
as the DB-level value. We also use these values for message retention
setting, so it helps maintain consistency.
2021-09-30 14:45:39 -07:00
Sahil Batra d1732fb9da settings: Remove realm-level default of enable_login_emails setting.
This commit does not remove the 'enable_login_emails' field from
RealmUserDefault table but it is just not used and cannot be
changed from UI or API similar to 'enable_marketing_emails' setting.
2021-09-28 12:41:09 -07:00
Aman Agrawal ef84224eed message_edit: Allow spectators to access raw message content.
We allow spectators to fetch the raw / original content of a
message which is used by the spectator to "View source" of
the message.
2021-09-28 10:07:36 -07:00
Sahil Batra c233ee9935 settings: Migrate twenty_four_hour_time setting to RealmUserDefault.
This commit removes the existing default_twenty_four_hour_time field in
Realm table which was used to set the twenty_four_hour_time setting of
new user on joining and instead we now use the twenty_four_hour_time
field of RealmUserDefault table for the same.

With some tweaks by tabbott to clarify the documentation.
2021-09-23 10:44:42 -07:00
Mateusz Mandera cacff28578 types: Fix declared type of custom profile field values.
None of the existing custom profile field types have the value as an
integer like declared in many places - nor is it a string like currently
decalred in types.py. The correct type is Union[str, List[int]]. Rather
than tracking this in so many places throughout the codebase, we add a
new ProfileDataElementValue type and insert it where appropriate.
2021-09-22 16:22:46 -07:00
Eeshan Garg 4118c4a56b find_team: Send one email per email address, not per organization.
With changes to the copy for these emails by tabbott.

Fixes #19659.
2021-09-21 15:56:50 -07:00
Aman Agrawal 5138652810 update_stream_backend: Add ability to make streams web public.
We allow clients to make existing streams web public via the API.

This feature is still disabled via settings in production
environments, because we may have additional policy rules or UI
warnings we wish to add to this sort of conversion.
2021-09-21 12:16:09 -07:00
Aman Agrawal 6a78112940 subscribe: Allow web public stream creation via the API.
User can now create web public stream via the /subscribe API.
So, when a web public stream present in the API request does not
exist, it will be created now by specifying the is_web_public
parameter. The parameter would have been ignored without this
commit.
2021-09-21 11:20:36 -07:00
Aman Agrawal 9566d06429 home: Use already imported redirect shortcut for redirects. 2021-09-21 10:05:30 -07:00
Aman Agrawal dff4ab0daf spectators: Add comments and assertions on security model.
These tweaks help make reasoning about the spectators security model
easier to understand.
2021-09-14 10:33:14 -07:00
Mateusz Mandera 3205f680c1 do_invite_users: Turn some args into kwargs-only. 2021-09-10 16:53:03 -07:00
shanukun 8c1ea78d7d invite: Extend invite api for handling expiration duration.
This extends the invite api endpoints to handle an extra
argument, expiration duration, which states the number of
days before the invitation link expires.

For prereg users, expiration info is attached to event
object to pass it to invite queue processor in order to
create and send confirmation link.
In case of multiuse invites, confirmation links are
created directly inside do_create_multiuse_invite_link(),

For filtering valid user invites, expiration info stored in
Confirmation object is used, which is accessed by a prereg
user using reverse generic relations.

Fixes #16359.
2021-09-10 16:53:03 -07:00
Mateusz Mandera c5806d9728 auth: Include user-input email in some error messages in the login form.
Fixes #13126.
2021-09-10 10:47:15 -07:00
Mateusz Mandera fb3864ea3c auth: Change the look of SOCIAL_AUTH_SUBDOMAIN when directly opened.
SOCIAL_AUTH_SUBDOMAIN was potentially very confusing when opened by a
user, as it had various Login/Signup buttons as if there was a realm on
it. Instead, we want to display a more informative page to the user
telling them they shouldn't even be there. If possible, we just redirect
them to the realm they most likely came from.
To make this possible, we have to exclude the subdomain from
ROOT_SUBDOMAIN_ALIASES - so that we can give it special behavior.
2021-09-10 10:47:15 -07:00
Sahil Batra c7cb983ebd settings: Move check_settings_values to user_settings.py.
This commit moves check_settings_values to user_settings.py
from validator.py such that we can import the functions at
the top without any issue of cyclic imports.
2021-09-09 15:03:55 -07:00
Sahil Batra 693d58265e realm: Add 'PATCH /realm/user_settings_defaults' endpoint.
The realm-level default value of settngs for new users will
be updated using this endpoint.
2021-09-09 10:55:18 -07:00
Sahil Batra 2eec0772fb user_settings: Extract setting values checks to a function.
We extract the checks for default_language, notification_sound,
and email_notifications_batching_period_seconds setting values
in json_change_settings to a new function check_settings_values.
2021-09-09 10:15:07 -07:00
Sahil Batra de4c0d9fe9 settings: Validate desktop_icon_display_count value in 'PATCH /settings'. 2021-09-08 11:04:44 -07:00
Sahil Batra 2f61afea9f settings: Remove do_change_notification_settings function.
This commit removes the do_change_notification_settings
function and we use do_change_user_setting directly to
change all the user settings now.
2021-09-08 11:04:44 -07:00
Sahil Batra 550d97a593 settings: Refactor callers of do_change_user_setting to pass acting_user. 2021-09-08 11:04:44 -07:00
Dinesh 9443e01a5d refactor: Rename do_set_user_display_setting to do_set_user_setting. 2021-09-07 10:16:42 -07:00
Dinesh 430c5cb8e7 user_settings: Create _legacy dicts for existing settings.
Since 84742a0, all settings are sent in the `user_settings` dictionary
which were previously sent inline with other fields in /register
response.

In order to simplify the process of adding new personal settings, we
want to transition to a world where new settings only need to consider
the `property_types` object, and code that needs to reference the
legacy behavior interacts with an object with `legacy` in its name.

This way, contributors working on new settings don't need to think
about the legacy code paths at all.

See https://chat.zulip.org/#narrow/stream/378-api-design/topic/user.20settings.20response.20in.20.2Fregister
to understand this better.
2021-09-07 10:14:45 -07:00
Aman Agrawal 34a37f55d6 spectators: Rename is_web_public_compatible functions. 2021-09-06 09:00:16 -07:00
PIG208 53888e5a26 request: Refactor ZulipRequestNotes to RequestNotes.
This utilizes the generic `BaseNotes` we added for multipurpose
patching. With this migration as an example, we can further support
more types of notes to replace the monkey-patching approach we have used
throughout the codebase for type safety.
2021-09-03 08:48:45 -07:00
Alex Vandiver 4d428490fd outgoing_http: Use OutgoingSession subclasses in more places.
This adds the X-Smokescreen-Role header to proxy connections, to track
usage from various codepaths, and enforces a timeout.  Timeouts were
kept consistent with their previous values, or set to 5s if they had
none previously.
2021-09-01 05:34:13 -07:00
Alex Vandiver 621d0e5123 send_email: Log IP address sources (and more realms) of send_email. 2021-08-31 16:36:42 -07:00
Anders Kaseorg 1ce12191aa docs: Update links for other repository branch renames.
GitHub redirects these, but we should use the canonical URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 15:28:01 -07:00
Anders Kaseorg 817146c28b python: Upgrade SQLAlchemy from 1.3.24 to 1.4.23.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 06:47:39 -07:00
Eeshan Garg c9861b3c74 marketing_emails: Ask for user's consent at sign-up. 2021-08-20 06:43:26 -07:00
Vishnu KS d11d454206 auth: Redirect password reset page to /accounts/go when required. 2021-08-20 06:18:49 -07:00
PIG208 3b11c36ed9 typing: Fix function signatures.
This fixes mypy errors for function signatures discovered with
django-stubs.
2021-08-20 05:54:19 -07:00
Eeshan Garg dea4187aff devlogin: Add support for creating development demo realms.
This is a part of #19523.
2021-08-19 03:09:36 -07:00
Anders Kaseorg 0d061f44c1 actions: Remove acting_client parameter from bulk_remove_subscriptions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
Anders Kaseorg 0b795e492f thumbnail: Remove unused is_camo_url parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
Alex Vandiver ebaafb32f3 send_email: Change clear_scheduled_emails to only take one user.
No codepath except tests passes in more than one user_profile -- and
doing so is what makes the deduplication necessary.

Simplify the API by making it only take one user_profile id.
2021-08-19 00:44:33 -07:00
PIG208 4905d73fe4 auth: Refactor oauth_redirect_to_root to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 0dac524ea4 registration: Refactor view functions in registration.py to use REQ. 2021-08-08 17:11:18 -07:00
PIG208 15eeb2cb25 message: Refactor send_message_backend to use REQ. 2021-08-08 17:11:18 -07:00
Gaurav Pandey 067a32925d api: Default `client_gravatar` to `true`.
Default `client_gravatar` to `true` for all endpoints passing the
parameter to avoid extra payload.

Fixes #11214.
2021-08-08 16:53:05 -07:00
Mateusz Mandera ddcfd9e2ee rate_limit: Rate limit the /accounts/find/ endpoint.
Closes #19287

This endpoint allows submitting multiple addresses so we need to "weigh"
the rate limit more heavily the more emails are submitted. Clearly e.g.
a request triggering emails to 2 addresses should weigh twice as much as
a request doing that for just 1 address.
2021-08-06 12:17:44 +02:00
Anders Kaseorg 5483ebae37 python: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg ad5f0c05b5 python: Remove default "utf8" argument for encode(), decode().
Partially generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Sahil Batra 0364d0c8ca events: Add "user_settings_object" to client_capabilities.
This commit adds "user_settings_object" field to
client_capabilities which will be used to determine
if the client needs 'update_display_settings' and
'update_global_notifications' event.
2021-08-01 15:30:17 -07:00
Anders Kaseorg aeb0dc7269 video_calls: Support separate testing credentials for Zoom.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-01 14:26:46 -07:00
Gaurav Pandey d89b4dcd0b api: Return zulip_merge_base alongside zulip_version.
Return zulip_merge_base alongside zulip_version
in `/register`, `/event` and `/server_settings`
endpoint so that the value can be used by other
clients.
2021-07-30 12:03:41 -07:00
Hemanth V. Alluri c00089ac28 drafts: Add support for toggling drafts synchronization.
With changes mostly to the API documentation by tabbott.
2021-07-30 10:00:27 -07:00
Anders Kaseorg 1f5fc12989 video_calls: Remove call to deprecated Zoom Data Compliance API.
https://marketplace.zoom.us/docs/guides/stay-up-to-date/announcements#data-compliance

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-30 09:19:22 -07:00
Hemanth V. Alluri f1f0a26c37 drafts: Extract all of the real work to lib functions.
The main reason why this is needed is because this seems to be
convention and because we can't easily test event creation without
doing this.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2021-07-28 16:33:01 -07:00
Mateusz Mandera 4161e0caeb rate_limit: Show html page when rate limited at /new/ endpoint.
Previously this showed a json error, but this is an endpoint that human
users use in the browser, so a proper HTML page is more appropriate.
2021-07-28 15:24:04 -07:00
Riken Shah 9fadd43830 user_status: Add backend changes to support status emoji.
In this commit:

* We update the `UserStatus` model to accept
`AbstractReaction` as a base class so, we can get all the
fields related to store status emoji.

* We update the user status endpoint
(`users/me/status`) to accept status emoji fields.

* We update the user status event to add status emoji
fields.

Co-authored-by: Yash Rathore <33805964+YashRE42@users.noreply.github.com>
2021-07-27 17:57:15 -07:00
sahil839 1ce09d9db6 realm: Extend add_custom_emoji_policy to use COMMON_POLICY_TYPES.
This commit adds full members and moderators option for
add_custom_emoji_policy by using COMMON_POLICY_TYPES.
2021-07-27 16:41:23 -07:00
sahil839 4b1313a92b models: Replace add_emoji_by_admins_only with add_custom_emoji_policy.
This commit replaces boolean field add_emoji_by_admins_only with an
integer field add_custom_emoji_policy as we would also add full members
and moderators option for this setting in further commits.
2021-07-27 16:41:22 -07:00
Anders Kaseorg 17749cb608 archive: Remove non-functional archive code.
This removes a bunch of non-functional duplicate JavaScript, HTML, and
CSS that was interfering with maintenance on the functional originals,
because it was never clear how to update the duplicates or how to
check that you’d updated the duplicates correctly.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-27 16:25:24 -07:00
Sahil Batra 355f6e9b53 settings: Move 'enter_sends' setting to property_types dict.
This commit moves "enter_sends" setting to property_types dict.
With this change, changing enter_sends setting also sends an
event of type "update_display_settings" and thus enables us
to live-update the UI.
2021-07-27 15:52:53 -07:00
PIG208 8121d2d58d typing: Fix misuse of HttpResponse.
Amend usage of HttpResponse when appropriate.
2021-07-27 14:31:19 +08:00
Suyash Vardhan Mathur ec6739ff29 openapi: Document get-subscribers endpoint. 2021-07-26 18:12:12 -07:00
Suyash Vardhan Mathur 2d2b089066 openapi: Document delete-topic endpoint. 2021-07-26 17:56:04 -07:00
sahil839 c7e756f984 setting: Allow only owners to change "Who can invite users" setting. 2021-07-26 17:48:01 -07:00
sahil839 8ced74192a invite: Add 'Nobody' option to invite_to_realm_policy in backend. 2021-07-26 17:48:01 -07:00
Mateusz Mandera a6924e23fa bots: Pass realm to get_system_bot calls in streams.py. 2021-07-26 15:33:13 -07:00
PIG208 7d1c475f69 typing: Use assertions for function arguments.
Utilize the assert_is_not_None helper to eliminate errors of
'Argument x to "Foo" has incompatible type "Optional[Bar]"...'
2021-07-26 14:48:45 -07:00
PIG208 cf8687662f confirmation: Use the correct type hints for create_confirmation_link.
Previously we annotate the first argument as `ContentType`, which
is wrong as suggested by django-stubs.
2021-07-26 14:46:45 -07:00
Mateusz Mandera 1c64bed8e4 rate_limiter: Rate limit the /new/ endpoint. 2021-07-24 15:52:06 -07:00
PIG208 66b1a4e7ca backend: Add None-checks with assertions and if-elses.
This fixes a batch of mypy errors of the following format:
'Item "None" of "Optional[Something]" has no attribute "abc"'
2021-07-24 15:00:21 -07:00
PIG208 fffd4ed8d5 invitations: Correctly compare realms for revoke_user_invite. 2021-07-24 14:59:19 -07:00
Tim Abbott 01ce58319d mypy: Fix most AnonymousUser type errors.
This commit fixes several mypy errors with Django stubs, by telling
mypy that we know in a given code path that the user is authenticated.
2021-07-24 14:55:46 -07:00
Hemanth V. Alluri b5cd232a2e drafts: Use enable_drafts_synchronization for access control.
If a user doesn't have enable_drafts_synchronization set to True, then
don't let them access the drafts API.  This will help protect us
against client bugs accidentally sending drafts to the server when the
feature is disabled.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2021-07-23 12:34:15 -07:00
Abhijeet Prasad Bodas 5db4fe8652 api: Allow setting email_notifications_batching_period_seconds.
We allow a maximum value of one week to make sure there aren't a huge
number of rows in the table for any user (this could happen if stream
notifications are enabled).

This commit also fixes a small error in the user_settings test.
2021-07-23 12:22:05 -07:00
Sahil Batra 318d71469b api: Remove '/users/me/enter-sends' endpoint.
We remove the '/users/me/enter-sends' endpoint
and 'enter_sends' setting will now be edited
using the '/settings' endpoint.
2021-07-22 11:01:43 -07:00
Erik Tews 1ecb87ec80 auth: show _OR_ during login only when other methods are available.
There might be good reasons to have other external authentication
methods such as SAML configured, but none of them is available.

This happens, for example, when you have enabled SAML so that Zulip is
able to generate the metadata in XML format, but you haven't
configured an IdP yet. This commit makes sure that the phrase _OR_ is
only shown on the login/account page when there are actually other
authentication methods available. When they are just configured, but
not available yet, the page looks like as if no external
authentication methods are be configured.

We achieve this by deleting any_social_backend_enabled, which was very
similar to page_params.external_authentication_methods, which
correctly has one entry per configured SAML IdP.
2021-07-20 14:29:48 -07:00
Tim Abbott 256091dc15 settings: Merge settings API endpoints.
This API change removes unnecessary complexity from a client that
wants to change a user's personal settings, and also saves developers
from needing to make decisions about what sort of setting something is
at the API level.

We preserve the old settings endpoints as mapping to the same function
as the new one for backwards-compatibility. We delete the
documentation for the old endpoints, though the documentation for the
merged /settings endpoint mentions how to use the old endpoints when
needed.

We migrate all backend tests to the new endpoints, except for
individual tests for each legacy endpoint to verify they still work.

Co-authored-by: sahil839 <sahilbatra839@gmail.com>
2021-07-19 21:14:32 -07:00
Tim Abbott 95606a7347 api: Return user IDs, not display emails, in subscribers endpoints.
Sometime in the deep past, Zulip the GET /users/me/subscriptions
endpoint started returning subscribers.  We noticed this and made it
optional via the include_subscribers parameter in
1af72a2745, however, we didn't notice
that they were being returned as emails rather than user IDs.

We migrated the core /register code paths to use subscriber IDs years
ago; this change completes that for the endpoints we forgot about.

The documentation allowed this error because we apparently had no
tests for this code path that used the actual API.
2021-07-18 11:32:28 -07:00
sahil839 d5a0c1ede5 api: Remove "full_name" and "account_email" from response of '/settings'.
We remove the "full_name" and "account_email" fields from the response
of 'PATCH /settings' endpoint. These fields were part of the response
to make sure that we tell that the parameters not present in response
were ignored.

We can remove these fields as 'ignored_parameters_unsupported' now
specifies which parameters were ignored and not supported by the
endpoint.
2021-07-16 16:10:15 -07:00
sahil839 05aff3f271 api: Add "ignored_parameters_unsupported" to response of '/settings'.
We add "ignored_parameters_unsupported" field to the response object
of 'PATCH /settings' endpoint. This will contain the parameters
passed to the endpoint which are not changed by the endpoint and are
ignored.

This will help in removing the other fields like "full_name" from
response which was essentially present to specify that only these
fields were updated by the endpoint and rest were ignored.

We will also change other endpoints to follow this in future.
2021-07-16 16:10:15 -07:00
Anders Kaseorg fb3ddf50d4 python: Fix mypy no_implicit_reexport errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-16 14:02:31 -07:00
Eeshan Garg faa695e86d registration: Collect organization type on sign-up. 2021-07-15 09:58:27 -07:00
sahil839 2f44a08b37 settings: Convert settings_overlay.html to handlebars template.
Fixes #18796.
2021-07-15 08:59:23 -07:00
Tim Abbott 4fb6f089e5 api: Remove error for no settings changes requested.
While this error message was vaguely useful, it wasn't reachable via
the UI at all, and it prevents merging this endpoint with related
endpoints.
2021-07-14 15:54:05 -07:00
sahil839 61acf68aa4 emoji: Allow emoji authors to delete the emoji always.
Previously, non-admin emoji authors were allowed to
delete the emoji only if add_emoji_by_admins_only
was false. But, as add_emoji_by_admins_only setting
is for who can add emoji and not delete emojis, it
should not affect the behavior of deleting emojis
and users should always be allowed to delete the
emojis which. they added themselves
2021-07-14 14:57:08 -07:00
sahil839 50240ca71b models: Use COMMON_POLICY_TYPES for user_group_edit_policy.
This commit adds moderators and full members options for
user_group_edit_policy by using COMMON_POLICY_TYPES.

Moderators do not require to be a member of user group in
order to edit or remove the user group if they are allowed
to do so according to user_group_edit_policy.
But full members need to be a member of user group to edit
or remove the user group.
2021-07-14 14:38:44 -07:00
PIG208 c03b9c95ad request: Store client information using ZulipRequestNotes.
This concludes the HttpRequest migration to eliminate arbitrary
attributes (except private ones that are belong to django) attached
to the request object during runtime and migrated them to a
separate data structure dedicated for the purpose of adding
information (so called notes) to a HttpRequest.
2021-07-14 12:01:07 -07:00
PIG208 8eb2c3ffdb request: Move realm from the request to ZulipRequestNotes. 2021-07-14 12:01:07 -07:00
PIG208 742c17399e request: Move miscellaneous attributes to ZulipRequestNotes.
This includes the migration of fields that require trivial changes
to be migrated to be stored with ZulipRequestNotes.

Specifically _requestor_for_logs, _set_language, _query, error_format,
placeholder_open_graph_description, saveed_response, which were all
previously set on the HttpRequest object at some point. This migration
allows them to be typed.
2021-07-14 12:01:07 -07:00
PIG208 5475334b16 request: Refactor to store requestor_for_logs in ZulipRequestNotes. 2021-07-14 12:01:07 -07:00
PIG208 da6e5ddcae request: Move log_data from HttpRequest to ZulipRequestNotes. 2021-07-14 12:01:05 -07:00
akshatdalton f5c4d51ed2 resolve topic: Add `is:resolved` search keyword/filtering support.
This commit adds the backend support for `is:resolved` search keyword.
In the next commit, I will add the frontend support for the same.
2021-07-13 23:18:41 -07:00
Suyash Vardhan Mathur 309f4ba145 openapi: Make endpoint operationId dash-separated.
The operationId is directly used in URLs of API doc pages
to find the OpenAPI data to render. However, this is dash-
separated in the URLs, and having underscore_separated IDs
in OpenAPI data doesn't allow direct comparison of the two.

This commit changes all OperationIDs from underscore_separated
to dash-separated.
2021-07-13 16:46:22 -07:00
aryanshridhar 7dc56fdcf2 settings_emoji: Allow only admins to override existing emojis.
Previously, even non-admins had the option to override built-in
emojis in the `Settings Emoji` UI.

This commits essentially limits the functionality of overriding
custom and allows only realm administrators to
override built-in emojis with their custom emojis by adding an
authorization check in the backend.

It also adds relevant tests in `test_realm_emoji` which tests
for the cases where an admin and non admin tries to override
the built-in emoji.

Fixes #18860.
2021-07-13 08:28:28 -07:00
Tim Abbott c804ab27d5 actions: Rename do_activate_user to do_activate_mirror_dummy_user.
This function had a confusing name, which could result in someone
using it unintentionally when they meant do_reactivate_user.

We also add docstrings for both functions.
2021-07-08 17:33:17 -07:00
sahil839 a535ec6262 settings: Remove timezone from UserProfile.property_types.
We remove timezone setting from UserProfile.property_types
so that we can directly use UserProfile.property_types for
implementation of realm-default values of various user
settings.
2021-07-08 12:52:29 -07:00
Ganesh Pawar d6711e3cd2 muting: Add support for muting deactivated users.
Fixes #19141
2021-07-07 10:44:18 -07:00
aryanshridhar c6828767fa navbar: Migrate to handlebars.
This commit migrates the `navbar.html` Django template
to handlebars by creating a new file as `navbar.hbs`
within `/static/templates` which is then rendered
using `ui_init` module.

As a part of migration, we also remove the `search_pills_enabled`
and `embedded` parameters from the context attribute as they
are no longer needed now.

Fixes part of #18792.
2021-07-06 16:46:50 -07:00
Tim Abbott 3fdfbb27f9 streams: Clean up compose_views logic.
Now that we're not using json_error and instead raising an exception,
we don't need this special logic to try to reraise an exception.
2021-07-06 15:36:39 -07:00
PIG208 8b9011dff8 json_error: Completely remove json_error.
This completes the migration from `return json_error` to
`raise JsonableError`.
2021-07-06 15:34:33 -07:00
Tim Abbott 331f34cc1f auth: Use standard classes for API key fetch exceptions.
This lets us reuse a bunch of code and error handling logic, resulting
in a more sensible and consistent API.
2021-07-06 15:28:38 -07:00
Eeshan Garg 5a94bfcb88 docs: Add proper backlink to the homepage from /help and /api.
Currently, the "Home" link at the top takes one to the doc root,
i.e., /help or /api. This is a little misleading since "Home"
seems to be more synonymous with the Zulip homepage.

This commit adds a proper backlink to the top logo that takes you to
the homepage and renames "Home" to be more specific. The text after
"|" will now take you to the doc root instead (/help or /api). Note
that this allows us to link the /help and /api pages from the
homepage while ensuring that backlinks allow the visitor to get back
to the homepage.
2021-07-06 10:35:41 -07:00
PIG208 43cca07b4b exceptions: Add ValidationFailureError. 2021-07-06 09:44:45 -07:00
aryanshridhar 13399833b0 right_sidebar: Migrate to handlebars.
This commit migrates the `right_sidebar.html` Django template
to handlebars by creating a new file as `right_sidebar.hbs`
which is then rendered using `ui_init` module.

It also removes the tests in `test_home` due to the template
migration, since these elements aren't rendered on the backend
anymore.

We also remove `test_compute_show_invites_and_add_streams*`.

Fixes part of #18792.
2021-07-05 18:58:24 -07:00
aryanshridhar 8fb7ff7637 left_sidebar: Migrate to handlebars.
This commit migrates the `left_sidebar.html` Django template
to handlebars by creating a new file as `left_sidebar.hbs`
which is then rendered using `ui_init` module.

These are the minor changes introduced by virtue of template
migration -
 - The `compute_show_invites_and_add_streams` function now
   only concerns with the invite_to_realm_policy.
 - Renamed the `compute_show_invites_and_add_streams` function
   to `compute_show_invites` due to the above change.
 - Fixes relevant `test_home.py` tests due to the above
   changes.

Fixes part of #18792.
2021-07-05 18:49:43 -07:00
Anders Kaseorg ee8724e436 docs: Correct Big Blue Button to BigBlueButton.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 16:59:00 -07:00
Anders Kaseorg 24c239d991 openapi: Add missing __init__.py; fix type errors hidden by its absence.
The absence of __init__.py was preventing mypy from following any of
the zerver.openapi imports.  These errors were being silenced by
ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:20:39 -07:00
PIG208 6a04648fd7 exceptions: Add ResourceNotFoundError. 2021-07-05 11:02:12 -07:00
PIG208 dcbb2a78ca python: Migrate most json_error => JsonableError.
JsonableError has two major benefits over json_error:
* It can be raised from anywhere in the codebase, rather than
  being a return value, which is much more convenient for refactoring,
  as one doesn't potentially need to change error handling style when
  extracting a bit of view code to a function.
* It is guaranteed to contain the `code` property, which is helpful
  for API consistency.

Various stragglers are not updated because JsonableError requires
subclassing in order to specify custom data or HTTP status codes.
2021-06-30 16:22:38 -07:00
sahil839 c8849f8fe3 logo: Compute realm-logo url in frontend only.
This commit fixes the bug of always showing
day-mode realm logo when color scheme display
setting is set to automatic but the OS setting
is dark theme. This is because we cannot check
the OS setting on backend and we need to set
the logo url accordingly in frontend only.
So, we remove the logo url computation from
backend completely and instead compute it in
the frontend only.

Fixes #18778.
2021-06-29 13:42:44 -07:00
Steve Howell c25dbf7020 widgets: Validate todo data on the backend. 2021-06-29 13:40:33 -07:00
PIG208 75cea329b4 markdown: Refactor out additional properties added to Message.
This adds a new class called MessageRenderingResult to contain the
additional properties we added to the Message object (like alert_words)
as well as the rendered content to ensure typesafe reference. No
behavioral change is made except changes in typing.

This is a preparatory change for adding django-stubs to the backend.

Related: #18777
2021-06-24 18:14:53 -07:00
Suyash Vardhan Mathur 23b991a801 api docs: Replace most markdown files with a common template.
This PR adds a basic .md template that is followed by lot of /api
pages. Since we have recently done the migration work to ensure that
our REST API documentation pages for individual endpoints are almost
all identical files following a common pattern, we can now get the
payoff of deleting them all in favor of a shared template.

This removes 2000 lines of somewhat finicky configuration from the
codebase, and thus should save significant effort when documenting new
API endpoints in the future.

The markdown files for endpoints or other pages which deviate from the
standard template remain, and the docs are instead generated from
those files using the existing system.
2021-06-24 10:42:08 -07:00
Suyash Vardhan Mathur 15123bca68 openapi: Refactor return value of get_path function.
The ArticlePath dataclass added can now be used
for the return value of get_path function to make
it more extensible and robust.
2021-06-24 10:38:44 -07:00
Suyash Vardhan Mathur 1375e99ae7 openapi: Add dataclass for holding article path fields.
The returned values of get_path function would be
expanded soon, and defining a dataclass would make
the code cleaner for returning and using the fields.
2021-06-24 10:38:44 -07:00
sahil839 828759d2ba models: Replace allow_community_topic_editing with edit_topic_policy.
This commit replaces the allow_community_topic_editing boolean with
integer field edit_topic_policy and includes both frontend and
backend changes.

We also update settings_ui.disable_sub_settings_onchange to not
change the color of label as we did previously when the setting
was a checkbox. But now as the setting is dropdown we keep the
label as it is and we don't do anything with label when disabling
dropdowns. Also, this function was used only here so we can safely
change this.
2021-06-16 14:59:36 -07:00
Aman Agrawal 64f04e9913 web_public_view: Replace `web_public_visitor` with `spectator`.
We use `spectator` term instead of `web_public_visitor` as it is
more concise and better represents this type of users.
2021-06-16 07:46:34 -07:00
Eeshan Garg 512229cf7d plans_page: Fix broken redirection to /accounts/login/.
When an unauthenticated user tries to access the /plans page, we
redirect to /accounts/login/?next=plans (note the missing slash
before "plans"). After the user is authenticated, they are then
redirected to /accounts/login/plans, which is an invalid URL. The
correct URL should be just /plans.

This commit solves this by prefixing the "plans" in the query
parameter with a forward slash, which results in the correct
redirect URL, i.e., /plans.
2021-06-15 16:49:01 -07:00
Riken Shah b2ab448b71 gear_menu: Extract the gear menu as a handlebars template.
For this extraction, we need to move some context
parameter (from home_real in `views/home.py`) to extra
page_params parameter (of
build_page_params_for_home_page_load in
`lib/home.py`) so handlebars template can access them.

While moving I confirmed that these parameters are not
used elsewhere if some parameter is used elsewhere
(like `apps_page_url`) then I didn't remove it from the
context list, I just added it to the page_params list.

Fixes: #18795.
2021-06-15 12:17:45 -07:00
Ganesh Pawar 1d4b6c1320 about_zulip: Migrate to handlebars.
This results in moving the `zulip_merge_base` parameter to
page_params, so that it's available to JavaScript.

Since this is technically a tiny overlay, it needs to be initialized
before hashchange.js.
2021-06-15 11:55:20 -07:00
Steve Howell e739bee00a poll widget: Add server validation. 2021-06-14 17:46:16 -07:00
Steve Howell 24ed5a2bb4 submessages: Add verify_submessage_sender.
Before this change a rogue actor could try to
widgetize another person's message. (The
rogue actor would already have access to read
the message.)
2021-06-14 17:46:16 -07:00
Vishnu KS 6e3d4e7e75 billing: Create is_free_trial_offer_enabled function. 2021-06-14 16:49:53 -07:00
Vishnu KS 5da3a69505 plans: Mention whether realm is on free trial. 2021-06-14 16:49:53 -07:00
Vishnu KS 44d8368003 plans: Don't hardcode plan_type values in template. 2021-06-14 16:49:53 -07:00
sahil839 aa5d2199b1 invite: Use handlebar template for invite-user overlay.
Fixes a part of #18794.
2021-06-14 13:46:57 -07:00
Suyash Vardhan Mathur 001435a297 refactor: Rename and move app_filters.py.
Moved the code in zerver/templatetags/app_filters.py
to more intuitive location and name at
zerver/lib/templates.py.
2021-06-11 07:43:22 -07:00
Eeshan Garg 50a6887d94 plans_page: Compute sponsorship link for subdomain pages correctly.
Currently, in the FAQ on our /plans page, when the user clicks on
the sponsorship link in the answer for the first question, they
are always taken to /accounts/go, causing them to have to input
their organization URL even if they are on a subdomain page.

This commit makes it so that when the user is on a subdomain page,
they are taken to /upgrade#sponsorship directly. On the other
hand, when they are on a root domain (/) page, they have to go
through /accounts/go and specify their organization's name.
2021-06-10 09:10:38 -07:00
Vishnu KS 1938076f67 billing: Enforce license limit for plans on manual license management. 2021-06-09 17:42:38 -07:00
Anders Kaseorg d8adeb66ca docs: Deemphasize Hubot as an integration bridge.
The Hubot project looks to be abandoned; it hasn’t been updated in
years and its own installation instructions don’t work anymore.
Remove our special placement of Hubot alongside Zapier and IFTTT.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 14:04:01 -07:00
Gaurav Pandey f82aba5a3d compatibility: Shift functions to new module.
Shift functions used for compatibility from
zerver.lib.home (is_outdated_server) and
zerver.view.compatibility (pop_numerals,
version_lt, find_mobile_os,
is_outdated_desktop_app, is_unsupported_browser)
to zerver.lib.compatibility module.
2021-06-05 07:33:25 -07:00
Abhijeet Prasad Bodas 90b6fa7881 reactions: Select Message FOR UPDATE when adding/removing.
This locks the message row while a reaction is being added/removed,
which will handle race conditions caused by deleting the message
at the same time.

We make sure that events work happens outside the transaction,
so that in case there's some problem with the queue processor, the
locks aren't held for too long.

As a nice side-effect, we also handle race conditions from double
adding reactions, because once the message is locked, a duplicate
request will wait till the earlier transaction commits, and hence
will not throw `IntegrityErrors`s (rather, will be handled in our
safety check in the /views code itself), which earlier had to be
handled explicitly.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas efc2f49e3c submessage: Select Message FOR UPDATE when creating.
This locks the message while creating a submessage, which
will handle race conditions caused by deleting the message
simultaneously.

We make sure that events work happens outside the transaction,
so that in case there's a problem with the queue processor,
the locks aren't held for too long.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 5f4113cf60 message delete: Select Message FOR UPDATE when archiving.
Further commits will start locking the message rows while
adding related fields like reactions or submessages,
to handle races caused by deleting the message itself at the
same time.

The message locking implemented then will create a possibility
of deadlocks, where the related field transaction holds a lock
on the message row, and the message-delete transaction holds a
lock on the database row of the related field (which will also
need to be deleted when the message is deleted), and both
transactions wait for each other.

To prevent such a deadlock, we lock the message itself while
it is being deleted, so that the message-delete transaction
will have to wait till the other transaction (which is about
to delete the related field, and also holds a lock on the
message row) commits.

https://chat.zulip.org/#narrow/near/1185943 has more details.
2021-06-04 08:18:17 -07:00
Suyash Vardhan Mathur d1ccf15cd3 apidocs: Fix invalid API page bug.
The current logic to get API pages' title using
OperationID should be used when the first line
of the file explicitly mentions so.

In cases where the files didn't begin with `#` but also
didn't need to get title from OpenAPI summary,
this logic fails and causes Server error.
This particularly happens when the article is invalid,
and the `missing.md` file doesn't need title to be
generated, but doesn't start with `#` either.

This commit fixes the logic of using the generated title and covers the bug.
2021-06-03 10:47:47 -07:00
Gilbert Bishop-White e4d9e58c7d emails: Make it obvious when registering creates new realm.
Checked the email looked OK in `/emails` for both creating realm and
registering within an existing one.

Not sure zerver/tests/test_i18n.py test has been suppressed correctly.

Fixes #17786.
2021-06-02 15:18:03 -07:00
Gaurav Pandey f67fc37f7d settings: Rename MAX_EMOJI_FILE_SIZE.
Rename MAX_EMOJI_FILE_SIZE to
MAX_EMOJI_FILE_SIZE_MIB reflecting
the size in mebibytes.
2021-06-02 13:55:15 -07:00
Gaurav Pandey 73059dd0c7 settings: Rename MAX_LOGO_FILE_SIZE.
Rename MAX_LOGO_FILE_SIZE to
MAX_LOGO_FILE_SIZE_MIB reflecting
the size in mebibytes.
2021-06-02 13:55:15 -07:00
Gaurav Pandey 2ddc62e9b3 settings: Rename MAX_ICON_FILE_SIZE.
Rename MAX_ICON_FILE_SIZE to
MAX_ICON_FILE_SIZE_MIB reflecting
size in mebibytes.
2021-06-02 13:55:15 -07:00
Gaurav Pandey 0f6bb210a6 settings: Rename MAX_AVATAR_FILE_SIZE.
Rename MAX_AVATAR_FILE_SIZE to
MAX_AVATAR_FILE_SIZE_MIB reflecting
size in mebibytes.
2021-06-02 13:55:15 -07:00
Mateusz Mandera e17758f8ad auth: Implement a generic OpenID Connect backend.
Fixes #11939.
2021-05-28 09:43:06 -07:00
Suyash Vardhan Mathur 52b9c96c5d apidocs: Add support for using OpenAPI title for /api pages.
The current logic of geneerating HTML titles requires the title to be
present as a heading in the first line of .md file. However, this will
shortly be no longer true for /api pages where these are
auto-generated from OpenAPI data.  Modified the code to fetch the
title from OpenAPI data in case of such pages.
2021-05-26 17:38:01 -07:00
Mateusz Mandera d17a4f64d6 registration: Return status code 404 if invalid confirmation key passed.
We were returning 200 before - 404 is more appropriate.
2021-05-20 17:29:09 -07:00
Mateusz Mandera 40169353ad auth: Validate multiuse invite link confirmation key in social auth. 2021-05-20 17:29:09 -07:00
Tim Abbott e1c885aeb9 home: Remove unused has_mobile_devices parameter.
This parameter has never been used, and causes an unnecessary database
query.

We keep the num_push_devices_for_user function, since we may have uses
for it down the line.

Fixes part of #14166.
2021-05-18 17:36:55 -07:00
Anders Kaseorg e015f3ed7d docs: Correct “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-14 13:15:38 -07:00
Anders Kaseorg 668b5137b0 version: Display Zulip version in About Zulip dialog.
We record Git details about the merge-base with upstream branches in
the zulip-git-version file, if the upstream repository is available.

Note that the first Git upgrade after merging the parent commit will
not include the merge-base details, since the upstream repository will
not have been available.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 11:36:12 -07:00
Vishnu KS 051dab58ea navbar: Add gear menu advertisement for sponsoring zulip. 2021-05-12 10:21:48 -07:00
Gaurav Pandey 96e035a2f0 api: Fix encoding of strings in streams endpoint.
* Remove unnecessary json_validator for string parameters.
* Update frontend to pass right parameter.

Bump api feature level and highlight the fix for `emojiset`
parameter of `settings/display` endpoint in zulip.yaml file.

Fixes part of #18035.
2021-05-10 10:29:22 -07:00
Gaurav Pandey 9abe1cafbe api: Fix encoding of strings in display settings endpoint.
* Remove unnecessary json validator for string validator.
* Update frontend to pass right validator.
* Update zulip.yaml to pass right parameter for curl request
  in openapi.
* Update python_examples to pass right paramater.

Fixes part of #18035.
2021-05-10 10:03:32 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
PIG208 7150fe5dc5 backend: Extract check_update_message from update_message_backend. 2021-05-09 20:44:04 -07:00
Gaurav Pandey 00647b6fa7 api: Fix encoding of strings in video calls endpoint.
This removes unnecessary json_validator for string parameters in the
BigBlueButton video calls endpoints.  Note that this breaks links to
video meetings sent before the upgrade; there's not much we can do
about that.

Since this is the last commit in this series, we update the
ZULIP_FEATURE_LEVEL for this batch of changes.

Fixes part of #18035.
2021-05-07 14:11:48 -07:00
Gaurav Pandey 36ad9b7d0e api: Fix encoding of strings in user settings endpoints.
* Remove unnecessary json_validator for string parameters.
* Remove unnecessary JSON encoding in frontend calls.  Structurally,
  JavaScript does correct encoding without explicit JSON encoding.

Fixes part of #18035.
2021-05-07 14:10:45 -07:00
Gaurav Pandey 232c7107eb api: Fix encoding of strings in tutorial endpoint.
* Remove unnecessary json validator for string parameter.
* Update frontend to pass right parameter.

Fixes part of #18035.
2021-05-07 14:06:54 -07:00
Gaurav Pandey d4219da254 api: Fix encoding of strings in realm domain endpoint.
* Remove unnecessary json_validator for string parameters.
* Update frontend to pass right parameters.

Fixes part of #18035.
2021-05-07 14:06:54 -07:00
Gaurav Pandey b2111f5753 api: Fix encoding of strings in default stream groups.
Remove unnecessary json_validator for string parameters.  This change
does not modify JavaScript because we don't have a frontend for these
API endpoints yet.

Fixes part of #18035.
2021-05-07 14:06:37 -07:00
Gaurav Pandey d6cf1ef27b api: Fix encoding of strings in hotspot endpoint.
* Remove unnecessary json_validator for string parameter.
* Update frontend to pass right parameters.

Fixes part of #18035.
2021-05-07 11:45:25 -07:00
Mateusz Mandera 4006bb5153 auth: Improve display of errors when user needs to reset password.
Raising jsonableError in the authentication form was non-ideal because
it took the user to an ugly page with the returned json.

We also add logging of this rare occurence of the scenario being
handled here.
2021-05-07 09:10:45 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Cyril Pletinckx e4ff372fc3 emails: Transform SMTPException into EmailNotDeliveredException.
Django's default SMTP implementation can raise various exceptions
when trying to send an email. In order to allow Zulip calling code
to catch fewer exceptions to handle any cause of "email not
sent", we translate most of them into EmailNotDeliveredException.
The non-translated exceptions concern the connection with the
SMTP server. They were not merged with the rest to keep some
details about the nature of these.

Tests are implemented in the test_send_email.py module.
2021-05-05 20:16:11 -07:00
Tim Abbott 97b08b77df registration: Improve error handling for source_realm_id.
This restores the previous behavior that we store None rather than
throwing an exception in the presence of invalid input.
2021-05-02 11:12:49 -07:00
Siddharth Asthana 55da3329ef registration: Rename source_realm field to source_realm_id.
Now that we are passing source realm's id instead of string_id in
source realm selector, it makes sense to rename the "source_realm" field
to "source_realm_id".
2021-05-02 11:12:49 -07:00
Siddharth Asthana 4262c04db1 registration: Encode source realm as an integer.
In the source realm selector, when we select a realm from which we want
to import the data, we pass the source realm's string_id. The problem
with this approach is that the string_id can be an empty string. This
commit makes the source_realm pass the realm's id instead of string_id.
Now, the source_realm's value will either be an integer or "" (empty
string) when we don't want to import settings from any realm.
2021-05-02 11:12:49 -07:00
sahil839 e669818544 invites: Allow user to be invited as a moderator.
This commit adds both frontend and backend code to invite a user as
moderator. We allow only existing owners and admins to invite a user
as a moderator.
2021-04-30 15:57:09 -07:00
sahil839 a4c3224328 users: Remove redundant get_role_for_new_user in lib/create_user.py.
The function get_role_for_new_user was added to get role from the
invited_as value, as invited_as values were one of (1,2,3,4)
previously, but it was then changed to be the actual role value,
i.e. one of (100, 200, 400, 600), in 1f8f227444.

So, we can safely remove this function now and use invited_as value
directly and handle realm_creation case by using an if condition.
2021-04-30 15:57:09 -07:00
Ganesh Pawar ddf2127035 widgets: Prevent edits to widgets.
As of now, editing a widget doesn't update the rendered content.
It's important to ensure that existing votes or options added later on
don't get deleted when rendered.
This seems more complex than it's worth.

For now, we just prevent edits to widgets.
This commit makes the UI clearer that editing widgets isn't allowed.

See also:
https://github.com/zulip/zulip/issues/14229
https://github.com/zulip/zulip/issues/14799

Fixes #17156
2021-04-30 09:55:25 -07:00
Anders Kaseorg 871e73ab8f mypy: Don’t use Iterable for values iterated multiple times.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:06:17 -07:00
Tim Abbott a965c77af6 unsubscribe: Configure as csrf_exempt.
The reason we didn't have this before is that a GET to one of these
URLs would suffice to unsubscribe the user; but with the
List-Unsubscribe system, we need to allow POST from a third-party page
as well.
2021-04-28 15:33:32 -07:00
PIG208 9c0ce19104 reactions: Extract check_add_reaction from add_reaction. 2021-04-28 09:11:08 -07:00
PIG208 35fae79422 backend: Use JsonableError instead for update_message_backend. 2021-04-28 08:32:21 -07:00
Tim Abbott 50f00d3e97 emails: Add unsubscribe link infrastructure for marketing emails.
This is intended to be used in any marketing emails that we send with
the send_custom_email infrastructure.
2021-04-27 21:36:09 -07:00
Dinesh 27e4f5da92 typing: Support sending stream/topic typing status.
This extends the /json/typing endpoint to also accept
stream_id and topic. With this change, the requests
sent to /json/typing should have these:
* `to`: a list set to
    - recipients for a PM
    - stream_id for a stream message
* `topic`, in case of stream message
along with `op`(start or stop).

On receiving a request with stream_id and topic, we send
typing events to clients with stream_typing_notifications set
to True for all users subscribed to that stream.
2021-04-27 20:52:21 -07:00
Dinesh 734d935d4a client_capabilities: Add stream_typing_notifications. 2021-04-27 20:52:21 -07:00
akshatdalton b573964bfa notifications: Add support for `None` option in `Notification sound`.
This commit adds support for a `None` option in the dropdown menu
of `Notification sound`. When this option is selected, no audible
notification is sent to the user.

`None` will appear as the first option in the dropdown menu, since
this is not categorized as a playable audio.

This new option is added so that folks can disable audio notifications
without losing their other notification configuration (like for PMs, mentions).

Necessary test case is added for this new option.

Fixes #16090.
2021-04-27 16:42:16 -07:00
sahil839 a84725de9b dev_login: Show 'shiva' in a separate Moderators section. 2021-04-26 17:14:06 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Anders Kaseorg e3f2ffa681 docs: Capitalize “Markdown” consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Riken Shah 0e5f6c21ad refactor: Split dev related code from `auth.py` into `dev_login.py`.
This refactor will help us make our development-related authentication
code to avoid production security impact.
2021-04-23 15:11:44 -07:00
Mateusz Mandera 1a8ad796f8 models: Replace __id syntax with _id where possible.
model__id syntax implies needing a JOIN on the model table to fetch the
id. That's usually redundant, because the first table in the query
simply has a 'model_id' column, so the id can be fetched directly.
Django is actually smart enough to not do those redundant joins, but we
should still avoid this misguided syntax.

The exceptions are ManytoMany fields and queries doing a backward
relationship lookup. If "streams" is a many-to-many relationship, then
streams_id is invalid - streams__id syntax is needed. If "y" is a
foreign fields from X to Y:
class X:
  y = models.ForeignKey(Y)

then object x of class X has the field x.y_id, but y of class Y doesn't
have y.x_id. Thus Y queries need to be done like
Y.objects.filter(x__id__in=some_list)
2021-04-22 14:53:00 -07:00
akshatdalton 6509c4f8f4 linkifiers: Add an API to support the editing of linkifier.
This commit adds an API to `zproject/urls.py` to edit/update
the realm linkifier. Its helper function to update the
database is added in `zerver/lib/actions.py`.

`zulip.yaml` is documented accordingly as well, clearly
stating that this API updates one linkifier at a time.

The tests are added for the API and helper function which
updates the realm linkifier.

Fixes #10830.
2021-04-19 18:01:45 -07:00
akshatdalton 5bbfa24d74 Refactor: Rename `Filter` to `Linkifier`.
Linkifier error message: `Filter not found` is
updated to `Linkifier not found.`.

Similarly, `filter_id` description is updated to:
`The ID of the linkifier that you want to remove.`,
renamed the term `filter` with `linkifier`, in `zulip.yaml`.
2021-04-19 18:01:45 -07:00
Aman Agrawal 70d444a8eb compose: Move to `hbs` file and append it to DOM on init.
We move compose.html to compose.hbs file while keeping
`#compose` still in `home.html` as a hanger
where append rest of the elements.

This will provide us with two benefits:
* We could share common elements between message_edit_form and
compose.
* We can insert compose directly in any element. We may decide to
do it for recent topics.
2021-04-19 16:53:30 -07:00
Sumanth V Rao f9b79999ed playgrounds: Avoid json encoding each field in POST /realm/playgrounds.
This prevents us from having to json encode every field in the POST
request to /realm/playgrounds, and keeps the client logic simpler
when adding a playground.
2021-04-19 11:23:09 -07:00
Riken Shah 1d9417f9a1 refactor: Rename `dev_panel.html` for more clarity.
This commit renames `dev_panel.html` to `integrations_dev_panel.html`.
2021-04-19 10:25:05 -07:00
Riken Shah bd9e6ae97e refactor: Move dev-only templates to `templates/zerver/development`.
This a prep commit for adding a check to lint to ensure
only development-related templates entrypoint name
starts with `dev`.
2021-04-19 10:25:05 -07:00
sahil839 2dc99aa90f settings: Add new setting for controlling who can move msgs to stream.
This commit adds a new setting 'move_messages_between_streams_policy`
for controlling who can move messages from one stream to other.
2021-04-16 15:10:39 -07:00
Anders Kaseorg dcdb00a5e6 python: Convert deprecated Django is_safe_url.
django.utils.http.is_safe_url is a deprecated alias of
django.utils.http.url_has_allowed_host_and_scheme as of Django 3.0,
and will be removed in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg e7ed907cf6 python: Convert deprecated Django ugettext alias to gettext.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Mateusz Mandera b4542cc059 message_edit: Verify the message is in a stream in move message API.
This wasn't being validated before. There wasn't any possibility to
actually succeed in moving a private message, because the codepath would
fail at assert message.is_stream_message() in do_update_message - but we
should have proper error handling for that case instead of internal
server errors.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 0c0e83eaff message_edit: Verify user has access to old stream when moving message.
Otherwise an admin can move a topic from a private stream they're no
longer a part of - including the newest messages in the topic, that
they're not supposed to have access to.
2021-04-14 12:37:34 -07:00
Tim Abbott 7856098bdb archive: Use access_web_public_stream helper.
This is just a refactor to delete potentially security-sensitive
duplicated code.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 3ba8348c51 CVE-2021-30487: Prevent admins from moving topics to disallowed streams.
A bug in the implementation of the topic moving API resulted in
organization administrators being able to move messages to streams they
shouldn't be allowed to - private streams they weren't subscribed to and
streams in other organization hosted by the same Zulip installation.

In our current model realm admins can't send messages to private streams
they're not subscribed to - and being able move messages to a
stream effectively allows to send messages to that stream and thus the
two need to be consistent.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 140655d69e CVE-2021-30479: Prevent guest user access to all_public_streams API.
A bug in the implementation of the all_public_streams API feature
resulted in guest users being able to receive message traffic to public
streams that should have been only accessible to members of the
organization.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 6e11754642 CVE-2021-30478: Prevent API super users from forging messages to other organizations.
A bug in the implementation of the can_forge_sender permission
(previously is_api_super_user) resulted in users with this permission
being able to send messages appearing as if sent by a system bots,
including to other organizations hosted by the same Zulip installation.

- The send message API had a bug allowing an api super user to
  use forging to send messages to other realms' streams, as a
  cross-realm bot. We fix this most directly by eliminating the
  realm_str parameter - it is not necessary for any valid current use
  case. The email gateway doesn't use this API despite the comment in
  that block suggesting otherwise.
- The conditionals inside access_stream_for_send_message are changed up
  to improve security. They were generally not ordered very well,
  allowing the function to successfully return due to very weak
  acceptance conditions - skipping the higher importance checks that
  should lead to raising an error.
- The query count in test_subs is decreased because
  access_stream_for_send_message returns earlier when doing its check
  for a cross-realm bot sender - some subscription checking queries are
  skipped.
- A linkifier test in test_message_dict needs to be changed. It didn't
  make much sense in the first place, because it was creating a message
  by a normal user, to a stream outside of the user's realm. That
  shouldn't even be allowed.
2021-04-14 12:37:34 -07:00
Aman Agrawal 802c450b3f realm: Add setting to configure GIPHY rating.
Organization admins can use this setting to restrict the maximum
rating of GIFs that will be retrieved from GIPHY. Also, there
is option to disable GIPHY too.
2021-04-14 10:29:39 -07:00
Abhijeet Prasad Bodas 3947b0c80a linkifiers: Update API to send data using dictionaries.
* This introduces a new event type `realm_linkifiers` and
a new key for the initial data fetch of the same name.
Newer clients will be expected to use these.

* Backwards compatibility is ensured by changing neither
the current event nor the /register key. The data which
these hold is the same as before, but internally, it is
generated by processing the `realm_linkifiers` data.
We send both the old and the new event types to clients
whenever the linkifiers are changed.
Older clients will simply ignore the new event type, and
vice versa.

* The `realm/filters:GET` endpoint (which returns tuples)
is currently used by none of the official Zulip clients.
This commit replaces it with `realm/linkifiers:GET` which
returns data in the new dictionary format.
TODO: Update the `get_realm_filters` method in the API
bindings, to hit this new URL instead of the old one.

* This also updates the webapp frontend to use the newer
events and keys.
2021-04-13 12:16:07 -07:00
Abhijeet Prasad Bodas 2b9f2cc8ff mute user: Add some comments on message fetch.
These explain why we don't consider user mutes
in message fetching/unread data.
2021-04-13 09:15:49 -07:00
Anders Kaseorg 9a2140d60e storage: Fix get_storage and remove_storage with empty keys list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 08:53:03 -07:00
Tim Abbott cdbcb43706 api: Fix encoding of strings in realm endpoint.
* Don't require strings to be unnecessarily JSON-encoded.
* Use check_capped_string rather than custom code for length checks.
* Update frontend to pass the right parameters.

With a much simplified populate_data_for_request design suggested by
Anders; we only support a handful of data types, all of which are
correctly encoded automatically by jQuery.

Fixes part of #18035.
2021-04-09 16:27:30 -07:00
Abhijeet Prasad Bodas 152508e346 mute user: Reduce two database fetches when unmuting to one.
Previously, when unmuting a user, we used to make
two database fetches - one to verify that the user
is has been muted before, and one while actually
unmuting the user.

This reduces that to one, by passing around the
`MutedUser` object fetched in the first round.

Since the new function returns `Optional[MutedUser]`,
we need to use a hack for events tests, because
mypy does not yet use the type inferred from
`assert foo is not None` in nested functions like lambdas.
See python/mypy@8780d45507.
2021-04-08 23:04:28 -07:00
Tim Abbott 1470dd9105 api: Delete ancient fetch_google_client_id endpoint.
This was used by the old native Zulip Android app
(zulip/zulip-android).  That app has been undeveloped for enough years
that we believe it no longer functions; as a result, there's no reason
to keep a prototype API endpoint for it (that we believe never worked).
2021-04-08 06:10:39 -07:00
Tim Abbott fdf44f19b2 api: Delete ancient generate_204 endpoint.
This endpoint was needed by the ancient pre-electron desktop app
written in QT; we removed support for that in practice a long time
ago, and even the custom error messages for it in
5a22e73cc6.

So we can delete this endpoint as well.
2021-04-08 06:10:39 -07:00
sahil839 39c9845cb0 streams: Raise same error for all values of invite_to_stream_policy.
We keep the error message same for all cases when a user is not
allowed to subscribe others for all values of invite_to_stream_policy.

We raise error with different message for guest cases because it
is handled by decorators. We aim to change this behavior in future.

Explaining the details in error message isn't much important as
we do not show errors probably in API only, as we do not the show
the options itself in the frontend.
2021-04-07 14:51:00 -07:00
sahil839 adec45777d invite: Raise same error for all values of invite_to_realm_policy.
We keep the error message same for all cases when a user is not
allowed to invite others for all values of invite_to_realm_policy.

We raise error with different message for guest cases because it
is handled by decorators. We aim to change this behavior in future.

Explaining the details in error message isn't much important as
we do not show errors probably in API only, as we do not the show
the options itself in the frontend.
2021-04-07 14:51:00 -07:00
Anders Kaseorg f0e655f1d8 request: Rename validator parameter of REQ to json_validator.
This makes it much more clear that this feature does JSON encoding,
which previously was only indicated in the documentation.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 14:13:06 -07:00
Anders Kaseorg 93d2ae8092 request: Remove redundant str_validator=check_string from REQ().
REQ(str_validator=check_string) is equivalent to the default behavior
of REQ().

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 14:13:03 -07:00
Tim Abbott 2c9a1e085c server: Add defensive assertions for permission checks. 2021-04-07 09:22:45 -07:00
sahil839 554ce5bf0d realm: Add moderators and full members option in invite_to_realm_policy.
We add moderators and full members option to invite_to_realm_policy
by using COMMON_POLICY_TYPES and use can_invite_others_to_realm helper
added in previous commit. This commit only does the backend work,
frontend work will be done in separate commit.
2021-04-07 09:05:16 -07:00
sahil839 4c8339fa8c settings: Replace invite_by_admins_policy with invite_to_realm_policy.
This commit replaces invite_by_admins_policy, which was a bool field,
with a new enum field invite_by_realm_policy.

Though the final goal is to add moderators and full members option
using COMMON_POLICY_TYPES, but this will be done in a separate
commit to make this easy for review.
2021-04-07 09:02:33 -07:00
Dinesh cd62b40f20 typing: Refactor view to support message_type.
For extending typing notifications support for streams.
2021-04-07 00:17:30 -07:00
Sumanth V Rao 1ac8fe7538 events/tests/api: Send realm_playground events to clients.
We send the whole data set as a part of the event rather than
doing an add/remove operation for couple of reasons:
    * This would make the client logic simpler.
    * The playground data is small enough for us to not worry
      about performance.

Tweaked both `fetch_initial_state_data` and `apply_events` to
handle the new playground event.

Tests added to validate the event matches the expected schema.

Documented realm_playgrounds sections inside /events and
/register to support our openapi validation system in test_events.

Tweaked other tests like test_event_system.py and test_home.py
to account for the new event being generated.

Lastly, documented the changes to the API endpoints in
api/changelog.md and bumped API_FEATURE_LEVEL.

Tweaked by tabbott to add an `id` field in RealmPlayground objects
sent to clients, which is essential to sending the API request to
remove one.
2021-04-06 20:56:58 -07:00
Sumanth V Rao d2e5b62dce realm/playground: Add API endpoint for deleting a playground entry.
Similar to the previous commit, we have added a `do_*` function
which does the deletion from the DB. The next commit handles sending
the events when both adding and deleting a playground entry.

Added the openAPI format data to zulip.yaml for DELETE
/realm/playgrounds/{playground_id}. Also added python and curl
examples to remove-playground.md.

Tests added.
2021-04-07 08:20:53 +05:30
Sumanth V Rao 251b415987 realm/playground: Add API endpoint for creating playground entry.
This endpoint will allow clients to create a playground entry
containing the name, pygments language and url_prefix for the
playground of their choice.

Introduced the `do_*` function in-charge of creating the entry in
the model. Handling the process of sending events which will be
done in a follow up commit.

Added the openAPI format data to zulip.yaml for POST
/realm/playgrounds. Also added python and curl examples for using
the endpoint in its markdown documented (add-playground.md).

Tests added.
2021-04-07 08:20:53 +05:30
Abhijeet Prasad Bodas 3bfcaa3968 mute user: Add backend infrastructure code.
Adds backend code for the mute users feature.
This is just infrastructure work (database
interactions, helpers, tests, events, API docs
etc) and does not involve any behavioral/semantic
aspects of muted users.

Adds POST and DELETE endpoints, to keep the
URL scheme mostly consistent in terms of `users/me`.

TODOs:
1. Add tests for exporting `zulip_muteduser` database table.
2. Add dedicated methods to python-zulip-api to be used
   in place of the current `client.call_endpoint` implementation.
2021-04-06 18:44:08 -07:00
Abhijeet Prasad Bodas 52a86d9604 linkifiers: Use dictionaries for internal structures.
This change does not affect the API in anyway.
All internal code now uses dictionaries to
denote a linkifier, instead of tuples.
2021-04-05 18:16:08 -07:00
Abhijeet Prasad Bodas 68fe912c63 refactor: Rename most of "filter" to "linkifier".
After this only the database table, events,
and API endpoints remain.
2021-04-05 18:14:07 -07:00
Abhijeet Prasad Bodas a49dc67d8e refactor: Rename backend files to use "linkifier".
This is a prep change to eventually completely
replace the term "filter" with "linkifier" in
the codebase.

This only renames files. Code changes will be
done in further commits.
2021-04-05 18:14:07 -07:00
Aman Agrawal 5e83965e80 giphy: Use GIPHY web SDK to allow inserting GIFs in compose box.
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.

We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
2021-04-05 15:04:49 -07:00
Siddharth Asthana ddbc6d7662 support: Create RealmAuditLog when realm reactivation email is sent. 2021-04-03 08:01:46 -07:00
shanukun 0bf067b681 refactor: Make acting_user a mandatory kwarg for bulk_add_subscriptions. 2021-04-02 14:44:41 -07:00
shanukun 4b00e5da72 refactor: Make acting_user a mandatory kwarg for do_deactivate_realm. 2021-04-02 14:44:41 -07:00
PIG208 8002c4ff10 python: Use the correct typing for update_message_backend.
This is likely a typo introduced in e2c14724d9.
2021-04-01 17:49:28 -07:00
Tim Abbott 08116a17b0 typing: Move to parameter validation to view code. 2021-04-01 08:30:47 -07:00
Dinesh ddca602123 typing_notifications: Do op validation in view function.
Instead of validating `op` value later, this commit does that
in `REQ`.

Also helps avoiding duplication of this validation when
stream typing notifications feature is added.
2021-04-01 07:50:02 -07:00
Riken Shah 08212ef74a puppeteer_tests: Remove login test.
This commit deletes the `01-login.ts` test because it was
redundant, We are already checking for log-in in all the
other tests.
2021-03-31 16:55:54 -07:00
Mateusz Mandera f329878376 migrations: Subscription.is_user_active denormalization - step one.
This adds the is_user_active with the appropriate code for setting the
value correctly in the future. In the following commit a migration to
backfill the value for existing Subscriptions will be added.

To ensure correct user_profile.is_active handling also in tests, we
replace all direct .is_active mutation with calls to appropriate
functions.
2021-03-30 09:19:03 -07:00
sahil839 bc42eab925 settings: Add moderators-only option for invite_to_stream_policy.
The moderators-only option was actually added in the previous
commit for create_stream_policy as we use the same function
'has_permission' for both the policies. But we add the error
handling code and tests for moderators-only option in this
commit.
2021-03-30 09:06:20 -07:00
Tim Abbott 459bf379c2 dev_panel: Use correct content helper for portico pages.
In 1a12e112d9, this page was converted
to use portico styling, but we intentionally left this page not using
the portico_content class since we didn't want the header/footer.

We still don't want the header/footer clutter, so instead, we achieve
that same goal using the isolated_page flag.
2021-03-26 09:40:40 -07:00
Tim Abbott f5b7e5d934 puppeteer_tests: Use POST for flush_caches call. 2021-03-25 12:58:36 -07:00
Riken Shah 1d23d13ef5 urls: Add /flush_caches URL to remove all the server-side cache.
This endpoint is introduced for the puppeteer test, to
clear all the caches after each run.
2021-03-25 12:58:36 -07:00
tushar912 98a6bdbd4c custom profile fields: Rename "SELECT" field validator.
Rename the "SELECT" field validator so that it can be reused
with the upcoming "SELECT_MULTIPLE" field.
2021-03-24 12:54:51 -07:00
tushar912 b220d29fed custom profile fields: Rename "CHOICE" to "SELECT" in backend.
Rename the "CHOICE" field to "SELECT" in backend. This is
done to improve readability as a prep for the upcoming
"SELECT_MULTIPLE" field.
2021-03-24 12:54:51 -07:00
Tushar912 55de66f944 api: Support user_id in get_user_presence_endpoint.
This is part of our general process of replacing emails, which are not
static with time, with user_ids when referring to users in the API.

We still keep the `email` reference option, since it can be useful for
linking third-party applications to Zulip on an intranet that might
have a user's corporate email handy and not want to do the extra round
trip to lookup the user.

The name of the parameter, user_id_or_email, was chosen to to make it
clear that the default/preferred option is user_id.

Fixes #14304.
2021-03-11 20:14:04 -08:00
Aman Agrawal e587c029f6 display_settings: Allow user to set default_view.
TextField is used to allow users to set long stream + topic narrow
names in the urls.

We currently restrict users to only set "all_messages" and
"recent_topics" as narrows.

This commit achieves 3 things:
* Removes recent topics as the default view which loads when
  hash is empty.
* Loads default_view when hash is empty.
* Loads default_view on pressing escape key when it is unhandled by
  other present UI elements.

NOTE: After this commit loading zulip with an empty hash will
automatically set hash to default_view.  Ideally, we'd just display
the default view without a hash, but that involves extra complexity.

One exception is when user is trying to load an overlay directly,
i.e. zulip is loaded with an overlay hash. In this case,
we render recent topics is background irrespective of default_view.

We consider this last detail to be a bug not important enough to block
adding this setting.
2021-03-11 18:09:08 -08:00
Mateusz Mandera 51d7f24d20 actions: Remove realm argument to internal_send_stream_message.
The argument is redundant.
2021-02-23 15:26:47 -08:00
Tushar912 dfafdda9b3 api: Add REST API endpoint for looking up a user by email address.
Add new rest api endpoint GET users/{email} for looking up a user by
email, which is useful especially for corporate API applications that
might already have a user's email address.

Fixes #14302.
2021-02-15 17:38:33 -08:00
Anders Kaseorg d001676728 streams: Fix compose_views type safety.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-15 17:05:28 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
m-e-l-u-h-a-n 0e6343c071 users: Clarify readability issues related to access_user_by_id.
zerver/lib/users.py has a function named access_user_by_id, which is
used in /users views to fetch a user by it's id. Along with fetching
the user this function also does important validations regarding
checking of required permissions for fetching the target user.

In an attempt to solve the above problem this commit introduces
following changes:
1. Make all the parameters except user_profile, target_user_id
   to be keyword only.
2. Use for_admin parameter instead of read_only.
3. Adds a documentary note to the function describing the reason for
   changes along with recommended way to call this function in future.
4. Changes in views and tests to call this function in this changed
   format.

Changes were tested using ./tools/test-backend.

Fixes #17111.
2021-02-05 17:31:45 -08:00
Alex Vandiver 3381fad258 registration: Stop enqueueing to the signups queue.
c2526844e9 removed the `signups` queue
worker, and the command-line tool that enqueues to it -- but not the
automated process that enqueues during signups itself.

Remove the signup, since it is no longer in use.
2021-01-24 09:42:55 -08:00
Alex Vandiver 08d716c741 registration: Re-use the redirect_to_email_login_url helper.
In the case of reusing a registration link, reuse the
redirect_to_email_login_url helper.  This does have the side effect of
now showing a "you've already registered" note, which did not happen
previously, but that seems probably for the best, since the user did
just click a "register" link.
2021-01-13 11:28:32 -08:00
Alex Vandiver ad3d25103b registration: Pre-fill the email when redirecting to login.
ecfafc05c0 shifted to using a different paramter name to hint that
the user had previously signed up -- and in so doing also stopped
pre-filling the "email" box.  Also send along the email box, to save
users time.
2021-01-13 11:28:32 -08:00
Tushar912 c60f48c889 registration: Move "already in realm" check outside of validation.
Checking for `validate_email_not_already_in_realm` again (after the
form already did so), but only in the case that the form fails to
validate, means that we may be spending time pushing totally invalid
emails to the DB to check.  In the case of emails containing nulls,
this can even trigger a 500 error from PostgreSQL.

Stop calling `validate_email_not_already_in_realm` in the form
validation. The form is currently only used in two places -- in
`accounts_home` and in `maybe_send_to_registration`.  The latter is
only called if the address is known to not currently have an account,
so checking in there is unnecessary; and in the former case, we wish
different behaviour (the redirect) than just validation failure, which
is all the validator can do.

Fixes #17015.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-01-13 11:28:32 -08:00
sushant52 6f0e8a9888 auth: Handle the case of invalid subdomain at various points.
Fixes #16770.
2021-01-11 22:29:50 -08:00
Mateusz Mandera 160cc5120a api: Require can_create_users permission to create users via API.
Allowing any admins to create arbitrary users is not ideal because it
can lead to abuse issues.  We should require something stronger that
requires the server operator's approval and thus we add a new
can_create_users permission.
2020-12-21 13:20:21 -08:00
Mateusz Mandera d0dc04a093 models: Rename is_api_super_user to can_forge_sender, 2020-12-21 13:15:39 -08:00
Anders Kaseorg a054f57af6 message: Bundle message stripping, validation, and truncation.
We always want to do these at the same time.  Previously, message
editing did too much stripping (fixes #16837) and failed to check for
NUL bytes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-18 17:44:13 -08:00
Siddharth Asthana daac7536f3 accounts/deactivated: Show deactivated_redirect url if present
If a user visits a realm which has been deactivated and it's
deactivated_redirect field is set, we should have a message telling the
user that the realm has moved to the deactivated_redirect url.
2020-12-14 21:04:52 -08:00
Sundar Guntnur cbb7fb8ac0 anchor_value: Fix parsing of large anchor values.
This handles the conditions when anchor values are larger than
LARGER_THAN_MAX_MESSAGE_ID by clamping them down to it.  Also added
tests for the function parse_anchor_value.

Fixes #16768.
2020-12-02 11:00:22 -08:00
Mateusz Mandera 43a0c60e96 exceptions: Make RateLimited into a subclass of JsonableError.
This simplifies the code, as it allows using the mechanism of converting
JsonableErrors into a response instead of having separate, but
ultimately similar, logic in RateLimitMiddleware.
We don't touch tests here because "rate limited" error responses are
already verified in test_external.py.
2020-12-01 13:40:56 -08:00
Anders Kaseorg 13e35bfa94 mypy: Use sqlalchemy-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Anders Kaseorg 8e0240300a message_fetch: Skip intermediate mutation in limit_query_to_range.
This avoids extra mypy annotations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Mateusz Mandera 4f47f35cb4 auth: Handle the case of invalid subdomain at /fetch_api_key endpoint. 2020-11-13 16:43:17 -08:00
Steve Howell e31326c823 refactor: Extract get_digest_context.
This eliminates the union type and boolean parameter,
and it makes it a bit easier to migrate to a
bulk-get approach.
2020-11-05 09:36:59 -08:00
Aman Agrawal 87cdd8433d home: Allow logged out user through home.
We allow user to load webapp without log-in. This is only
be enabled for developed purposes now. Production setups will
see no changes.
2020-11-02 17:07:12 -08:00
Anders Kaseorg 18d0e4664c python: Replace binascii with bytes.hex to skip some decode operations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 86e8d81c7f python: Skip unnecessary decode before JSON parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
sahil839 b29d39195c streams: Do not allow default streams to be private.
We now do not allow to make a stream private which is already
a default stream.
2020-10-29 15:47:32 -07:00
sahil839 557ca0802c streams: Do not allow private streams to be set as default.
We now do not allow to set a private stream as default.
2020-10-29 15:43:37 -07:00
Hemanth V. Alluri 99cf37dc51 drafts: Make the ID of the draft a part of the draft dict.
Then because the ID is now part of the draft dict, we can
(and do) change the structure of the "drafts" parameter
returned from `GET /drafts` from an object (mapping ID to
data) to an array.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
Vishnu KS fdea49742c apps: Use GitHub API for generating the web app download link. 2020-10-28 23:04:14 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Anders Kaseorg 1352f2f233 python: Replace manual quote_plus usage with urlencode.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 0b288f92c9 timezone: Remove get_timezone wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 0134112b51 timezone: Remove get_all_timezones wrapper.
Both callers want a set, and pytz already provides all_timezones_set.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 96dee7e40b decorator: Unfork redirect_to_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-25 14:58:53 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg f461a64a6b i18n: Fix some ineffective calls to ugettext at top level.
Translation has no effect when we don’t yet know what language we’re
translating for.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-18 14:31:15 -07:00
Steve Howell e1bcf6124f refactor: Remove recipient from access_stream_by_name. 2020-10-16 12:58:11 -07:00
Steve Howell a51b483f1a performance: Remove recipient from access_stream_by_id.
The Recipient table is now kind of useless for
stream-related operations, since we have
recipient_id on Stream now.
2020-10-16 12:58:11 -07:00
Steve Howell bfd6e2b1fd refactor: Use recipient_id to get topic history. 2020-10-16 12:58:11 -07:00
Steve Howell 3685fcc701 refactor: Remove recipient arg for do_mute_topic. 2020-10-16 12:58:11 -07:00
Steve Howell 65dbee4837 minor: Ask for recipient_id, not recipient. 2020-10-16 12:58:11 -07:00
Steve Howell 31eb97ddde performance: Fix do_mark_stream_messages_as_read.
This function no longer asks for data that it
doesn't need.
2020-10-16 12:58:11 -07:00
Steve Howell 116a441bc5 refactor: Introduce SubInfo class.
This class removes a lot of the annoying tuples
we were passing around.

Also, by including the user everywhere, which
is easily available to us when we make instances
of SubInfo, it sets the stage to remove
select_related('user_profile').
2020-10-14 10:53:10 -07:00
Steve Howell c29ba75135 refactor: Extract send_messages_for_new_subscribers.
This is a pure extraction, except that I remove a
redundant check that `len(principals) > 0`.  Whenever
that value is false, then `new_subscriptions` will
only have one possible entry, which is the current
user, and we skip that in the loop.
2020-10-13 18:54:55 -04:00
Steve Howell 9df9934ed6 refactor: Pass realm to bulk_add_subscriptions.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.

This may be overkill--I would also be happy if we
just included the assertions from this commit.
2020-10-13 11:28:17 -07:00
Anders Kaseorg 7f69c1d3d5 python: Catch specific exceptions from requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 17ac17286c python: Catch specific exceptions from subprocess.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg aabef3d9be python: Catch specific exceptions from orjson.
Followup to #16120.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 1346c5397a zephyr: Use correct shell quoting for ssh.
ssh always runs its command through a shell (after naïvely joining
multiple arguments with spaces), so it needs an extra level of shell
quoting.  This should have no effect because we already validated user
with a regex, but it’s better for escaping to be locally correct in
case the context changes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 82593338ba report: Show Git commit in a way that works for merges.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
sahil839 4e8f5b5b31 streams: Change access_stream_for_delete_or_update to also return sub.
We modify access_stream_for_delete_or_update function to return
Subscription object also along with stream. This change will be
helpful in avoiding an extra query to get subscription object in
code for updating subscription role.
2020-10-08 17:07:30 -07:00
Mateusz Mandera 6e83bcc0d5 custom_profile_fields: Don't allow leading/trailing whitespaces.
Allowing such whitespaces can lead to hard to debug issues e.g. with
ldap sync.
2020-10-02 14:58:06 -07:00
Tim Abbott 351d73ac5a home: Pass realm to build_page_params_for_home_page_load.
This is preparation for this needing to be a separate parameter from
the user.
2020-10-01 15:00:36 -07:00
sahil839 78b98d8067 realm: Add wildcard_mention_policy setting.
We add a new wildcard_mention_policy setting to handle wildcard
mentions in large streams, with a wide range of policies available to
organizations.

We set the default to the safe option for preventing accidental spam:
only stream administrators being able to use wildcard mentions in
large streams.
2020-10-01 12:18:03 -07:00
Steve Howell c199571112 mypy: Add StreamDict.
This requires us to rework the view code a little
bit to explicitly assign fields.
2020-09-29 16:49:10 -07:00
Dinesh 232eb8b7cf auth: Render config error page on configuration error.
We previously used to to redirect to config error page with
a different URL. This commit renders config error in the same
URL where configuration error is encountered. This way when
conifguration error is fixed the user can refresh to continue
normally or go back to login page from the link provided to
choose any other backend auth.

Also moved those URLs to dev_urls.py so that they can be easily
accessed to work on styling etc.

In tests, removed some of the asserts checking status code to be 200
as the function `assert_in_success_response` does that check.
2020-09-25 16:16:17 -07:00
Vishnu KS a888e65ea4 i18n: Don't include email tags in translation strings. 2020-09-25 15:53:26 -07:00
Anders Kaseorg faf600e9f5 urls: Remove unused URL names and shorten others.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Aman Agrawal 37e77c77e2 message_fetch: Better handle unauthorized requests.
We raise MissingAuthenticationError now, which adds
`www_authenticate=session` header to the error response. This
stops modern web-browsers from displaying a login form everytime
a 401 response it sent to the client.
2020-09-21 16:07:43 -07:00
Aman Agrawal 2bc3924672 move_topic_to_stream: Allow moving to/between/from private streams.
Fixes #16284.

Most of the work for this was done when we implemented correct
behavior for guest users, since they treat public streams like private
streams anyway.

The general method involves moving the messages to the new stream with
special care of UserMessage.

We delete UserMessages for subs who are losing access to the message.
For private streams with protected history, we also create UserMessage
elements for users who are not present in the old stream, since that's
important for those users to access the moved messages.
2020-09-14 15:00:55 -07:00
Anders Kaseorg 463929f349 urls: Migrate re_path routes to path.
Django treats path("<name>") like re_path(r"(?P<name>[^/]+)") and
path("<path:name>") like re_path(r"(?P<name>.+)").

This is more readable and consistent than the mix of slightly
different regexes we had before, and fixes various bugs:

• The r'apps/(.*)$' regex was missing a start anchor ^, so it
  incorrectly matched all URLs that included apps/ as a substring
  anywhere.
• The r'accounts/login/(google)/$' regex was missing a start anchor ^,
  so it incorrectly matched all URLs that ended with
  accounts/login/google/.
• The type annotation of zerver.views.realm_export.delete_realm_export
  takes export_id as an int, but it was previously passed as a string.
• The type annotation of zerver.views.users.avatar takes medium as a
  bool, but it was previously passed as a string.
• The [0-9A-Za-z]+ pattern for uidb64 was missing the - and _
  characters that can validly be part of a base64url encoded
  string (although I think the id is actually a decimal integer here,
  in which case only 012345ADEIMNOQTUYcgjkwxyz are present in its
  base64url encoding).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 14:29:26 -07:00
Aman Agrawal ed8796f23f report: Allow anonymous queries to report (un)narrow times. 2020-09-11 16:57:27 -07:00
Anders Kaseorg b7b7475672 python: Use standard secrets module to generate random tokens.
There are three functional side effects:

• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.

• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.

• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.

(The base32 alphabet is A-Z 2-7.  We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-09 15:52:57 -07:00
Aman c114447e11 topic_history: Allow anonymous access in web-public streams.
For web-public streams, clients can access full topic history
without being authenticated. They only need to additionally
send "streams:web-public" narrow with their request like all
the other web-public queries.
2020-09-04 10:17:56 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bef46dab3c python: Prefer kwargs form of dict.update.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Aman Agrawal d9431a5e66 exceptions: Raise InvalidSubdomainError when realm is invalid.
When user requests for a realm that doesn't exists, we raise
a InvalidSubdomainError.

This reduces our effort at repeatedly ensuring realm is valid
in request in web-public queries.
2020-09-03 10:45:14 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Tim Abbott 992c6126a8 docs: Update many references to Casper. 2020-08-30 17:16:02 -07:00
Kartik Srivastava b9b9a79382 user_groups: Use 'path_only=True' in REQ for 'user_group_id' in views.
'user_groups' endpoints are currently under 'pending_endpoints' in
test_openapi.py (even after being documented except one), due to the
'user_group_id' and 'group_id' parameter name mismatch in the
url config and the view functions.

This commit includes 'path_only=True' for 'user_group_id' parameter in
views to avoid the failure of 'test_openapi_arguments', in
test_openapi.py, which excludes the path parameters. This is a prep
commit for renaming 'group_id' to 'user_group_id' in the documentation
and removing the 'user_groups' endpoints from 'pending_endpoints'.
2020-08-26 15:40:19 -07:00
Aman c3a8492697 topic: Rename get_topic_history_for_web_public_stream. 2020-08-25 17:01:12 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00
Tim Abbott 28b43b4edc message_fetch: Refactor get_base_query_for_search. 2020-08-12 16:33:59 -07:00
Tim Abbott 221cd496c6 NarrowBuilder: self.user_profile.realm -> self.realm. 2020-08-12 16:33:59 -07:00
Tim Abbott 35bd14574d message_fetch: Refactor public-streams query logic. 2020-08-12 16:33:59 -07:00
Tim Abbott d2e191b94e NarrowBuilder: Rename self.user_realm to self.realm.
The previous name will be very confusing in upcoming commits where the
UserProfile becomes Optional but Realm is not (E.g. for web-public
queries).
2020-08-12 16:33:59 -07:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Alex Vandiver 153f16ee6a links: Flatten the set into a list before serializing into the queue.
orjson does not transparently do this set-to-list translation, unlike
ujson.
2020-08-12 11:42:24 -07:00
Kartik Srivastava 63173d5554 api: Return 'user_id' in 'POST /users' response.
This adds 'user_id' to the simple success response for 'POST /users'
api endpoint, to make it convenient for API clients to get details
about users they just created.  Appropriate changes have been made in
the docs and test_users.py.

Fixes #16072.
2020-08-11 16:40:12 -07:00
Aman Agrawal 94839c9492 NarrowBuilder: Directly use denormalized 'recipient_id' of streams.
'recipient_id' was is now directly accessible and we can now use
it instead of doing an extra query to Recipient model.
2020-08-11 11:22:22 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 114b0a2982 message_edit: Output a list, not a reversed iterator.
A list_reverseiterator object is not JSON serializable, and orjson
enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Alex Vandiver 816f91ae27 auth: Remove redirect on invalid realm during password reset.
This redirect was never effective -- because of the
HostDomainMiddleware, all requests to invalid domains have their
actual results thrown away, and replaced by an "Invalid realm" 404.
These lines are nonetheless _covered_ by coverage, because they do
run; the redirect is simply ineffective.  This can be seen by the test
that was added with them, in c8edbae21c, actually testing the contents
for the invalid realm wording, not the "find your accounts" wording.
2020-08-11 10:27:29 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Anders Kaseorg 36bae2e86c message_fetch: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 80c0e60edc integrations: Decode webhook responses to str.
bytes is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 28143c0ee2 drafts: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Mateusz Mandera bb46b21123 find_account: Fix the email search query.
The search should be case-insensitive.
2020-08-02 12:37:20 -07:00
Clara Dantas a9af80d7a2 streams: Make /streams endpoint return also web-public streams.
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
2020-07-29 17:52:36 -07:00
Hemanth V. Alluri 21c9adbb3c drafts: Add an API endpoint for fetching drafts.
This endpoint will allow a user to fetch their drafts.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 630fd10be1 drafts: Add an API endpoint for deleting drafts.
This endpoint will allow a user to delete a single draft.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 2d307c760e drafts: Add an API endpoint for editing drafts.
This endpoint will allow a user to edit a single draft.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri a0f71b7458 drafts: Add an API endpoint for creating drafts.
This endpoint will allow a user to create drafts in bulk.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Dinesh 9583554d44 auth: Add check_config for apple auth.
Apple has some other obligatory settings other than key and secret.
To handle that this commit adds a function check_config() similar
to that of SAML.
2020-07-28 17:12:49 -07:00
Gittenburg 87d8a54b91 help: Stop collapsing sidebar sections.
Three reasons:

1. The sliding was disorienting.
2. The collapsing disallowed searching for other pages with Ctrl+F.
3. The collapsing mechanism wasn't accessible (not usable with the
   keyboard / no ARIA tags).

Tweaked by tabbott to center the left sidebar on the selected page.

Part of #15948.
2020-07-28 16:28:58 -07:00
majordwarf ad23c6e808 emoji: Switch to `user_profile.realm` for cross-realm reactions.
A temporary fix to enable cross-realm emoji reactions till `Message.realm`
is denormalized.

Fixes #15866.
2020-07-24 13:11:27 -07:00
arpit551 0d79b55b2e audit_log: Record RealmAuditLog in do_change_notification_settings.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:04:40 -07:00
arpit551 94d2de8b4a audit_log: Record RealmAuditLog while changing default streams.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:31 -07:00
arpit551 753713fde7 audit_log: Record RealmAuditLog in do_change_subscription_property.
Removed logging with log_subscription_property_change
and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:29 -07:00
arpit551 e6f54a3278 audit_log: Record RealmAuditLog in do_change_icon_source.
Removed logging with  log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 11:56:40 -07:00
arpit551 ccdf52fef6 audit_log: Log RealmAuditLog for realm notification properties.
Log RealmAuditLog for do_set_realm_notifications_stream and
do_set_realm_signup_notifications_stream function.
Added tests for the same.
2020-07-24 11:51:47 -07:00
arpit551 8f6a1c3f40 actions.py: Removed do_set_realm_message_deleting function.
Used do_set_realm_property function instead of
do_set_realm_message_deleting and removed it.
2020-07-24 11:51:47 -07:00
arpit551 422fa0ff7d audit_log: Log RealmAuditLog for do_set_realm_message_editing.
Log RealmAuditLog for do_set_realm_message_editing. Added tests for
same.
2020-07-24 11:51:46 -07:00
Vishnu KS 9e0ff58a6d team: Rename contrib to contributors in page_params. 2020-07-23 10:22:28 -07:00
Steve Howell 1fa6ae1e16 refactor: Extract build_page_params_for_home_page_load. 2020-07-22 17:15:03 -07:00
Steve Howell 27072289ce refactor: Extract zerver/lib/home.py.
The two functions extracted here are mostly
copied verbatim, but we use dataclasses
to marshal the values back.
2020-07-22 17:15:02 -07:00
Steve Howell 26e4d34e81 refactor: Move code up higher in home view.
The two pieces of code here don't need to be
intermingled with the bigger task of building
page_params.
2020-07-22 17:13:53 -07:00
Vishnu KS 67bacd6e31 billing: Don't allow guest users to upgrade. 2020-07-22 16:57:49 -07:00
Tim Abbott 9efec1f929 auth: Clean up DevAuthBackend error messages.
We had a user confused by these error messages, which suggested they
needed to enable something rather than using a development
environment.
2020-07-21 12:55:11 -07:00
Tim Abbott 6be3fca037 auth: Remove short_name from LDAP API.
As best I can tell, we fetched this field and then ignored it, so
unlike the last few commits, this is more a code cleanup than a
functional change.
2020-07-17 11:28:08 -07:00
Steve Howell c44500175d database: Remove short_name from UserProfile.
A few major themes here:

    - We remove short_name from UserProfile
      and add the appropriate migration.

    - We remove short_name from various
      cache-related lists of fields.

    - We allow import tools to continue to
      write short_name to their export files,
      and then we simply ignore the field
      at import time.

    - We change functions like do_create_user,
      create_user_profile, etc.

    - We keep short_name in the /json/bots
      API.  (It actually gets turned into
      an email.)

    - We don't modify our LDAP code much
      here.
2020-07-17 11:15:15 -07:00
Steve Howell c60f4236a9 api: Do not require short_name to create user.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.

An upcoming commit will eliminate it from the
database.
2020-07-17 11:15:15 -07:00
arpit551 a2428017e9 audit_log: Log RealmAuditLog in do_set_realm_authentication_methods.
Log RealmAuditLog in do_set_realm_authentication_methods and added
tests for it.
2020-07-06 17:38:20 -07:00
arpit551 ba92666dbd audit_log: Log RealmAuditLog in do_set_realm_property.
Log RealmAuditLog in do_set_realm_property and do_remove_realm_domain.
Tests for the changes are written in test_events because it will save
duplicate code for test_change_realm_property.
2020-07-06 17:38:19 -07:00
arpit551 6af337c07e audit_log: Log RealmAuditLog when stream is deactivated. 2020-07-06 17:32:11 -07:00
arpit551 87aaa84b42 audit_log: Log acting_user in do_change_user_role. 2020-07-06 17:32:11 -07:00
arpit551 01f12b9fc2 audit_log: Log acting_user in user creation and user activation. 2020-07-06 17:32:09 -07:00
arpit551 653928bdfe audit_log: Log acting_user in do_change_avatar_fields. 2020-07-06 17:24:18 -07:00
arpit551 2279fef316 audit_log: Log acting_user in do_change_logo_source. 2020-07-06 17:24:18 -07:00
Anders Kaseorg 2fb96e94f1 registration: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Steve Howell 50cbe1b19b minor: Rename var to max_message_id.
We are phasing out the concept of pointer.
2020-07-03 13:08:40 +00:00
Vishnu KS cbd73a6dc1 home: Use get_customer_by_realm for getting customer. 2020-07-01 16:45:38 -07:00
Vishnu KS 4c6350fa4b billing: Add option to request a sponsorship in /upgrade. 2020-07-01 16:45:38 -07:00
Vishnu KS a74cdf3123 18n: Cleanup strings to be translated in add_subscriptions_backend.
Also expand the test to check the details of the user who created the
stream.
2020-07-01 11:03:29 -07:00
Vishnu KS eb50ecf07a 18n: Cleanup strings to be translated in you_were_just_subscribed_message. 2020-07-01 11:03:29 -07:00
Anders Kaseorg f2e7076e2a decorator: Replace type: ignore with cast, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg dcbc8e66fb decorator: Remove authenticated_json_post_view.
It’s effectively a combination of require_post with
authenticated_json_view and has one use.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg d6ecb67d75 report: Split hostname from port more carefully.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-29 22:19:47 -07:00
Mohit Gupta bbec1f72a8 refactor: Rename bugdown to markdown in message_edit.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 15:03:20 -07:00
Mateusz Mandera 9392e8bec7 emails: Add a custom header specifying the organization when possible.
Closes #15135.
2020-06-28 13:33:02 -07:00
Steve Howell 69be97e365 pointer: Remove pointer from API and page_params.
There is still some miscellaneous cleanup that
has to happen for things like analytics queries
and dead code in node tests, but this should
remove the main use of pointers in the backend.

(We will also still need to drop the DB field.)
2020-06-27 16:44:38 -07:00
Mohit Gupta 3f5fc13491 refactor: Rename zerver.lib.bugdown to zerver.lib.markdown .
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
2020-06-26 17:08:37 -07:00
Vishnu KS cc0b3a08c9 i18n: Set the correct language for translation in add_subscriptions_backend. 2020-06-26 14:57:18 -07:00
Tim April 8e2a79095d mobile: Add support for alternative mobile URI.
Due to authentication restrictions, a deployment may need to direct
traffic for mobile applications to an alternate uri to take advantage
alternate authentication mechansism. By default the standard realm URI
will be usedm but if overridden in the settings file, an alternate uri
can be substituted.
2020-06-26 12:13:26 -07:00
Tim Abbott def6189d53 docs: Document local echo paramters for sending messages. 2020-06-25 14:44:16 -07:00
Steve Howell 2fb67b3f32 refactor: Extract add/remove_subscriptions_schema.
Now we are consistent about validating color/description.

Ideally we wouldn't need to validate the
`streams_raw` parameters multiple times per
request, but the outer function here changes
the error messages to explicitly reference
the "delete" and "add" request variables.

And for the situation where the user-supplied
parameters are correct, the performance penalty
for checking them twice is extremely negligible.

So it's probably fine for now to just make sure
we use the same validators in all the relevant
places.

There's probably some deeper refactor that we
can do to eliminate the whole `compose_views`
scheme.  And it's also not entirely clear to
me that we really need to support the update
endpoint.  But that's all out of the scope of
this commit.
2020-06-25 13:52:59 -07:00
Steve Howell f960df04e8 narrows: Validate negated field. 2020-06-25 13:52:59 -07:00
Steve Howell 80c057d91d REQ: Use check_dict_only in update_user_backend.
Update the REQ check for profile_data in
update_user_backend by tweaking `check_profile_data`
to use `check_dict_only`.

Here is the relevant URL:

    path('users/<int:user_id>', rest_dispatch,
         {'GET': 'zerver.views.users.get_members_backend',

It would be nice to unify the validator
for these two views, but they are different:

    update_user_backend
    update_user_custom_profile_data

It's not completely clear to me why update_user_backend
seems to support a superset of the functionality
of `update_user_custom_profile_data`, but it has
this code to allow you to remove custom profile fields:

    clean_profile_data = []
    for entry in profile_data:
        assert isinstance(entry["id"], int)
        if entry["value"] is None or not entry["value"]:
            field_id = entry["id"]
            check_remove_custom_profile_field_value(target, field_id)
        else:
            clean_profile_data.append({
                "id": entry["id"],
                "value": entry["value"],
            })

Whereas the other view is much simpler:

def update_user_custom_profile_data(
    <snip>
) -> HttpResponse:

    validate_user_custom_profile_data(user_profile.realm.id, data)
    do_update_user_custom_profile_data_if_changed(user_profile, data)
    # We need to call this explicitly otherwise constraints are not check
    return json_success()
2020-06-25 10:54:15 -07:00
Steve Howell 92a552d2ad REQ: Check value in update_user_custom_profile_data.
This tightens our checking of user-supplied data
for this endpoint:

    path('users/me/profile_data', rest_dispatch,
         {'PATCH': 'zerver.views.custom_profile_fields.update_user_custom_profile_data',
         ...

We now explicitly require the `value` field
to be present in the dicts being passed in
here, as part of `REQ`.  There is no reason
that our current clients would be sending
extra fields here, and we would just ignore
them anyway, so we also move to using
check_dict_only.

Here is some relevant webapp code (see settings_account.js):

    fields.push({id: field.id, value: user_ids});
    update_user_custom_profile_fields(fields, channel.patch);

    settings_ui.do_settings_change(method, "/json/users/me/profile_data",
                                   {data: JSON.stringify([field])}, spinner_element);

The webapp code sends fields one at a time
as one-element arrays, which is strange, but
that is out of the scope of this change.
2020-06-25 10:54:15 -07:00
orientor 8ab6182683 fetch_api_key: Return `email` in json_response.
`/api/v1/fetch_api_key`'s response had a key `email` with the user's
delivery email. But its JSON counterpart `/json/fetch_api_key`, which
has a completely different implementation, did not return `email` in
its success response.

So to avoid confusion, the non-API endpoint, `/json/fetch_api_key`
response has been made identical with it's `/api` counterpart by
adding the `email` key. Also it is safe to send as the calling user
will only see their own email.
2020-06-24 15:13:31 -07:00
Tim Abbott b7b173d2ae realm: Fix type-checking for message_retention_days.
The best way to handle this is to have the potentially string-typed
value be a separate variable.
2020-06-24 11:01:34 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 7b3f4755ff streams: Fix bug that allows 0 value for message_retention_days. 2020-06-24 10:38:58 -07:00
Hashir Sarwar 9b8521faee people: Enable `user_avatar_url_field_optional` in webapp.
In 5200598a31, we introduced a new
client capability that can be used to avoid unreasonable network
bandwidth consumed sending avatar URLs of long term idle users in
organizations with 10,000s members.

This commit enables this feature and adds support for it to the web
client.
2020-06-24 10:06:35 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 48719cb8c4 message_send: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg f6c73ed45a auth: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg feeb39a55f timezone: Strengthen get_timezone type.
pytz.BaseTzInfo is a subclass of datetime.tzinfo that adds the
localize and normalize methods.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 1d545efac2 custom_profile_fields: Replace a type: ignore with an assert.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg a3a39494e3 video_calls: Annotate a type: ignore with a specific bug URL.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 30ab0524b5 video_calls: Don’t request a scope for the Zoom OAuth session.
As requested by a Zoom reviewer for unclear reasons.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 18:48:35 -07:00
Felix a389c7390d calls: Add Big Blue Button as a Video Call Provider.
Big Blue Button needs an API secret so communication to creating a
room has to be done server side.

Fixes #14763.
2020-06-22 16:19:07 -07:00
Tim Abbott 866965f162 views: Rename messages.py to message_fetch.py.
This completes to effort to split zerver.views.messages into smaller
modules focused on the different core type of work.
2020-06-22 15:08:34 -07:00
Tim Abbott 3657717ade views: Extract message_send.py for sending views. 2020-06-22 15:08:34 -07:00
Tim Abbott 55448ecf24 views: Extract message_flags.py for flags views. 2020-06-22 15:08:34 -07:00
Tim Abbott 4d7550d705 views: Extract message_edit.py for message editing views.
This is a pretty clean extraction of files that lets us shrink one of
our largest files.
2020-06-22 15:08:34 -07:00
MariaGkoulta b10f156250 settings: Add automatic theme detection feature.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.

This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.

Fixes: #14451.

Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
2020-06-21 01:09:01 -07:00
sahil839 63389b3bd3 invite: Add option to invite user as an organization owner.
We can now invite new users as realm owners. We restrict only
owners to invite new users as owners both for single invite
and multiuse invite link. Also, only owners can revoke or resend
owner invitations.
2020-06-20 23:04:59 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c0fa42dd8c complete_zoom_user: Validate state type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 64038163e7 update_user_backend: Validate profile_data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c584462b32 update_storage: Validate storage type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 717e349834 events_register_backend: Validate client_capabilities type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 475fc5910f update_user_custom_profile_data: Validate data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Steve Howell fff2a81536 mypy: Fix payload_url. 2020-06-20 17:09:15 -07:00
Steve Howell 82036741b9 mypy: Fix service_interface 2020-06-20 17:09:15 -07:00
Steve Howell 704c84502b mypy: Fix authentication_methods.
We know the key of the dictionary is a string
here, since we get it via json.
2020-06-20 17:09:15 -07:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell 2e9f689add mypy: Fix annotation for principals. 2020-06-19 21:23:23 -07:00
Tim Abbott 14c0a9a228 messages: Deduplicate non-search code path in messages_in_narrow.
The non-search code path here was simulating the response and escaping
logic from get_search_fields by duplicating what it would do with an
empty set of highlight locations.

We can produce much more readable code by just passing an empty list
of locations in this case.
2020-06-19 15:09:57 -07:00
Steve Howell 7c6ddf90ae mypy: Fix default_language annotation. 2020-06-19 12:35:47 -04:00
Hashir Sarwar 5200598a31 events: Don't send avatar URLs of long term idle users.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.

Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.

This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.

Fixes #15287.
2020-06-18 21:35:16 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Tim Abbott f8ea5f3769 pointer: Remove pointer from GET /users/me.
This cleans up a bit of mess in the Zulip API.
2020-06-18 12:55:59 -07:00
Mateusz Mandera 8d2d64c100 CVE-2020-14215: Fix validation in PreregistrationUser queries.
The most import change here is the one in maybe_send_to_registration
codepath, as the insufficient validation there could lead to fetching
an expired PreregistrationUser that was invited as an administrator
admin even years ago, leading to this registration ending up in the
new user being a realm administrator.

Combined with the buggy migration in
0198_preregistrationuser_invited_as.py, this led to users incorrectly
joining as organizations administrators by accident.  But even without
that bug, this issue could have allowed a user who was invited as an
administrator but then had that invitation expire and then joined via
social authentication incorrectly join as an organization administrator.

The second change is in ConfirmationEmailWorker, where this wasn't a
security problem, but if the server was stopped for long enough, with
some invites to send out email for in the queue, then after starting it
up again, the queue worker would send out emails for invites that
had already expired.
2020-06-16 23:35:39 -07:00
Clara Dantas ddbde66af5 realm: Remove Google Hangouts integration.
Google  has removed the Google Hangouts brand, thus we are removing
them as video chat provider option.
This commit removes Google Hangouts integration and make a migration
that sets all realms that are using Hangouts as their video chat
provider to the default, jitsi.

With changes by tabbott to improve the overall video call documentation.

Fixes: #15298.
2020-06-16 17:02:27 -07:00
Mateusz Mandera 2ac6a8f829 auth: Change the "continue in browser" link in desktop flow end page.
Fixes #14828.
Giving the /subdomain/<token>/ url there could feel buggy if the user
ended up using the token in the desktop app, and then tried clicking the
"continue in browser" link - which had the same token that would now be
expired. It's sufficient to simply link to /login/ instead.
2020-06-16 16:27:53 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Mateusz Mandera f467ab046e messages: Remove redundant "assert is not None" for mypy.
Apparently new mypy is able to infer correctly and doesn't need this
assert anymore.
2020-06-15 16:20:43 -07:00
Aman Agrawal ccc0c8706c delete_message: Support `bulk_message_deletion`.
Fixes #15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
2020-06-15 10:41:12 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Aman Agrawal cda7b2f539 deletion: Add support for bulk message deletion events.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.

Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).

Backend implementation of #15285.
2020-06-14 22:34:00 -07:00
Aman Agrawal 21ac1fb32c register: Refactor to pass client_capabilities directly.
We extract values from client_capabilities directly in
do_events_register where we decide how to process the
extracted variables.
2020-06-14 21:42:50 -07:00
Tim Abbott d97c891afe realm owners: Remove unnecessary duplicate strings. 2020-06-14 21:32:10 -07:00
sahil839 18429cfd29 realm: Allow only owners to set message retention policy for a realm.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.

Dropdown for changing retention policy is disabled in UI for admins also.
2020-06-14 21:23:53 -07:00
sahil839 87e72ac8e2 realm: Allow only owners to configure auth methods for a realm.
This commit adds the restriction on configuring auth methods for
admins. We now allow only owners to configure the auth methods
for realm.
2020-06-14 21:23:51 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Graham Bleaney 461d5b1a3e pysa: Introduce sanitizers, models, and inline marking safe.
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
2020-06-11 12:57:49 -07:00
Graham Bleaney 56345f98cb registration: Incorporate redirect URL building helper function.
This commit adds 'add_query_to_redirect_url' to one additional
function which had not yet been written when
'add_query_to_redirect_url' was introduced. This helper centralizes
URL manipulation for redirects, making it easier to add Pysa
sanitization in subsequent commits.
2020-06-11 12:57:49 -07:00
Kartik Srivastava 8c39ddfd28 api: Add GET /users/{user_id}/subscription/{stream_id} endpoint.
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.

Fixes #14966.
2020-06-10 17:59:14 -07:00
Steve Howell d5cc29755e typing: Accept only user_ids in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.

There may be a very tiny number of mobile
clients that still use the old emails API.

This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.

Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.

The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:

    validator=check_list(check_int)

This also allows us to remove a test hack
related to the API documentation.  (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
2020-06-10 17:50:32 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
sahil839 86b52ef7bf users: Owners can only be deactivated by other organization owners.
This commit adds restriction on deactivation of organization owners.
Only owners can deactivate other organization owners.
2020-06-10 17:33:02 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
sahil839 6d667dbe53 realm owner: Add ability to change realm owner status of user.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.

This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.

We need to apply that restriction both in the role change code path
and the deactivate code path.
2020-06-10 12:49:37 -07:00
Anders Kaseorg 5c9d56d2f7 get_safe_redirect_to: Remove incorrect allowed_hosts setting.
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:51:52 -07:00
Dinesh dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
Tim Abbott 2beaf2cab2 pointer: Remove have_initial_messages code.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history.  However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
2020-06-08 22:36:18 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Anders Kaseorg b1b34b6f6a dev_login: Move owners to the top.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 17:26:34 -07:00
sahil839 7de23b8b5c users: Remove short_name and client_id from get_profile_backend.
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.

* short_name hasn't been present anywhere else in the Zulip API for
  several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
  returned by the register_event_queue process.

The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
2020-06-08 17:01:08 -07:00
sahil839 1f5778bad7 users: Refactor get_profile_backend to be based on format_user_row.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.

To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.

This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
2020-06-08 16:57:44 -07:00
Anders Kaseorg 8e4f22c184 auth: Require algorithms setting for JWT auth.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning.  This is for protecting against
symmetric/asymmetric key confusion attacks.

This is a backwards-incompatible configuration change.

Fixes #15207.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 16:22:25 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 139cb8026f auth: Accept next as POST parameter in POST requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 11:07:32 -07:00
Anders Kaseorg 22178c169e request: Replace type argument hack with a different workaround.
This works in mypy 0.770 and is needed for mypy 0.780.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
sahil839 43bff1f67a streams: Extract removing_someone_else logic to a function.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.

This is a follow up to 2187c84.
2020-06-05 16:04:28 -07:00
Aman Agrawal a096f34cab move_topic_to_stream: Add option to disable breadcrumb messages.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.

User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
2020-06-05 12:28:51 -07:00
Anders Kaseorg 4d04fa3118 compose: Rewrite Zoom video call integration to use OAuth.
This reimplements our Zoom video call integration to use an OAuth
application.  In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.

Fixes #11672.

Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-06-03 16:39:12 -07:00
Anders Kaseorg 7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
sahil839 0b432e26b6 test_subs: Add tests for unsubscribing multiple users from stream.
This commit adds tests for unsusbcribing multiple users from a
stream and fixes the missing coverage issue introduced in 2187c84.
2020-06-02 16:27:22 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Mateusz Mandera 3e7fc17788 auth: Delegate RemoteUser SSO to browser when using the desktop app. 2020-06-02 13:00:17 -07:00
Tim Abbott fbcf6a7d72 streams: Add nocoverage to fix CI. 2020-06-01 16:12:13 -07:00
sahil839 2187c84ed9 stream: Add support for sending user_id to endpoint when subscribing users.
This commit modifies the backend to accept user ids when subscribing
users to streams.

It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
2020-06-01 15:59:04 -07:00
sahil839 9ef1c5b1a6 users: Add is_owner field to user objects returned by get endpoints.
This commit adds 'is_owner' field to the user object returned by
'/users', 'users/{user_id}', and '/users/me' endpoints.
2020-06-01 15:33:51 -07:00
sahil839 bd9e891e1c dev_login: Change dev env login page to show Desdemona as owner.
This commits changes the dev login page to show Desdemona as
owner.
2020-06-01 15:22:44 -07:00
Mateusz Mandera 4b10a07c3e auth: Fix social signup with the desktop app.
Only start_social_login had the code for handling the desktop
application, it was missing from start_social_signup.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 676305f6ab auth: Extract handle_desktop_flow wrapper. 2020-06-01 14:14:58 -07:00
sahil839 9fa60672e6 users: Modify update user API endpoint to accept role as parameter.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.

User role dropdown in user info modal is also modified to use
"dropdown_options_widget".

Modified by tabbott to document the API change.
2020-05-29 14:29:17 -07:00
clarammdantas f0c85c5796 registration: Show error if invalid key is posted on /accounts/register.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
2020-05-27 15:37:16 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00