Commit Graph

12155 Commits

Author SHA1 Message Date
Mohit Gupta 97872591ef refactor: rename ZEPHYR_MIRROR_BUGDOWN_KEY and DEFAULT_BUGDOWN_KEY
rename ZEPHYR_MIRROR_BUGDOWN_KEY to ZEPHYR_MIRROR_MARKDOWN_KEY and
DEFAULT_BUGDOWN_KEY tp DEFAULT_MARKDOWN_KEY.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:20:40 -07:00
Mohit Gupta 0241111aef refactor: Rename bugdown_logger to markdown_logger.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:20:40 -07:00
Tim Abbott 81d587d771 test_markdown: Import markdown the normal way. 2020-06-26 17:20:14 -07:00
Tim Abbott 99d49e88f7 test_markdown: Import FencedBlockPreprocessor the normal way. 2020-06-26 17:14:03 -07:00
Tim Abbott 6742ba54c2 test_markdown: Import mdiff the normal way. 2020-06-26 17:11:33 -07:00
Mohit Gupta a8f52a53c9 refactor: Rename markdown variable to markdown_input.
Preparatory commit before removing bugdown alias for markdown. This
will prevent same variable name errors when name markdown is used
instead of bugdown.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 2dde51053f refactor: Rename class name Bugdown to Markdown.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 5a79c48603 refactor: Rename BugdownListPreprocessor to MarkdownListPreprocessor.
This commits changes class name of MarkdownListPreprocessor to
MarkdownListPreprocessor. It also changes corresponding references
in tests.
This is part of series of commits which aims for renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 0578a918e6 refactor: Rename test_bugdown.py to test_markdown.py.
Rename the file and all the refrences to file and module test_bugdown.py
to test_markdown.py.
This commit is part of series of commit that renames bugdown to markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 05cce86670 refactor: Change BugdownRenderingException to MarkdownRenderingException.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 3f5fc13491 refactor: Rename zerver.lib.bugdown to zerver.lib.markdown .
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
2020-06-26 17:08:37 -07:00
Steve Howell 7c2be24cf8 tests: Refine external_authentication_methods.
This removes our last use of `check_dict` in
`zerver/tests`, except for `test_decorators.py`,
which is testing the function itself.
2020-06-26 17:00:30 -07:00
Steve Howell a46c28eea2 tests: Improve authentication_methods schema.
This also removes the last use of the "loose"
check_dict() validator in the events tests.
2020-06-26 17:00:30 -07:00
Steve Howell 5faea773b1 tests: Improve schema check in test_realm_emoji_events. 2020-06-26 17:00:30 -07:00
sahil839 6eb8442a59 invites: Send user_id of the referrer instead of email in invites dict.
We send user_id of the referrer instead of email in the invites dict.
Sending user_ids is more robust, as those are an immutable reference
to a user, rather than something that can change with time.

Updates to the webapp UI to display the inviters for more convenient
inspection will come in a future commit.
2020-06-26 16:57:50 -07:00
orientor 5629dcc8a6 openapi_docs: Display deprecated parameters with a `deprecated` tag.
In zulip.yaml, add `deprecated` tags to all parameters/keys with
`Deprecated` in the description. Then add tests to ensure that deprecated
parameters/keys will always have the `deprecated` key. Also, in
the API docs, sort the parameters according to presence of `deprecated`
key, presenting the `deprecated` keys at the end and add a `deprecated`
tag next to them.
2020-06-26 16:05:41 -07:00
Vishnu KS 0a36f04c20 i18n: Mark notification bot message in queue_processors for translation. 2020-06-26 14:57:18 -07:00
Vishnu KS cc0b3a08c9 i18n: Set the correct language for translation in add_subscriptions_backend. 2020-06-26 14:57:18 -07:00
Vishnu KS 28f5e86c7c i18n: Set the correct language in notify_topic_moved_streams. 2020-06-26 14:57:18 -07:00
Vishnu KS 5178c58209 i18n: Mark notification bot message in do_create_realm for translation. 2020-06-26 14:57:18 -07:00
Vishnu KS d42515df1f i18n: Set the correct language in do_rename_stream. 2020-06-26 14:57:18 -07:00
Vishnu KS e27921dbe3 i18n: Set the correct language for translation in send_pm_if_empty_stream. 2020-06-26 14:57:17 -07:00
Vishnu KS a174d8b755 i18n: Mark notification bot message in process_new_human_user for translation. 2020-06-26 14:57:17 -07:00
Vishnu KS ce6203906f i18n: Mark notification bot message in notify_new_user for translation. 2020-06-26 14:57:16 -07:00
orientor f188708b20 attachments: Change data type and make variable names more accurate.
Change variable `name` to `date_sent` as `name` actually stores
the date sent. Also change the data types of `name` and `create_time`
to integer. As they actually have empty decimal value.
2020-06-26 14:39:18 -07:00
Tim April 8e2a79095d mobile: Add support for alternative mobile URI.
Due to authentication restrictions, a deployment may need to direct
traffic for mobile applications to an alternate uri to take advantage
alternate authentication mechansism. By default the standard realm URI
will be usedm but if overridden in the settings file, an alternate uri
can be substituted.
2020-06-26 12:13:26 -07:00
Aman Agrawal 984c2d2777 push_notification: Remove notification if user no longer mentioned.
We send a remove mobile push notification to the users who were
no longer mentioned after the content of the message was edited.

This also corrects the notification count for the mobile apps
where a user was prior mentioned in a muted stream / topic and the
message was edited and the user is no longer mentioned now.
Hence, fixing the case where user has read all his unreads
but the notification badge on the app is still positive.

Fixes #15428.
2020-06-26 11:45:28 -07:00
Aman Agrawal 4612ee511f clear_push_notification: Upgrade method to accept multiples users.
do_clear_mobile_push_notifications_for_ids can now be used to
clear push_notification for multiple users at once. This method
loops over users, so no performance optimization is gained.
2020-06-26 11:25:59 -07:00
Tim Abbott a5be2a30fa events: Fix buggy realm_user/update events during user creation.
We've been seeing an exception in server_event_dispatch.js in
production where in large organizations, sometimes when a new user
joined, every other browser in the organization would throw an
exception processing some sort of realm_user/update event.

It turns out the cause was that when a user copies their profile from
an existing user account with a user-uploaded avatar, the code path we
reused to set the avatar properly send a realm_user/update event about
the avatar change -- for a user that hadn't been fully created and
certainly hadn't have the realm_user/add event sent for.

We fix this and add tests and comments to prevent it recurring.

(Removed an incorrect docstring while working on this).
2020-06-26 11:21:11 -07:00
Steve Howell bc2ed25d2d pointer tests: Use restart for test_collapse_event.
The restart event was always handled pretty similarly
to pointer, so I use restart events now for this
test (in preparation of eliminating pointer events).
2020-06-26 10:02:37 -07:00
Steve Howell 677f9361fe tests: Simplify test_event_collapsing.
We now use update_message_flags instead of
pointer events.
2020-06-26 10:02:37 -07:00
Steve Howell 93899c1d98 pointer tests: Fix test_one_event. 2020-06-26 10:02:37 -07:00
Anders Kaseorg 47913fb091 CVE-2020-15070: Replace eval with ast.literal_eval.
This eval function performs the inverse of the implicit
stringification that’s implied by this type-incorrect assignment in
do_update_user_custom_profile_data_if_changed:

field_value.value = field['value']

We believe there’s sufficient validation for the data being passed to
this eval that it could only have been exploited by a PostgreSQL
administrator editing the database manually.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-25 17:17:07 -07:00
Chris Heald 42f2399155 markdown: Escape HTML entities in inline code blocks.
This fixes an issues that causes HTML entities inside of inline code
blocks to be converted rather than being displayed literally.

The upstream python-markdown now handles this correctly, so we just use
their implementation with our changes for removing .strip(). As a result
of this migration, we switch backtick pattern to an inline processor
too.

Fixes #12056.

For the codeblock counterpart of this issue, we should follow the
upstream PR https://github.com/Python-Markdown/markdown/pull/990.

Co-authored-by: Rohitt Vashishtha <aero31aero@gmail.com>
2020-06-25 14:46:33 -07:00
Tim Abbott def6189d53 docs: Document local echo paramters for sending messages. 2020-06-25 14:44:16 -07:00
Tim Abbott 6412ea6413 api docs: Document changes in API topic encoding. 2020-06-25 14:44:04 -07:00
Tim Abbott 0ecdc663b9 api docs: Correct errors in the stream creation documentation.
* Reordered the settings relevant without stream creation to the top.
* Removed useless/misleading defaults for optional parameters.
* Clarified description of the announce and authorization_errors_fatal settings.
* Clarified that `invite_only` only applies for stream creation.
  (It's annoying to do so for its friends because they are including
   common description content and OpenAPI doesn't have a way to have
   extra content in a place you included something)

Fixes #14705.
2020-06-25 14:34:10 -07:00
Steve Howell 2fb67b3f32 refactor: Extract add/remove_subscriptions_schema.
Now we are consistent about validating color/description.

Ideally we wouldn't need to validate the
`streams_raw` parameters multiple times per
request, but the outer function here changes
the error messages to explicitly reference
the "delete" and "add" request variables.

And for the situation where the user-supplied
parameters are correct, the performance penalty
for checking them twice is extremely negligible.

So it's probably fine for now to just make sure
we use the same validators in all the relevant
places.

There's probably some deeper refactor that we
can do to eliminate the whole `compose_views`
scheme.  And it's also not entirely clear to
me that we really need to support the update
endpoint.  But that's all out of the scope of
this commit.
2020-06-25 13:52:59 -07:00
Steve Howell 6b910ff3b4 widgets: Make type checks more explicit.
Note that I don't actually convert the
checker from check_dict to check_dict_only,
because that would be a user-facing change,
but I think we can sweep a lot of things
like this after the next release.
2020-06-25 13:52:59 -07:00
Steve Howell f960df04e8 narrows: Validate negated field. 2020-06-25 13:52:59 -07:00
Steve Howell 0039c858a4 test: Extract basic_stream_fields.
This avoids some code duplication as well
as adding some missing fields.

We also use check_dict_only to prevent
folks from adding new fields to the
relevant events without updating these
tests.  (A bigger sweep comes later.)
2020-06-25 13:52:59 -07:00
Steve Howell e0ebc1307a tests: Extract ad_hoc_config_data_schema.
As the code comment indicates, we just
use a strict check here rather than
pretending that the test exercises a
more complicated schema for the config
data, which is dynamic in nature.

Cleaning up config_data is outside the
scope of this PR; my main goal is to
eliminate check_dict calls (usually in favor
of check_dict_only).
2020-06-25 13:51:24 -07:00
Steve Howell 3f385ca799 tests: Use check_dict_only and check_events_dict for message. 2020-06-25 13:51:24 -07:00
Steve Howell 69126ca809 tests: Use check_dict_only for custom_profile_field. 2020-06-25 13:51:24 -07:00
Anders Kaseorg ebb2efa664 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-25 13:14:37 -07:00
Anders Kaseorg 6363c49e3f test_auth_backends: Add request parameter to patched_authenticate.
This is required by social-auth-app-django 4.0.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-25 13:14:37 -07:00
Anders Kaseorg 30c6797239 test_runner: Fix SENDFILE_ROOT.
This is required by django-sendfile2 0.6.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-25 13:14:37 -07:00
Tim Abbott cb1321d0d2 lint: Harden various checks for URLs.
Because of other validation on these values, I don't believe any of
these does anything different, but these changes improve readability
and likely make GitHub's code scanners happy.
2020-06-25 12:10:45 -07:00
Clara Dantas d2da9827ac tests: Use get_account_data_dict helper in some github tests.
The helper should be used instead of constructing the dict manually.

Change get_account_data_dict, on GitHubAuthBackendTest
class, so it has a third argument, user_avatar_url.

This is a preparation for support using GitHub avatar
upon user resgistration (when the user logs using
GitHub).
2020-06-25 11:13:16 -07:00
Steve Howell 80c057d91d REQ: Use check_dict_only in update_user_backend.
Update the REQ check for profile_data in
update_user_backend by tweaking `check_profile_data`
to use `check_dict_only`.

Here is the relevant URL:

    path('users/<int:user_id>', rest_dispatch,
         {'GET': 'zerver.views.users.get_members_backend',

It would be nice to unify the validator
for these two views, but they are different:

    update_user_backend
    update_user_custom_profile_data

It's not completely clear to me why update_user_backend
seems to support a superset of the functionality
of `update_user_custom_profile_data`, but it has
this code to allow you to remove custom profile fields:

    clean_profile_data = []
    for entry in profile_data:
        assert isinstance(entry["id"], int)
        if entry["value"] is None or not entry["value"]:
            field_id = entry["id"]
            check_remove_custom_profile_field_value(target, field_id)
        else:
            clean_profile_data.append({
                "id": entry["id"],
                "value": entry["value"],
            })

Whereas the other view is much simpler:

def update_user_custom_profile_data(
    <snip>
) -> HttpResponse:

    validate_user_custom_profile_data(user_profile.realm.id, data)
    do_update_user_custom_profile_data_if_changed(user_profile, data)
    # We need to call this explicitly otherwise constraints are not check
    return json_success()
2020-06-25 10:54:15 -07:00
Steve Howell 92a552d2ad REQ: Check value in update_user_custom_profile_data.
This tightens our checking of user-supplied data
for this endpoint:

    path('users/me/profile_data', rest_dispatch,
         {'PATCH': 'zerver.views.custom_profile_fields.update_user_custom_profile_data',
         ...

We now explicitly require the `value` field
to be present in the dicts being passed in
here, as part of `REQ`.  There is no reason
that our current clients would be sending
extra fields here, and we would just ignore
them anyway, so we also move to using
check_dict_only.

Here is some relevant webapp code (see settings_account.js):

    fields.push({id: field.id, value: user_ids});
    update_user_custom_profile_fields(fields, channel.patch);

    settings_ui.do_settings_change(method, "/json/users/me/profile_data",
                                   {data: JSON.stringify([field])}, spinner_element);

The webapp code sends fields one at a time
as one-element arrays, which is strange, but
that is out of the scope of this change.
2020-06-25 10:54:15 -07:00
Mateusz Mandera 85d4536486 docs: Update some comments for the new release versioning scheme.
With the new scheme, the equivalent of 2.3 is 4.0.
2020-06-25 10:33:03 -07:00
Anders Kaseorg 58bc175526 models: Avoid undocumented sre_constants module.
>>> sre_constants.error is re.error
True

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-24 23:18:17 -07:00
Tim Abbott e46bbf18eb docs: Change next planned major release to 3.0.
After some discussion, everyone seems to agree that 3.0 is the more
appropriate version number for our next major release.  This updates
our documentation to reflect that we'll be using 3.0 as our next major
release.
2020-06-24 16:27:27 -07:00
orientor 8ab6182683 fetch_api_key: Return `email` in json_response.
`/api/v1/fetch_api_key`'s response had a key `email` with the user's
delivery email. But its JSON counterpart `/json/fetch_api_key`, which
has a completely different implementation, did not return `email` in
its success response.

So to avoid confusion, the non-API endpoint, `/json/fetch_api_key`
response has been made identical with it's `/api` counterpart by
adding the `email` key. Also it is safe to send as the calling user
will only see their own email.
2020-06-24 15:13:31 -07:00
Steve Howell 2dd1ccef15 check_list: Make sub_validator a required argument.
This allows us to avoid the overload decorators.
2020-06-24 15:04:59 -07:00
Steve Howell de2285222d tests: Use check_tuple for realm filters.
This also sets the stage to simplify
our check_list validator.
2020-06-24 15:01:57 -07:00
Steve Howell c7b82d3ece mypy: Use tuples for muted_topics.
We now have our muted topics use tuples internally,
which allows us to tighten up the annotation
for get_topic_mutes, as well as our schema
checking.

We want to deprecate sub_validator=None
for check_list, so we also introduce
check_tuple here.  Now we also want to deprecate
check_tuple, but it's at least isolated now.

We will use this for data structures that are tuples,
but which are sent as lists over the wire.  Fortunately,
we don't have too many of those.

The plan is to convert tuples to dictionaries,
but backward compatibility may be tricky in some
places.
2020-06-24 15:01:57 -07:00
Steve Howell 8e8228535e tests: Use check_dict for external_authentication_methods
This is still imperfect, but the only goal for now is
to make sure that `check_list` always get a sub_validator.
2020-06-24 15:01:57 -07:00
sahil839 fe35f1e9bd invites: Do not return multi-use invites to non-admin users.
This commit changes do_get_user_invites function to not return
multiuse invites to non-admin users. We should only return multiuse
invites to admins, as we only allow admins to create them.
2020-06-24 15:01:41 -07:00
Tim Abbott 2a6ef2b9eb models: Add missing migrations from invite_as change.
I neglected to `git add` after doing `renumber-migrations` for
1f8f227444.
2020-06-24 11:39:01 -07:00
sahil839 1f8f227444 models: Update values of PreregistrationUser.invite_as dict.
This commit changes the PreregistrationUser.invite_as dict to have
same set of values as we have for UserProfile.role.

This also adds a data migration to update the already exisiting
PreregistrationUser and MultiuseInvite objects.
2020-06-24 11:09:07 -07:00
Tim Abbott b7b173d2ae realm: Fix type-checking for message_retention_days.
The best way to handle this is to have the potentially string-typed
value be a separate variable.
2020-06-24 11:01:34 -07:00
Mateusz Mandera 890cafac11 retention: Use batch size of 100 for stream messages.
Streams can have lots of subscribers, meaning that the archiving process
will be moving tons of UserMessages per message. For that reason, using
a smaller batch size for stream messages is justified.

Some personal messages need to be added in test_scrub_realm to have
coverage of do_delete_messages_by_sender after these changes.
2020-06-24 10:41:00 -07:00
Mateusz Mandera 0c6497d43a retention: Add restore_retention_policy_deletions_for_stream function. 2020-06-24 10:40:38 -07:00
Mateusz Mandera 468f8cf488 retention: Improve logging of transactions. 2020-06-24 10:40:38 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 7b3f4755ff streams: Fix bug that allows 0 value for message_retention_days. 2020-06-24 10:38:58 -07:00
Mateusz Mandera 7a03e2a7fe retention: Replace Realm.message_retention_days None value with -1.
To be more consistent with the meaning in the Stream model, and to make
it easier to have a reasonable settings API, we get rid of the None
value for Realm.message_retention_days in favor of the value -1 to
represent the "don't delete messages" default policy.
2020-06-24 10:33:21 -07:00
Hashir Sarwar 9b8521faee people: Enable `user_avatar_url_field_optional` in webapp.
In 5200598a31, we introduced a new
client capability that can be used to avoid unreasonable network
bandwidth consumed sending avatar URLs of long term idle users in
organizations with 10,000s members.

This commit enables this feature and adds support for it to the web
client.
2020-06-24 10:06:35 -07:00
Steve Howell bc53fe1606 mypy: Use equals([]) to check services.
We are trying to deprecate sub_validator=None
for check_list.
2020-06-24 10:35:36 -04:00
Mateusz Mandera 7fe52bbb9e tests: Clean up the subdomain argument to social_auth_test.
subdomain=None didn't make much sense as a value, and wasn't actually in
use anywhere, except one test where it was accidental. All tests specify
the subdomain explicitly, so we should change the type to str, and make
it an obligatory kwarg.
2020-06-23 17:14:31 -07:00
Brainrecursion 30eaed0378 saml: Add option to restrict subdomain access based on SAML attributes.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.

Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-23 17:14:31 -07:00
Anders Kaseorg d6655689f5 validator: Remove unused type_structure system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
wowol a3c475f137 urls: Migrate dev urls to use modern django patterns. 2020-06-23 15:02:44 -07:00
Kartik Srivastava 3b8ee2a30f js_examples: Migrate and test update_message example. 2020-06-23 14:42:03 -07:00
Kartik Srivastava 8a00b1f1aa js_examples: Migrate and test update_message_flags example. 2020-06-23 14:42:03 -07:00
Kartik Srivastava cc4086e98a js_examples: Migrate and test remove_subscriptions example. 2020-06-23 14:35:08 -07:00
Kartik Srivastava db303ccaef js_examples: Migrate and test add_subscriptions example. 2020-06-23 14:35:08 -07:00
Kartik Srivastava cf6e1de0de js_examples: Migrate and test set_typing_status example. 2020-06-23 14:35:08 -07:00
Kartik Srivastava a1f6540cec js_examples: Migrate and test render_message example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 5a4d31825c js_examples: Migrate and test register_queue example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 4aec4cbfae js_examples: Migrate and test get_users example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 69a7d7fd30 js_examples: Migrate and test get_subscriptions example.
Also, `send_message` example is altered to send a message to the
stream 'social' to avoid getting a "first_message_id: null"
in the response for `get_subscriptions` example, that caused
`validate_against_openapi_schema` to throw an error.
2020-06-23 14:35:07 -07:00
Kartik Srivastava cc7671e602 js_examples: Migrate and test get_stream_topics example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 2c5815edbb js_examples: Migrate and test get_stream_id example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 63127d4990 js_examples: Migrate and test get_own_user example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 4615519d77 js_examples: Migrate and test get_messages example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava 57e27ff7db js_examples: Migrate and test delete_queue example. 2020-06-23 14:35:07 -07:00
Kartik Srivastava f96c86ad63 js_examples: Migrate and test get_custom_emoji example. 2020-06-23 14:35:07 -07:00
Anders Kaseorg 48719cb8c4 message_send: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg f6c73ed45a auth: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg 5733a1bcd4 validate_user_access_to_subscribers: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
orientor 5ac411aeaa openapi: Document `delivery_email` property. 2020-06-23 12:30:14 -07:00
orientor 358d9fc35f openapi: Document various properties in `EXCLUDED_PROPERTIES`.
The `EXCLUDE_PROPERTIES` is a dictionary in `zerver/openapi/openapi.py`
which holds the undocumented properties of our API. Document all
properties other than:

*`delivery_email` which is in another PR.
*'events' and 'register'.
*'/setting/notification' since its response is about to undergo heavy
  changes.
2020-06-23 12:21:58 -07:00
Anders Kaseorg 8e8e34c000 test_runner: Fix a type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg b7241f20d5 timezone: Remove dead assignment with type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 3ffed617a2 mypy: Type simple generators as Iterator, not Iterable.
A generator that yields values without receiving or returning them is
an Iterator.  Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg c242d176ba rest: Fix a type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 58942e2616 test_embedded_bot_system: Replace type: ignore annotations with an assert.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 123b53ae86 test_push_notifications: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg feeb39a55f timezone: Strengthen get_timezone type.
pytz.BaseTzInfo is a subclass of datetime.tzinfo that adds the
localize and normalize methods.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 1d545efac2 custom_profile_fields: Replace a type: ignore with an assert.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg a3a39494e3 video_calls: Annotate a type: ignore with a specific bug URL.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 57b87269ec test_signup: Remove dead code with type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg ca1d9603cb decorator: Fix type of signature-changing decorators.
In a decorator annotated with generic type (ViewFuncT) -> ViewFuncT,
the type variable ViewFuncT = TypeVar(…) must be instantiated to
the *same* type in both places.  This amounts to a claim that the
decorator preserves the signature of the view function, which is not
the case for decorators that add a user_profile parameter.

The corrected annotations enforce no particular relationship between
the input and output signatures, which is not the ideal type we might
get if mypy supported variadic generics, but is better than enforcing
a relationship that is guaranteed to be wrong.

This removes a bunch of ‘# type: ignore[call-arg] # mypy doesn't seem
to apply the decorator’ annotations.  Mypy does apply the decorator,
but the decorator’s incorrect annotation as signature-preserving made
it appear as if it didn’t.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 95c6d44a4a test_runner: Remove unused type: ignore comments.
Also enable warn_unused_ignores.  I think the fact that there are so
few of these is good evidence that it’s not a significant burden for
people fixing type errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 30ab0524b5 video_calls: Don’t request a scope for the Zoom OAuth session.
As requested by a Zoom reviewer for unclear reasons.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 18:48:35 -07:00
Anders Kaseorg 1e4fc8f2bf users: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg 086a3ef18b check_remove_custom_profile_field_value: Fix type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg b62c82c3e1 tests: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg 579f05f3ed queue_processors: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg f33bfaf545 import_realm: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:13:48 -07:00
Anders Kaseorg 78f4482867 addressee: Remove redundant casts.
Also enable warn_redundant_casts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:11:42 -07:00
Tim Abbott 16cdc3c66b migrations: Fix migration 290 not being marked atomic.
Migrations that add items to a table in the same migration htat
craetes the table require atomic=False.

This fixes this exception:

Running migrations:
  Applying zerver.0287_clear_duplicate_reactions... OK
  Applying zerver.0288_reaction_unique_on_emoji_code... OK
  Applying zerver.0289_tighten_attachment_size... OK
  Applying zerver.0290_remove_night_mode_add_color_scheme...Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-22-23-20-36/zulip-py3-venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/zulip/deployments/2020-06-22-23-20-36/zerver/lib/db.py", line 33, in execute
    return wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2020-06-22-23-20-36/zerver/lib/db.py", line 20, in wrapper_execute
    return action(sql, params)
psycopg2.errors.ObjectInUse: cannot ALTER TABLE "zerver_userprofile" because it has pending trig
2020-06-22 17:08:32 -07:00
Felix a389c7390d calls: Add Big Blue Button as a Video Call Provider.
Big Blue Button needs an API secret so communication to creating a
room has to be done server side.

Fixes #14763.
2020-06-22 16:19:07 -07:00
Tim Abbott 866965f162 views: Rename messages.py to message_fetch.py.
This completes to effort to split zerver.views.messages into smaller
modules focused on the different core type of work.
2020-06-22 15:08:34 -07:00
Tim Abbott 3657717ade views: Extract message_send.py for sending views. 2020-06-22 15:08:34 -07:00
Tim Abbott 55448ecf24 views: Extract message_flags.py for flags views. 2020-06-22 15:08:34 -07:00
Tim Abbott 4d7550d705 views: Extract message_edit.py for message editing views.
This is a pretty clean extraction of files that lets us shrink one of
our largest files.
2020-06-22 15:08:34 -07:00
Tim Abbott dc2a045d3b docs: Add documentation on mypy @overload.
We only use this in a few places, but they're really important places
for understanding the types in the codebase, and so it's worth having
a bit of expository documentation explaining how we use it.

(And I expect we'll add more with time).
2020-06-21 10:51:47 -07:00
Mateusz Mandera 60c4896c3a import: Fix import and export parallelism defaults.
Fixes #14960.
The default of 6 thread may not be appropriate in certain
configurations. Taking half of the numer of CPUs available to the
process will be more flexible.
2020-06-21 09:39:31 -07:00
orientor c4ccf0d6b3 openapi: Fix wrong schemas.
In a recent commit, due to conflicts with old schemas the new
schemas were added as oneOf. But the old schemas were wrong. So
change them accordingly.
2020-06-21 01:42:40 -07:00
Tim Abbott 3b4a9f8a9c test_middleware: Remove night mode test.
We removed the quoted string this test was for.
2020-06-21 01:17:42 -07:00
MariaGkoulta b10f156250 settings: Add automatic theme detection feature.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.

This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.

Fixes: #14451.

Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
2020-06-21 01:09:01 -07:00
sahil839 63389b3bd3 invite: Add option to invite user as an organization owner.
We can now invite new users as realm owners. We restrict only
owners to invite new users as owners both for single invite
and multiuse invite link. Also, only owners can revoke or resend
owner invitations.
2020-06-20 23:04:59 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 596bfb47c4 papertrail: Replace custom validator with check_dict.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 114f859d3a api_travis_webhook: Fix message type annotation.
Not all values of this dict are strings, just the ones we use.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c0fa42dd8c complete_zoom_user: Validate state type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 64038163e7 update_user_backend: Validate profile_data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c584462b32 update_storage: Validate storage type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 717e349834 events_register_backend: Validate client_capabilities type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 475fc5910f update_user_custom_profile_data: Validate data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 6e33b9c6f2 get_events_backend: Validate narrow type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Tim Abbott d503549f0b retention: Add some more system documentation. 2020-06-20 17:35:07 -07:00
Steve Howell fff2a81536 mypy: Fix payload_url. 2020-06-20 17:09:15 -07:00
Steve Howell 82036741b9 mypy: Fix service_interface 2020-06-20 17:09:15 -07:00
Steve Howell 704c84502b mypy: Fix authentication_methods.
We know the key of the dictionary is a string
here, since we get it via json.
2020-06-20 17:09:15 -07:00
orientor fbf647283b openapi: Fix validate_against_openapi_schema nested object validation.
We had a bug in `validate_against_openapi_schema` that prevented it
from correctly inspecting nested arrays.

Fix the bug and address all the exceptions, either via
EXCLUDE_PROPERTIES or fixing them when simple.  Also add a test case
for nested verification.
2020-06-20 13:23:39 -07:00
Tim Abbott f98d244ed6 models: Change attachment.size to not be nullable.
Attachment objects in production are only created in one place, which
passses a size.  Additionally, I verified in multiple production
environments with old data that this never actually happens (or has
happened).

So we should make the data model correctly reflect the possibilities here.
2020-06-20 13:15:02 -07:00
Steve Howell 22ead64d54 tests: Extract USING_TYPE_STRUCTURE. 2020-06-20 05:22:35 -04:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell fbe45fa889 test helpers: Tweak common_subscribe_to_streams.
We assert that the post was successful, to give
more immediate feedback for tests that don't
bother to check the return value and may be
implicitly assuming this method just works in
all cases.

And we also make it more convenient for tests
that are happy-path tests--they don't have to
do the assertion themselves.  (And they're still
free to do deeper checks on the json.)

We opt out with allow_fail=True.  We probably want
a more direct API eventually for tests that are
clearly trying to test the failure path for
subscribing to streams.

It's possible that a couple tests here that I added
allow_fail=True to just have flawed data setup--
I don't have time to investigate all cases, but
hopefully they will at least stand out more.
2020-06-19 22:03:12 -07:00
Steve Howell 2e9f689add mypy: Fix annotation for principals. 2020-06-19 21:23:23 -07:00
Tim Abbott a4fdc6f826 api docs: Fix a few misformatted elements. 2020-06-19 17:22:30 -07:00
Tim Abbott 1b620c7e9f openapi: Fix duplicated declarations for GET /messages response.
There were 2 copies of this block.
2020-06-19 15:28:10 -07:00
akashaviator 03325bd615 api docs: Document GET /messages/matches_narrow endpoint.
With extensive tweaks from tabbott to provide clear explanations for
the features.
2020-06-19 15:28:10 -07:00
Tim Abbott 14c0a9a228 messages: Deduplicate non-search code path in messages_in_narrow.
The non-search code path here was simulating the response and escaping
logic from get_search_fields by duplicating what it would do with an
empty set of highlight locations.

We can produce much more readable code by just passing an empty list
of locations in this case.
2020-06-19 15:09:57 -07:00
Steve Howell 00e60c0c91 events: Use stream_id for peer_add/peer_remove.
Two things were broken here:
    * we were using name(s) instead of id(s)
    * we were always sending lists that only
      had one element

Now we just send "stream_id" instead of "subscriptions".

If anything, we should start sending a list of users
instead of a list of streams.  For example, see
the code below:

    if peer_user_ids:
        for new_user_id in new_user_ids:
            event = dict(type="subscription", op="peer_add",
                         stream_id=stream.id,
                         user_id=new_user_id)
            send_event(realm, event, peer_user_ids)

Note that this only affects the webapp, as mobile/ZT
don't use this.
2020-06-19 14:48:42 -07:00
orientor 58b93c3e83 openapi: Show API argument type in docs.
Currently the API docs do not specify whether a given API parameter
is to be specified in `query` or in `path`. Edit the docs so as
to show the type of argument right beside argument name.
2020-06-19 14:42:01 -07:00
orientor 1505f77ca3 openapi: Parse {!api-admin-only.md!} in OpenAPI markdown extension.
Currently, the OpenAPI extension for rendering description in docs
cannot parse {!api-admin-only.md!}. Edit order of markdown extensions
in app_filters.py so that rendering of OpenAPI elements takes place
before substitution of files using `include`.
2020-06-19 11:04:46 -07:00
Tim Abbott a66a8419b9 bankruptcy: Remove broken push notifications loop.
The loop I added here in 5b49839b08 was
ill-conceived.  The critical issue was that despite its name,
do_clear_mobile_push_notifications_for_ids does not immediately clear
push notifications (Except in our test suite, where `send_event`
immediately calls into the queue worker code!).

Instead, it queues work to clear those push notifications.  Which
means that the first user to declare bankruptcy with a large number of
unreads will fill the queue, and then this will just be an infinite
loop adding more work to the queue.
2020-06-19 10:54:14 -07:00
arpit551 c7d0192755 reaction: Fix missing unique constraint on Reactions model.
This fixes a missing unique constraint on the Reactions data model
state when using multiple aliases for an emoji code.  As with any
missing unique constraints, we first need to apply a migration that
eliminates violations of the rule; in this case, deleting the
duplicates is correct.

Added unique constraint for "user_profile", "message",
"reaction_type", "emoji_code".

Fixes #15347.
2020-06-19 10:07:42 -07:00
Steve Howell 7c6ddf90ae mypy: Fix default_language annotation. 2020-06-19 12:35:47 -04:00
Steve Howell 9cde534cf4 mypy: Fix event_types annotation. 2020-06-19 12:35:47 -04:00
Tim Abbott bfc30e4214 api docs: Improve headings and organization for endpoints.
Mostly, this is a change in ordering to make more sense, but we also
fix several names that were clearly confusing.

We restore the convention that each endpoint has the same title at the
top of the page as what we have in the sidebar menu, which appears to
have been violated in many recent updates to API documentation.
2020-06-18 22:38:05 -07:00
orientor d8ef853545 openapi: Make api-docs filename match with operationIds.
api docs filenames are basically the operationId of their endpoint
in zulip.yaml with `_` replaced by `-`. But some operationIds have
changed, so change the affected filenames. Make changes in other
files accordingly.
2020-06-18 22:38:05 -07:00
Hashir Sarwar 5200598a31 events: Don't send avatar URLs of long term idle users.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.

Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.

This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.

Fixes #15287.
2020-06-18 21:35:16 -07:00
Hashir Sarwar 9911ec3e6d cache: Add `long_term_idle` in `realm_user_dict_fields`.
We need this field to avoid O(N) database operations
while fetching realm user data for clients with
`user_avatar_url_field_optional` flag enabled.

Part of #15287.
2020-06-18 20:23:10 -07:00
Karim Tayie 7233608d4e tests: Extend test_get_accounts_for_email.
This extends get_accounts_for_email test by adding a deactivated
user and assert that get_accounts_for_email doesn't return any accounts
for that deactivated user.

Fixes #14807.
2020-06-18 20:19:08 -07:00
Tim Abbott 0b6ebb4fbb upload: Remove unused get_realm_for_filename. 2020-06-18 17:55:13 -07:00
Tim Abbott 5962d1ea14 upload: Avoid fetching bucket objects repeatedly.
This takes of advantage of saving the bucket object on the
UploadBackend class to deduplicate a bunch of redundant code getting
buckets.
2020-06-18 17:55:13 -07:00
Wyatt Hoodes 2ef791fc21 upload.py: Support using non S3-providers.
With #14378, we regressed back to the state of that
prior to 7e0ea61b00.

We fix this by getting our avatar bucket on
object initialization, and use the appropriate means
of gathering the network location for the urls.

Fixes #14484.
2020-06-18 17:55:13 -07:00
Vishnu KS eb829e6dce realm_logo: Return default logo for realms on LIMITED plan.
This completes our effort to avoid destroying realm logo state while
providing a path to dowgrade from paid plans to LIMITED.
2020-06-18 17:46:26 -07:00
Vishnu KS d0220fc5fc import tests: Pass realm to _setup_export_files.
_setup_export_files modifies the zulip realm. We used to
call realm.refresh_from_db in tests after _setup_export_files was
called to make sure that the change is reflected. But sometimes
calling refresh_from_db was missed out here and there.

This commit makes calling refresh_from_db after _setup_export_files
unnecessary.
2020-06-18 17:45:35 -07:00
Vishnu KS 0a9e114014 import tests: Run test_import_files in billing disabled mode.
We don't really care about the plan specific details
in these tests. It's taken care in test_plan_type.
2020-06-18 17:45:35 -07:00
Tim Abbott dbe9f09dae lint: Fix extra blank lines. 2020-06-18 17:45:35 -07:00
Vishnu KS cde4486f8c billing: Support switching from monthly to annual plan. 2020-06-18 17:34:03 -07:00
Siddharth Varshney bbb07aed38 portico: Add hint text for invite only realm on /login.
Fixes: #10542

This will make obvious for users what to do when they are not able to login.

Test added by hackerkid.
2020-06-18 17:23:11 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Anders Kaseorg 3916ea23a9 python: Combine some split import groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:54:11 -07:00
Anders Kaseorg b2cb992d9f bugdown: Use standard library datetime.timezone.utc.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:47:20 -07:00
Tim Abbott 3ec068f10c bugdown: Import datetime our normal way. 2020-06-18 14:14:53 -07:00
Rohitt Vashishtha 6ea3816fa6 markdown: Use html5 <time> tag for timestamps.
Previously, we had implemented:
    <span class="timestamp" data-timestamp="unix time">Original text</span>
The new syntax is:
    <time timestamp="ISO 8601 string">Original text</time>
    <span class="timestamp-error">Invalid time format: Original text</span>

Since python and JS interpretations of the ISO format are very
slightly different, we force both of them to drop milliseconds
and use 'Z' instead of '+00:00' to represent that the string is
in UTC. The resultant strings look like: 2011-04-11T10:20:30Z.

Fixes #15431.
2020-06-18 14:11:33 -07:00
Dinesh 0445311430 auth: Make apple log in and sign up buttons consistent with others. 2020-06-18 13:06:10 -07:00
Tim Abbott f8ea5f3769 pointer: Remove pointer from GET /users/me.
This cleans up a bit of mess in the Zulip API.
2020-06-18 12:55:59 -07:00
Tim Abbott b77da6b834 pointer: Remove fix_pre_pointer fixing logic.
This fundamentally requires the pointer to exist to do anything
useful.
2020-06-18 12:55:59 -07:00
orientor 817ee5c6b2 openapi_docs: Replace `argument` with `parameter`.
The term `parameter` is a better word than `argument` for data passed
to an API endpoint; this is why OpenAPI uses in their terminology.
Replace `argument` with `parameter` in the API docs to improve their
readability.

Fixes #15435.
2020-06-18 09:18:27 -07:00
Aman Agrawal b7e5b7f193 do_update_messages: Delete user_message if guest user loses access.
Fixes #14498.
When a topic is moved to a different stream, the message may no
longer be reachable to guest user, if the user is not subscribed
to the new stream.

We used to send message update event to the client in these cases,
which seems to be confusing both to the client updating the message
and the server sending push_notifications for it.

Now, we delete the UserMessage entry for these messages for the
user and send a delete message event to the client; which makes
both push_notification and the event handling client think that
the message was deleted and hence no confusion in the code is
raised.
2020-06-18 01:00:04 -07:00
Hashir Sarwar ab6be2a711 push_notifications: Store tokens locally even when bouncer is used.
This makes the system store and track PushDeviceToken objects on
the local Zulip server when using the push notifications bouncer
and includes tests for this.

This is something we need to implement end-to-end encryption for
push notifications. We'll add the encryption key as an additional
property on the local PushDeviceToken object.

It also likely adds some value in the case that a server were to
switch between using the bouncer service and sending notifications
directly, though in practice that's unlikely to happen.
2020-06-17 18:44:59 -07:00
akashaviator ce571048b9 openapi: Document GET /attachments endpoint. 2020-06-17 18:35:44 -07:00
Tim Abbott bca164c1b6 api docs: Use better language for no parameters allowed. 2020-06-17 18:20:23 -07:00
Tim Abbott 3a46f6ecca i18n: Add regex for nested handlebars i18n.
This is required for our i18n search code to find the "Upload icon"
category of strings we're about to restore recently regressed i18n
for.
2020-06-17 17:56:57 -07:00
Tim Abbott a9877d6765 migrations: Add merged migration following 2.1.5 release. 2020-06-16 23:35:39 -07:00
Tim Abbott 9f4905d7e3 CVE-2020-14215: Add migration to clear INVITED_AS_REALM_ADMIN.
This migration fixes any PreregistrationUser objects that might have
been already corrupted to have the administrator role by the buggy
original version of migration 0198_preregistrationuser_invited_as.

Since invitations that create new users as administrators are rare, it
is cleaner to just remove the role from all PreregistrationUser
objects than to filter for just those older invitation objects that
could have been corrupted by the original migration.
2020-06-16 23:35:39 -07:00
Tim Abbott 3f55bfc53b CVE-2020-14215: Fix migration 0198_preregistrationuser_invited_as.
This migration incorrectly swapped the role associated with invitation
objects between members and organization administrators, resulting in
most invitation objects that existed before the upgrade to Zulip
2.0.0-rc1 or later to be incorrectly administrator invitations.

Fixing the migration is safe and will help those installations
upgrading directly from 1.9.x to 2.1.5 or later.

A migration to fix the corrupted records will appear in an upcoming
commit.
2020-06-16 23:35:39 -07:00
Mateusz Mandera 8d2d64c100 CVE-2020-14215: Fix validation in PreregistrationUser queries.
The most import change here is the one in maybe_send_to_registration
codepath, as the insufficient validation there could lead to fetching
an expired PreregistrationUser that was invited as an administrator
admin even years ago, leading to this registration ending up in the
new user being a realm administrator.

Combined with the buggy migration in
0198_preregistrationuser_invited_as.py, this led to users incorrectly
joining as organizations administrators by accident.  But even without
that bug, this issue could have allowed a user who was invited as an
administrator but then had that invitation expire and then joined via
social authentication incorrectly join as an organization administrator.

The second change is in ConfirmationEmailWorker, where this wasn't a
security problem, but if the server was stopped for long enough, with
some invites to send out email for in the queue, then after starting it
up again, the queue worker would send out emails for invites that
had already expired.
2020-06-16 23:35:39 -07:00
Anders Kaseorg 87f7874a79 CVE-2020-12759: Fix reflected XSS vulnerability in Dropbox webhook.
Also check the challenge argument’s presence before using it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-16 22:46:16 -07:00
Clara Dantas ddbde66af5 realm: Remove Google Hangouts integration.
Google  has removed the Google Hangouts brand, thus we are removing
them as video chat provider option.
This commit removes Google Hangouts integration and make a migration
that sets all realms that are using Hangouts as their video chat
provider to the default, jitsi.

With changes by tabbott to improve the overall video call documentation.

Fixes: #15298.
2020-06-16 17:02:27 -07:00
Tim Abbott f4ac4be851 sentry webhook: Check for platform support early.
Otherwise, we don't know whether the event format might not have the
`filename` or other parameters.
2020-06-16 16:49:31 -07:00
Mateusz Mandera 2ac6a8f829 auth: Change the "continue in browser" link in desktop flow end page.
Fixes #14828.
Giving the /subdomain/<token>/ url there could feel buggy if the user
ended up using the token in the desktop app, and then tried clicking the
"continue in browser" link - which had the same token that would now be
expired. It's sufficient to simply link to /login/ instead.
2020-06-16 16:27:53 -07:00
Sara Gulotta 1cb040647b markdown: Add support for spoilers.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.

We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.

Fixes #5802.

Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
2020-06-16 16:14:10 -07:00
Tim Abbott 54604257e0 sentry: Provide more clarity around unsupported platforms. 2020-06-16 14:04:16 -07:00
Aman Agrawal d7ca5bad62 do_delete_messages: Create a TypedDict declaration for the event.
This will help us ensure that other functions defining delete_message
event use the same formatting.
2020-06-16 11:40:45 -07:00
Hashir Sarwar ecd35b9565 push_notifications: Add support for setting counts in iOS.
This adds a new function `get_apns_badge_count()` to
fetch count value for a user push notification and
then sends that value with the APNs payload.

Once a message is read from the web app, the count is
decremented accordingly and a push notification with
`event: remove` is sent to the iOS clients.

Fixes #10271.
2020-06-16 11:26:36 -07:00
Hashir Sarwar 2bc34bb3ff test_push_notifications: Remove mocking of `get_base_payload()`.
Mocking `get_base_payload()` verifies the wrong output
when the code is actually correct. So, its better that
we call the real function here, especially when we are
adding the Apple case.
2020-06-16 11:26:36 -07:00
Tim Abbott 59b68aaa98 events: Remove incorrect line from new-stream handler.
This line was effectively hardcoding a specific stream_post_policy,
overriding the value already present in the event, to no purpose.

(I believe it got here via cargo-culting induced by #13787.)
2020-06-16 10:41:50 -07:00
sahil839 791e5de5de api: Remove is_old_stream property from the stream objects.
This commit removes is_old_stream property from the stream objects
returned by the API. This property was unnecessary and is essentially
equivalent to 'stream_weekly_traffic != null'.

We compute sub.is_old_stream in stream_data.update_calculated_fields
in frontend code and it is used to check whether we have a non-null
stream_weekly_traffic or not.

Fixes #15181.
2020-06-16 10:26:33 -07:00
Hashir Sarwar 31b6867360 push_notifications: Remove redundant `user_profile` assignment. 2020-06-15 22:32:12 -07:00
Tim Abbott 5b49839b08 bankruptcy: Clear mobile push notifications first.
This likely fix a bug that can leak thousands of messages into the
invalid state where:
* user_message.flags.active_mobile_push_notification is True
* user_message.flags.read is True

which is intended to be impossible except during the transient process
between marking messages as read sending the "remove push
notifications" event.

The bug is that if a user who is declaring bankruptcy with 10,000s of
unreads ends up having the database query to mark all of those as read
take 60s, the Django/uwsgi request will time out and kill the process.
If the postgres transaction still completes, we'll end up with the
second half of this function never being run.

A safer ordering is to do the smaller queries first.

We do this in a loop for correctness in the unlikely event there are
more than 10,000 of these.
2020-06-15 17:43:43 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Mateusz Mandera eda396376c actions: Remove redundant "assert is None" in do_update_message_flags.
The type of messages arg has changed since this was added.
2020-06-15 16:20:49 -07:00
Mateusz Mandera f467ab046e messages: Remove redundant "assert is not None" for mypy.
Apparently new mypy is able to infer correctly and doesn't need this
assert anymore.
2020-06-15 16:20:43 -07:00
Tim Abbott 05455f432e sentry: Enable the javascript name for that platform.
Hopefully this will mean our Sentry integration properly handles
javascript events now.
2020-06-15 13:01:39 -07:00
Aman Agrawal ccc0c8706c delete_message: Support `bulk_message_deletion`.
Fixes #15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
2020-06-15 10:41:12 -07:00
Tim Abbott f448ee404a lint: Fix a new % format string that should be fstring. 2020-06-15 00:12:08 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1ed2d9b4a0 logging: Use logging.exception and exc_info for unexpected exceptions.
logging.exception() and logging.debug(exc_info=True),
etc. automatically include a traceback.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 4b6d2cf25f logging: Pass more format arguments to logging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Hemanth V. Alluri 545f40b2c2 webhooks/circleci: Add support for super minimal payloads.
After merging PR #15355 we found that CircleCI may send
super minimal payloads. This does not seem to depend on
.circleci/config.yml since we received two different
types of payloads off of the same configuration.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-14 22:35:41 -07:00
Aman Agrawal cda7b2f539 deletion: Add support for bulk message deletion events.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.

Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).

Backend implementation of #15285.
2020-06-14 22:34:00 -07:00
Aman Agrawal 21ac1fb32c register: Refactor to pass client_capabilities directly.
We extract values from client_capabilities directly in
do_events_register where we decide how to process the
extracted variables.
2020-06-14 21:42:50 -07:00
Tim Abbott d97c891afe realm owners: Remove unnecessary duplicate strings. 2020-06-14 21:32:10 -07:00
sahil839 18429cfd29 realm: Allow only owners to set message retention policy for a realm.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.

Dropdown for changing retention policy is disabled in UI for admins also.
2020-06-14 21:23:53 -07:00
sahil839 87e72ac8e2 realm: Allow only owners to configure auth methods for a realm.
This commit adds the restriction on configuring auth methods for
admins. We now allow only owners to configure the auth methods
for realm.
2020-06-14 21:23:51 -07:00
arpit551 c4b5d09283 analytics: Add LoggingCount for messages read stats.
Whenever we use API queries to mark messages as read we now increment
two new LoggingCount stats, messages_read::hour and
messages_read_interactions::hour.

We add an early return in do_increment_logging_stat function if there
are no changes (increment is 0), as an optimization to avoid
unnecessary database queries.

We also log messages_read_interactions::hour Logging stat
as the number of API queries to mark messages as read.

We don't include tests for the case where do_update_pointer is called
because do_update_pointer will most likely be removed from the
codebase in the near future.
2020-06-14 21:15:27 -07:00
Hashir Sarwar 47ece353e6 test_unread: Change flag condition in get_mobile_push_notification_ids.
This uses `where_active_push_notification()` instead of
the flag condition to keep it consistent with the app
code as we use the same function to filter active push
notifications everywhere else.
2020-06-14 20:43:04 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg bff3dcadc8 email: Migrate to new Python ≥ 3.3 email API.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Dinesh d308c12ae2 auth: Add native flow support for Apple authentication.
Overrides some of internal functions of python-social-auth
to handle native flow.

Credits to Mateusz Mandera for the overridden functions.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-14 16:20:12 -07:00
orientor 61c64b3df9 tests: Use validate_against_openapi_schema in testing.
This adds a powerful end-to-end test for Zulip's API documentation:
For every documented API endpoint (with a few declared exceptions that
we hope to remove), we verify that every API response received by our
extensive backend test suite matches the declared schema.

This is a critical step towards being able to have complete, high
quality API documentation.

Fixes #15340.
2020-06-14 15:05:52 -07:00
Hemanth V. Alluri 5bbfaa225b webhooks/circleci: Overhaul the CircleCI integration.
This change should add a lot more information to
our CircleCI integration's output.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-14 13:14:55 -07:00
Aman Agrawal 2e5f860d41 message_edit: Do case-insensitive exact match when editing topics.
When doing query for same topic names in a stream, we should do a
case-insensitive exact match for the topic, since that's the data
model for topics in Zulip.
2020-06-13 16:36:29 -07:00
Aman Agrawal be40a3da99 test_topic_edit: Add test for topic edit case sensitivity.
This is a test case verifying the current codebase produces incorrect
results.  All the messages should have been edited in this case
regardless of the topic's case unless they belong to a different
stream.
2020-06-13 16:35:47 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Anders Kaseorg 46d0018fbb types: Convert ProfileDataElement to Python 3.6 style and total.
The Python 3.6 style does support non-total and even partially-total
TypedDict, but total gives us better guarantees.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:37:25 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg edf411718c openapi: Fix documentation of narrow parameter.
The narrow parameter was incorrectly documented as a one-level array
rather than an array of arrays, and the tests incorrectly expected
this due to a combination of design and implementation bugs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg cfcbf58cd1 do_render_markdown: Remove unused message_user_ids parameter.
It’s unused since commit 7c5f316cb8
(#11586).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Steve Howell 09f5090d1f test: Simplify test_multi_user_subscription.
We triage on the event's "op" field instead
of using the type check.

And then we still get the implicit assertion
that we any "add" event produces a dict
for "subscriptions", since we dereference it
in the assertion for "subscribers".
2020-06-12 14:00:53 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg bf7c421351 test_signup: Remove spurious bare tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 15:36:43 -07:00
Anders Kaseorg aca065e1c1 logging_handlers: Remove spurious bare tuples.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 15:36:43 -07:00
Anders Kaseorg 03e147d5e1 python: Replace NamedTuple with dataclass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 15:19:31 -07:00
Graham Bleaney 461d5b1a3e pysa: Introduce sanitizers, models, and inline marking safe.
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
2020-06-11 12:57:49 -07:00
Graham Bleaney 56345f98cb registration: Incorporate redirect URL building helper function.
This commit adds 'add_query_to_redirect_url' to one additional
function which had not yet been written when
'add_query_to_redirect_url' was introduced. This helper centralizes
URL manipulation for redirects, making it easier to add Pysa
sanitization in subsequent commits.
2020-06-11 12:57:49 -07:00
Anders Kaseorg ca4357fd64 python: Use standard NoReturn (Python ≥ 3.6).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 12:56:52 -07:00
Kartik Srivastava 8c39ddfd28 api: Add GET /users/{user_id}/subscription/{stream_id} endpoint.
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.

Fixes #14966.
2020-06-10 17:59:14 -07:00
Steve Howell d5cc29755e typing: Accept only user_ids in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.

There may be a very tiny number of mobile
clients that still use the old emails API.

This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.

Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.

The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:

    validator=check_list(check_int)

This also allows us to remove a test hack
related to the API documentation.  (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
2020-06-10 17:50:32 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
sahil839 86b52ef7bf users: Owners can only be deactivated by other organization owners.
This commit adds restriction on deactivation of organization owners.
Only owners can deactivate other organization owners.
2020-06-10 17:33:02 -07:00
orientor e989c87a18 openapi: Fix response validation bug.
Currently there are no checks in validate_against_openapi_schema
to check whether the `content` it received actually matched with
the `response code`. For example during testing if a certain endpoint
was returning 400 but it was expected to return 200, then it would
pass schema validation as it would only have `msg` and `result` keys.
Add this validation and fix the wrong response returning points.
2020-06-10 16:45:36 -07:00
orientor 03ef5e0b31 openapi: Add response example validation.
Add test to validate example responses in zulip.yaml. Also change
zulip.yaml for some wrong examples or for cases which were not
covered by `test-api`. Also enhance `validate_against_openapi_schema`.
2020-06-10 15:45:50 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
Hemanth V. Alluri 7b2255597c outgoing-webhook: Fix check for empty content.
Prevent `JsonableError(_("Missing content"))` from
ever being triggered.

That error wasn't handle by anything, and thus just threw a 500, as
it's not a response to an HTTP request.

The right fix is to adjust the caller to ban the empty string in
content (or content that strips to the empty string).

Closes #15145.
2020-06-10 14:46:04 -07:00
Dinesh 0c45e0118f tests: Clear remove/remock approach for ACCESS_TOKEN_URL endpoint.
ACCESS_TOKEN_URL works a different for apple authentication, so,
we removed and remocked the ACCESS_TOKEN_URL mock in
`register_extra_endpoints` override of apple auth test class.
It is cleaner to have it as generic feature of `social_auth_test`.

So, this commit adds a function that returns token_data_dict that
we had earlier and is called in the ACCESS_TOKEN_URL mock.
This function is overriden in apple auth test class to generate
payload of the format that apple auth expects.

Thanks to Mateusz Mandera for the simple idea.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-10 14:38:49 -07:00
Tim Abbott 94d0c330e4 management: Display organization owners in show_admins. 2020-06-10 14:07:38 -07:00
sahil839 22a188f12e migration: Add migration to set all realm admins as realm owners.
This migration is added to set all the realm admins as realm owners
and also creates RealmAuditLog entries for modified users.
2020-06-10 14:07:38 -07:00
sahil839 5cc937f670 events: Add 'is_owner' to page_params and raw_users dict of page_params.
This commit adds 'is_owner' field to page_params and also to the
raw_users dict of page_params.
2020-06-10 14:07:38 -07:00
sahil839 6d667dbe53 realm owner: Add ability to change realm owner status of user.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.

This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.

We need to apply that restriction both in the role change code path
and the deactivate code path.
2020-06-10 12:49:37 -07:00
sahil839 2c8c641556 realm owner: Make first user as realm owner on realm creation.
This commit sets the role of the user creating the realm as
realm owner after the realm is created.

Previously, the role of user creating the realm was set as admin.
But now we want it to be owner because owners have the highest
privilege level.
2020-06-10 12:44:52 -07:00
Anders Kaseorg d3e8af4ad2 python: Replace silly uses of filter().
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 23:31:59 -07:00
Anders Kaseorg 5c9d56d2f7 get_safe_redirect_to: Remove incorrect allowed_hosts setting.
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:51:52 -07:00
Anders Kaseorg 9824405842 dev_settings: Set ZULIP_ADMINISTRATOR.
Fixes this error in the dev environment:

$ ./manage.py checkconfig
Error: You must set ZULIP_ADMINISTRATOR in /etc/zulip/settings.py.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:15:34 -07:00
Anders Kaseorg 04919528e7 models: Parameterize .extra(where=["… IN …"]) with tuple/list adapter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg de22f7a378 migrations: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 859c8b169e stream_recipient: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 68e78c809f get_recent_private_conversations: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 3aab9c03a9 fix_unreads: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 14bbfe6ffb bulk_get_subscriber_user_ids: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 2604ebba38 import_message_data: Use psycopg2.extras.execute_values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 6649c8e35f bulk_insert_ums: Use psycopg2.extras.execute_values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Dinesh dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
David Wood f0f42f7a94 integrations: Add Thinkst Canary integration.
This commit adds an integration for Thinkst Canaries - physical, VM and
cloud-based canaries for detecting attackers to a network. Thinkst
Canaries can send webhook alerts when canaries have been tripped, and
this integration will post Zulip messages when these webhooks are
received.

Signed-off-by: David Wood <david@davidtw.co>
2020-06-09 16:28:28 -07:00
Tim Abbott e242ddc848 page_params: Add community_topic_editing_limit_seconds.
This was previously hardcoded with agreement between the Zulip backend
and frontend as 86400 seconds (1 day).  Now, it's still hardcoded in
the backend, but arranged in a way where we could add a setting
without any changes to the mobile and terminal apps to update logic.

Fixes #15278.
2020-06-09 14:40:12 -07:00
Tim Abbott cb8290ac57 test_home: Stop checking for removed string. 2020-06-08 23:09:21 -07:00
Tim Abbott 2beaf2cab2 pointer: Remove have_initial_messages code.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history.  However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
2020-06-08 22:36:18 -07:00
Anders Kaseorg 47b4e45931 markdown_test_cases: Update encoded zulipchat.com links too.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 19:47:07 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 95be7dcbab portico: Remove mystery hunt landing page.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
2020-06-08 18:10:45 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Dinesh d30f11888a logging: Set up a different logger for each backend.
Adds a top-level logger in `settings.LOGGING` `zulip.auth`
with the default handlers `DEFAULT_ZULIP_HANDLERS` and
an extra hanlder that writes to `/var/log/zulip/auth.log`.

Each auth backend uses it's own logger, `self.logger` which
is in form 'zulip.auth.<backend name>'.

This way it's clear which auth backend generated the log
and is easier to look for all authentication logs in one file.

Besides the above mentioned changes, `name` attribute is added to
`ZulipAuthMixin` so that these logging kind of calls wouldn't raise
any issues when logging is tried in a class without `name` attribute.

Also in the tests we use a new way to check if logger calls are made
i.e. we use `assertLogs` to test if something is logged.

Thanks to Mateusz Mandera for the idea of having a seperate logger
for auth backends and suggestion of using `assertLogs`.
2020-06-08 17:42:07 -07:00
Anders Kaseorg b1b34b6f6a dev_login: Move owners to the top.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 17:26:34 -07:00
sahil839 7de23b8b5c users: Remove short_name and client_id from get_profile_backend.
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.

* short_name hasn't been present anywhere else in the Zulip API for
  several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
  returned by the register_event_queue process.

The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
2020-06-08 17:01:08 -07:00
sahil839 1f5778bad7 users: Refactor get_profile_backend to be based on format_user_row.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.

To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.

This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
2020-06-08 16:57:44 -07:00
sahil839 2f7d684a84 slack_import: Map slack owners to zulip realm owners.
Slack owners and primary owners will be mapped to zulip
realm owners on import.

Previously, we mapped the owner and primary owner roles of slack
to realm admins in zulip. As we have added ROLE_REALM_OWNER in
8bbc074, we now map slack owners and primary owners to owners in
zulip.

Tests are modified for checking all the 3 cases-
 - Slack workspace primary owner
 - Slack workspace owner
 - Slack workspace admin

This commit also has docs changes in 'import-from-slack.md'.
2020-06-08 16:22:54 -07:00
Anders Kaseorg 8e4f22c184 auth: Require algorithms setting for JWT auth.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning.  This is for protecting against
symmetric/asymmetric key confusion attacks.

This is a backwards-incompatible configuration change.

Fixes #15207.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 16:22:25 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 8c15081069 python: Further pyupgrade changes.
Generated by pyupgrade --py3-plus --keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:23:10 -07:00
Anders Kaseorg 444fbbf964 python: Whitespace fixes from autopep8.
Generated by autopep8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:21:30 -07:00
Anders Kaseorg 549fbeac72 test_narrow: Convert a triple-quoted format string to single-quoted.
This works around a semgrep bug following the conversion to f-strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:19:59 -07:00
Anders Kaseorg 139cb8026f auth: Accept next as POST parameter in POST requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 11:07:32 -07:00
Johannes Hochreiter d78d9099d1 gitlab: added pipeline number to event notification
- link to pipeline now also features the pipeline number
- updated tests accordingly
2020-06-07 14:01:04 -07:00
Johannes Hochreiter 60077f3097 gitlab: Added artifact link to pipeline event body
- if artifact was built file name, download link and browse link will be added to pipeline event body
- added test for pipeline with built artifacts
2020-06-07 14:01:04 -07:00
Aman Agrawal e2ed155310 do_update_message: Send old topic with stream edit event.
Old topic of the msg edit event can be used to help the client
calculate useful information such as if a change
in current narrow is required.

This fixes our re narrow logic after a stream edit of a topic, with
no change in topic name itself, since the original topic was not
present in the event received and hence the `orig_topic` was
undefined in this case.
2020-06-07 11:20:34 -07:00
sahil839 c394337118 tests: Refactor 'attempt_unsubscribe_of_principal` to accept list of users.
This adds 'target_users' parameter to 'attempt_unsubscribe_of_principal`
function in test_subs.py, which accepts list of UserProfile objects to be
unsubscribed, instead of defining users in the function itself.

This change makes the code cleaner and more readable.

Also, 'other_user_subbed' parameter is changed to 'target_users_subbed'
to clearly depict the use of this parameter.
2020-06-07 11:18:37 -07:00
Anders Kaseorg 19cc22e5ab queue: Fix types to reflect that Pika channels transmit bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:09:24 -07:00
anshgit101 7f4aad4511 populate_db:Add uppercase topic name randomly. 2020-06-06 09:25:13 -04:00
Anders Kaseorg a236256d64 test_slack_importer: Fix mocked_requests_get type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
Anders Kaseorg 4b44d87477 test_push_notifications: Fix mock_apns type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
Anders Kaseorg 22178c169e request: Replace type argument hack with a different workaround.
This works in mypy 0.770 and is needed for mypy 0.780.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
Tim Abbott 42f1cb3444 management: Fix delete_old_unclaimed_attachments. 2020-06-05 17:06:19 -07:00
Kartik Srivastava 5cb1edcadc openapi/python_examples: Update examples for add/remove subscriptions.
This updates the examples to use user ids to add or remove subscriptions
from a stream.
2020-06-05 16:11:04 -07:00
sahil839 43bff1f67a streams: Extract removing_someone_else logic to a function.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.

This is a follow up to 2187c84.
2020-06-05 16:04:28 -07:00
Aman Agrawal a096f34cab move_topic_to_stream: Add option to disable breadcrumb messages.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.

User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
2020-06-05 12:28:51 -07:00
Aman Agrawal ec257f6ec3 to_dict_uncached: Pipe realm_id from the caller functions.
We pipe realm_id through functions where it is available,
this helps us avoid doing query for realm_id in loop when
multiple messages are being processed.
2020-06-05 12:03:31 -07:00
Aman Agrawal 4b6ce8473e test_messages: Check number of queries when a topic is moved. 2020-06-05 12:03:31 -07:00
Mateusz Mandera 5b746b4128 docs: Add missing step to dropbox webhook instructions. 2020-06-05 10:21:25 -07:00
Anders Kaseorg 0bcd4a6281 email_mirror: Avoid parameters that shadow names in their own types.
They confuse semgrep, and also, like, people in general.

https://github.com/returntocorp/semgrep/issues/923

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:35:11 -07:00
Anders Kaseorg 8d20d1e632 models: Set a timezone on the MutedTopic.date_muted default.
Fixes warnings like this:

/srv/zulip-py3-venv/lib/python3.8/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField MutedTopic.date_muted received a naive datetime (2020-01-01 00:00:00) while time zone support is active.
 warnings.warn("DateTimeField %s received a naive datetime (%s)"

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Anders Kaseorg 0797626911 bugdown: Avoid deprecated ElementTree.getchildren method.
Fixes warnings like these with python -Wd:

/home/circleci/zulip/zerver/lib/bugdown/__init__.py:327: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  for child in currElementPair.value.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:328: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  if child.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:282: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  for child in currElement.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:283: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  if child.getchildren():

https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-04 14:17:08 -07:00
Anders Kaseorg b94b350ff0 bot_config: Avoid deprecated ConfigParser.readfp method.
Fixes this warning with python -Wd:

/home/circleci/zulip/zerver/lib/bot_config.py:69: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  config.readfp(conf)

https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.readfp

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-04 14:16:17 -07:00
Anders Kaseorg 95f29b5409 tests: Avoid deprecated TestCase method aliases.
Fixes these warnings with python -Wd:

/home/circleci/zulip/zerver/tests/test_middleware.py:43: DeprecationWarning: Please use assertRegex instead.
  self.assertRegexpMatches(
/home/circleci/zulip/zerver/webhooks/travis/tests.py:40: DeprecationWarning: Please use assertNotEqual instead.
  self.assertNotEquals(msg.topic_name(), self.TOPIC)

https://docs.python.org/3/library/unittest.html#deprecated-aliases

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-04 14:15:52 -07:00
Anders Kaseorg e91050b674 migrations: Replace deprecated django.db.backends.postgresql_psycopg2.
This has been a deprecated alias for django.db.backends.postgresql
since Django 1.9, removed in Django 3.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 67dbb3b2a9 bugdown: Replace deprecated markdown members with md.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg f65af9cdb7 bugdown: Replace deprecated markdown.util.etree with ElementTree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 01ccb75259 bugdown: Replace deprecated cElementTree with ElementTree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg aad0b8b2da bugdown: Add assert to work around type problem.
url_to_a returns Union[Element, str], but str cannot be appended to
Element; that would raise TypeError at runtime.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 10805abd6e python: Fix errors from pycodestyle 2.6.0.
zerver/lib/i18n.py:34:28: E741 ambiguous variable name 'l'
zerver/lib/webhooks/common.py:103:34: E225 missing whitespace around operator
zerver/tests/test_queue_worker.py:563:9: E306 expected 1 blank line before a nested definition, found 0

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Hemanth V. Alluri 9a9c1e0794 webhooks/stripe: Fix the invoice_created event.
The previous code for this event was using a key that's not actually
a part of the payload. So here we simple remove the usage of that key
and add a (previously missing) test for this event.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-03 17:18:43 -07:00
Tim Abbott 7dbdfe9a97 models: Remove database columns from old Zoom integration. 2020-06-03 16:39:12 -07:00
Anders Kaseorg 4d04fa3118 compose: Rewrite Zoom video call integration to use OAuth.
This reimplements our Zoom video call integration to use an OAuth
application.  In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.

Fixes #11672.

Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-06-03 16:39:12 -07:00
Anders Kaseorg 7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
Anders Kaseorg fa9bddcfb2 emoji: Add workaround for collectstatic step of build-release-tarball.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
orientor 1227b65066 openapi: Forbid opaque objects from OpenAPI documentation.
Objects whose properties are not described were validated by the
current validator. Edit it so that objects with no `properties`
or `additionalProperties` attribute i.e. opaque objects get
invalidated.

Also make changes in zulip.yaml to fix any opaque objects (tweaked by
tabbott to edit the documentation for better clarity).
2020-06-02 16:45:06 -07:00
orientor 8224400166 openapi_responses: Improve response validation tests.
Currently, `validate_against_openapi_schema` checks only the top
level of the response dictionary. Improve it so that it can
validate objects and arrays at all levels. Also edit zulip.yaml
accordingly. And for new response keys which were not defined
before add VERY basic documentation.
2020-06-02 16:34:54 -07:00
sahil839 0b432e26b6 test_subs: Add tests for unsubscribing multiple users from stream.
This commit adds tests for unsusbcribing multiple users from a
stream and fixes the missing coverage issue introduced in 2187c84.
2020-06-02 16:27:22 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Hemanth V. Alluri 18f8859baa refactor: Remove unnecessary request_data param in outgoing_webhook.
request_retry and notify_bot_owner don't use request_data so might
as well not send it to them at all.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-02 15:19:27 -07:00
Hemanth V. Alluri 9c172036bb refactor: Make OutgoingWebhookServiceInterface more interface like.
Using the Python Standard Library's abc library and NotImplementedError
we can better define interfaces (this is mainly to improve readability
and consistency).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-02 15:19:27 -07:00
Mateusz Mandera 3e7fc17788 auth: Delegate RemoteUser SSO to browser when using the desktop app. 2020-06-02 13:00:17 -07:00
Tim Abbott f93c19ec62 openapi: Fix various errors in OpenAPI declarations.
Based on original work by Aditya Verma to tag places where our API
documentation was inaccurate / incomplete.
2020-06-02 11:54:19 -07:00
Puneeth Chaganti 788203778d tools: Use the correct logo path to generate bot avatar.
Integrations can be configured with specific logo paths, which weren't
correctly being used for generating the bot avatars.
2020-06-01 18:08:23 -07:00
Puneeth Chaganti dfd4b4d051 integrations: Store logo_path as an attribute along with logo_url.
Integrations can be supplied a logo parameter which is used to contruct
their `logo_url`. It would be useful to store this parameter, instead of
computing the path from the URL.
2020-06-01 18:08:23 -07:00
Puneeth Chaganti 752fe58f4f tools: Allow specifying & capturing screenshots for all integrations. 2020-06-01 18:08:22 -07:00
Puneeth Chaganti b533420d3b tools: Allow specifying extra query params in screenshot config. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 56c31a8647 tools: Allow specifying the bot name for notifications. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 55444c33a0 tools: Allow passing payload as a query param. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti d4c6195bca tools: Allow specifying use of basic auth in ScreenshotConfig. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 5ffc51382d tools: Allow specifying custom headers in ScreenshotConfig. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti bb37e4ef00 integrations: Remove corrupted png image in bitbucket docs. 2020-06-01 18:03:06 -07:00
sahil839 c7cbf66553 tests: Fix tests to use role instead of is_admin for updating user.
This commit fixes the tests to use role instead of is_admin in
update user endpoint. These changes got missed in the original
commit 9fa6067 which included the change of using role in update user
endpoint and were also not caught in tests.
2020-06-01 16:13:05 -07:00
sahil839 9a0202c0c1 test_users: Remove redundant lines from test for changing full name.
This commit removes redundant lines from the test for changing
full name in test_users.py. Those lines were passing is_admin=False
for already non admin user and were added in 41fbb16, but these lines
are of no use now.
2020-06-01 16:13:05 -07:00
Tim Abbott fbcf6a7d72 streams: Add nocoverage to fix CI. 2020-06-01 16:12:13 -07:00
sahil839 2187c84ed9 stream: Add support for sending user_id to endpoint when subscribing users.
This commit modifies the backend to accept user ids when subscribing
users to streams.

It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
2020-06-01 15:59:04 -07:00
sahil839 9ef1c5b1a6 users: Add is_owner field to user objects returned by get endpoints.
This commit adds 'is_owner' field to the user object returned by
'/users', 'users/{user_id}', and '/users/me' endpoints.
2020-06-01 15:33:51 -07:00
sahil839 bd9e891e1c dev_login: Change dev env login page to show Desdemona as owner.
This commits changes the dev login page to show Desdemona as
owner.
2020-06-01 15:22:44 -07:00
sahil839 2fe8971204 populate_db: Change role of 'desdemona' to realm owner.
Modified by tabbott to update PROVISION_VERSION.
2020-06-01 15:22:27 -07:00
sahil839 169bd3082a users: Modify format_user_row to set is_admin as True for owners also. 2020-06-01 15:14:16 -07:00
Mateusz Mandera 90b2f933b0 tests: Change self.client_post to client_get in remote sso tests.
GET is the intended way to use this endpoint, this is how the mobile and
desktop apps pass their params.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 4b10a07c3e auth: Fix social signup with the desktop app.
Only start_social_login had the code for handling the desktop
application, it was missing from start_social_signup.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 676305f6ab auth: Extract handle_desktop_flow wrapper. 2020-06-01 14:14:58 -07:00
sahil9001 328caf1ad5 api: Simplify format of realm_emoji author data.
There's no reason to send data beyond the user `id` of the uploader,
and reason not to, as the previous model was both awkward when
`author=None` and resulted in unecessary parsing complexity for
clients.

Modified by tabbott to add the frontend changes and API documentation.

Fixes #15115.
2020-05-31 17:44:50 -07:00
Tim Abbott c78d728812 events: Fix test_events race handling of role changes.
Previously, our apply_events logic (verified by test_events) did not
properly handle changing a user's role to and from being a guest.
2020-05-31 17:31:33 -07:00
sahil839 2ab6767b73 events: Update person dict in event for do_change_user_role to send role.
This commit changes the person dict in event sent by do_change_user_role
to send role instead of is_admin or is_guest.

This makes things much more straightforward for our upcoming primary
owners feature.
2020-05-31 17:22:50 -07:00
Mateusz Mandera 200ce821a2 user_activity: Put client id instead of name in event dicts.
This saves the completely unnecessary work of mapping the Client name
to its ID.  Because we had in-process caching of the immutable Client
objects, this isn't a material performance win, but it will eventually
let us delete that caching logic and have a simpler system.
2020-05-29 15:19:55 -07:00
Mateusz Mandera acfb378245 tests: Change some assertTrue usage where assertEqual was meant.
These were typos, causing the asserts to pass always, because the syntax
was meant for assertEqual.
2020-05-29 15:15:52 -07:00
orientor 5b5095ad9f openapi: Use correct `operationId` atrribute for endpoints.
Update `operationId` of some endpoints to their correct version.
2020-05-29 15:04:50 -07:00
orientor d2ee99a2fd openapi: Add markdown extension for rendering return values in API docs.
Currently response return values have to be written twice, once in
the docs and once in zulip.yaml. Create a markdown extension so
that the return values in api docs are rendered using content from
zulip.yaml
2020-05-29 15:02:56 -07:00
orientor 98903df639 openapi: Fix some zulip.yaml return values description.
There are several places where the return values description is better
in the api docs than in the zulip.yaml file. Import such changes to
zulip.yaml before we replace the current hacky API docs with the
zulip.yaml content.
2020-05-29 15:02:43 -07:00
sahil839 9fa60672e6 users: Modify update user API endpoint to accept role as parameter.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.

User role dropdown in user info modal is also modified to use
"dropdown_options_widget".

Modified by tabbott to document the API change.
2020-05-29 14:29:17 -07:00
Alex Vandiver 107fe3d3b4 tests: Remove references to trac.zulip.net.
That host has not existed for some time; rename the tests to use a
URL that is clearly nonfunctional.
2020-05-28 17:24:35 -07:00
Anders Kaseorg ba7906a3c6 bugdown: Revert NamedTuple declarations to old style.
This reverts part of commit 8bcdf4ca97
(#15093), to work around a mypy caching bug:
https://github.com/python/mypy/issues/7281.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-28 16:47:08 -07:00
Alex Vandiver 0288002b84 models: Comment call to python2-esque `iteritems()`.
This method is called on a `bitfield.types.BitHandler` object, not a dict; as
such, it is not a runtime failure.

This was missed in 035ed93111 and adjacent commits because it was not
`six.iteritems()` but just a bare `iteritems()` -- having been added
after 9e3c3e14f5 moved all other calls from `iteritems()` to
`six.iteritems()`.

Comment on why this is not a leftover python2-ism.

Grep finds no other remaining callsites to `iteritems` or friends:
```
$ rg '\.iter\w+\(' -t py
zerver/lib/url_preview/preview.py
44:        content = next(response.iter_content(1000))

zerver/lib/email_notifications.py
49:    for link_info in fragment.iterlinks():

zerver/lib/push_notifications.py
508:        nested_levels = len(list(ol.iterancestors('ol')))

zerver/lib/bugdown/__init__.py
1364:            parsed = etree.iterparse(StringIO(rendered))

zerver/lib/bugdown/nested_code_blocks.py
47:                if parent.text is None and len(list(parent)) == 1 and len(list(parent.itertext())) == 1:
```
2020-05-28 15:56:00 -07:00
Imran Iqbal 465947d1bb
webhooks: Fix travis display emoji for `Pending` status.
The 🔄 emoji is spelled  :counterclockwise: in Zulip.
2020-05-28 15:02:30 -07:00
Hemanth V. Alluri 37d34dda1c auth: Make sure that we enforce max_length during password reset.
Make sure that we use the max password length defined in
RegistrationForm.MAX_PASSWORD_LENGTH when validating the
password for the password reset form.

Fixes #15087.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-28 15:01:33 -07:00
Tim Abbott 4d2b1673f8 docs: Replace support@zulipchat.com with support@zulip.com.
The new address is cleaner and shorter.
2020-05-28 08:14:30 +00:00
sahil839 7d9d2414ca actions: Modify do_change_user_role for realm owner.
This commit changes do_change_user_role to support adding or removing
the realm owner status of user and sending an event.

We also extend the existing test for do_change_user_role to do a bit
more validation to confirm the audit log records all values of role.
2020-05-27 15:52:39 -07:00
sahil839 8bbc07474f models: Add realm owner role and is_realm_owner property for user.
The new realm_owner role is added as option for role field in
UserProfile model and is_realm_owner is added as property for the user
profile.

Aside from some basic tests validating the logic, this has no effect
as users cannot end up with set as realm owners.
2020-05-27 15:47:42 -07:00
clarammdantas f0c85c5796 registration: Show error if invalid key is posted on /accounts/register.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
2020-05-27 15:37:16 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
clarammdantas 923c5e3063 test_signup: Check if user is already registered.
This tests if a user, that is already registered, is
redirected to the login page when they click on an
invitation.

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
clarammdantas 8869f05c20 test_signup: Attach prereg_user to confirmation obj.
Tests attached a UserProfile to confirmation objects,
which is not very valid as this is the only place
where this is done. Now we attach PreregUser to
the confirmation object, making the tests correct.

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
orientor 7e1bc9bf2f openapi_responses: Fix get_user_presence. 2020-05-27 13:23:37 -07:00
orientor cffb927f07 openapi_responses: Fix get_own_user. 2020-05-27 13:23:36 -07:00
orientor 1273545637 openapi_responses: Fix get_messages. 2020-05-27 13:21:04 -07:00
orientor 4096e0e6b9 openapi_responses: Fix list_subscriptions. 2020-05-27 13:15:51 -07:00
orientor a4cf77e5dc openapi_responses: Fix get_message_history. 2020-05-27 13:15:51 -07:00
orientor 29ef17c999 openapi_responses: Fix get_realm_emoji. 2020-05-27 13:15:51 -07:00
orientor b9c31942b5 openapi_responses: Fix get_user_groups. 2020-05-27 13:15:51 -07:00
orientor 74e9019922 openapi_responses: Fix get_stream_topics. 2020-05-27 11:27:27 -07:00
orientor ae8167cd09 openapi_responses: Fix get_users and get_user. 2020-05-27 11:27:26 -07:00
orientor bc50308799 openapi_responses: Fix send_message. 2020-05-27 11:23:14 -07:00
orientor f50aab6740 openapi_responses: Fix update_subscription_settings. 2020-05-27 11:23:12 -07:00
orientor 6d438cbba4 openapi_responses: Fix get_streams. 2020-05-27 21:29:14 +05:30
whoodes cea7d713cd requirements: Upgrade boto to boto3.
Fixes: #3490

Contributors include:

Author:    whoodes <hoodesw@hawaii.edu>
Author:    zhoufeng1989 <zhoufengloop@gmail.com>
Author:    rht <rhtbot@protonmail.com>
2020-05-26 23:18:07 -07:00
sahil9001 94dfe4bed3 portico: Fix apps page webapp link being useless.
The /apps page webapp link now takes the user to /accounts/go to find
their organization's login page, rather than failing to do anything.

Fixes #14977.
2020-05-26 21:50:31 -07:00
arunikaydav42 4680d504de invite: Fix invitations removed from list too soon.
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.

Fixes: #12281
2020-05-26 21:45:44 -07:00
Tim Abbott 368e9572cc billing: Clean up copy on free trial pages.
This fixes some issues with unclear terminology and visual styling in
the pages for the new free trial.

There's probably more we can and should usefully do in the future.
2020-05-26 21:31:07 -07:00
Vishnu KS 8784539d53 free trial: Send users to /upgrade after realm creation. 2020-05-26 17:01:32 -07:00
Anders Kaseorg cf923b49d3 python: Remove extra pass statements with autoflake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 8bcdf4ca97 python: Convert TypedDict declarations to Python 3.6 style.
A subset of the diff generated by pyupgrade --py36-plus
--keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg f5b33f9398 python: Further pyupgrade changes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Tim Abbott 463f1503fc Revert "markdown: Process fenced code blocks in blockquotes."
This reverts commit 7002f98ea1.

This failed tests due to some sort of conflict with a recent
python-markdown upgrade.
2020-05-25 18:13:03 -07:00
Rohitt Vashishtha 7002f98ea1 markdown: Process fenced code blocks in blockquotes.
We handle fenced code blocks in a preprocessor, and > style blockquotes
are parsed in a blockprocessor. Pymarkdown doesn't run the preprocessors
again on any blocks that it is parsing, and is unlikely to accept our
solution upstream; they intend to convert fenced_code to a block parser.

We simply run all the preprocessors on the text again, with the exception
of NormalizeWhitespace which removed delimiters used by HtmlStash to mark
preprocessed html code. To counter this, we subclass NormalizeWhitespace
and use our customized version for when it is called from a blockparser.

Upstream issue: https://github.com/Python-Markdown/markdown/issues/53

Fixes #12800.
2020-05-25 17:35:10 -07:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
Mateusz Mandera b66dc9de50 saml: Support IdP-initiated SSO. 2020-05-25 16:09:30 -07:00
Mateusz Mandera f2d052bff8 tests: Flush session before a simulated cross-domain POST in saml tests.
This is important, because lack of this meant that the POST request in
our tests still had the old session, with various params stored in it.
This mechanism doesn't work in reality in SAML, so the backend uses
redis to store and recover the params from redis. Without flushing the
session, these tests would fail to catch some breakages in the
redis-based mechanism.
2020-05-25 15:53:15 -07:00
Mateusz Mandera 13c3eaf086 rate_limit: Restrict tornado backend to explicitly specified domains.
This will protect us in case of some kinds of bugs that could allow
making requests such as password authentication attempts to tornado.
Without restricting the domains to which the in-memory backend can
be applied, such bugs would lead to attackers having multiple times
larger rate limits for these sensitive requests.
2020-05-25 15:40:00 -07:00
Vishnu KS 8b3eb9a846 landing: Add option to show custom navbar message. 2020-05-24 17:34:03 -07:00