Commit Graph

5597 Commits

Author SHA1 Message Date
Anders Kaseorg dbd1b56362 remote_server: Fix send_to_push_bouncer type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-05 11:53:45 -07:00
Steve Howell 0b65abcdf5 pointer: Remove pointer from UserProfile.
Most of the changes here are just that we no
longer need to provide a value for pointer
when we create UserProfile objects.
2020-07-03 13:08:40 +00:00
Anders Kaseorg c08ee904d8 models: Add explicit id fields for better type checking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 13:28:10 -07:00
Anders Kaseorg 9b7c6828ec models: Annotate most field types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 13:28:10 -07:00
Anders Kaseorg 8c64a14a31 markdown: Fix get_user_by_id type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 13:28:10 -07:00
Tim Abbott 0c48c75920 push_notifications: Fix events with no message_ids.
This fixes our triggering a RabbitMQ event to send a push notification
to remove the empty set of push notifications, resulting from not
using the correct data structure to determine which message IDs to look at.

This was causing a lot of visible exceptions when running
the `test_messages.py` test suite.
2020-07-01 14:58:43 -07:00
Tim Abbott c6638a62dc message_edit: Clean up comments around moving topics. 2020-07-01 14:55:17 -07:00
Tim Abbott 03cf3edf42 message_edit: Fix typo in old_stream_unsubbed_guests. 2020-07-01 14:54:36 -07:00
Tim Abbott 7e11be0429 message_edit: s/subscribers_ids/subscriber_ids/. 2020-07-01 14:51:58 -07:00
Tim Abbott bda7116f86 message_edit: Reuse delete_event_notify_user_ids.
This better describes semantically what we have in mind here -- every
user should get one of the two events.
2020-07-01 14:50:32 -07:00
Aman Agrawal 3f42d15168 move_topic_to_stream: Delete UserMessage for new stream unsubs.
For users who are unsubscribed from the new stream but are in
the old stream, we delete the UserMessage.

We send the delete_message event only to guest users,
who have completely lost asses to the moved messages, for other
users we send the normal update_message event which moves
the messages to the new unsubed stream which
otherwise would look broken to the
user without reloading to the webpage.
2020-07-01 14:42:01 -07:00
orientor 529da34513 openapi: Use third-party validator for schema validation.
Our previous OpenAPI schema validator that we implemented ourselves
was useful training wheels for our understanding OpenAPI properly, and
was mostly correct.  But given that we've finally reached the point
where our OpenAPI file accurately describes the API, it makes sense to
switch to use an official OpenAPI validator.  We lose some ability to
do exclude rules for particular elements, but those were primarily
important for us when we had a lot of them.

As part of this change, we need to add `additionalProperties: false`
for all of our dictonaries/objects where we've documented every
parameter; otherwise the OpenAPI schema checker won't know that we
expect every parameter to be documented.
2020-07-01 11:21:41 -07:00
Vishnu KS 908e1b6bea 18n: Cleanup strings to be translated in do_rename_stream. 2020-07-01 11:03:29 -07:00
Vishnu KS c7ce9097d0 18n: Cleanup strings to be translated in send_pm_if_empty_stream. 2020-07-01 11:03:29 -07:00
Anders Kaseorg 1b96af2987 cache: Fix type: ignore issues.
This was hiding an actual type error in test_cache: a mismatch between
the object ID type, which is str, and the default id_fetcher, which
returns int.

Mypy’s insufficient support for default generic arguments basically
means we can’t use them without a lot of overloading, and there are
not enough callers here to justify that.

https://github.com/python/mypy/issues/3737

We avoid this being super messy where the code calls this by adding
some less generic wrappers for generic_bulk_cached_fetch.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:51:15 -07:00
Anders Kaseorg bb8dcb9b1e test_runner: Remove customized run_test, TestSuite, and TestLoader.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg e40cdc6c2c test_runner: Move bounce_key_prefix_for_testing to ZulipTestCase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg a7c0cca704 test_runner: Replace full_test_name with TestCase.id.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg 3d917ddfde test_runner: Remove unused ReturnT variable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg 92d0290dc5 tests: Remove slow tests detection.
According to @showell:

> All the slow decorators can die. That was a failed experiment of
> mine from 2014 days.  I have meaning to kill them for a couple years
> now.  I wrote this with the best of intentions, but I believe it's
> now just cruft.  We never made a "fast" mode, for one. And we kept
> writing more and more slow tests, haha.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 23:34:53 -07:00
Anders Kaseorg ec70aab138 test_service_bot_system: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 83b6884d8b test_decorators: Remove tests for conditions already excluded by mypy.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 5fbccbf64d test_helpers: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 8a7cfd05a1 sessions: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg bc9663d57f webhooks: Fix a type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg f2e7076e2a decorator: Replace type: ignore with cast, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg e582bbea4a decorator: Strengthen types of signature-preserving decorators.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 96cf928d27 models: Fix user_profile_id type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg b655029ced rest: Strengthen default_never_cache_responses type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 8667e632c8 python: Use isinstance with a tuple for testing multiple types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Aman Agrawal 5b7917da5f notifications: Remove support for unbatched push removal events.
We remove support for the old clients which required an event for
each message to clear notification.

This is justified since it has been around 1.5 years since we started
supporting the bulk operation (and so essentially nobody is using a
mobile app version so old that it doesn't support the batched
approach) and the unbatched approach has a maintenance and reliability
cost.
2020-06-30 10:12:27 -07:00
Tim Abbott f682008585 markdown: Delete unused log_markdown_error.
Its used was removed in 50ed378dd6.
2020-06-29 15:17:27 -07:00
Mohit Gupta 08e74558a9 refactor: Rename remaining bugdown word to markdown in .py files.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 15:03:20 -07:00
Mohit Gupta 66d54d86ad refactor: Rename remaining bugdown_convert to markdown_convert.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 15:03:20 -07:00
Mohit Gupta 0552936c9b refactor: Rename bugdown to markdown in message.py and test_messages.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta e13691aa10 refactor: Rename log_bugdown_error to log_markdown_error.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta c1b6fbbc7d refactor: Rename bugdown to markdown in import_realm.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta bb8cafdcf1 refactor: Rename remaining bugdown refrences to markdown in actions.py.
This commit removes bugdown alias and do proper imports from markdown
module. Also remove bugdown word and replace it with markdown in
comments.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta 6cca775557 refactor: Rename alias bugdown_verion to markdown_version.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Steve Howell 987408db68 test_events: Extract BaseAction. 2020-06-29 12:38:26 -07:00
Mateusz Mandera 9392e8bec7 emails: Add a custom header specifying the organization when possible.
Closes #15135.
2020-06-28 13:33:02 -07:00
Aman Agrawal 5f82e1a984 push_notif: Don't clear notif if stream notif enabled.
If a message was edited to remove a user mention, we don't
remove the push_notification for the user if the user has
stream_push_notfications enabled.
2020-06-28 11:51:50 -07:00
Steve Howell 69be97e365 pointer: Remove pointer from API and page_params.
There is still some miscellaneous cleanup that
has to happen for things like analytics queries
and dead code in node tests, but this should
remove the main use of pointers in the backend.

(We will also still need to drop the DB field.)
2020-06-27 16:44:38 -07:00
Mohit Gupta df701ba779 refactor: Rename remaining bugdown to markdown in markdown/__init__.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:23:15 -07:00
Mohit Gupta 44d68c1840 refactor: Rename bugdown words to markdown in stats related functions.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:20:40 -07:00
Mohit Gupta 97872591ef refactor: rename ZEPHYR_MIRROR_BUGDOWN_KEY and DEFAULT_BUGDOWN_KEY
rename ZEPHYR_MIRROR_BUGDOWN_KEY to ZEPHYR_MIRROR_MARKDOWN_KEY and
DEFAULT_BUGDOWN_KEY tp DEFAULT_MARKDOWN_KEY.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:20:40 -07:00
Mohit Gupta 0241111aef refactor: Rename bugdown_logger to markdown_logger.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:20:40 -07:00
Mohit Gupta 2dde51053f refactor: Rename class name Bugdown to Markdown.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 5a79c48603 refactor: Rename BugdownListPreprocessor to MarkdownListPreprocessor.
This commits changes class name of MarkdownListPreprocessor to
MarkdownListPreprocessor. It also changes corresponding references
in tests.
This is part of series of commits which aims for renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 05cce86670 refactor: Change BugdownRenderingException to MarkdownRenderingException.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 3f5fc13491 refactor: Rename zerver.lib.bugdown to zerver.lib.markdown .
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
2020-06-26 17:08:37 -07:00
sahil839 6eb8442a59 invites: Send user_id of the referrer instead of email in invites dict.
We send user_id of the referrer instead of email in the invites dict.
Sending user_ids is more robust, as those are an immutable reference
to a user, rather than something that can change with time.

Updates to the webapp UI to display the inviters for more convenient
inspection will come in a future commit.
2020-06-26 16:57:50 -07:00
orientor 5629dcc8a6 openapi_docs: Display deprecated parameters with a `deprecated` tag.
In zulip.yaml, add `deprecated` tags to all parameters/keys with
`Deprecated` in the description. Then add tests to ensure that deprecated
parameters/keys will always have the `deprecated` key. Also, in
the API docs, sort the parameters according to presence of `deprecated`
key, presenting the `deprecated` keys at the end and add a `deprecated`
tag next to them.
2020-06-26 16:05:41 -07:00
Vishnu KS 28f5e86c7c i18n: Set the correct language in notify_topic_moved_streams. 2020-06-26 14:57:18 -07:00
Vishnu KS 5178c58209 i18n: Mark notification bot message in do_create_realm for translation. 2020-06-26 14:57:18 -07:00
Vishnu KS d42515df1f i18n: Set the correct language in do_rename_stream. 2020-06-26 14:57:18 -07:00
Vishnu KS e27921dbe3 i18n: Set the correct language for translation in send_pm_if_empty_stream. 2020-06-26 14:57:17 -07:00
Vishnu KS a174d8b755 i18n: Mark notification bot message in process_new_human_user for translation. 2020-06-26 14:57:17 -07:00
Vishnu KS ce6203906f i18n: Mark notification bot message in notify_new_user for translation. 2020-06-26 14:57:16 -07:00
Aman Agrawal 984c2d2777 push_notification: Remove notification if user no longer mentioned.
We send a remove mobile push notification to the users who were
no longer mentioned after the content of the message was edited.

This also corrects the notification count for the mobile apps
where a user was prior mentioned in a muted stream / topic and the
message was edited and the user is no longer mentioned now.
Hence, fixing the case where user has read all his unreads
but the notification badge on the app is still positive.

Fixes #15428.
2020-06-26 11:45:28 -07:00
Aman Agrawal 4612ee511f clear_push_notification: Upgrade method to accept multiples users.
do_clear_mobile_push_notifications_for_ids can now be used to
clear push_notification for multiple users at once. This method
loops over users, so no performance optimization is gained.
2020-06-26 11:25:59 -07:00
Tim Abbott a5be2a30fa events: Fix buggy realm_user/update events during user creation.
We've been seeing an exception in server_event_dispatch.js in
production where in large organizations, sometimes when a new user
joined, every other browser in the organization would throw an
exception processing some sort of realm_user/update event.

It turns out the cause was that when a user copies their profile from
an existing user account with a user-uploaded avatar, the code path we
reused to set the avatar properly send a realm_user/update event about
the avatar change -- for a user that hadn't been fully created and
certainly hadn't have the realm_user/add event sent for.

We fix this and add tests and comments to prevent it recurring.

(Removed an incorrect docstring while working on this).
2020-06-26 11:21:11 -07:00
Chris Heald 42f2399155 markdown: Escape HTML entities in inline code blocks.
This fixes an issues that causes HTML entities inside of inline code
blocks to be converted rather than being displayed literally.

The upstream python-markdown now handles this correctly, so we just use
their implementation with our changes for removing .strip(). As a result
of this migration, we switch backtick pattern to an inline processor
too.

Fixes #12056.

For the codeblock counterpart of this issue, we should follow the
upstream PR https://github.com/Python-Markdown/markdown/pull/990.

Co-authored-by: Rohitt Vashishtha <aero31aero@gmail.com>
2020-06-25 14:46:33 -07:00
Steve Howell 6b910ff3b4 widgets: Make type checks more explicit.
Note that I don't actually convert the
checker from check_dict to check_dict_only,
because that would be a user-facing change,
but I think we can sweep a lot of things
like this after the next release.
2020-06-25 13:52:59 -07:00
Anders Kaseorg 30c6797239 test_runner: Fix SENDFILE_ROOT.
This is required by django-sendfile2 0.6.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-25 13:14:37 -07:00
Tim Abbott cb1321d0d2 lint: Harden various checks for URLs.
Because of other validation on these values, I don't believe any of
these does anything different, but these changes improve readability
and likely make GitHub's code scanners happy.
2020-06-25 12:10:45 -07:00
Steve Howell 2dd1ccef15 check_list: Make sub_validator a required argument.
This allows us to avoid the overload decorators.
2020-06-24 15:04:59 -07:00
Steve Howell c7b82d3ece mypy: Use tuples for muted_topics.
We now have our muted topics use tuples internally,
which allows us to tighten up the annotation
for get_topic_mutes, as well as our schema
checking.

We want to deprecate sub_validator=None
for check_list, so we also introduce
check_tuple here.  Now we also want to deprecate
check_tuple, but it's at least isolated now.

We will use this for data structures that are tuples,
but which are sent as lists over the wire.  Fortunately,
we don't have too many of those.

The plan is to convert tuples to dictionaries,
but backward compatibility may be tricky in some
places.
2020-06-24 15:01:57 -07:00
sahil839 fe35f1e9bd invites: Do not return multi-use invites to non-admin users.
This commit changes do_get_user_invites function to not return
multiuse invites to non-admin users. We should only return multiuse
invites to admins, as we only allow admins to create them.
2020-06-24 15:01:41 -07:00
Mateusz Mandera 890cafac11 retention: Use batch size of 100 for stream messages.
Streams can have lots of subscribers, meaning that the archiving process
will be moving tons of UserMessages per message. For that reason, using
a smaller batch size for stream messages is justified.

Some personal messages need to be added in test_scrub_realm to have
coverage of do_delete_messages_by_sender after these changes.
2020-06-24 10:41:00 -07:00
Mateusz Mandera 0c6497d43a retention: Add restore_retention_policy_deletions_for_stream function. 2020-06-24 10:40:38 -07:00
Mateusz Mandera 468f8cf488 retention: Improve logging of transactions. 2020-06-24 10:40:38 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Mateusz Mandera 7a03e2a7fe retention: Replace Realm.message_retention_days None value with -1.
To be more consistent with the meaning in the Stream model, and to make
it easier to have a reasonable settings API, we get rid of the None
value for Realm.message_retention_days in favor of the value -1 to
represent the "don't delete messages" default policy.
2020-06-24 10:33:21 -07:00
Anders Kaseorg d6655689f5 validator: Remove unused type_structure system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
wowol a3c475f137 urls: Migrate dev urls to use modern django patterns. 2020-06-23 15:02:44 -07:00
Anders Kaseorg f6c73ed45a auth: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg 5733a1bcd4 validate_user_access_to_subscribers: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg 8e8e34c000 test_runner: Fix a type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg b7241f20d5 timezone: Remove dead assignment with type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 3ffed617a2 mypy: Type simple generators as Iterator, not Iterable.
A generator that yields values without receiving or returning them is
an Iterator.  Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg c242d176ba rest: Fix a type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg feeb39a55f timezone: Strengthen get_timezone type.
pytz.BaseTzInfo is a subclass of datetime.tzinfo that adds the
localize and normalize methods.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 95c6d44a4a test_runner: Remove unused type: ignore comments.
Also enable warn_unused_ignores.  I think the fact that there are so
few of these is good evidence that it’s not a significant burden for
people fixing type errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 1e4fc8f2bf users: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg 086a3ef18b check_remove_custom_profile_field_value: Fix type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg b62c82c3e1 tests: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:18:19 -07:00
Anders Kaseorg f33bfaf545 import_realm: Avoid unchecked cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:13:48 -07:00
Anders Kaseorg 78f4482867 addressee: Remove redundant casts.
Also enable warn_redundant_casts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-22 17:11:42 -07:00
Felix a389c7390d calls: Add Big Blue Button as a Video Call Provider.
Big Blue Button needs an API secret so communication to creating a
room has to be done server side.

Fixes #14763.
2020-06-22 16:19:07 -07:00
Tim Abbott dc2a045d3b docs: Add documentation on mypy @overload.
We only use this in a few places, but they're really important places
for understanding the types in the codebase, and so it's worth having
a bit of expository documentation explaining how we use it.

(And I expect we'll add more with time).
2020-06-21 10:51:47 -07:00
MariaGkoulta b10f156250 settings: Add automatic theme detection feature.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.

This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.

Fixes: #14451.

Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
2020-06-21 01:09:01 -07:00
sahil839 63389b3bd3 invite: Add option to invite user as an organization owner.
We can now invite new users as realm owners. We restrict only
owners to invite new users as owners both for single invite
and multiuse invite link. Also, only owners can revoke or resend
owner invitations.
2020-06-20 23:04:59 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Tim Abbott d503549f0b retention: Add some more system documentation. 2020-06-20 17:35:07 -07:00
Steve Howell 22ead64d54 tests: Extract USING_TYPE_STRUCTURE. 2020-06-20 05:22:35 -04:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell fbe45fa889 test helpers: Tweak common_subscribe_to_streams.
We assert that the post was successful, to give
more immediate feedback for tests that don't
bother to check the return value and may be
implicitly assuming this method just works in
all cases.

And we also make it more convenient for tests
that are happy-path tests--they don't have to
do the assertion themselves.  (And they're still
free to do deeper checks on the json.)

We opt out with allow_fail=True.  We probably want
a more direct API eventually for tests that are
clearly trying to test the failure path for
subscribing to streams.

It's possible that a couple tests here that I added
allow_fail=True to just have flawed data setup--
I don't have time to investigate all cases, but
hopefully they will at least stand out more.
2020-06-19 22:03:12 -07:00
Steve Howell 00e60c0c91 events: Use stream_id for peer_add/peer_remove.
Two things were broken here:
    * we were using name(s) instead of id(s)
    * we were always sending lists that only
      had one element

Now we just send "stream_id" instead of "subscriptions".

If anything, we should start sending a list of users
instead of a list of streams.  For example, see
the code below:

    if peer_user_ids:
        for new_user_id in new_user_ids:
            event = dict(type="subscription", op="peer_add",
                         stream_id=stream.id,
                         user_id=new_user_id)
            send_event(realm, event, peer_user_ids)

Note that this only affects the webapp, as mobile/ZT
don't use this.
2020-06-19 14:48:42 -07:00
orientor 58b93c3e83 openapi: Show API argument type in docs.
Currently the API docs do not specify whether a given API parameter
is to be specified in `query` or in `path`. Edit the docs so as
to show the type of argument right beside argument name.
2020-06-19 14:42:01 -07:00
Tim Abbott a66a8419b9 bankruptcy: Remove broken push notifications loop.
The loop I added here in 5b49839b08 was
ill-conceived.  The critical issue was that despite its name,
do_clear_mobile_push_notifications_for_ids does not immediately clear
push notifications (Except in our test suite, where `send_event`
immediately calls into the queue worker code!).

Instead, it queues work to clear those push notifications.  Which
means that the first user to declare bankruptcy with a large number of
unreads will fill the queue, and then this will just be an infinite
loop adding more work to the queue.
2020-06-19 10:54:14 -07:00
Hashir Sarwar 5200598a31 events: Don't send avatar URLs of long term idle users.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.

Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.

This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.

Fixes #15287.
2020-06-18 21:35:16 -07:00
Hashir Sarwar 9911ec3e6d cache: Add `long_term_idle` in `realm_user_dict_fields`.
We need this field to avoid O(N) database operations
while fetching realm user data for clients with
`user_avatar_url_field_optional` flag enabled.

Part of #15287.
2020-06-18 20:23:10 -07:00
Tim Abbott 0b6ebb4fbb upload: Remove unused get_realm_for_filename. 2020-06-18 17:55:13 -07:00
Tim Abbott 5962d1ea14 upload: Avoid fetching bucket objects repeatedly.
This takes of advantage of saving the bucket object on the
UploadBackend class to deduplicate a bunch of redundant code getting
buckets.
2020-06-18 17:55:13 -07:00
Wyatt Hoodes 2ef791fc21 upload.py: Support using non S3-providers.
With #14378, we regressed back to the state of that
prior to 7e0ea61b00.

We fix this by getting our avatar bucket on
object initialization, and use the appropriate means
of gathering the network location for the urls.

Fixes #14484.
2020-06-18 17:55:13 -07:00
Vishnu KS eb829e6dce realm_logo: Return default logo for realms on LIMITED plan.
This completes our effort to avoid destroying realm logo state while
providing a path to dowgrade from paid plans to LIMITED.
2020-06-18 17:46:26 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Anders Kaseorg 3916ea23a9 python: Combine some split import groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:54:11 -07:00
Anders Kaseorg b2cb992d9f bugdown: Use standard library datetime.timezone.utc.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:47:20 -07:00
Tim Abbott 3ec068f10c bugdown: Import datetime our normal way. 2020-06-18 14:14:53 -07:00
Rohitt Vashishtha 6ea3816fa6 markdown: Use html5 <time> tag for timestamps.
Previously, we had implemented:
    <span class="timestamp" data-timestamp="unix time">Original text</span>
The new syntax is:
    <time timestamp="ISO 8601 string">Original text</time>
    <span class="timestamp-error">Invalid time format: Original text</span>

Since python and JS interpretations of the ISO format are very
slightly different, we force both of them to drop milliseconds
and use 'Z' instead of '+00:00' to represent that the string is
in UTC. The resultant strings look like: 2011-04-11T10:20:30Z.

Fixes #15431.
2020-06-18 14:11:33 -07:00
Tim Abbott b77da6b834 pointer: Remove fix_pre_pointer fixing logic.
This fundamentally requires the pointer to exist to do anything
useful.
2020-06-18 12:55:59 -07:00
Aman Agrawal b7e5b7f193 do_update_messages: Delete user_message if guest user loses access.
Fixes #14498.
When a topic is moved to a different stream, the message may no
longer be reachable to guest user, if the user is not subscribed
to the new stream.

We used to send message update event to the client in these cases,
which seems to be confusing both to the client updating the message
and the server sending push_notifications for it.

Now, we delete the UserMessage entry for these messages for the
user and send a delete message event to the client; which makes
both push_notification and the event handling client think that
the message was deleted and hence no confusion in the code is
raised.
2020-06-18 01:00:04 -07:00
Hashir Sarwar ab6be2a711 push_notifications: Store tokens locally even when bouncer is used.
This makes the system store and track PushDeviceToken objects on
the local Zulip server when using the push notifications bouncer
and includes tests for this.

This is something we need to implement end-to-end encryption for
push notifications. We'll add the encryption key as an additional
property on the local PushDeviceToken object.

It also likely adds some value in the case that a server were to
switch between using the bouncer service and sending notifications
directly, though in practice that's unlikely to happen.
2020-06-17 18:44:59 -07:00
Tim Abbott bca164c1b6 api docs: Use better language for no parameters allowed. 2020-06-17 18:20:23 -07:00
Mateusz Mandera 8d2d64c100 CVE-2020-14215: Fix validation in PreregistrationUser queries.
The most import change here is the one in maybe_send_to_registration
codepath, as the insufficient validation there could lead to fetching
an expired PreregistrationUser that was invited as an administrator
admin even years ago, leading to this registration ending up in the
new user being a realm administrator.

Combined with the buggy migration in
0198_preregistrationuser_invited_as.py, this led to users incorrectly
joining as organizations administrators by accident.  But even without
that bug, this issue could have allowed a user who was invited as an
administrator but then had that invitation expire and then joined via
social authentication incorrectly join as an organization administrator.

The second change is in ConfirmationEmailWorker, where this wasn't a
security problem, but if the server was stopped for long enough, with
some invites to send out email for in the queue, then after starting it
up again, the queue worker would send out emails for invites that
had already expired.
2020-06-16 23:35:39 -07:00
Clara Dantas ddbde66af5 realm: Remove Google Hangouts integration.
Google  has removed the Google Hangouts brand, thus we are removing
them as video chat provider option.
This commit removes Google Hangouts integration and make a migration
that sets all realms that are using Hangouts as their video chat
provider to the default, jitsi.

With changes by tabbott to improve the overall video call documentation.

Fixes: #15298.
2020-06-16 17:02:27 -07:00
Sara Gulotta 1cb040647b markdown: Add support for spoilers.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.

We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.

Fixes #5802.

Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
2020-06-16 16:14:10 -07:00
Aman Agrawal d7ca5bad62 do_delete_messages: Create a TypedDict declaration for the event.
This will help us ensure that other functions defining delete_message
event use the same formatting.
2020-06-16 11:40:45 -07:00
Hashir Sarwar ecd35b9565 push_notifications: Add support for setting counts in iOS.
This adds a new function `get_apns_badge_count()` to
fetch count value for a user push notification and
then sends that value with the APNs payload.

Once a message is read from the web app, the count is
decremented accordingly and a push notification with
`event: remove` is sent to the iOS clients.

Fixes #10271.
2020-06-16 11:26:36 -07:00
Tim Abbott 59b68aaa98 events: Remove incorrect line from new-stream handler.
This line was effectively hardcoding a specific stream_post_policy,
overriding the value already present in the event, to no purpose.

(I believe it got here via cargo-culting induced by #13787.)
2020-06-16 10:41:50 -07:00
sahil839 791e5de5de api: Remove is_old_stream property from the stream objects.
This commit removes is_old_stream property from the stream objects
returned by the API. This property was unnecessary and is essentially
equivalent to 'stream_weekly_traffic != null'.

We compute sub.is_old_stream in stream_data.update_calculated_fields
in frontend code and it is used to check whether we have a non-null
stream_weekly_traffic or not.

Fixes #15181.
2020-06-16 10:26:33 -07:00
Hashir Sarwar 31b6867360 push_notifications: Remove redundant `user_profile` assignment. 2020-06-15 22:32:12 -07:00
Tim Abbott 5b49839b08 bankruptcy: Clear mobile push notifications first.
This likely fix a bug that can leak thousands of messages into the
invalid state where:
* user_message.flags.active_mobile_push_notification is True
* user_message.flags.read is True

which is intended to be impossible except during the transient process
between marking messages as read sending the "remove push
notifications" event.

The bug is that if a user who is declaring bankruptcy with 10,000s of
unreads ends up having the database query to mark all of those as read
take 60s, the Django/uwsgi request will time out and kill the process.
If the postgres transaction still completes, we'll end up with the
second half of this function never being run.

A safer ordering is to do the smaller queries first.

We do this in a loop for correctness in the unlikely event there are
more than 10,000 of these.
2020-06-15 17:43:43 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Mateusz Mandera eda396376c actions: Remove redundant "assert is None" in do_update_message_flags.
The type of messages arg has changed since this was added.
2020-06-15 16:20:49 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1ed2d9b4a0 logging: Use logging.exception and exc_info for unexpected exceptions.
logging.exception() and logging.debug(exc_info=True),
etc. automatically include a traceback.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 4b6d2cf25f logging: Pass more format arguments to logging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Aman Agrawal cda7b2f539 deletion: Add support for bulk message deletion events.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.

Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).

Backend implementation of #15285.
2020-06-14 22:34:00 -07:00
Aman Agrawal 21ac1fb32c register: Refactor to pass client_capabilities directly.
We extract values from client_capabilities directly in
do_events_register where we decide how to process the
extracted variables.
2020-06-14 21:42:50 -07:00
arpit551 c4b5d09283 analytics: Add LoggingCount for messages read stats.
Whenever we use API queries to mark messages as read we now increment
two new LoggingCount stats, messages_read::hour and
messages_read_interactions::hour.

We add an early return in do_increment_logging_stat function if there
are no changes (increment is 0), as an optimization to avoid
unnecessary database queries.

We also log messages_read_interactions::hour Logging stat
as the number of API queries to mark messages as read.

We don't include tests for the case where do_update_pointer is called
because do_update_pointer will most likely be removed from the
codebase in the near future.
2020-06-14 21:15:27 -07:00
Anders Kaseorg a803e68528 email-mirror-postfix: Handle 8-bit messages correctly.
Since JSON can’t represent bytes, we encode them with base64.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
Anders Kaseorg bff3dcadc8 email: Migrate to new Python ≥ 3.3 email API.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:24:06 -07:00
orientor 61c64b3df9 tests: Use validate_against_openapi_schema in testing.
This adds a powerful end-to-end test for Zulip's API documentation:
For every documented API endpoint (with a few declared exceptions that
we hope to remove), we verify that every API response received by our
extensive backend test suite matches the declared schema.

This is a critical step towards being able to have complete, high
quality API documentation.

Fixes #15340.
2020-06-14 15:05:52 -07:00
Hemanth V. Alluri 5bbfaa225b webhooks/circleci: Overhaul the CircleCI integration.
This change should add a lot more information to
our CircleCI integration's output.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-14 13:14:55 -07:00
Aman Agrawal 2e5f860d41 message_edit: Do case-insensitive exact match when editing topics.
When doing query for same topic names in a stream, we should do a
case-insensitive exact match for the topic, since that's the data
model for topics in Zulip.
2020-06-13 16:36:29 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Anders Kaseorg 46d0018fbb types: Convert ProfileDataElement to Python 3.6 style and total.
The Python 3.6 style does support non-total and even partially-total
TypedDict, but total gives us better guarantees.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:37:25 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg cfcbf58cd1 do_render_markdown: Remove unused message_user_ids parameter.
It’s unused since commit 7c5f316cb8
(#11586).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 03e147d5e1 python: Replace NamedTuple with dataclass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 15:19:31 -07:00
Graham Bleaney 461d5b1a3e pysa: Introduce sanitizers, models, and inline marking safe.
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
2020-06-11 12:57:49 -07:00
Anders Kaseorg ca4357fd64 python: Use standard NoReturn (Python ≥ 3.6).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 12:56:52 -07:00
Steve Howell d5cc29755e typing: Accept only user_ids in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.

There may be a very tiny number of mobile
clients that still use the old emails API.

This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.

Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.

The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:

    validator=check_list(check_int)

This also allows us to remove a test hack
related to the API documentation.  (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
2020-06-10 17:50:32 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
Hemanth V. Alluri 7b2255597c outgoing-webhook: Fix check for empty content.
Prevent `JsonableError(_("Missing content"))` from
ever being triggered.

That error wasn't handle by anything, and thus just threw a 500, as
it's not a response to an HTTP request.

The right fix is to adjust the caller to ban the empty string in
content (or content that strips to the empty string).

Closes #15145.
2020-06-10 14:46:04 -07:00
sahil839 5cc937f670 events: Add 'is_owner' to page_params and raw_users dict of page_params.
This commit adds 'is_owner' field to page_params and also to the
raw_users dict of page_params.
2020-06-10 14:07:38 -07:00
sahil839 6d667dbe53 realm owner: Add ability to change realm owner status of user.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.

This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.

We need to apply that restriction both in the role change code path
and the deactivate code path.
2020-06-10 12:49:37 -07:00
sahil839 2c8c641556 realm owner: Make first user as realm owner on realm creation.
This commit sets the role of the user creating the realm as
realm owner after the realm is created.

Previously, the role of user creating the realm was set as admin.
But now we want it to be owner because owners have the highest
privilege level.
2020-06-10 12:44:52 -07:00
Anders Kaseorg d3e8af4ad2 python: Replace silly uses of filter().
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 23:31:59 -07:00
Anders Kaseorg 04919528e7 models: Parameterize .extra(where=["… IN …"]) with tuple/list adapter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 859c8b169e stream_recipient: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 68e78c809f get_recent_private_conversations: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 3aab9c03a9 fix_unreads: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 14bbfe6ffb bulk_get_subscriber_user_ids: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 2604ebba38 import_message_data: Use psycopg2.extras.execute_values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 6649c8e35f bulk_insert_ums: Use psycopg2.extras.execute_values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
David Wood f0f42f7a94 integrations: Add Thinkst Canary integration.
This commit adds an integration for Thinkst Canaries - physical, VM and
cloud-based canaries for detecting attackers to a network. Thinkst
Canaries can send webhook alerts when canaries have been tripped, and
this integration will post Zulip messages when these webhooks are
received.

Signed-off-by: David Wood <david@davidtw.co>
2020-06-09 16:28:28 -07:00
Tim Abbott e242ddc848 page_params: Add community_topic_editing_limit_seconds.
This was previously hardcoded with agreement between the Zulip backend
and frontend as 86400 seconds (1 day).  Now, it's still hardcoded in
the backend, but arranged in a way where we could add a setting
without any changes to the mobile and terminal apps to update logic.

Fixes #15278.
2020-06-09 14:40:12 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 444fbbf964 python: Whitespace fixes from autopep8.
Generated by autopep8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:21:30 -07:00
Aman Agrawal e2ed155310 do_update_message: Send old topic with stream edit event.
Old topic of the msg edit event can be used to help the client
calculate useful information such as if a change
in current narrow is required.

This fixes our re narrow logic after a stream edit of a topic, with
no change in topic name itself, since the original topic was not
present in the event received and hence the `orig_topic` was
undefined in this case.
2020-06-07 11:20:34 -07:00
Anders Kaseorg 19cc22e5ab queue: Fix types to reflect that Pika channels transmit bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:09:24 -07:00
Anders Kaseorg 22178c169e request: Replace type argument hack with a different workaround.
This works in mypy 0.770 and is needed for mypy 0.780.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 20:05:18 -07:00
Aman Agrawal a096f34cab move_topic_to_stream: Add option to disable breadcrumb messages.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.

User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
2020-06-05 12:28:51 -07:00
Aman Agrawal ec257f6ec3 to_dict_uncached: Pipe realm_id from the caller functions.
We pipe realm_id through functions where it is available,
this helps us avoid doing query for realm_id in loop when
multiple messages are being processed.
2020-06-05 12:03:31 -07:00
Anders Kaseorg 0bcd4a6281 email_mirror: Avoid parameters that shadow names in their own types.
They confuse semgrep, and also, like, people in general.

https://github.com/returntocorp/semgrep/issues/923

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:35:11 -07:00
Anders Kaseorg 1f565a9f41 timezone: Use standard library datetime.timezone.utc consistently.
datetime.timezone is available in Python ≥ 3.2.  This also lets us
remove a pytz dependency from the PostgreSQL scripts.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-05 09:34:17 -07:00
Anders Kaseorg 0797626911 bugdown: Avoid deprecated ElementTree.getchildren method.
Fixes warnings like these with python -Wd:

/home/circleci/zulip/zerver/lib/bugdown/__init__.py:327: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  for child in currElementPair.value.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:328: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  if child.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:282: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  for child in currElement.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:283: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  if child.getchildren():

https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-04 14:17:08 -07:00
Anders Kaseorg b94b350ff0 bot_config: Avoid deprecated ConfigParser.readfp method.
Fixes this warning with python -Wd:

/home/circleci/zulip/zerver/lib/bot_config.py:69: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  config.readfp(conf)

https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.readfp

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-04 14:16:17 -07:00
Anders Kaseorg 67dbb3b2a9 bugdown: Replace deprecated markdown members with md.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg f65af9cdb7 bugdown: Replace deprecated markdown.util.etree with ElementTree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 01ccb75259 bugdown: Replace deprecated cElementTree with ElementTree.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg aad0b8b2da bugdown: Add assert to work around type problem.
url_to_a returns Union[Element, str], but str cannot be appended to
Element; that would raise TypeError at runtime.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 10805abd6e python: Fix errors from pycodestyle 2.6.0.
zerver/lib/i18n.py:34:28: E741 ambiguous variable name 'l'
zerver/lib/webhooks/common.py:103:34: E225 missing whitespace around operator
zerver/tests/test_queue_worker.py:563:9: E306 expected 1 blank line before a nested definition, found 0

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-03 17:23:20 -07:00
Anders Kaseorg 4d04fa3118 compose: Rewrite Zoom video call integration to use OAuth.
This reimplements our Zoom video call integration to use an OAuth
application.  In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.

Fixes #11672.

Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-06-03 16:39:12 -07:00
Anders Kaseorg fa9bddcfb2 emoji: Add workaround for collectstatic step of build-release-tarball.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
orientor 1227b65066 openapi: Forbid opaque objects from OpenAPI documentation.
Objects whose properties are not described were validated by the
current validator. Edit it so that objects with no `properties`
or `additionalProperties` attribute i.e. opaque objects get
invalidated.

Also make changes in zulip.yaml to fix any opaque objects (tweaked by
tabbott to edit the documentation for better clarity).
2020-06-02 16:45:06 -07:00
Sahil Batra 77d4be56a4
users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Hemanth V. Alluri 18f8859baa refactor: Remove unnecessary request_data param in outgoing_webhook.
request_retry and notify_bot_owner don't use request_data so might
as well not send it to them at all.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-02 15:19:27 -07:00
Hemanth V. Alluri 9c172036bb refactor: Make OutgoingWebhookServiceInterface more interface like.
Using the Python Standard Library's abc library and NotImplementedError
we can better define interfaces (this is mainly to improve readability
and consistency).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-02 15:19:27 -07:00
Puneeth Chaganti 788203778d tools: Use the correct logo path to generate bot avatar.
Integrations can be configured with specific logo paths, which weren't
correctly being used for generating the bot avatars.
2020-06-01 18:08:23 -07:00
Puneeth Chaganti dfd4b4d051 integrations: Store logo_path as an attribute along with logo_url.
Integrations can be supplied a logo parameter which is used to contruct
their `logo_url`. It would be useful to store this parameter, instead of
computing the path from the URL.
2020-06-01 18:08:23 -07:00
Puneeth Chaganti 752fe58f4f tools: Allow specifying & capturing screenshots for all integrations. 2020-06-01 18:08:22 -07:00
Puneeth Chaganti b533420d3b tools: Allow specifying extra query params in screenshot config. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 56c31a8647 tools: Allow specifying the bot name for notifications. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 55444c33a0 tools: Allow passing payload as a query param. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti d4c6195bca tools: Allow specifying use of basic auth in ScreenshotConfig. 2020-06-01 18:03:27 -07:00
Puneeth Chaganti 5ffc51382d tools: Allow specifying custom headers in ScreenshotConfig. 2020-06-01 18:03:27 -07:00
sahil839 9ef1c5b1a6 users: Add is_owner field to user objects returned by get endpoints.
This commit adds 'is_owner' field to the user object returned by
'/users', 'users/{user_id}', and '/users/me' endpoints.
2020-06-01 15:33:51 -07:00
sahil839 169bd3082a users: Modify format_user_row to set is_admin as True for owners also. 2020-06-01 15:14:16 -07:00
Tim Abbott c78d728812 events: Fix test_events race handling of role changes.
Previously, our apply_events logic (verified by test_events) did not
properly handle changing a user's role to and from being a guest.
2020-05-31 17:31:33 -07:00
sahil839 2ab6767b73 events: Update person dict in event for do_change_user_role to send role.
This commit changes the person dict in event sent by do_change_user_role
to send role instead of is_admin or is_guest.

This makes things much more straightforward for our upcoming primary
owners feature.
2020-05-31 17:22:50 -07:00
orientor d2ee99a2fd openapi: Add markdown extension for rendering return values in API docs.
Currently response return values have to be written twice, once in
the docs and once in zulip.yaml. Create a markdown extension so
that the return values in api docs are rendered using content from
zulip.yaml
2020-05-29 15:02:56 -07:00
Anders Kaseorg ba7906a3c6 bugdown: Revert NamedTuple declarations to old style.
This reverts part of commit 8bcdf4ca97
(#15093), to work around a mypy caching bug:
https://github.com/python/mypy/issues/7281.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-28 16:47:08 -07:00
sahil839 7d9d2414ca actions: Modify do_change_user_role for realm owner.
This commit changes do_change_user_role to support adding or removing
the realm owner status of user and sending an event.

We also extend the existing test for do_change_user_role to do a bit
more validation to confirm the audit log records all values of role.
2020-05-27 15:52:39 -07:00
sahil839 8bbc07474f models: Add realm owner role and is_realm_owner property for user.
The new realm_owner role is added as option for role field in
UserProfile model and is_realm_owner is added as property for the user
profile.

Aside from some basic tests validating the logic, this has no effect
as users cannot end up with set as realm owners.
2020-05-27 15:47:42 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
whoodes cea7d713cd requirements: Upgrade boto to boto3.
Fixes: #3490

Contributors include:

Author:    whoodes <hoodesw@hawaii.edu>
Author:    zhoufeng1989 <zhoufengloop@gmail.com>
Author:    rht <rhtbot@protonmail.com>
2020-05-26 23:18:07 -07:00
arunikaydav42 4680d504de invite: Fix invitations removed from list too soon.
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.

Fixes: #12281
2020-05-26 21:45:44 -07:00
Anders Kaseorg cf923b49d3 python: Remove extra pass statements with autoflake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 8bcdf4ca97 python: Convert TypedDict declarations to Python 3.6 style.
A subset of the diff generated by pyupgrade --py36-plus
--keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Tim Abbott 463f1503fc Revert "markdown: Process fenced code blocks in blockquotes."
This reverts commit 7002f98ea1.

This failed tests due to some sort of conflict with a recent
python-markdown upgrade.
2020-05-25 18:13:03 -07:00
Rohitt Vashishtha 7002f98ea1 markdown: Process fenced code blocks in blockquotes.
We handle fenced code blocks in a preprocessor, and > style blockquotes
are parsed in a blockprocessor. Pymarkdown doesn't run the preprocessors
again on any blocks that it is parsing, and is unlikely to accept our
solution upstream; they intend to convert fenced_code to a block parser.

We simply run all the preprocessors on the text again, with the exception
of NormalizeWhitespace which removed delimiters used by HtmlStash to mark
preprocessed html code. To counter this, we subclass NormalizeWhitespace
and use our customized version for when it is called from a blockparser.

Upstream issue: https://github.com/Python-Markdown/markdown/issues/53

Fixes #12800.
2020-05-25 17:35:10 -07:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
Mateusz Mandera 13c3eaf086 rate_limit: Restrict tornado backend to explicitly specified domains.
This will protect us in case of some kinds of bugs that could allow
making requests such as password authentication attempts to tornado.
Without restricting the domains to which the in-memory backend can
be applied, such bugs would lead to attackers having multiple times
larger rate limits for these sensitive requests.
2020-05-25 15:40:00 -07:00
Tim Abbott a53daa6f8c message: Fix malformed reaction data.
After a message was reset in our caches cache via message editing or
adding/removing a reaction, we were sending corrupt data to the cache
because build_message_dict (and thus build_dict_from_raw_db_row) was
improperly being called before sewing in the reaction data.

As a result, we were sending raw database data in the reaction
dictionaries, rather than the reformatted version expected by the API.

Bug introduced in 2a4c62a326.

Fixing this correctly required moving the rendering_realm_id logic one
step higher in the call chain, which is a useful refactoring anyway
(since we're no longer passing a `Message` object down)
2020-05-21 14:09:53 -07:00
Rohitt Vashishtha 88367a129c markdown: Disable tex and latex for math rendering.
We now parse tex and latex as regular languages, highlighting them
with pygments. We only allow 'math' to trigger latex rendering,
which is in line with the documentation.
2020-05-21 12:30:27 -07:00
Rohitt Vashishtha 52c25a9301 markdown-timestamp: Use data-timestamp attribute.
This commit shifts our timestamp syntax to be of the form:

    <span class="timestamp data-timestamp="123456"></span>

since value is not a valid attribute of span elements.
2020-05-20 14:28:08 -07:00
Rohitt Vashishtha b062e8332f markdown: Add timestamp syntax to markdown processors.
This adds support for syntax like: !time(Jun 7 2017, 6:30 PM) so that
everyone sees the time in their own local timezone. This can be used
when scheduling online meetings, etc.

This adds some hardcoded values for timezones, because of there
being no sureshot way of determining the timezone easily. However,
since the main way of using the feature should be a typeahead for
entering the time, this shouldn't be cause of much concern.

Fixes #5176.
2020-05-20 14:23:55 -07:00
Dinesh 2711619376 redis_utils: Extend `put_dict_in_redis` to accept token as param.
This extends `put_dict_in_redis` to take token as an argument
and return that with the as a `key` with following key format.

Also, edit regex for token to include uppercase letters as
a token sent during apple authentication contains uppercase
letters.

Useful for Adding "Sign in with Apple" support.
2020-05-20 13:30:44 -07:00
Vishnu KS 4e0d8953b4 models: Create ensure_not_on_limited_plan instance method. 2020-05-19 14:48:59 -07:00
sahil839 9b78a73e36 populate_db: Add new admin user as 'Desdemona'.
This commit adds a second admin user named 'Desdemona' to dev and
test database.
2020-05-19 11:42:27 -07:00
Aman Agrawal 2a4c62a326 update_to_dict_cache: Use bulk queries when preparing msgs for cache.
During events such as stream / topic name edit for a topic, we were
running queries to db in loop for each message for reactions,
submessages and realm_id. This commit reduces the queries to be
done only for realm_id, which is yet to be fixed.

This is accomplished by building messages with empty reactions
and submessages and then updating them in the messages using bulk
queries.
2020-05-19 10:30:03 -07:00
Aman Agrawal b8fe6245e3 message: Extract method to sew submessages and reactions to msg. 2020-05-19 10:30:03 -07:00
sahil839 46ef6816b6 stream: Allow non admins to set stream post policy when creating streams.
This commit allows non admins to set stream post policy while creating
streams.

Restriction was there to prevent user from creating a stream in which
the user cannot post himself but this will be taken care of with
stream admin feature.
2020-05-16 14:53:22 -07:00
Mateusz Mandera b234fe8ccb retention: Pass optional realm argument to move_messages_to_archive.
This allows having the realm field of ArchiveTransaction set instead of
NULL when using move_messages_to_archive.
2020-05-16 14:46:56 -07:00
Mateusz Mandera 7d8a3581a5 retention: Clarify the status of cross-realm huddles in a comment. 2020-05-16 14:42:40 -07:00
Tim Abbott f10f2600e0 retention: Fix OOM issues when deleting large numbers of transactions.
For unknown reasons, deleting 10,000s of ArchiveTransaction objects
results in rapidly growing memory in the job making the request in the
Django process, eventually leading to an OOM kill.

I don't understand why Django behaves that way; I would have expected
the failure mode to instead be a serious load problem on the database
server, but perhaps the way Django's internal deletion logic handles
cascading the deletes to many millions of ArchiveMessages and other
ForeignKey objects requires tracking a lot of data in memory.

The solution is the same in any case, which is to batch the deletions
to execute a reasonable number of them at once.  Doing a single
ArchiveTransaction at a time would likely result in huge numbers of
database queries in a loop, which performs very poorly.  So we balance
by batching deletions in groups of 100 ArchiveTransactions; testing
this in production, I saw no spike of memory usage materially beyond
that of a normal Django process, and each bulk-deletion transaction
takes several seconds to process (meaning per-transaction overhead is
negligible).
2020-05-15 17:10:19 -07:00
Tim Abbott bd3c0aef9b presence: Optimize performance of mobile_query.
I'm not sure exactly what series of history got us here, but we were
fetching the mobile_user_ids data for all users in the organization,
regardless of whether they were recently active (and thus relevant for
the main presence data set).  And doing so in a sloppy fashion
(sending every user ID over the wire, rather than just having the
database join on Realm).

Fixing this saves a factor of 4-5 on the total runtime of a presence
request on organizations with 10Ks of users like chat.zulip.org; more
like 25% in an organization with 150.  Since large organizations are
very heavily weighted in the overall cost of presence, this is a huge
win.

Fixes part of #13734.
2020-05-14 23:23:37 -07:00
orientor 9170931da3 openapi: Add test for validating examples.
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.

Part of #14100.
2020-05-12 23:03:06 -07:00
Tim Abbott 35139ac559 api: Remove email field from realm_user and realm_bot events.
The `email` field for identifying the user being modified in these
events was not used by either the webapp or other official Zulip
clients.  Instead, it was legacy data from before we switched years
ago to sending user_id fields as the correct way to uniquely identify
a user.
2020-05-12 17:46:13 -07:00
cestrell e3f0b2f20f integrations: Add Gogs webhook for release event.
Extends Gogs integrations in order to support a published release.
Tested on my local Ubuntu development server running on WSL2.

Fixes #14746.
2020-05-12 17:39:38 -07:00
Tim Abbott f642a0fc89 addressee: Remove obsolete parsing functions.
These became unused in c971576b00.
2020-05-12 17:33:11 -07:00
clarammdantas 7e9024a39c popovers.js: Add version to user avatar request.
When a user changes its avatar image, the user's avatar in popovers
wasn't being correctly updated, because of browser caching of the
avatar image.  We added a version on the request to get the image in
the same format we use elsewhere, so the browser knows when to use the
cached image or to make a new request to the server.

Edited by Tim to preserve/fix sort orders in some tests, and update
zulip_feature_level.

Fixes: #14290
2020-05-12 11:09:01 -07:00
Steve Howell 9137726e38 bots: Remove `owner` (email) from bot-related payloads.
We remove the `owner` field from `page_params/realm_bots`
and bot-related events.

In the recent commit 155f6da8ba
we added `owner_id`, which we now use everywhere we need
bot owners for.

We also bump the `API_FEATURE_LEVEL` to 5 here.  We
had already documented this in the prior commit to
add `owner_id`.

Note that we don't have to worry about mobile/ZT clients
here--we only deal with bot data in the webapp.
2020-05-12 09:48:37 -07:00
Steve Howell 155f6da8ba bots: Add owner_id to bot-related payloads.
For the below payloads we want `owner_id` instead
of `owner`, which we should deprecate.  (The
`owner` field is actually an email, which is
not a stable key.)

    page_params.realm_bots

    realm_bot/add

    realm_bot/update

IMPORTANT NOTE: Some of the data served in
these payloads is cached with the key
`bot_dicts_in_realm_cache_key`.

For page_params, we get the new field
via `get_owned_bot_dicts`.

For realm_bot/add, we modified
`created_bot_event`.

For realm_bot/update, we modified
`do_change_bot_owner`.

On the JS side, we no longer
look up the bot's owner directly in
`server_events_dispatch` when we get
a realm_bot/update event. Instead, we
delegate that job to `bot_data.js`.
I modified the tests accordingly.
2020-05-11 16:16:58 -07:00
Rohitt Vashishtha 26238426cd message-edit: Process user groups for mentioned user ids.
When editing a message where we mention a usergroup, we would remove
the 'mentioned' flag from messages, resulting in the message being
hidden from your mentions in the UI. This was reported by Greg Price in
https://chat.zulip.org/#narrow/stream/9-issues/topic/missing.20mention.

We add the same code that we use in do_send_messages to calculate the
updated mentions_user_ids. We add some tests alongside other user group
mention tests in test_bugdown.
2020-05-11 14:52:11 -07:00
Jenny Ghose 180c16c80e integrations: Add an incoming webhook for Grafana.
Tweaked by tabbott to use formatted suggested in one of the various
duplicate PRs for this issue, showing the rule name clearly.

Fixes #12951.
2020-05-11 00:27:38 -07:00
Tim Abbott a0c2121958 docs: Advertise Slack-compatible webhook a bit.
This should make it discoverable enough that users will try it out and
send us feedback.
2020-05-11 00:07:28 -07:00
Chris Heald c80e913c7a webhooks: Add a webhook capable of parsing Slack payloads.
This adds a webhook that can be used to interpret standard Slack
payloads. Since there are a ton of existing Slack integrations out
there, having a webhook which can accept standard Slack payloads can
significantly ease transition pains. Obviously this can't do everything
that Slack payloads can (particularly WRT their widgets/interactions),
but we can ingest text and parse out multi-block payloads into a message
relatively reasonably.
2020-05-11 00:07:28 -07:00
shubhamgupta2956 9cd8644c7c uploads: Add support for ".jpe" file extension.
Currently when the user uploads files with ".jpe" file extension, the
markdown is converted to link but the image is not embedded.

This commit adds the support for ".jpe" file extension.

Fixes #14863
2020-05-10 22:55:52 -07:00
Pragati Agrawal 60a762704a org settings: Fix limited plan realm can change message_retention_days.
These changes should be included in bd9b74436c,
as it makes sure that Zulip limited plan realm won't be able to change the
`message_retention_days` setting.
2020-05-09 16:40:50 -07:00
Anders Kaseorg 78c70b1424 bugdown: Leave link titles alone until clean_user_content_links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Anders Kaseorg 32f3fd1c77 bugdown: Fix ElementPair typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Anders Kaseorg 6aaeab75bc bugdown: Fix ResultWithFamily typing.
It needs to be a full class because a generic NamedTuple doesn’t work
in Python 3.6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Aman Agrawal 898e717b9d register:api: Provide additional fields that can be useful to clients.
Fixes #14166

* Bumped the api version to feature level 4.
See https://github.com/zulip/zulip/issues/14166#issuecomment-598961462
for details about implementation and future steps.
2020-05-08 14:54:54 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Pragati Agrawal 12504075ee org settings: Rename plan_includes_wide_organization_logo to more generic.
It's a preliminary step to enable message_retention_setting in org settings
UI, which is a non-limited plan only feature. So we require a page_param
property that tells us the limited-plan state of the Zulip realm.
2020-05-08 14:04:21 -07:00
Hashir Sarwar 6364d27ed5 topic: Remove 7 days restriction for editing & moving topics.
Previously, we had a restriction that we could only
edit and move the topics of 7 days old messages.
This buggy behaviour is now removed as in this
commit.

Fixes #14492.
Part of #13912.
2020-05-08 12:57:50 -07:00
wowol ffed6b87dd urls: Migrate urls to use modern django pattern.
New path() function changed the way a regex pattern
is created from urls - it adds escape backslashes,
so for testing purposes we need to take care of them
and remove them, to check if urls were tested.

Additionaly, regex patterns from urls can have
[^/]+ instead of [^/]*, so we need to take care
of it too.
2020-05-07 16:28:58 -07:00
Mateusz Mandera 812ac4714f retention: Optimize fetching of realms and streams with retention policy. 2020-05-07 16:28:05 -07:00
Steve Howell ebb16e7a68 error logging: Simplify deployment data.
We no longer have intermediate constants of
`git_described` and `zulip_version_const`.

Instead, we make a `deployment_data` dictionary
that is grep-friendly, and we just let
`deployment_repr` do simple formatting
without translating string constants.

This is pretty easy to test:

    - set DEBUG_ERROR_REPORTING = True
    - modify some code to throw an exception
    - see error output in #errors
    - use "/emails" with text-only option to view
      errors
2020-05-06 16:29:53 -07:00
Steve Howell 710e3144e7 error logging: Remove `zulip_version_file` code.
This code was bitrotted--we no longer have a file
called `version`.

The info that was probably reported when that feature
was originally written probably lives now
in `zulip-git-version`, although I didn't research
all the history here.  Here is the relevant
excerpt from `version.py`:

    zulip_git_version_file = os.path.join(
        os.path.dirname(os.path.abspath(__file__)),
        'zulip-git-version')

    if os.path.exists(zulip_git_version_file):
        with open(zulip_git_version_file) as f:
            version = f.read().strip()
            if version:
                ZULIP_VERSION = version

The file gets written as follows:

    $ cat tools/cache-zulip-git-version
    #!/usr/bin/env bash
    set -e

    cd "$(dirname "$0")/.."
    git describe --tags --match='[0-9]*' > zulip-git-version || true

Here is what that might look like:

    2.2-dev-2102-gf256ea39eb

Here is an excerpt from one of our recent error reports,
which demonstrates that the code I eliminated here was not
functioning (the third field is missing):

    Deployed code:
    - git: 2.2-dev-2028-g99ce96d49b-dirty
    - ZULIP_VERSION: 2.2-dev-2028-g99ce96d49b

This fixes the main problem reported on #7868.  I think
we may just want to close the issue, since the other
`nocoverage` stuff seems harmless to me.
2020-05-06 16:29:53 -07:00
Tim Abbott 1c5aa10147 soft_deactivation: Fix buggy error handling.
There is no such thing as `max()` on the manager object.  We meant
.last().

Introduced in 37189e1f9d, so my bug, in
a rare untested code path.
2020-05-06 10:46:54 -07:00
orientor 3ffc9466c9 openapi: Combine two similar openapi markdown extensions.
Previously api_description and api_code_examples were two independent
markdown extensions for displaying OpenAPI content used in the same
places.  We combine them into a single markdown extension (with two
processors) and move them to the openapi folder to make the codebase
more readable and better group the openapi code in the same place.
2020-05-05 21:42:31 -07:00
Aman Agrawal a4495dbc6b populate_db: Generate topics using config fixture.
Instread of using stream_name + Intergers as topics, we now
generate topics using pos in `config.generate_data.json`.

This helps us create and test more realistic topics.
2020-05-05 16:11:09 -07:00
Aman Agrawal 28375e82c5 generate_test_data: Remove unused get_stream_title function. 2020-05-05 16:11:09 -07:00
Anders Kaseorg 36d50cc465 test_fixtures: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:35:30 -07:00
Anders Kaseorg a9651e3e43 import_realm: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:35:30 -07:00
Anders Kaseorg fd65511fe9 retention: Improve move_rows escaping correctness with psycopg2.sql.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:35:30 -07:00
Anders Kaseorg ffe5402c49 migrate: Improve do_batch_update escaping correctness with psycopg2.sql.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:33:03 -07:00
Anders Kaseorg cebac3f35a db: Fix types to accept psycopg2.sql.Composable queries, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:33:03 -07:00
Anders Kaseorg d0b40cd7a3 test_runner: Remove _destroy_test_db monkey patch.
This was fixed upstream in Django 1.11.

https://code.djangoproject.com/ticket/27690

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:33:03 -07:00
Anders Kaseorg b405780561 do_delete_messages_by_sender: Add missing list conversion.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-04 09:33:03 -07:00
Tim Abbott 341787a5e0 retention: Use logging API in a more standard way. 2020-05-03 10:57:23 -07:00
Mateusz Mandera 0d7cbc71dd retention: Make logging less unnecessarily verbose.
For realms with no retention policy on themselves or any of their
streams, no archiving happens, but 3 lines of logs would be generated.
That's redundant and we make changes in this commit to avoid logging
those lines if nothing of interest is happening.
2020-05-03 19:24:00 +02:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
sahil839 7cfbf660e6 actions: Remove deactivated stream from default stream groups.
Stream after deactivation should be removed from all the
default stream groups that it belongs to.

Fixes #14760.
2020-05-01 17:45:06 -07:00
jagansivam28 bbf5a5efed invitation: Make Member to see invitations sent by him/her.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.

Fixes #14007.
2020-05-01 17:42:49 -07:00
Rohitt Vashishtha 7d3a31cd8b bugdown: Support hanging_lists preprocessor for indented lists.
Previously, hanging_lists preprocessor didn't consider anything
indented at 4 or above spaces to be a list. This meant that when
we had a list like:

1. 1
  2. 2
    3. 3
  2. 2a
1. 1a

We would insert a newline between 3. 3 and 2. 2a. This resulted
in the block processor breaeking down 1 list into 2 blocks, which
messed up the nesting and indentation for the second block.
2020-04-30 17:54:40 -07:00
Abhishek-Balaji 43e39718c4 user_name: Prevent users from setting name ending with |number.
We've had bugs in the past where users with a name in the format
"Alice|999" would confuse our markdown rendering or typeahead.  While
that's a fully solvable problem, there's no real use case for that, so
it's probably simpler to just prevent users from setting their name
that way.

Fixes #13923.
2020-04-30 15:59:12 -07:00
Wyatt Hoodes 5a58b7c549 data exports: Keep deleted export in UI table.
It makes sense to keep a deleted export in the table,
along with the time of deletion, for auditing reasons.
2020-04-30 13:00:59 -07:00
Wyatt Hoodes 82e7ad8e25 data exports: Handle pending and failed exports.
Prior to this change, there were reports of 500s in
production due to `export.extra_data` being a
Nonetype.  This was reproducible using the s3
backend in development when a row was created in
the `RealmAuditLog` table, but the export failed in
the `DeferredWorker`.  This left an entry lying
about that was never updated with an `extra_data`
field.

To fix this, we catch any exceptions in the
`DeferredWorker`, and then update `extra_data` to
encode the failure.  We also fix the fact that we
never updated the export UI table with pending exports.

These changes also negated the use for the somewhat
hacky `clear_success_banner` logic.
2020-04-30 13:00:59 -07:00
Aman Agrawal 5fa1dbf5b3 test_fixtures: Add DB function to write database digest.
This will give help up write new digest only if the db rebuild
succeeds. We were relying on the caller to
be successful in building db, this was hacky and unreliable.

We write new db digest once the caller succeeds, this ensures
that we write new digest after every successful attempt.

This fixes the anomality we were facing that Databases were rebuild
on the 2nd provision attempt with no changes to files or migrations.
This was happening because we didn't write a new digest for db
after the first provision (The case of DB didn't exist).

During the 1st provision, we check the template_status() of
Database both Dev and Test, but database_exists() of Databases
obviously returned false, and we rebuild the database,
but forgot to write_new_digest and hence the anomaly in the
second provision explained above.
2020-04-30 10:46:53 -07:00
Kartik Srivastava 7be52449a8 lib/bugdown/api_code_examples: Fix buggy rendering of print statement.
This fixes the buggy rendering of the print statement in the
python code examples.
2020-04-29 23:21:42 -07:00
Tim Abbott 8bba5dc6b6 test_fixtures: Include LOCAL_DATABASE_PASSWORD in digest.
This ensures that if one deletes `zproject/dev-secrets.conf`, we end
up rebuilding the databases from scratch (which, critically, will
ensure the password that gets setup matches what's in the current
version of the configuration file).

This should address a category of issue we've had where deleting
`zproject/dev-secrets.conf` would result in provision failing.
2020-04-29 22:57:37 -07:00
sahil839 3b99e3440e actions: Avoid unnecessary fake email address changes.
The logic in do_set_realm_property would previously "change" the email
addrssees of every user in the realm, even if they hadn't actually
changed.

We fix this by skipping the logic when it's unnecessary.
2020-04-29 16:50:08 -07:00
sahil839 ce6ea76110 actions: Use bulk_update when changing email_address_visibility.
bulk_update is used to update the email of user_profile objects in
database when email_address_visibility is changed.

This helps resolve the problem of timeout errors in realms with large
number of users due to large number of database queries run in a
loop.

Since bulk_update doesn't flush caches, we need our own bit of code to
do that.

Fixes a part of #14600.
2020-04-29 16:45:11 -07:00
Steve Howell 2c63130195 api: Always return zulip_version/zulip_feature_level.
We no longer make these conditional to simplify writing clients;
there's no cost to including them, and a real cost to not doing so.
2020-04-29 11:58:22 -07:00
Hashir Sarwar 3a110bd647 actions: Remove redundant 'url_embed_preview_enabled' check.
We add URLs to the `links_for_embed set`, only when
the `url_embed_preview_enabled` flag is turned on.
So, it is sufficient to check if `links_for_embed`
is not empty.
2020-04-28 22:41:00 -07:00
Mateusz Mandera f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
Vishnu KS 3f69500765 billing: Rename downgrade_realm_for_deactivation to downgrade_now. 2020-04-28 22:05:49 -07:00
Steve Howell cf78cb0d6e templates: Use `<hr>` and `<br>` consistently.
We now prevent these variations:

    * <hr/>
    * <hr />
    * <br/>
    * <br />

We could enforce similar consistency for other void
tags, if we wished, but these two are particularly
prevalent.
2020-04-28 17:05:48 -07:00
Puneeth Chaganti 88c297dffe tools: Allow specifying more options when capturing screenshot.
The name and directory into which the screenshot should be captured can be
specified now, apart from specifying the fixture file to use.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 2c325e34ac integrations: Remove mutable default argument for config_options. 2020-04-28 13:00:09 -07:00
Puneeth Chaganti 39fe206289 integrations: Change legacy arg to a bool from Optional[bool]. 2020-04-28 13:00:09 -07:00
orientor 64c6bab276 openapi: Create markdown extension for rendering endpoint descriptions.
Add function in openapi.py to access endpoint descriptions written
in zulip.yaml. Use this function for creating a markdown extension
for rendering endpoint descriptions written in zulip.yaml.

We use this extension for a single endpoint to get test coverage.
2020-04-28 12:57:19 -07:00
Steve Howell 1aaef75bc5 test databases: Fix dev/test typo.
This line was changed to the wrong tool by
accident in 23f09fadfa.
2020-04-28 07:13:22 -04:00
Tim Abbott 03fecba917 alert_words: Fix cache flushing behavior and add tests.
The post_init cache-flushing behavior in the original alert words
migration was subtly wrong; while it may have passed tests, it didn't
have the right ordering for unlikely races.

We use post_save rather than post_init hooks precisely because they
ensure that we flush the cache after we know the database has been
updated and any future reads from the database will have the latest
state.
2020-04-27 11:48:32 -07:00
Tim Abbott 8e5b0351b3 alert_words: Fix case-sensitivity of alert words.
Previously, alert words were case-insensitive in practice, by which I
mean the Markdown logic had always been case-insensitive; but the data
model was not, so you could create "duplicate" alert words with the
same words in different cases.  We fix this inconsistency by making
the database model case-insensitive.

I'd prefer to be using the Postgres `citext` extension to have
postgres take care of case-insensitive logic for us, but that requires
installing a postgres extension as root on the postgres server, which
is a pain and perhaps not worth the effort to arrange given that we
can achieve our goals with transaction when adding alert words.

We take advantage of the migrate_alert_words migration we're already
doing for all users to effect this transition.

Fixes #12563.
2020-04-27 11:31:51 -07:00
Abhishek-Balaji 052368bd3e alert_words: Move alert_words from UserProfile to separate model.
Previously, alert words were a JSON list of strings stored in a
TextField on user_profile.  That hacky model reflected the fact that
they were an early prototype feature.

This commit migrates from that to a separate table, 'AlertWord'.  The
new AlertWord has user_profile, word, id and realm(denormalization so
we can provide a nice index for fetching all the alert words in a
realm).

This transition requires moving the logic for flushing the Alert Words
caches to their own independent feature.

Note that this commit should not be cherry-picked without the
following commit, which fixes case-sensitivity issues with Alert Words.
2020-04-27 11:29:50 -07:00
Abhishek-Balaji 818776faae alert bugdown: Change name of AlertWordsNotificationProcessor.
This is a precursor commit to change the name of
AlertWordNotificationProcessor to AlertWordsNotificationProcessor
to match the change from UserProfile.alert_words to Alertword.
2020-04-27 10:45:18 -07:00
Rohitt Vashishtha 087b54aa45 markdown: Keep only 'text' for a no-op pygments rendering.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.

For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
2020-04-27 10:40:29 -07:00
rebtung f7fbe3419f api: Improve consistency of reactions API.
Previously, the message and event APIs represented the user differently
for the same reaction data. To make this more consistent, I added a
user_id field to the reaction dict for both messages and events. I
updated the front end to use the user_id field rather than the user
dict. Lastly, I updated front end and back end tests that used user
info.

I primarily tested this by running my local Zulip build and
adding/removing reactions from messages.

Fixes #12049.
2020-04-26 10:35:55 -07:00
Puneeth Chaganti 2a65be2bf5 url preview: Use Chrome's user agent instead of a Zulip one.
Some sites don't render correctly unless you are one of the latest browsers.
YouTube Music, for instance, changes the page title to "Your browser is
deprecated, please upgrade.", which makes our URL previews look bad.
2020-04-26 10:16:43 -07:00
Rohitt Vashishtha ce6c57b714 markdown: Set default code block language for quotes and latex as well.
In the original implementation, we were checking for the default language
inside format_code, which resulted in the setting being ignored when set to
quote, math, tex or latex. We shift the validation to `check_for_new_fence`

We also update the tests to use a saner naming scheme for the variables.
2020-04-23 17:51:01 -07:00
Vishnu KS 22b5297e32 actions: Mark strings in send_welcome_bot_response for translation. 2020-04-23 17:19:27 -07:00
Vishnu KS 03286012d2 onboarding: Mark strings in send_initial_realm_messages for translation. 2020-04-23 17:18:39 -07:00
Vishnu KS 9a2ee29ac7 onboarding: Mark strings in send_initial_pms for translation. 2020-04-23 17:15:31 -07:00
sahil839 3a7de8ad3b models: Add has_permission as a generic function for different policies.
This commit removes can_create_streams and can_subscribe_other_users
to use has_permission as a generic function in UserProfile model for
these settings policies.

Relevant changes are made to events.py to avoid duplication at some
places.
2020-04-22 17:48:52 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 679f092a75 integrations: Extract code to get logo path for integrations. 2020-04-22 17:45:30 -07:00
Steve Howell 02252c255a db tools: Use common scheme for digests.
We have two different digest schemes to make
sure we keep the database up to date.  There
is the migration digest, which is NOT in the
scope of this commit, and which already
used the mechanism we use for other tools.

Here we are talking about the digest for
important files like `populate_db.py`.

Now our scheme is more consistent with how we
check file changes for other tools (as
well as the aformentioned migration files).

And we only write one hash file, instead of
seven.

And we only write the file when things have
actually changed.

And we are explicit about side effects.

Finally, we include a couple new bot settings
in the digest:

    INTERNAL_BOTS
    DISABLED_REALM_INTERNAL_BOTS

NOTE: This will require a one-time transition,
where we rebuild both databases (dev/test).
It takes a little over two minutes for me,
so it's not super painful.

I bump the provision version here, even
though you don't technically need it (since
the relevant tools are actually using the
digest files to determine if they need to
rebuild the database).  I figure it's just
good to explicitly make this commit trigger
a provision, and the user will then see
the one-time migration of the hash files
with a little bit less of a surprise.

And I do a major bump, not a minor bump,
because when we go in the reverse direction,
the old code will have to rebuild the
database due to the legacy hash files not
being around, so, again, I just prefer it
to be explicit.
2020-04-22 14:41:42 -07:00
Mateusz Mandera ec6022a186 logging_util: Handle record.status_code missing in skip_200_and_304.
Fixes #14595.
Invalid HTTP requests could end up in an unhandled exception in
skip_200_and_304 due the record not having the status_code attribute
set. With this change we'll avoid the exception

Example:
curl  -X POST -H 'Transfer-Encoding : chunked' --data-binary 'a' 'http://zulipdev.com:9991/json/messages/57'

2020-04-21 10:56:22.007 WARN [django.server] "POST /json/messages/57 HTTP/1.1" 405 95
2020-04-21 10:56:22.007 INFO [django.server] code 400, message Bad request syntax ('a')
2020-04-21 10:56:22.008 WARN [django.server] "a" 400 -
2020-04-22 11:26:13 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Steve Howell 239474124e test-backend: Remove generate_fixtures option.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).

The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.

The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.

And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.

This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.

The other caller wasn't using `use_force`.
2020-04-22 09:24:42 -07:00
Steve Howell 23f09fadfa refactor: Use run in update_test_databases_if_required.
Use `run` to run the tools, and take advantage
that `rebuild-dev-database` is really the same
as `generate-fixtures --force`.
2020-04-22 09:24:42 -07:00
Steve Howell b10be1f8b7 refactor: Early-exit in update_test_databases_if_required.
Just make each conditional run what it needs to run.  The
simplicity that this provides will be more apparent
soon.
2020-04-22 09:24:42 -07:00
Steve Howell 67d0349239 refactor: Use precise names for migration helpers.
Somewhat confusingly, we have two types of different
digests related to databases.  The migration digests
are pragmatic, since changes to migrations are a bit
more frequent for certain use cases and don't
necessitate a complete rebuild of the database.

Anyway, these are just more specific names.
2020-04-22 09:24:42 -07:00
Anders Kaseorg 088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Kartik Srivastava 18b577f600 lib/bugdown/api_code_examples: Refactor extract_python_code_example.
This refactors `extract_python_code_example` to accept an
`example_regex` parameter. It can now be used to extract code examples
from javascript_examples.py.
2020-04-21 17:31:07 -07:00
Tim Abbott 703fae8980 send_custom_email: Use a special .gitignored directory.
Previously, the send_custom_email code path leaked files in paths that
were not `.gitignored`, under templates/zerver/emails.

This became problematic when we added automated tests for this code
path, as it meant we leaked these files every time `test-backend` ran.

Fix this by ensuring all the files we generate are in this special
subdirectory.
2020-04-21 16:50:11 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Steve Howell ca49f38619 provision: Extract helpers for paths to hash.
I make these all functions for consistency,
and in particular I want to continue to avoid
`glob.glob` calls until we are actually
computing hashes.

This is mostly a prep to allow us to do
hashing in two separate places:

    - check hashes
    - update hashes

We would only update hashes **after** running
processes anew.

For `provision_inner` I considered using a
class to put the three path-related helpers
into a mini namespace, but it felt too heavy.

It wouldn't be completely implausible here
to extract something like a JSON config
file that has a list of globs for each
process that we do path-hashing for, but I
want to clean up other stuff first.
2020-04-20 15:06:47 -07:00
Steve Howell 2dd6e6f568 refactor: Add Database.database_exists(). 2020-04-20 15:06:47 -07:00
Steve Howell 4822f8d7d6 refactor: Add Database.template_status.
This is mostly a pure code move from
template_database_status().
2020-04-20 15:06:47 -07:00
Steve Howell 0ea4f727d4 refactor: Change params to template_database_status(). 2020-04-20 15:06:47 -07:00
Steve Howell 108b43c873 refactor: Add Database.what_to_do_with_migrations().
This is purely a code move and s/database/self/.
2020-04-20 15:06:47 -07:00
Steve Howell cce223965b refactor: Tweak args to what_to_do_with_migrations.
This is a minor prep commit--we'll move it into
the class next.
2020-04-20 15:06:47 -07:00
Steve Howell 5c5d85cf19 test databases: Add Database.run_db_migrations().
We can reduce some code duplication by having this
on the class.
2020-04-20 15:06:47 -07:00
Steve Howell 1795c06a53 tests databases: Clean up Database class.
We now remove the `Type` and `_TYPE` suffixes,
as we will start treating this like a real
class with behavior, instead of a glorified
struct.

We pass in `platform_type`, so that we can
just derive some of our data from that,
where naming conventions apply.

And we use the name `migrations_status_path`,
instead of the name `migration_status`, which
had two different meanings before this change.
2020-04-20 15:06:47 -07:00
Steve Howell 33cbb4f688 provision: Early-exit in template_database_status.
This is a pure refactor, and we just early-exit
in case the datbase doesn't exist (knowing that
that can be a bit of a lie now--see the comment
I added.)
2020-04-20 15:06:47 -07:00
Mateusz Mandera 4018dcb8e7 upload: Include filename at the end of temporary access URLs. 2020-04-20 10:25:48 -07:00
Udit107710 cc542a607e refactor: Making onboarding independent of actions.
Moved missing_any_realm_internal_bots from actions.py to
onboarding.py since it wa only being used by it.
2020-04-18 21:48:41 -07:00
Udit107710 db30cf470c refactor: Making email_mirror independent of actions.
Moved truncate_body, truncate_content and truncate_topic
to message.py.
2020-04-18 16:58:29 -07:00
Udit107710 16218d6de3 streams: Remove dependency of streams on actions.
Refactored code in actions.py and streams.py to move stream related
functions into streams.py and remove the dependency on actions.py.

validate_sender_can_write_to_stream function in actions.py was renamed
to access_stream_for_send_message in streams.py.
2020-04-18 16:56:59 -07:00
Abhishek-Balaji c83f147a9a alert_words: Remove unnecessary do_set_alert_words.
This function was only in test_bugdown.py and did the same thing as
add_alert_words in that context.
2020-04-18 16:18:59 -07:00
Tim Abbott e1849b63c1 send_email: Use CommandError for user-facing command line errors.
This provides much nicer error output (not a traceback).
2020-04-18 13:30:03 -07:00
wowol 507f889901 send_custom_email: Add support for emailing all admins.
This provides a convenient way to send a custom email to just the
administrators of an organization.

Fixes part of #13413.
2020-04-18 13:27:30 -07:00
Anders Kaseorg 11194873ca requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Anders Kaseorg d3c55c166e requirements: Upgrade mypy from 0.761 to 0.770.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Tim Abbott a25b38cd0e i18n: Fix translated strings for topic-move notices.
These strings were improperly using positional string substitution,
which doesn't work with i18n for languages with a different word order.
2020-04-17 13:46:39 -07:00
Steve Howell c0af648b0c db checks: Remove options from database_exists().
The original commit had `options` without actually
using it:

    dbeab6aa6f

We still aren't using it, so I removed the needless
confusion.
2020-04-17 09:53:28 -07:00
Steve Howell 067196c508 provision: Simplify `is_force` codepaths.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.

If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.

Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.

For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.

We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
2020-04-17 09:45:59 -07:00
Tim Abbott 0ccc0f02ce upload: Support requesting a temporary unauthenticated URL.
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).

The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.

Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-04-17 09:08:10 -07:00
Tim Abbott 7f582b3861 upload: Increase the lifetime of signed upload URLs.
For some mobile use cases, 15 seconds is potentially too short for a
busy+slow device to open a browser and fetch the URL.  60 seconds is
plenty, and doesn't carry a materially increased security risk.
2020-04-17 09:08:10 -07:00
Vishnu KS a2781e6364 emails: Set correct language for email in send_email_to_admins.
Previously the emails were translated to the default_language of
admin[0] in build_email function. Now we use realm.default_language
instead.
2020-04-16 19:31:08 -07:00
Puneeth Chaganti 4d2ce607c9 tools: Add script to trigger webhook notification using fixtures.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.

Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.

Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
2020-04-16 19:25:13 -07:00
Ryan Rehman 9340cd1a0b muting: Send muted_topic's date_muted field to frontend. 2020-04-15 15:48:25 -07:00