Commit Graph

11560 Commits

Author SHA1 Message Date
sahil839 bd9e891e1c dev_login: Change dev env login page to show Desdemona as owner.
This commits changes the dev login page to show Desdemona as
owner.
2020-06-01 15:22:44 -07:00
sahil839 2fe8971204 populate_db: Change role of 'desdemona' to realm owner.
Modified by tabbott to update PROVISION_VERSION.
2020-06-01 15:22:27 -07:00
sahil839 169bd3082a users: Modify format_user_row to set is_admin as True for owners also. 2020-06-01 15:14:16 -07:00
Mateusz Mandera 90b2f933b0 tests: Change self.client_post to client_get in remote sso tests.
GET is the intended way to use this endpoint, this is how the mobile and
desktop apps pass their params.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 4b10a07c3e auth: Fix social signup with the desktop app.
Only start_social_login had the code for handling the desktop
application, it was missing from start_social_signup.
2020-06-01 14:14:58 -07:00
Mateusz Mandera 676305f6ab auth: Extract handle_desktop_flow wrapper. 2020-06-01 14:14:58 -07:00
sahil9001 328caf1ad5 api: Simplify format of realm_emoji author data.
There's no reason to send data beyond the user `id` of the uploader,
and reason not to, as the previous model was both awkward when
`author=None` and resulted in unecessary parsing complexity for
clients.

Modified by tabbott to add the frontend changes and API documentation.

Fixes #15115.
2020-05-31 17:44:50 -07:00
Tim Abbott c78d728812 events: Fix test_events race handling of role changes.
Previously, our apply_events logic (verified by test_events) did not
properly handle changing a user's role to and from being a guest.
2020-05-31 17:31:33 -07:00
sahil839 2ab6767b73 events: Update person dict in event for do_change_user_role to send role.
This commit changes the person dict in event sent by do_change_user_role
to send role instead of is_admin or is_guest.

This makes things much more straightforward for our upcoming primary
owners feature.
2020-05-31 17:22:50 -07:00
Mateusz Mandera 200ce821a2 user_activity: Put client id instead of name in event dicts.
This saves the completely unnecessary work of mapping the Client name
to its ID.  Because we had in-process caching of the immutable Client
objects, this isn't a material performance win, but it will eventually
let us delete that caching logic and have a simpler system.
2020-05-29 15:19:55 -07:00
Mateusz Mandera acfb378245 tests: Change some assertTrue usage where assertEqual was meant.
These were typos, causing the asserts to pass always, because the syntax
was meant for assertEqual.
2020-05-29 15:15:52 -07:00
orientor 5b5095ad9f openapi: Use correct `operationId` atrribute for endpoints.
Update `operationId` of some endpoints to their correct version.
2020-05-29 15:04:50 -07:00
orientor d2ee99a2fd openapi: Add markdown extension for rendering return values in API docs.
Currently response return values have to be written twice, once in
the docs and once in zulip.yaml. Create a markdown extension so
that the return values in api docs are rendered using content from
zulip.yaml
2020-05-29 15:02:56 -07:00
orientor 98903df639 openapi: Fix some zulip.yaml return values description.
There are several places where the return values description is better
in the api docs than in the zulip.yaml file. Import such changes to
zulip.yaml before we replace the current hacky API docs with the
zulip.yaml content.
2020-05-29 15:02:43 -07:00
sahil839 9fa60672e6 users: Modify update user API endpoint to accept role as parameter.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.

User role dropdown in user info modal is also modified to use
"dropdown_options_widget".

Modified by tabbott to document the API change.
2020-05-29 14:29:17 -07:00
Alex Vandiver 107fe3d3b4 tests: Remove references to trac.zulip.net.
That host has not existed for some time; rename the tests to use a
URL that is clearly nonfunctional.
2020-05-28 17:24:35 -07:00
Anders Kaseorg ba7906a3c6 bugdown: Revert NamedTuple declarations to old style.
This reverts part of commit 8bcdf4ca97
(#15093), to work around a mypy caching bug:
https://github.com/python/mypy/issues/7281.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-28 16:47:08 -07:00
Alex Vandiver 0288002b84 models: Comment call to python2-esque `iteritems()`.
This method is called on a `bitfield.types.BitHandler` object, not a dict; as
such, it is not a runtime failure.

This was missed in 035ed93111 and adjacent commits because it was not
`six.iteritems()` but just a bare `iteritems()` -- having been added
after 9e3c3e14f5 moved all other calls from `iteritems()` to
`six.iteritems()`.

Comment on why this is not a leftover python2-ism.

Grep finds no other remaining callsites to `iteritems` or friends:
```
$ rg '\.iter\w+\(' -t py
zerver/lib/url_preview/preview.py
44:        content = next(response.iter_content(1000))

zerver/lib/email_notifications.py
49:    for link_info in fragment.iterlinks():

zerver/lib/push_notifications.py
508:        nested_levels = len(list(ol.iterancestors('ol')))

zerver/lib/bugdown/__init__.py
1364:            parsed = etree.iterparse(StringIO(rendered))

zerver/lib/bugdown/nested_code_blocks.py
47:                if parent.text is None and len(list(parent)) == 1 and len(list(parent.itertext())) == 1:
```
2020-05-28 15:56:00 -07:00
Imran Iqbal 465947d1bb
webhooks: Fix travis display emoji for `Pending` status.
The 🔄 emoji is spelled  :counterclockwise: in Zulip.
2020-05-28 15:02:30 -07:00
Hemanth V. Alluri 37d34dda1c auth: Make sure that we enforce max_length during password reset.
Make sure that we use the max password length defined in
RegistrationForm.MAX_PASSWORD_LENGTH when validating the
password for the password reset form.

Fixes #15087.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-28 15:01:33 -07:00
Tim Abbott 4d2b1673f8 docs: Replace support@zulipchat.com with support@zulip.com.
The new address is cleaner and shorter.
2020-05-28 08:14:30 +00:00
sahil839 7d9d2414ca actions: Modify do_change_user_role for realm owner.
This commit changes do_change_user_role to support adding or removing
the realm owner status of user and sending an event.

We also extend the existing test for do_change_user_role to do a bit
more validation to confirm the audit log records all values of role.
2020-05-27 15:52:39 -07:00
sahil839 8bbc07474f models: Add realm owner role and is_realm_owner property for user.
The new realm_owner role is added as option for role field in
UserProfile model and is_realm_owner is added as property for the user
profile.

Aside from some basic tests validating the logic, this has no effect
as users cannot end up with set as realm owners.
2020-05-27 15:47:42 -07:00
clarammdantas f0c85c5796 registration: Show error if invalid key is posted on /accounts/register.
If the key paramenter on POST isn't correct we won't be
able to find the confirmation object, which will lead to
an exception. To deal with it more gracefully, we are
catching the exception and redirecting to the
confirmation_link_expired_error page.
2020-05-27 15:37:16 -07:00
clarammdantas edcf4f0ea2 invitations: Revoke remaining invitations after user registers.
If a user receives more than one invite to join a
realm, after that user registers, all the remaining
invitations should be revoked, preventing them to be
listed in active invitations on admin panel.
To do this, we added a new prereg_user status,
STATUS_REVOKED.

We also added a confirmation_link_expired_error page
in case the user tries click on a revoked invitaion.
This page has a link to login page.

Fixes: #12629

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
clarammdantas 923c5e3063 test_signup: Check if user is already registered.
This tests if a user, that is already registered, is
redirected to the login page when they click on an
invitation.

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
clarammdantas 8869f05c20 test_signup: Attach prereg_user to confirmation obj.
Tests attached a UserProfile to confirmation objects,
which is not very valid as this is the only place
where this is done. Now we attach PreregUser to
the confirmation object, making the tests correct.

Co-authored-by: Arunika <arunikayadav42@gmail.com>
2020-05-27 15:37:16 -07:00
orientor 7e1bc9bf2f openapi_responses: Fix get_user_presence. 2020-05-27 13:23:37 -07:00
orientor cffb927f07 openapi_responses: Fix get_own_user. 2020-05-27 13:23:36 -07:00
orientor 1273545637 openapi_responses: Fix get_messages. 2020-05-27 13:21:04 -07:00
orientor 4096e0e6b9 openapi_responses: Fix list_subscriptions. 2020-05-27 13:15:51 -07:00
orientor a4cf77e5dc openapi_responses: Fix get_message_history. 2020-05-27 13:15:51 -07:00
orientor 29ef17c999 openapi_responses: Fix get_realm_emoji. 2020-05-27 13:15:51 -07:00
orientor b9c31942b5 openapi_responses: Fix get_user_groups. 2020-05-27 13:15:51 -07:00
orientor 74e9019922 openapi_responses: Fix get_stream_topics. 2020-05-27 11:27:27 -07:00
orientor ae8167cd09 openapi_responses: Fix get_users and get_user. 2020-05-27 11:27:26 -07:00
orientor bc50308799 openapi_responses: Fix send_message. 2020-05-27 11:23:14 -07:00
orientor f50aab6740 openapi_responses: Fix update_subscription_settings. 2020-05-27 11:23:12 -07:00
orientor 6d438cbba4 openapi_responses: Fix get_streams. 2020-05-27 21:29:14 +05:30
whoodes cea7d713cd requirements: Upgrade boto to boto3.
Fixes: #3490

Contributors include:

Author:    whoodes <hoodesw@hawaii.edu>
Author:    zhoufeng1989 <zhoufengloop@gmail.com>
Author:    rht <rhtbot@protonmail.com>
2020-05-26 23:18:07 -07:00
sahil9001 94dfe4bed3 portico: Fix apps page webapp link being useless.
The /apps page webapp link now takes the user to /accounts/go to find
their organization's login page, rather than failing to do anything.

Fixes #14977.
2020-05-26 21:50:31 -07:00
arunikaydav42 4680d504de invite: Fix invitations removed from list too soon.
On invitations panel, invites were being removed when
the user clicked on invitation's link. Now we only remove
it when the user completes registration.

Fixes: #12281
2020-05-26 21:45:44 -07:00
Tim Abbott 368e9572cc billing: Clean up copy on free trial pages.
This fixes some issues with unclear terminology and visual styling in
the pages for the new free trial.

There's probably more we can and should usefully do in the future.
2020-05-26 21:31:07 -07:00
Vishnu KS 8784539d53 free trial: Send users to /upgrade after realm creation. 2020-05-26 17:01:32 -07:00
Anders Kaseorg cf923b49d3 python: Remove extra pass statements with autoflake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 8bcdf4ca97 python: Convert TypedDict declarations to Python 3.6 style.
A subset of the diff generated by pyupgrade --py36-plus
--keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg f5b33f9398 python: Further pyupgrade changes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Tim Abbott 463f1503fc Revert "markdown: Process fenced code blocks in blockquotes."
This reverts commit 7002f98ea1.

This failed tests due to some sort of conflict with a recent
python-markdown upgrade.
2020-05-25 18:13:03 -07:00
Rohitt Vashishtha 7002f98ea1 markdown: Process fenced code blocks in blockquotes.
We handle fenced code blocks in a preprocessor, and > style blockquotes
are parsed in a blockprocessor. Pymarkdown doesn't run the preprocessors
again on any blocks that it is parsing, and is unlikely to accept our
solution upstream; they intend to convert fenced_code to a block parser.

We simply run all the preprocessors on the text again, with the exception
of NormalizeWhitespace which removed delimiters used by HtmlStash to mark
preprocessed html code. To counter this, we subclass NormalizeWhitespace
and use our customized version for when it is called from a blockparser.

Upstream issue: https://github.com/Python-Markdown/markdown/issues/53

Fixes #12800.
2020-05-25 17:35:10 -07:00
sahil839 1aebf3cab9 actions: Merge do_change_is_admin and do_change_is_guest.
This commit merges do_change_is_admin and do_change_is_guest to a
single function do_change_user_role which will be used for changing
role of users.

do_change_is_api_super_user is added as a separate function for
changing is_api_super_user field of UserProfile.
2020-05-25 16:17:10 -07:00
Mateusz Mandera b66dc9de50 saml: Support IdP-initiated SSO. 2020-05-25 16:09:30 -07:00
Mateusz Mandera f2d052bff8 tests: Flush session before a simulated cross-domain POST in saml tests.
This is important, because lack of this meant that the POST request in
our tests still had the old session, with various params stored in it.
This mechanism doesn't work in reality in SAML, so the backend uses
redis to store and recover the params from redis. Without flushing the
session, these tests would fail to catch some breakages in the
redis-based mechanism.
2020-05-25 15:53:15 -07:00
Mateusz Mandera 13c3eaf086 rate_limit: Restrict tornado backend to explicitly specified domains.
This will protect us in case of some kinds of bugs that could allow
making requests such as password authentication attempts to tornado.
Without restricting the domains to which the in-memory backend can
be applied, such bugs would lead to attackers having multiple times
larger rate limits for these sensitive requests.
2020-05-25 15:40:00 -07:00
Vishnu KS 8b3eb9a846 landing: Add option to show custom navbar message. 2020-05-24 17:34:03 -07:00
Dinesh 288921d425 auth: Log when a user tries to login with deactivated account.
Helps to see if users are often trying to login with deactived
accounts.
A use case: Trackdown whether any deactivated bot users are still
trying to access the API.

This implementation adds a new key `inactive_user_id`
to `return_data` in the function `is_user_active` which
check if a `user_profile` is active. This reduces the effort
of getting `user_id` just before logging.

Modified tests for line coverage.
2020-05-24 17:27:19 -07:00
Aman Agrawal 91c7ea2275 test_reactions: Cache stores reactions data after formatting.
Reactions data should always be formatted before it saved in cache,
otherwise clients receive malformed data.

This is followup on a53daa6f8c
2020-05-24 17:03:30 -07:00
Mateusz Mandera dac4a7a70b saml: Figure out the idp from SAMLResponse.
Instead of plumbing the idp to /complete/saml/ through redis, it's much
more natural to just figure it out from the SAMLResponse, because the
information is there.
This is also a preparatory step for adding IdP-initiated sign in, for
which it is important for /complete/saml/ to be able to figure out which
IdP the request is coming from.
2020-05-24 16:40:28 -07:00
Mateusz Mandera c74f8363e2 saml: Gracefully handle bad SAMLResponses. 2020-05-24 16:40:28 -07:00
Tim Abbott a53daa6f8c message: Fix malformed reaction data.
After a message was reset in our caches cache via message editing or
adding/removing a reaction, we were sending corrupt data to the cache
because build_message_dict (and thus build_dict_from_raw_db_row) was
improperly being called before sewing in the reaction data.

As a result, we were sending raw database data in the reaction
dictionaries, rather than the reformatted version expected by the API.

Bug introduced in 2a4c62a326.

Fixing this correctly required moving the rendering_realm_id logic one
step higher in the call chain, which is a useful refactoring anyway
(since we're no longer passing a `Message` object down)
2020-05-21 14:09:53 -07:00
Hemanth V. Alluri ab03659095 webhooks/github: Provide more detailed info for unexpected events.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-21 12:31:43 -07:00
Rohitt Vashishtha 88367a129c markdown: Disable tex and latex for math rendering.
We now parse tex and latex as regular languages, highlighting them
with pygments. We only allow 'math' to trigger latex rendering,
which is in line with the documentation.
2020-05-21 12:30:27 -07:00
Mateusz Mandera e2262b0b64 queue_processors: Log time spent getting data for url in embed_links. 2020-05-21 12:13:46 -07:00
Tim Abbott e59a2a23c3 portico: Fix a few issues with /for/research. 2020-05-20 17:57:22 -07:00
Rohitt Vashishtha 52c25a9301 markdown-timestamp: Use data-timestamp attribute.
This commit shifts our timestamp syntax to be of the form:

    <span class="timestamp data-timestamp="123456"></span>

since value is not a valid attribute of span elements.
2020-05-20 14:28:08 -07:00
Rohitt Vashishtha b062e8332f markdown: Add timestamp syntax to markdown processors.
This adds support for syntax like: !time(Jun 7 2017, 6:30 PM) so that
everyone sees the time in their own local timezone. This can be used
when scheduling online meetings, etc.

This adds some hardcoded values for timezones, because of there
being no sureshot way of determining the timezone easily. However,
since the main way of using the feature should be a typeahead for
entering the time, this shouldn't be cause of much concern.

Fixes #5176.
2020-05-20 14:23:55 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-05-20 13:47:20 -07:00
Dinesh 2711619376 redis_utils: Extend `put_dict_in_redis` to accept token as param.
This extends `put_dict_in_redis` to take token as an argument
and return that with the as a `key` with following key format.

Also, edit regex for token to include uppercase letters as
a token sent during apple authentication contains uppercase
letters.

Useful for Adding "Sign in with Apple" support.
2020-05-20 13:30:44 -07:00
orientor c7a2c48bb7 openapi: Add tests for checking presence of various attributes.
The swagger validator is a basic tool to check whether our
openapi specification file follows the basic syntax. But to ensure
that our zulip.yaml file is not only syntactically compatible but
also describes our API well, we need to add custom tests. This
commit currently checks whether each endpoint has an `operationId`
and a valid tag. It also makes it easier to check for custom rules
in the future.
2020-05-20 13:10:58 -07:00
Imran Iqbal da084528db feat(travis): display emoji for `Pending` status
* Encountered `No emoji specified for status 'Pending'` when using `on_start: always`:

```yaml
notifications:
  webhooks:
    ...
    on_start: always    # default: never
```
2020-05-20 13:09:30 -07:00
Rohitt Vashishtha 1e4ad0bdfc js-api: Migrate and test send-message example. 2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 203b722624 js-api: Migrate and test create-user example. 2020-05-20 10:18:29 -07:00
Kartik Srivastava 5f3849b517 openapi/markdown_extension: Avoid shelling out to node for JS snippets.
This refactors render_javascript_code_example to avoid shelling out to
node and parse the javascript file with python instead, to get example
code snippets.
2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 6dd4030e67 js-api: Add hooks to run tests and render docs of JS API examples.
This commit adds python code to call javascript_examples.js in its
two supported modes. tools/test-api asserts that the example output
is as expected, whereas the API markdown extension is used to render
these examples in the docs.
2020-05-20 10:18:29 -07:00
Kartik Srivastava 642d1a20d0 openapi/markdown_extension: Refactor extract_code_example.
This refactors `extract_code_example` to return a nested list
of code snippets between '{code_example|start/end}' instead of
returing a list of all the lines between '{code_example|start/end}'
markers in the code examples.

Appropriate changes have been made to render_python_code_example.
2020-05-20 10:18:29 -07:00
Kartik Srivastava 302906211d js-api: Refactor ExamplesHandler to avoid running examples in a loop.
This refactors `ExamplesHandler` to avoid running examples in a loop
and add result objects to `response_data` array one by one with
`generate_validation_data`.
2020-05-20 10:18:29 -07:00
Rohitt Vashishtha 31d04fb370 js-api: Add module to create and run JS API examples.
This file will act as the container for all JS API examples to use
in our documentation, similar to our python and curl API testing
and examples generation code.

This module has two modes of operation:

- node javascript_examples.js generate-responses

   This mode runs all the examples against a server and prints the JSON
   output of all the examples we ran.

- node javascript_examples.js generate-example <endpoint>

   This mode prints example code for endpoints like: /users:post. We then
   want to render this full example code in our docs.
2020-05-20 10:18:29 -07:00
Mateusz Mandera 2f5fd272aa auth: Gracefully handle bad http responses from IdP in social auth.
If the IdP authentication API is flaky for some reason, it can return
bad http responses, which will raise HTTPError inside
python-social-auth. We don't want to generate a traceback
in those cases, but simply log the exception and fail gracefully.
2020-05-20 09:30:56 -07:00
orientor 915d801327 openapi: Add 'tags' attribute to endpoints.
'tags' attribute is helpful in differentiating and grouping the
endpoints on basis of their usage. For example tags like 'messages'
help in grouping all endpoints related to messages and thus make the
api specification more user-friendly. So give tags to the endpoints
on the basis of what heading they are under in the API docs.
2020-05-19 23:17:31 -07:00
orientor 3467d2fe68 openapi: Add 'operationId' attribute to endpoints.
'operationId' helps code generators in naming functions and other purposes.
So name operationId of endpoints as their function names in python-zulip-api
if it exists else use most appropriate function name.

Part of #14100 .
2020-05-19 23:13:39 -07:00
Kartik Srivastava 59ac3ca95f openapi/python_examples: Add reactivate_user example. 2020-05-19 23:10:50 -07:00
Kartik Srivastava 4af7ad70e7 openapi: Add OpenAPI data for /users/{user_id}/reactivate. 2020-05-19 23:10:50 -07:00
Vishnu KS 9129da88af tests: Add test for STANDARD plan_type in test_message_retention_days. 2020-05-19 14:48:59 -07:00
Vishnu KS a699f39050 models: Use UPGRADE_TEXT_STANDARD in ensure_not_on_limited_plan. 2020-05-19 14:48:59 -07:00
Vishnu KS 4e0d8953b4 models: Create ensure_not_on_limited_plan instance method. 2020-05-19 14:48:59 -07:00
sahil839 9b78a73e36 populate_db: Add new admin user as 'Desdemona'.
This commit adds a second admin user named 'Desdemona' to dev and
test database.
2020-05-19 11:42:27 -07:00
Aman Agrawal 2a4c62a326 update_to_dict_cache: Use bulk queries when preparing msgs for cache.
During events such as stream / topic name edit for a topic, we were
running queries to db in loop for each message for reactions,
submessages and realm_id. This commit reduces the queries to be
done only for realm_id, which is yet to be fixed.

This is accomplished by building messages with empty reactions
and submessages and then updating them in the messages using bulk
queries.
2020-05-19 10:30:03 -07:00
Aman Agrawal b8fe6245e3 message: Extract method to sew submessages and reactions to msg. 2020-05-19 10:30:03 -07:00
Aman Agrawal 21acc65489 test_messages: Test number of queries performed by `to_dict_uncached`.
`to_dict_uncached` performs queries to db in a loop which should be
replaced with bulk db queries.
2020-05-19 10:30:03 -07:00
Hemanth V. Alluri c8f4177872 webhooks/github: Add a support for team/edited events.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-16 14:59:08 -07:00
sahil839 46ef6816b6 stream: Allow non admins to set stream post policy when creating streams.
This commit allows non admins to set stream post policy while creating
streams.

Restriction was there to prevent user from creating a stream in which
the user cannot post himself but this will be taken care of with
stream admin feature.
2020-05-16 14:53:22 -07:00
Vishnu Ks 66b1ad7002 billing: User FREE_TRIAL_DAYS instead of FREE_TRIAL_MONTHS. 2020-05-16 14:52:01 -07:00
Mateusz Mandera b234fe8ccb retention: Pass optional realm argument to move_messages_to_archive.
This allows having the realm field of ArchiveTransaction set instead of
NULL when using move_messages_to_archive.
2020-05-16 14:46:56 -07:00
Kartik Srivastava 112f539034 openapi/python_examples: Update update_user example.
This updates the `update_user` example to use `update_user_by_id`.
2020-05-16 14:45:47 -07:00
Kartik Srivastava d2ab5673fd openapi/python_examples: Update deactivate_user example.
This updates the `deactivate_user` example to use
`deactivate_user_by_id`.
2020-05-16 14:45:47 -07:00
Mateusz Mandera 7d8a3581a5 retention: Clarify the status of cross-realm huddles in a comment. 2020-05-16 14:42:40 -07:00
Tim Abbott f10f2600e0 retention: Fix OOM issues when deleting large numbers of transactions.
For unknown reasons, deleting 10,000s of ArchiveTransaction objects
results in rapidly growing memory in the job making the request in the
Django process, eventually leading to an OOM kill.

I don't understand why Django behaves that way; I would have expected
the failure mode to instead be a serious load problem on the database
server, but perhaps the way Django's internal deletion logic handles
cascading the deletes to many millions of ArchiveMessages and other
ForeignKey objects requires tracking a lot of data in memory.

The solution is the same in any case, which is to batch the deletions
to execute a reasonable number of them at once.  Doing a single
ArchiveTransaction at a time would likely result in huge numbers of
database queries in a loop, which performs very poorly.  So we balance
by batching deletions in groups of 100 ArchiveTransactions; testing
this in production, I saw no spike of memory usage materially beyond
that of a normal Django process, and each bulk-deletion transaction
takes several seconds to process (meaning per-transaction overhead is
negligible).
2020-05-15 17:10:19 -07:00
Tim Abbott bd3c0aef9b presence: Optimize performance of mobile_query.
I'm not sure exactly what series of history got us here, but we were
fetching the mobile_user_ids data for all users in the organization,
regardless of whether they were recently active (and thus relevant for
the main presence data set).  And doing so in a sloppy fashion
(sending every user ID over the wire, rather than just having the
database join on Realm).

Fixing this saves a factor of 4-5 on the total runtime of a presence
request on organizations with 10Ks of users like chat.zulip.org; more
like 25% in an organization with 150.  Since large organizations are
very heavily weighted in the overall cost of presence, this is a huge
win.

Fixes part of #13734.
2020-05-14 23:23:37 -07:00
Puneeth Chaganti 734d651b45 home: Fix furthest read time when user has multiple clients.
The query to fetch the latest user activity was missing an
`.order_by('last_visit')`. This meant that the results were being
ordered by the `id`, which resulted in us getting `update_message_flags`
action performed on the client that the user installed last, instead of
being client agnostic and fetching the "global" last
`update_message_flags` action performed by the user.
2020-05-14 13:38:09 -07:00
Hemanth V. Alluri 7e7847b55a webhooks/stripe: Add support for pyr_ events.
"pyr_" events are like refund events some source called it a pseudo
refund event (https://stackoverflow.com/questions/46296374/how-can-
i-get-the-original-charge-and-refund-ids-of-an-automatic-payout).
Though due to the lack of any documentation on this event I'm not
confident if this is the right fix.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-14 12:02:30 -07:00
Hemanth V. Alluri 20b39bd54e webhooks/stripe: Update refund updated event message.
Also add missing test coverage for this event.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-14 12:02:30 -07:00
orientor 9170931da3 openapi: Add test for validating examples.
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.

Part of #14100.
2020-05-12 23:03:06 -07:00
orientor fab2ec9e63 openapi: Rectify wrong examples and schemas.
Some examples mentioned in zulip.yaml did not match their schema.
Change either the schema or the example so that all examples are
valid with respect to their schemas.
2020-05-12 23:03:06 -07:00
Tim Abbott 35139ac559 api: Remove email field from realm_user and realm_bot events.
The `email` field for identifying the user being modified in these
events was not used by either the webapp or other official Zulip
clients.  Instead, it was legacy data from before we switched years
ago to sending user_id fields as the correct way to uniquely identify
a user.
2020-05-12 17:46:13 -07:00
cestrell 297185cc12 integrations: Use new function in Github releases.
Use get_release_event_message from webhooks/git.py to format release
events using the newly implemented release message template.

Tweaked by tabbott to handle name=None.

Builds on #14746. Proposed in #14934.
2020-05-12 17:42:49 -07:00
cestrell e3f0b2f20f integrations: Add Gogs webhook for release event.
Extends Gogs integrations in order to support a published release.
Tested on my local Ubuntu development server running on WSL2.

Fixes #14746.
2020-05-12 17:39:38 -07:00
Tim Abbott f642a0fc89 addressee: Remove obsolete parsing functions.
These became unused in c971576b00.
2020-05-12 17:33:11 -07:00
Tim Abbott 496c08e26c slack import: Fix DefaultStream import of deactivated #random.
If the #random channel in Slack is deactivated, we should follow
Zulip's data model of not allowing deactivated, default streams.

This had apparently happened in zulipchat.com for a few organizations,
resulting in weird exceptions trying to invite new users.
2020-05-12 17:18:57 -07:00
Rohan Prashant 44a56a66d0 tests-backend: Bring stream_recipient.py to 100 percent coverage.
Fixes part of #7089.
2020-05-12 16:43:01 -07:00
clarammdantas 7e9024a39c popovers.js: Add version to user avatar request.
When a user changes its avatar image, the user's avatar in popovers
wasn't being correctly updated, because of browser caching of the
avatar image.  We added a version on the request to get the image in
the same format we use elsewhere, so the browser knows when to use the
cached image or to make a new request to the server.

Edited by Tim to preserve/fix sort orders in some tests, and update
zulip_feature_level.

Fixes: #14290
2020-05-12 11:09:01 -07:00
Steve Howell 9137726e38 bots: Remove `owner` (email) from bot-related payloads.
We remove the `owner` field from `page_params/realm_bots`
and bot-related events.

In the recent commit 155f6da8ba
we added `owner_id`, which we now use everywhere we need
bot owners for.

We also bump the `API_FEATURE_LEVEL` to 5 here.  We
had already documented this in the prior commit to
add `owner_id`.

Note that we don't have to worry about mobile/ZT clients
here--we only deal with bot data in the webapp.
2020-05-12 09:48:37 -07:00
Hemanth V. Alluri cd69da26dc webhooks/github: Ignore the label event. 2020-05-12 09:32:17 -07:00
Hemanth V. Alluri 4dd957faa2 webhooks/github: Add support for pull request ready for review.
GitHub supports opening a draft/WIP pull request and then marking it
as ready for review later on. This PR supports the ready_for_review
action for pull_request events.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-12 09:32:17 -07:00
Anders Kaseorg 4362cceffb portico: Add setting to put Google Analytics on selected portico pages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-11 23:22:50 -07:00
Vishnu KS f1b1bf5a0d billing: Add support for Zulip Standard free trial. 2020-05-11 17:20:54 -07:00
Steve Howell 155f6da8ba bots: Add owner_id to bot-related payloads.
For the below payloads we want `owner_id` instead
of `owner`, which we should deprecate.  (The
`owner` field is actually an email, which is
not a stable key.)

    page_params.realm_bots

    realm_bot/add

    realm_bot/update

IMPORTANT NOTE: Some of the data served in
these payloads is cached with the key
`bot_dicts_in_realm_cache_key`.

For page_params, we get the new field
via `get_owned_bot_dicts`.

For realm_bot/add, we modified
`created_bot_event`.

For realm_bot/update, we modified
`do_change_bot_owner`.

On the JS side, we no longer
look up the bot's owner directly in
`server_events_dispatch` when we get
a realm_bot/update event. Instead, we
delegate that job to `bot_data.js`.
I modified the tests accordingly.
2020-05-11 16:16:58 -07:00
Tim Abbott d3c7579730 gitlab: Fix missing coverage/bug for job hooks corner case.
Apparently, the change and test I added before didn't quite cover the
corner case that was broken.  This does, and exposes a second bug as
well, which we fix.
2020-05-11 16:13:23 -07:00
Tim Abbott d4f875f7c5 webhooks: Add missing gitlab test fixture. 2020-05-11 15:04:34 -07:00
Rohitt Vashishtha 26238426cd message-edit: Process user groups for mentioned user ids.
When editing a message where we mention a usergroup, we would remove
the 'mentioned' flag from messages, resulting in the message being
hidden from your mentions in the UI. This was reported by Greg Price in
https://chat.zulip.org/#narrow/stream/9-issues/topic/missing.20mention.

We add the same code that we use in do_send_messages to calculate the
updated mentions_user_ids. We add some tests alongside other user group
mention tests in test_bugdown.
2020-05-11 14:52:11 -07:00
Tim Abbott 11027161c3 webhooks: Fix handling of GitLab Job Hook events.
This fixes an exception for these events.

Fixture from https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#job-events.
2020-05-11 14:37:01 -07:00
arpit551 f3ab8e66dc ci: add configuration for posting to czo whenever CI fails in master.
Since notify is mostly depreciated  in the latest versions of CircleCI.
Although we can use use notify in CircleCI 2.0 but currently
there is no documentation regarding it.

We could have use notify here rather than this hacky solution but
if we use notify it was not possible to trigger CircleCI webhook only
for the main Zulip repository.

Also corrected the circle ci webhook for the case where we don't receive
previous in post request
2020-05-11 13:45:22 -07:00
Rohitt Vashishtha 9506be0f4f slack-import: Downgrade Slack legacy-token check failure to warning.
Slack has disabled creation of legacy tokens, which means we have to use other
tokens for importing the data. Thus, we shouldn't throw an error if the token
doesn't match the legacy token format.

Since we do not have any other validation for those tokens yet, we log a warning
but still try to continue with the import assuming that the token has the right
scopes.

See https://api.slack.com/changelog/2020-02-legacy-test-token-creation-to-retire.
2020-05-11 13:41:50 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Jenny Ghose 180c16c80e integrations: Add an incoming webhook for Grafana.
Tweaked by tabbott to use formatted suggested in one of the various
duplicate PRs for this issue, showing the rule name clearly.

Fixes #12951.
2020-05-11 00:27:38 -07:00
Tim Abbott a0c2121958 docs: Advertise Slack-compatible webhook a bit.
This should make it discoverable enough that users will try it out and
send us feedback.
2020-05-11 00:07:28 -07:00
Chris Heald c80e913c7a webhooks: Add a webhook capable of parsing Slack payloads.
This adds a webhook that can be used to interpret standard Slack
payloads. Since there are a ton of existing Slack integrations out
there, having a webhook which can accept standard Slack payloads can
significantly ease transition pains. Obviously this can't do everything
that Slack payloads can (particularly WRT their widgets/interactions),
but we can ingest text and parse out multi-block payloads into a message
relatively reasonably.
2020-05-11 00:07:28 -07:00
shubhamgupta2956 9cd8644c7c uploads: Add support for ".jpe" file extension.
Currently when the user uploads files with ".jpe" file extension, the
markdown is converted to link but the image is not embedded.

This commit adds the support for ".jpe" file extension.

Fixes #14863
2020-05-10 22:55:52 -07:00
Pragati Agrawal 60a762704a org settings: Fix limited plan realm can change message_retention_days.
These changes should be included in bd9b74436c,
as it makes sure that Zulip limited plan realm won't be able to change the
`message_retention_days` setting.
2020-05-09 16:40:50 -07:00
Anders Kaseorg 78c70b1424 bugdown: Leave link titles alone until clean_user_content_links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Anders Kaseorg 32f3fd1c77 bugdown: Fix ElementPair typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Anders Kaseorg 6aaeab75bc bugdown: Fix ResultWithFamily typing.
It needs to be a full class because a generic NamedTuple doesn’t work
in Python 3.6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Tim Abbott 524d347c39 i18n: Fix makemessages not walking subdirectories.
Fixes an issue noticed in #14881.
2020-05-08 17:03:16 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Tim Abbott 200209995b test_signup: Update email content for last design tweak. 2020-05-08 15:18:21 -07:00
Puneeth Chaganti ded3b00760 home: Use last update_message_flags activity instead of pointer.
The pointer doesn't get updated when a user is only reading messages in
narrowed views. But, we use the pointer position to determine the
furthest read time, which causes the bankruptcy banner to show up even
for users who have been actively reading and sending messages.

This commit switches to using the time of the last update_message_flags
activity by a user to determine the time of last activity.
2020-05-08 15:06:06 -07:00
Puneeth Chaganti 1adb88cf62 home: Extract code to get the furthest read time. 2020-05-08 15:01:08 -07:00
Puneeth Chaganti a937ea531e test_home: Return message id when sending test message. 2020-05-08 15:01:08 -07:00
Aman Agrawal 898e717b9d register:api: Provide additional fields that can be useful to clients.
Fixes #14166

* Bumped the api version to feature level 4.
See https://github.com/zulip/zulip/issues/14166#issuecomment-598961462
for details about implementation and future steps.
2020-05-08 14:54:54 -07:00
Aman Agrawal 0c2b25cab3 home/page_params: Add suffix `_mib` to clarify the size units.
The docs say "megabytes" or "MB", but client software needs to
know the actual meaning, which is in mebibytes.
2020-05-08 14:54:54 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Pragati Agrawal 12504075ee org settings: Rename plan_includes_wide_organization_logo to more generic.
It's a preliminary step to enable message_retention_setting in org settings
UI, which is a non-limited plan only feature. So we require a page_param
property that tells us the limited-plan state of the Zulip realm.
2020-05-08 14:04:21 -07:00
Tim Abbott a920544bc3
invite emails: Ensure user-controlled input is always in links.
Popular email clients like Gmail will automatically linkify link-like
content present in an HTML email they receive, even if it doesn't have
links in it.  This made it possible to include what in Gmail will be a
user-controlled link in invitation emails that Zulip sends, which a
spammer/phisher could try to take advantage of to send really bad spam
(the limitation of having the rest of the invitation email HTML there
makes it hard to do something compelling here).

We close this opportunity by structuring our emails to always show the
user's name inside an existing link, so that Gmail won't do new
linkification, and add a test to help ensure we don't remove this
structure in a future design change.

Co-authored-by: Anders Kaseorg <andersk@mit.edu>
2020-05-08 14:02:41 -07:00
Hashir Sarwar 6364d27ed5 topic: Remove 7 days restriction for editing & moving topics.
Previously, we had a restriction that we could only
edit and move the topics of 7 days old messages.
This buggy behaviour is now removed as in this
commit.

Fixes #14492.
Part of #13912.
2020-05-08 12:57:50 -07:00
Tim Abbott a702894e0e middleware: Stop using X_REAL_IP.
The comment was wrong, in that REMOTE_ADDR is where the real external
IP was; X_REAL_IP was the loadbalancer's IP.
2020-05-08 11:40:54 -07:00
wowol ffed6b87dd urls: Migrate urls to use modern django pattern.
New path() function changed the way a regex pattern
is created from urls - it adds escape backslashes,
so for testing purposes we need to take care of them
and remove them, to check if urls were tested.

Additionaly, regex patterns from urls can have
[^/]+ instead of [^/]*, so we need to take care
of it too.
2020-05-07 16:28:58 -07:00
Mateusz Mandera 812ac4714f retention: Optimize fetching of realms and streams with retention policy. 2020-05-07 16:28:05 -07:00
Tim Abbott f9a61aa645 i18n: Update translation data from Transifex.
With a fix for a test.
2020-05-07 16:09:16 -07:00
Tim Abbott 73b2fcd618 runtornado: Remove some unnecessary startup logging.
This significantly cleans up the run-dev.py output.
2020-05-07 11:54:58 -07:00
Hemanth V. Alluri fb757e91c1 webhooks/sentry: Add back support for the legacy integrations.
When the integration was originally rewritten, support for the
deprecated webhook payloads was removed. We later noticed that some
people using Zulip were still using versions of Sentry that required
the older integration code.

Thus this commit adds back the older integration code and whenever the
Sentry webhook payload does not have a "data" field (which must be
present in all modern payloads as per the documentation at
https://docs.sentry.io/workflow/integrations/integration-platform/webhooks)
we will use the older Sentry integration code.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-05-07 11:26:19 -07:00
Anders Kaseorg 8411f7f884 version: Update for Zulip Desktop v5.2.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-07 11:25:15 -07:00