Commit Graph

5167 Commits

Author SHA1 Message Date
Anders Kaseorg aaa7b766d8 python: Use universal_newlines to get str from subprocess.
We can replace ‘universal_newlines’ with ‘text’ when we bump our
minimum Python version to 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 7c4f68d9cf python: Skip unnecessary decode before BeautifulSoup parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 86e8d81c7f python: Skip unnecessary decode before JSON parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 1802a50cc9 python: Use requests.Response.text instead of decoding content.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
sahil839 b29d39195c streams: Do not allow default streams to be private.
We now do not allow to make a stream private which is already
a default stream.
2020-10-29 15:47:32 -07:00
sahil839 557ca0802c streams: Do not allow private streams to be set as default.
We now do not allow to set a private stream as default.
2020-10-29 15:43:37 -07:00
m-e-l-u-h-a-n cbfd6464a5 logging: replace mock.patch() for logging with assertLogs()
This commit removes mock.patch with assertLogs().

* Adds return value to do_rest_call() in outgoing_webhook.py, to
  support asserting log output in test_outgoing_webhook_system.py.

* Logs are not asserted in test_realm.py because it would require to users
  to be queried using users=User.objects.filter(realm=realm) and the order
  of resulting queryset varies for each run.

* In test_decorators.py, replacement of mock.patch is not done because
  I'm not sure if it's worth the effort to replace it as it's a return
  value of a function.

Tweaked by tabbott to set proper mypy types.
2020-10-29 15:37:45 -07:00
Hemanth V. Alluri 99cf37dc51 drafts: Make the ID of the draft a part of the draft dict.
Then because the ID is now part of the draft dict, we can
(and do) change the structure of the "drafts" parameter
returned from `GET /drafts` from an object (mapping ID to
data) to an array.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
Hemanth V. Alluri 8d59fd2f45 tests/drafts: Simplify create_and_check_drafts_for_success.
Sometimes we don't need to specify the expected_drafts field.
So by removing it, we can reduce the clutter a bit.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
Hemanth V. Alluri e60925b3e8 drafts: Change "timestamp" from float to integer.
Now the timestamp returned in a draft dict will always be an int.
The endpoints will still accept either an int or a float.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-10-29 11:06:04 -07:00
m-e-l-u-h-a-n be7a70e742 logging: Remove unnecessary mock.patch() for logging.
Our test-backend validation confirms that we don't log anything to
stdout in the tests, so the fact that CI passes with this removes
shows there was nothing being logged.
2020-10-28 23:15:27 -07:00
Vishnu KS fdea49742c apps: Use GitHub API for generating the web app download link. 2020-10-28 23:04:14 -07:00
Alex Vandiver f4eae83542 export: Only include real, active humans in the displayed count. 2020-10-28 18:31:06 -07:00
Anders Kaseorg 1352f2f233 python: Replace manual quote_plus usage with urlencode.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 4e9d587535 python: Pass query parameters as a dict when making GET requests.
This provides automatic URL-encoding.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 41f509170b users: Canonicalize the timezone identifier.
While working on shifting toward native browser time zone APIs
(#16451), it was found that all but very recent Chrome and Node
versions reject certain legacy timezone aliases like US/Pacific
(https://crbug.com/364374).

For now, we only canonicalize the timezone property returned in user
objects and not the timezone setting itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Anders Kaseorg 0b288f92c9 timezone: Remove get_timezone wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:42:54 -07:00
Tim Abbott 6d7cd351a3 events: Optimize creating streams for new users.
During the new user creation code path, there can be no existing
active clients for the user being created, so we can skip the code to
send events to that user's clients.

The tests here reflect that we need to send fewer events, and do fewer
queries that would have been spent computing data for these..

Fixes #16503, combined with the long series of recent changes by Steve
Howell to fix super-linear behavior in this code path.
2020-10-26 12:47:15 -07:00
Steve Howell 88a7a1b002 events: Optimize peer_add/peer_remove for public streams.
We no bulk up peer_add/peer_remove events by user if the
same user has subscribed to multiple streams (and just
that single user).

This mostly optimizes the new-user codepath, but the
algorithm is a bit more general in nature.
2020-10-26 12:33:28 -07:00
Alex Vandiver 7cf737988d queue: Be more explicit about test/real queue division. 2020-10-26 12:32:47 -07:00
Anders Kaseorg 31d0141a30 python: Close opened files.
Fixes various instances of ‘ResourceWarning: unclosed file’ with
python -Wd.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-26 12:31:30 -07:00
Steve Howell 3ad1335a97 tests: Clear ContentType cache for user test.
This keeps the number of queries predictable.
2020-10-26 07:18:08 -04:00
Steve Howell 5ef01b3ad8 tests: Fix test_create_user_with_multiple_streams.
This test was flaky due to some date-related
non-determinism.  I make all the Message objects
current to make add_new_user_history reliably
try to bulk-update UserMessage rows to read.
2020-10-26 07:18:08 -04:00
Harsh Srivastava 9b31df009b openapi: Fix excessively large test_events failure output.
Because of the very large `oneOf` clause of the formats of events
possible in Zulip's `GET /events` system, we had issues with
`test-backend` failures for missing documentation for a new event
format being like 1000 lines of output, which was very much unhelpful.

Fix this by limiting the output use only the oneOf variants that are
broadly similar to the actual payload received.

Fixes #16023.
2020-10-23 17:00:17 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg b9fd49a2c6 mypy: Correct mistaken *args type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:29:13 -07:00
Anders Kaseorg d295da676b test_message_fetch: Clean up obsolete PGroonga bug workaround.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-22 23:27:23 -07:00
sahil839 571bb62e3d events: Update subscriber list on peer_add for unsubscribed streams.
We update the subscriber list on peer_add event for unsubscribed
streams as well.
2020-10-22 15:12:32 -07:00
sahil839 733d26aef2 events: Update subscriber list on peer_remove for never subscribed stream.
We now update the subscriber list on peer_remove event for never
subscribed streams also.
2020-10-22 15:12:32 -07:00
sahil839 af9b153ee3 events: Update subscriber list on peer_remove for unsubscribed stream.
We update the subscriber list on peer_remove event for unsubscribed
streams also.
2020-10-22 15:12:32 -07:00
sahil839 709edd29d4 test_events: Fix comment in do_test_subscribe_events.
The comment still pointed to 'vacate' event flow, but
we have removed the vacate event in a9356508ca.
This commit fixes the comment to depict the correct
purpose of below lines, i.e. to test the remove
event flow.
2020-10-22 15:12:32 -07:00
sahil839 e578742b02 test_events: Remove 'realm_user' from event_types in subscription test.
We were including 'realm_user' in event_types along with 'subscription',
but we don't send event of type 'realm_user' when subscribing to a new
stream. This was added in 1c332f5d6a.

This commit removes 'realm_user' from event_types.
2020-10-22 15:12:32 -07:00
sahil839 d0f5537fb2 actions: Modify check_message for handling wildcard_mention_policy setting.
This commit adds enforcement for sending messages containing wildcard
mentions according to wildcard_mention_policy.
2020-10-22 14:46:32 -07:00
sahil839 25f32d461e tests: Add tests for all the values of wildcard_mention_policy. 2020-10-22 12:08:22 -07:00
Mateusz Mandera 48f80fcb0a auth: Expect name in request params in Apple auth.
The name used to be included in the id_token, but this seems to have
been changed by Apple and now it's sent in the `user` request param.

https://github.com/python-social-auth/social-core/pull/483 is the
upstream PR for this - but upstream is currently unmaintained, so we
have to monkey patch.

We also alter the tests to reflect this situation. Tests no longer put
the name in the id_token, but rather in the `user` request param in the
browser flow, just like it happens in reality.

An adaptation has to be made in the native flow - since the name won't
be included by Apple in the id_token anymore, the app, when POSTing
to the /complete/apple/ endpoint,
can (and should for better user experience)
add the `user` param formatted as json of
{"email": "hamlet@zulip.com", "name": {"firstName": "Full", "lastName": "Name"}}
dict. This is also reflected by the change in the
native flow tests.
2020-10-22 12:07:46 -07:00
Steve Howell 7ff3859136 subscriber events: Change schema for peer_add/peer_remove.
We now can send an implied matrix of user/stream tuples
for peer_add and peer_remove events.

The client code basically does this:

    for stream_id in event['stream_ids']:
        for user_id in event['user_ids']:
            update_sub(stream_id, user_id)

We used to send individual events, which gets real
expensive when you are creating new streams. For
the case of copy-to-stream case, we should see
events go from U to 1, where U is the number of users
added.

Note that we don't yet fully optimize the potential
of this schema.  For adding a new user with lots
of default streams, we still send S peer_add events.

And if you subscribe a bunch of users to a bunch of
private streams, we only go from U * S to S; we can't
optimize it down to one event easily.
2020-10-22 11:19:53 -07:00
Steve Howell 85ed6f332a performance: Avoid Recipient lookup for stream messages.
All the fields of a stream's recipient object can
be inferred from the Stream, so we just make a local
object.  Django will create a Message object without
checking that the child Recipient object has been
saved.  If that behavior changes in some upgrade,
we should see some pretty obvious symptom, including
query counts changing.

Tweaked by tabbott to add a longer explanatory comment, and delete a
useless old comment.
2020-10-20 11:47:23 -07:00
Steve Howell 7bbcc2ac96 refactor: Compute peers for public streams later.
This saves us a query for edge cases like when
you try to unsubscribe from a public stream
that you have already unsubscribed from.

But this is mostly to prep for upcoming
optimizations.
2020-10-20 11:31:22 -07:00
akshatdalton 287c4ed2bb markdown: Fix Youtube and Vimeo preview overriding markdown link titles bug.
Initially markdown titles were overridden by Youtube and Vimeo preview titles.
But now it will check if any markdown title is present to replace Youtube or
Vimeo preview titles, if preview of linked websites is enabled.
Fixes #16100
2020-10-19 12:06:13 -07:00
Anders Kaseorg d81a93cdf3 requirements: Upgrade markdown to 3.3.1.
Upstream has slightly changed the whitespace around stashes.  Take
this opportunity to clean up the extra blank lines we were outputting.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Steve Howell 4dce34ab8b refactor: Simplify call to bulk_get_subscriber_user_ids.
The way we were computing the dictionary was very
convoluted--all we need is a set of subscribed user
ids.
2020-10-18 14:27:31 -07:00
Steve Howell 0ca07ffd3c peformance: Eliminate StreamRecipientMap.
That class is an artifact of when Stream
didn't have recipient_id.  Now it's simpler
to deal with stream subscriptions.

We also save a query during page load (and
other places where we get subscriber
info).
2020-10-18 14:27:31 -07:00
Steve Howell 2f8ba383ef tests: Test overhead for creating new users. 2020-10-18 14:27:31 -07:00
Mateusz Mandera 716df658fa queue_processors: Don't run test queues with run-dev.py. 2020-10-18 14:07:31 -07:00
Steve Howell e1bcf6124f refactor: Remove recipient from access_stream_by_name. 2020-10-16 12:58:11 -07:00
Steve Howell a51b483f1a performance: Remove recipient from access_stream_by_id.
The Recipient table is now kind of useless for
stream-related operations, since we have
recipient_id on Stream now.
2020-10-16 12:58:11 -07:00
Steve Howell 3685fcc701 refactor: Remove recipient arg for do_mute_topic. 2020-10-16 12:58:11 -07:00
Steve Howell 378062cc83 performance: Avoid call to access_stream_by_id.
We already trust ids that are put on our queue
for deferred work. For example, see the code for
"mark_stream_messages_as_read_for_everyone"

We now pass stream_recipient_id when we queue
up work for do_mark_stream_messages_as_read.

This generally saves about 3 queries per
user when we unsubscribe them from a stream.
2020-10-16 12:58:11 -07:00
Steve Howell 2256d72015 minor: Add comment to subscriber test. 2020-10-16 12:58:11 -07:00
Steve Howell 31eb97ddde performance: Fix do_mark_stream_messages_as_read.
This function no longer asks for data that it
doesn't need.
2020-10-16 12:58:11 -07:00