Commit Graph

40213 Commits

Author SHA1 Message Date
Steve Howell 1d1e45e9ec digests: Use UserActivityInterval for user activity.
Note that we are much more efficient about finding
active users here:

    - we do one query per realm (instead of per-user)
    - we pass the cutoff date to the database
    - we get back just a list of distinct ids
2020-11-16 08:59:29 -08:00
Steve Howell b52f56080e performance: Just get user_ids to queue digest emails. 2020-11-16 08:59:29 -08:00
Steve Howell e13e5d104d refactor: Only require user_id for inactive_since().
This function is going away completely soon.  It is
querying everybody's entire UserActivity history instead
of passing the cutoff date to the database!
2020-11-16 08:59:29 -08:00
Steve Howell d0260392f7 digests: Get user objects from the database.
The query counts increase here for somewhat
contrived reasons.  The tests before this
commit reflected a successful trip to the
UserProfile cache, but that's not actually
realistic in practice.
2020-11-16 08:59:29 -08:00
Steve Howell 7737413cec digest tests: Improve gather_new_streams test.
We don't need to mock the dates here.  We also
explicitly clear out all streams first, and then
we explicitly test with both the stream being
current and the stream being old.
2020-11-16 08:59:28 -08:00
Steve Howell 9538edde06 digest tests: Simplify bots test.
We can use the _enqueue_emails_for_realm helper
to avoid all the Tuesday-related logic here.

We also don't bother to create UserActivity
records, since the bot gets excluded by virtue
of its being a bot.  (Also, the date ranges
here were sketchy due to the time mocking.)
2020-11-16 08:59:28 -08:00
Steve Howell 0624833af6 digest tests: Improve Tuesday tests.
If we're mocking time, we should do it consistently.
2020-11-16 08:59:28 -08:00
Steve Howell 2f4d7a6171 tests: Fix test_inactive_users_queued_for_digest.
We can avoid all the date mocking now for all
but a couple tests that exercise the is-it-Tuesday
logic.

And this test now correctly tests that we exclude
recently active users.

And this allows us to remove the other test.
2020-11-16 08:59:28 -08:00
Steve Howell e49a482baf email digests: Make transactions atomic. 2020-11-16 08:59:28 -08:00
Steve Howell cf6bcfb84a digest emails: Exclude users who had recent digests.
This code protects us in case we ever need to re-run
email digests twice in the same day.
2020-11-16 08:59:28 -08:00
Steve Howell fb3d4c1618 digest tests: Avoid warnings about naive time. 2020-11-16 08:59:28 -08:00
Steve Howell 4271442fba email digests: Write RealmAuditLog rows. 2020-11-16 08:59:28 -08:00
Alex Vandiver 90ca06d873 puppet: Allow unattended upgrades of -updates in addition to -security.
This ensures that software will be fully up-to-date, not just with
security patches.
2020-11-13 16:45:05 -08:00
Mateusz Mandera 4f47f35cb4 auth: Handle the case of invalid subdomain at /fetch_api_key endpoint. 2020-11-13 16:43:17 -08:00
Alex Vandiver 2e20ab1658 puppet: Log the "Host" header and total response time.
Logging `Host` is useful for determining access patterns to realms,
especially if ROOT_DOMAIN_LANDING_PAGE is set.  Total response time is
useful in debugging access and performance patterns.
2020-11-13 16:42:32 -08:00
Anders Kaseorg 1275613812 requirements: Upgrade mypy to 0.790.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 29050918a6 mypy: Exclude broken typeshed stubs for markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 254758733e mypy: Remove unused yamole stub.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 8ba95063d5 test_markdown: Construct FencedBlockPreprocessor with a real Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg e7e1fde6ec fenced_code: Use immutable type for codehilite_conf.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg fbf8ce0305 markdown: Add types for extra Markdown members.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg b48bdc65b9 markdown: Fix AlertWordNotificationProcessor.run type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 9573f6dc00 markdown: Fix build_block_parser type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 4398eecd2b markdown: Use immutable type for extension config.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 060036dfd5 markdown: Merge build_engine into Markdown constructor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 08c64f5cfa markdown: Fix imports for compatibility with typeshed stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg f294688157 test_stripe: Add CallableT cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 2a8a59f548 test_queue_worker: Simplify worker_queue_names computation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
Anders Kaseorg dc84e9696c mypy: Fix types for redis.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
Anders Kaseorg bcb571c187 test-backend: Fix argparse error type.
ArgumentError takes two arguments.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 14:17:22 -08:00
Aman Agrawal 9eb82f31c9 wsl_dev_setup: Warn user about cloning inside windows mounted disk.
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
2020-11-10 00:21:41 -08:00
Nikhil 9c3ff29d35 docs: Added number of selected participants in GSoC 2020. 2020-11-09 20:26:56 -08:00
Anders Kaseorg 3a8cf869db python: Convert os.open(…, O_EXCL) to open(…, "x").
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-09 14:31:01 -08:00
Puneeth Chaganti 358f1f9ba7 webhooks/sentry: Support integration configured as webhook.
Sentry allows adding simple webhooks without going through the process
of creating an Internal Integration in Sentry's Integration
Platform[1] (which our docs recommend).

The payload from sent from such a (simple) webhook integration is
slightly different from the payload sent by an Internal Integration
webhook. This commit tries to wrangle this payload into a form that is
usable by our webhook handler to send a notification message.

[1]: https://sentry.io/integration-platform/
2020-11-09 12:02:49 -08:00
Mateusz Mandera 47228f3a95 actions: Implement do_delete_user.
To have a reasonable way of creating the dummy user without duplicating
code, we need change create_user to have the optional force_id argument.
2020-11-09 11:58:02 -08:00
YashRE42 d9b14b5b74 widgetize: Don't warn for deleted tictactoe widget. 2020-11-09 06:01:39 -05:00
Tim Abbott 909b484d64 docs: Improve headings for changelog. 2020-11-06 16:50:28 -08:00
akshatdalton 806c1a0b8b markdown: Fix flickering of embedded link inside Italic.
This commit fixes a bug in marked.js which caused it to double-escape
HTML when rendering messages of the form: *[text](url)*.

This fixes a bug introduced in
3bdc8bbaa5, where an unnecessary
escape() call was added for the <em> code path, likely just because it
was adjacent to the others that needed it in the file.

Fix this, and add tests to verify that things are still being escaped
once after removing this extra escape.

Fixes #14845.
2020-11-06 10:09:15 -08:00
Steve Howell 5da4332620 minor: Add order-by-id to digest message query.
The order-by-id is now explicit, and I add
comments to explain the select_related tables.
2020-11-06 10:05:46 -08:00
Steve Howell 936171d258 refactor: Extract DigestTopic class.
This gets us away from a lot of dictionary soup.
2020-11-06 10:05:46 -08:00
Steve Howell e8b6c56322 refactor: Simplify get_hot_topics().
The code we deleted here was no longer
doing anything.

Maybe the code was always dead, or maybe it
was written during a time when topics_by_diversity
and topics_by_length actually had different keys.

But now it's clearly cruft.

If we have 4 or more topics, then the code above
it would already have populated the list with 4
elements, and the `if num_convos < 4` condition
would evaluate to False.

And if we had 3 or fewer topics, then we would
have already put all possible topics into our
result, and the `topics_by_diversity[num_convos:4]`
slice would be empty.

It's possible that we should just have a simple
heuristic for topic hotness like `10*num_senders
+ messages`, so we don't have to maintain this
fiddly function, and we can just do something like
`topics_by_score[:4]`.
2020-11-06 10:05:46 -08:00
Vishnu KS bae9dc5f2f billing: Use name selector for overriding the spin button for license input. 2020-11-06 09:59:33 -08:00
Steve Howell c5dc9d386f refactor: Use sets of stream_ids for email digests.
I now use sets for stream_ids in more of the digest
code.

As part of this I replaced exclude_subscription_modified_streams
with streams_recently_modified_for_user.

It's easier for the caller to just ask for ids
to delete from its callee than it is to pass
in a set/list to mutate.

The simpler boundary between the functions makes
the tests easier to write--you can see the
`filtered_streams` logic goes away in this diff.

I also make the tests a bit more thorough by using
combinations of Cordelia/Othello and Verona/Denmark
to try to find multiple possible flaws.

And I make the time intervals longer than 1s to
avoid false negatives from slow CI boxes.
2020-11-05 17:42:43 -08:00
Suyash Vardhan Mathur 6b48fb3c08 streams: Fix live update of stream name when renamed.
Currently, the Stream Name change isn't reflected in the streams
sidebar when a stream is renamed if the order of streams in the
sidebar remains unchanged, because the optimization to avoid
rerendering when nothing changes about the order prevents the
rerendering code from running.

We fix by this adding a flag in build_stream_list and
update_streams_sidebar functions to force a rerender, and pass that
when a stream is renamed.

Fixes #16026.
2020-11-05 11:45:44 -08:00
Tim Abbott 2d847728a3 message_fetch: Fix node tests for new on-load query.
This adds missing unit test updates from
ebea17b9a6.
2020-11-05 11:03:35 -08:00
Tim Abbott ebea17b9a6 message_fetch: Add an extra fetch for Recent Topics.
The comment explains the problem statement in some detail, but
basically this algorithm ensures that the top items in "Recent Topics"
on page load are always the very most recent topics the user has
received messages in (well, ignoring muted topics in this iteration).
2020-11-05 10:38:45 -08:00
Vishnu KS d6743da139 docs test: Verify zulip/zulip file and directory links locally. 2020-11-05 10:07:43 -08:00
Vishnu KS cba292b4bb lint: Ensure that docs don't link directly to code lines. 2020-11-05 10:07:43 -08:00
Vishnu KS f2d18010c4 docs: Don't link directly to the python file line from user doc. 2020-11-05 10:07:43 -08:00
Vishnu KS e94989bc75 docs: Don't link directly to the python file line from integration doc. 2020-11-05 10:07:43 -08:00