Commit Graph

40054 Commits

Author SHA1 Message Date
Steve Howell cf6bcfb84a digest emails: Exclude users who had recent digests.
This code protects us in case we ever need to re-run
email digests twice in the same day.
2020-11-16 08:59:28 -08:00
Steve Howell fb3d4c1618 digest tests: Avoid warnings about naive time. 2020-11-16 08:59:28 -08:00
Steve Howell 4271442fba email digests: Write RealmAuditLog rows. 2020-11-16 08:59:28 -08:00
Alex Vandiver 90ca06d873 puppet: Allow unattended upgrades of -updates in addition to -security.
This ensures that software will be fully up-to-date, not just with
security patches.
2020-11-13 16:45:05 -08:00
Mateusz Mandera 4f47f35cb4 auth: Handle the case of invalid subdomain at /fetch_api_key endpoint. 2020-11-13 16:43:17 -08:00
Alex Vandiver 2e20ab1658 puppet: Log the "Host" header and total response time.
Logging `Host` is useful for determining access patterns to realms,
especially if ROOT_DOMAIN_LANDING_PAGE is set.  Total response time is
useful in debugging access and performance patterns.
2020-11-13 16:42:32 -08:00
Anders Kaseorg 1275613812 requirements: Upgrade mypy to 0.790.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 29050918a6 mypy: Exclude broken typeshed stubs for markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 254758733e mypy: Remove unused yamole stub.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 8ba95063d5 test_markdown: Construct FencedBlockPreprocessor with a real Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg e7e1fde6ec fenced_code: Use immutable type for codehilite_conf.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg fbf8ce0305 markdown: Add types for extra Markdown members.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg b48bdc65b9 markdown: Fix AlertWordNotificationProcessor.run type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 9573f6dc00 markdown: Fix build_block_parser type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 4398eecd2b markdown: Use immutable type for extension config.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 060036dfd5 markdown: Merge build_engine into Markdown constructor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 08c64f5cfa markdown: Fix imports for compatibility with typeshed stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg f294688157 test_stripe: Add CallableT cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 2a8a59f548 test_queue_worker: Simplify worker_queue_names computation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
Anders Kaseorg dc84e9696c mypy: Fix types for redis.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
Anders Kaseorg bcb571c187 test-backend: Fix argparse error type.
ArgumentError takes two arguments.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 14:17:22 -08:00
Aman Agrawal 9eb82f31c9 wsl_dev_setup: Warn user about cloning inside windows mounted disk.
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
2020-11-10 00:21:41 -08:00
Nikhil 9c3ff29d35 docs: Added number of selected participants in GSoC 2020. 2020-11-09 20:26:56 -08:00
Anders Kaseorg 3a8cf869db python: Convert os.open(…, O_EXCL) to open(…, "x").
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-09 14:31:01 -08:00
Puneeth Chaganti 358f1f9ba7 webhooks/sentry: Support integration configured as webhook.
Sentry allows adding simple webhooks without going through the process
of creating an Internal Integration in Sentry's Integration
Platform[1] (which our docs recommend).

The payload from sent from such a (simple) webhook integration is
slightly different from the payload sent by an Internal Integration
webhook. This commit tries to wrangle this payload into a form that is
usable by our webhook handler to send a notification message.

[1]: https://sentry.io/integration-platform/
2020-11-09 12:02:49 -08:00
Mateusz Mandera 47228f3a95 actions: Implement do_delete_user.
To have a reasonable way of creating the dummy user without duplicating
code, we need change create_user to have the optional force_id argument.
2020-11-09 11:58:02 -08:00
YashRE42 d9b14b5b74 widgetize: Don't warn for deleted tictactoe widget. 2020-11-09 06:01:39 -05:00
Tim Abbott 909b484d64 docs: Improve headings for changelog. 2020-11-06 16:50:28 -08:00
akshatdalton 806c1a0b8b markdown: Fix flickering of embedded link inside Italic.
This commit fixes a bug in marked.js which caused it to double-escape
HTML when rendering messages of the form: *[text](url)*.

This fixes a bug introduced in
3bdc8bbaa5, where an unnecessary
escape() call was added for the <em> code path, likely just because it
was adjacent to the others that needed it in the file.

Fix this, and add tests to verify that things are still being escaped
once after removing this extra escape.

Fixes #14845.
2020-11-06 10:09:15 -08:00
Steve Howell 5da4332620 minor: Add order-by-id to digest message query.
The order-by-id is now explicit, and I add
comments to explain the select_related tables.
2020-11-06 10:05:46 -08:00
Steve Howell 936171d258 refactor: Extract DigestTopic class.
This gets us away from a lot of dictionary soup.
2020-11-06 10:05:46 -08:00
Steve Howell e8b6c56322 refactor: Simplify get_hot_topics().
The code we deleted here was no longer
doing anything.

Maybe the code was always dead, or maybe it
was written during a time when topics_by_diversity
and topics_by_length actually had different keys.

But now it's clearly cruft.

If we have 4 or more topics, then the code above
it would already have populated the list with 4
elements, and the `if num_convos < 4` condition
would evaluate to False.

And if we had 3 or fewer topics, then we would
have already put all possible topics into our
result, and the `topics_by_diversity[num_convos:4]`
slice would be empty.

It's possible that we should just have a simple
heuristic for topic hotness like `10*num_senders
+ messages`, so we don't have to maintain this
fiddly function, and we can just do something like
`topics_by_score[:4]`.
2020-11-06 10:05:46 -08:00
Vishnu KS bae9dc5f2f billing: Use name selector for overriding the spin button for license input. 2020-11-06 09:59:33 -08:00
Steve Howell c5dc9d386f refactor: Use sets of stream_ids for email digests.
I now use sets for stream_ids in more of the digest
code.

As part of this I replaced exclude_subscription_modified_streams
with streams_recently_modified_for_user.

It's easier for the caller to just ask for ids
to delete from its callee than it is to pass
in a set/list to mutate.

The simpler boundary between the functions makes
the tests easier to write--you can see the
`filtered_streams` logic goes away in this diff.

I also make the tests a bit more thorough by using
combinations of Cordelia/Othello and Verona/Denmark
to try to find multiple possible flaws.

And I make the time intervals longer than 1s to
avoid false negatives from slow CI boxes.
2020-11-05 17:42:43 -08:00
Suyash Vardhan Mathur 6b48fb3c08 streams: Fix live update of stream name when renamed.
Currently, the Stream Name change isn't reflected in the streams
sidebar when a stream is renamed if the order of streams in the
sidebar remains unchanged, because the optimization to avoid
rerendering when nothing changes about the order prevents the
rerendering code from running.

We fix by this adding a flag in build_stream_list and
update_streams_sidebar functions to force a rerender, and pass that
when a stream is renamed.

Fixes #16026.
2020-11-05 11:45:44 -08:00
Tim Abbott 2d847728a3 message_fetch: Fix node tests for new on-load query.
This adds missing unit test updates from
ebea17b9a6.
2020-11-05 11:03:35 -08:00
Tim Abbott ebea17b9a6 message_fetch: Add an extra fetch for Recent Topics.
The comment explains the problem statement in some detail, but
basically this algorithm ensures that the top items in "Recent Topics"
on page load are always the very most recent topics the user has
received messages in (well, ignoring muted topics in this iteration).
2020-11-05 10:38:45 -08:00
Vishnu KS d6743da139 docs test: Verify zulip/zulip file and directory links locally. 2020-11-05 10:07:43 -08:00
Vishnu KS cba292b4bb lint: Ensure that docs don't link directly to code lines. 2020-11-05 10:07:43 -08:00
Vishnu KS f2d18010c4 docs: Don't link directly to the python file line from user doc. 2020-11-05 10:07:43 -08:00
Vishnu KS e94989bc75 docs: Don't link directly to the python file line from integration doc. 2020-11-05 10:07:43 -08:00
Vishnu KS 2a17dd61aa docs: Link session length and idle timeouts to GitHub search.
The line number was outdated and was linking to totally unrelated
section. I think the best way to handle this case would be to
link directly to search.
2020-11-05 10:07:43 -08:00
Steve Howell 88a57ed4ac bulk digest: Get stream subscriptions in bulk.
If we have multiple users, this reduces the amount
of queries we need to do, because we get all
subscriptions for all users in a single query
to Subscription.

For the single-user case, we are introducing an
extra query hop, but the database is doing
roughly the same work, because we are just breaking
up this complex query into two hops:

    messages =
        select ...  from message
        where recipient__type_id in (
            select stream_id from subscription
            where ...
        )

Now it's more like:

    stream_ids =
        select stream_id from subscription
        where ...

    messages =
        select ... from message
        where recipient__type_id in stream_ids
2020-11-05 09:36:59 -08:00
Steve Howell c83db37161 email digests: Introduce bulk methods for digest.
Note that we are not changing anything semantically
or algorithmically yet.  The only overhead here
for the single-user case is boxing and unboxing
data into single-item dicts and lists.

The interfaces for callers in the view and the
queue processor remain the same for now.
2020-11-05 09:36:59 -08:00
Steve Howell 7c89e46731 minor: Clean up some code formatting. 2020-11-05 09:36:59 -08:00
Steve Howell 4bd02eea19 minor: Use user, not user_profile, in some digest code. 2020-11-05 09:36:59 -08:00
Steve Howell 0e2d02b0a2 digest tests: Count cache tries. 2020-11-05 09:36:59 -08:00
Steve Howell 127f4e1291 digest tests: Add more users to bulk digest test. 2020-11-05 09:36:59 -08:00
Steve Howell 89cb3fa841 digest tests: Localize mocks.
We didn't need the enough-traffic mock.

We also continue to prep for testing multiple users.

I also finally remove a comment that is about to
be addressed (and which inaccurately refers to huddles).
2020-11-05 09:36:59 -08:00
Steve Howell 1ec16dd1da digest tests: Prep to test bulk digests.
All this does, essentially, is put the logic
we used to test for othello inside of a loop.

We'll add more users in the next commit.
2020-11-05 09:36:59 -08:00