Commit Graph

40135 Commits

Author SHA1 Message Date
Mateusz Mandera e3be6db73a retention: Eliminate redundant userprofile JOIN from cross-realm query. 2021-01-18 21:40:37 -08:00
Harsh kumar 3cb07bc998
docs: Fix type in remote.md.
spelling errors in remote.md
2021-01-17 21:05:48 -08:00
Tim Abbott 5a02b33f2e digest: Add a large block comment on correctness. 2021-01-17 11:37:59 -08:00
Steve Howell 1040fb7219 email digests: Remove handle_digest_email shim.
The previous commit made it so we only call the
shim in tests, so now we completely remove it.
2021-01-17 11:28:30 -08:00
Steve Howell bfa0bdf3d6 email digests: Process users in chunks of 30.
This should make the queue empty more quickly,
because we do bulk queries to prevent database
hops.
2021-01-17 11:28:30 -08:00
Steve Howell e0b451730a email digests: Extract get_new_streams.
This makes us more efficient when handling
multiple users.  We don't have to keep
sending the same two queries to the database.

Note that as part of this we eliminated
a failure mode for the obscure population
of users from whom both `user.is_guest` and
`user.can_access_public_streams()` returns
False.  We know this would have only affected
Zephyr users (by looking at the code), and
we know we don't actually process Zephyr
users for email digests (or else we would
have raised exceptions in the old code).
2021-01-17 11:28:30 -08:00
Steve Howell 23de94504f email digests: Query streams for messages up front.
This should save us many hops to the database when
we process users in bulk.
2021-01-17 11:28:30 -08:00
Steve Howell 3662bf2dcb minor: Rename stream_map -> user_stream_map. 2021-01-17 11:28:30 -08:00
Steve Howell 11c93aced5 minor: Rename user_profile -> user and avoid shadowing. 2021-01-17 11:28:30 -08:00
Steve Howell f8bbb7fea9 email digests: Use select_related("realm").
We mostly need realm_id, but when we go to build
message lists, we need realm.uri.

We could probably be more aggresive about using
`only` here, but for now I am just trying to
reduce hops to the database.
2021-01-17 11:28:29 -08:00
Steve Howell bb56f0ec0e minor: Move get_stream_map to module level.
This is a pure code move.
2021-01-17 11:28:29 -08:00
Steve Howell 52e2d5a733 email digests: Avoid long_term_idle check.
We want to exclude users with recent subscription
activity from emails, regardless of whether
the long_term_idle flag is set.
2021-01-17 11:28:29 -08:00
Steve Howell 162b372b93 email digests: Do one query for recent streams.
This is another way to limit hops to the database
when we process users in bulk.
2021-01-17 11:28:29 -08:00
rht 16583fd5b4 docs: Say that certbot wildcard has been released. 2021-01-17 11:17:09 -08:00
Alex Vandiver c2526844e9 worker: Remove SignupWorker and friends.
ZULIP_FRIENDS_LIST_ID and MAILCHIMP_API_KEY are not currently used in
production.

This removes the unused 'signups' queue and worker.
2021-01-17 11:16:35 -08:00
Alex Vandiver 01658e39a9 sentry: Verify version is supported, first.
Raven SDK does not send a `title` field.
2021-01-17 11:15:40 -08:00
Alex Vandiver d688e18de2 errors: Remove references to "deployment", use "host".
The `deployment` key was only set in `do_report_error`, which is now
only used in one codepath (the queue worker).  The logging handlers on
staging call notify_server_error directly, which omits the
`deployment` key.

Remove the odd one-of key, and instead simply do dispatch in
`do_report_error`.
2021-01-17 11:08:12 -08:00
Aman Agrawal 9b23e540df message_edit: Stop stream_header_colorblock from overlapping others. 2021-01-17 11:07:32 -08:00
Steve Howell 2382fa7a19 refactor: Pass stream_ids to is_subscriber_subset.
After this change all peer_data functions consistently
use stream_id rather than some "sub" object whose
data type is complicated by all sort of fields that
don't really concern how we track subscribers.
2021-01-17 10:40:17 -08:00
Steve Howell 355f44ef13 refactor: Only pass stream_id for set_subscribers.
The goal here is to make all our peer_data functions
basically work in id space.  Passing a full `sub`
to these functions is a legacy of when subscriber
info was attached to a full stream "sub" object,
but we don't care about anything sub-related
(color, description, name, etc.) when we are
dealing with subscriptions.

When callers pass in stream_id, you can be more
confident in a quick skim of the code that we're
not mutating anything in the "sub".
2021-01-17 10:40:17 -08:00
Steve Howell 6cc880c858 refactor: Extract peer_data.js.
This de-clutters stream_data a bit.  Since our
peer data is our biggest performance concern,
I want to contain any optimizations to a fairly
well-focused module.

The name `peer_data` is a bit of a compromise,
since we already have `subs.js` and we use
`sub` as a variable name for stream records
throughout our code, but it's consistent with
our event nomenclature (peer/add, peer/remove)
and it's short while still being fairly easy
to find with grep.
2021-01-17 10:40:17 -08:00
Steve Howell 5c2f6321f2 node tests: Eliminate set_subscribers mock. 2021-01-17 10:40:17 -08:00
Steve Howell 58dcc70a35 refactor: Remove subscribers from stream_data subs.
This sets us up to use better system-wide data structures
for tracking subscribers.

Basically, instead of storing subscriber data on the
"sub" objects in stream_data.js, we instead have a
parallel data structure called stream_subscribers.

We also have stream_create, stream_edit, and friends
use helper functions rather than accessing
sub.subscribers directly.
2021-01-17 10:40:17 -08:00
Steve Howell a175ce65cf minor: Inline add_sub into create_sub_from_server_data.
We now use add_sub only in tests.

The line to defensively initialize subscribers does
not get copied from add_sub, since we know that
create_sub_from_server_data always initializes
subscribers via set_subscribers.
2021-01-17 10:40:17 -08:00
Mateusz Mandera 3623681d30 message_edit: Don't rely on .recipient_id change not affecting recipient.
The codepath for moving a topic changes the message.recipient_id to the
id of the new recipient, but later, in update_messages_for_topic_edit,
it uses message.recipient when querying for messages with the matching
topic in the *old* stream (because those are the other messages that
need to be moved). This is a bug which happens to work fine, because in
Django 2, if message.recipient gets fetched first and then
message.recipient_id is mutated, message.recipient will not be altered
and thus will retain the outdated, previously fetched value.

In Django 3 changing .recipient_id causes .recipient to be updated to
the new Recipient objects, which is the Recipient of the *new* stream.
That will cause the bug to manifest.

This is a bugfix preparing for the upgrade to Django 3.
2021-01-17 10:39:46 -08:00
Mateusz Mandera f76202dd59 django3: Save language preference in a cookie rather than the session.
Support for saving it in the session is dropped in django3, the cookie
is the mechanism that needs to be used. The relevant i18n code doesn't
have access to the response objects and thus needs to delegate setting
the cookie to LocaleMiddleware.

Fixes the LocaleMiddleware point of #16030.
2021-01-17 10:38:58 -08:00
Steve Howell 04b6108e71 minor: Require keywords for verify_action. 2021-01-17 12:31:04 -05:00
Steve Howell 3df507be73 refactor: Clean up args for fetch_initial_state_data.
We now require explicit keywords for all arguments
to fetch_initial_state_data except user_profile.

We provide reasonable defaults to keep the test
code concise.
2021-01-17 12:31:04 -05:00
Alex Vandiver 08d716c741 registration: Re-use the redirect_to_email_login_url helper.
In the case of reusing a registration link, reuse the
redirect_to_email_login_url helper.  This does have the side effect of
now showing a "you've already registered" note, which did not happen
previously, but that seems probably for the best, since the user did
just click a "register" link.
2021-01-13 11:28:32 -08:00
Alex Vandiver ad3d25103b registration: Pre-fill the email when redirecting to login.
ecfafc05c0 shifted to using a different paramter name to hint that
the user had previously signed up -- and in so doing also stopped
pre-filling the "email" box.  Also send along the email box, to save
users time.
2021-01-13 11:28:32 -08:00
Tushar912 c60f48c889 registration: Move "already in realm" check outside of validation.
Checking for `validate_email_not_already_in_realm` again (after the
form already did so), but only in the case that the form fails to
validate, means that we may be spending time pushing totally invalid
emails to the DB to check.  In the case of emails containing nulls,
this can even trigger a 500 error from PostgreSQL.

Stop calling `validate_email_not_already_in_realm` in the form
validation. The form is currently only used in two places -- in
`accounts_home` and in `maybe_send_to_registration`.  The latter is
only called if the address is known to not currently have an account,
so checking in there is unnecessary; and in the former case, we wish
different behaviour (the redirect) than just validation failure, which
is all the validator can do.

Fixes #17015.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-01-13 11:28:32 -08:00
Tushar912 410bb8ad89 imports: Add better checking for subdomains.
Add a `--allow-reserved-subdomain` flag which allows creation of
reserved keyword domains.  This also always enforces that the domain
is not in use, which was removed in 0258d7d.

Fixes #16924.
2021-01-12 17:54:01 -08:00
rht d514b799ad slack-import: Explain current implementation about threads. 2021-01-12 10:21:32 -08:00
marchantia f03e4b9f04 docs: Fix typo in emoji.md.
Change emoji-map.json to emoji_map.json.
2021-01-12 10:20:35 -08:00
sushant52 6f0e8a9888 auth: Handle the case of invalid subdomain at various points.
Fixes #16770.
2021-01-11 22:29:50 -08:00
Siddharth Asthana 6c888977a6 change_subdomain: Create a deactivated realm on updating subdomain.
When changing the subdomain of a realm, create a deactivated realm with
the old subdomain of the realm, and set its deactivated_redirect to the
new subdomain.
Doing this will help us to do the following:
- When a user visits the old subdomain of a realm, we can tell the user
that the realm has been moved.
- During the registration process, we can assure that the old subdomain
of the realm is not used to create a new realm.

If the subdomain is changed multiple times, the deactivated_redirect
fields of all the deactivated realms are updated to point to the new
uri.
2021-01-07 14:15:22 -08:00
Aman Agrawal 42dfd98607 stream_sidebar_row: Show full stream name on hover.
This will help users see the full stream name for truncated
stream names in the left sidebar more easily.
2021-01-04 18:19:48 -08:00
Aman Agrawal e566e985e4 topic_edit: Store edit history in all the message affected.
Instead of just storing the edit history in the message which
triggered the topic edit, we store the edit history in all
the messages that changed. This helps users track the edit history
of a message more reliably.
2021-01-04 18:18:05 -08:00
cozyrohan 16d1ab3d5f webhooks/github: Fix repeating description for edits and updates.
This change updates the GitHub Integration webhook
get_opened_or_update_pull_request_body method so that
the description is only printed if it actually changes.
If the update event is a result of some other
attribute update, such as an asignee change, then the
description is not included in the message sent to
the zulip stream.

Fixes #16345
2021-01-04 14:34:17 -08:00
Aman Agrawal 6f31a2d49f node_tests: Fix suddenly failing tests for unknown reasons. 2021-01-02 09:32:47 -05:00
Greg Price c7d329544b shared: Bump version to 0.0.4. 2020-12-30 15:02:13 -08:00
Greg Price 0de1d03dd4 typing_status: Mark recipient array as allowing number subtypes.
Concretely, we'll use this with a `UserId` type which is an
"opaque type alias" of `number` -- it's secretly implemented as
simply `number`, and it can be consumed by anything that wants a
`number` (in other words, it's a subtype of `number`), but the
fact that it secretly just is `number` is private to the module
that defines the type.

As far as the typing_status code is concerned, allowing this to
be a subtype of `number` just means that the code doesn't ever
try to inject new numbers of its own into the recipients arrays
that it passes around.
2020-12-30 14:55:24 -08:00
Greg Price 9a15301f4a typing_status: Mark recipient array as read-only for Flow.
This type means that code consuming this value promises not to
mutate it.  It's useful partly for the sake of simply controlling
mutation, so that arrays can be passed around without making
defensive copies; and partly because it makes the type covariant
in the elements, rather than invariant.

That is, if a function takes a plain Array<number | null>, then you
can't pass it an Array<number>, because it might add a `null` to it.
But if it takes $ReadOnlyArray<number | null>, then you can.
In general, Array<S> <: $ReadOnlyArray<S> <: $ReadOnlyArray<T>
for any S <: T, where `<:` means "is a subtype of".

Marking this type as read-only means we can pass in a read-only
array without adding a fixme (equivalent to a mypy type-ignore) to
locally disable the type-checker, nor a redundant defensive copy.
2020-12-30 14:55:24 -08:00
Anders Kaseorg 93101c953f alerts: Prevent icons from spilling out of alerts.
Fixes #16911.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-30 07:55:29 -08:00
Anders Kaseorg 2ba22d273a requirements: Delete unused pip2.in symlink.
It’s unused since commit a486872a8e
(#14278).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-30 07:54:19 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Aman Agrawal 62d721e859 docs: Remove HipChat migration guide.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
2020-12-23 15:43:13 +05:30
Anders Kaseorg df9e40491d ui_init: Unnest an if statement.
This is equivalent because we cannot have delta < 0 && delta > 0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-22 16:44:46 -08:00
Vishnu KS 9fe39646fa analytics: Specify exact end_time in realm summary query.
Fetchings rows with end_time within the last 25 hours would result
in the realmcount queries returning two rows for each realm
if the analytics page was opened within an hour since the
count stats were updated.
2020-12-22 16:44:31 -08:00
Vishnu KS 9d5a1271d4 analytics: Make last_successful_fill handle FillState.STARTED case properly.
Subtracting an hour from end_time is correct only for CountStats with
hourly frequency. For daily frequency we should subtract a day instead.
2020-12-22 16:44:31 -08:00