Commit Graph

14521 Commits

Author SHA1 Message Date
Steve Howell 8dd3c1038f exports: Rename parent_key to include_rows.
Even though Django usually treats foo__in
and foo_id__in identically for filters where
foo is a ForeignKey type, we want to insist
on somewhat more consistent syntax, because
we have the odd combo of type and type_id
in Recipient, where type_id is kinda like a
foreign key, but not a ForeignKey.

So we assert for now that all our include_rows
values end in "_id__in".
2021-12-08 15:58:07 -08:00
Steve Howell 02207f47d5 minor: Move code blocks to be alphabetical. 2021-12-08 15:58:07 -08:00
Steve Howell aae9f1b6f5 export: Make Config errors more clear. 2021-12-08 15:58:07 -08:00
Nikhil Maske 091772b534 hotspots: Remove intro_reply hotspot.
Zulip shows two guides on How to reply, first one by
the welcome bot and second one is intro_reply hotspot.
To simply and avoid redundancy, intro_reply hotspot is
removed.

Fixes #20482.
2021-12-07 21:55:59 -08:00
Nipunn Koorapati 0ca49bc93a emoji reactions: Order reactions query results by id.
Force postgres to give reactions in ID order - which
is generally chronological order. Results in frontend
displaying reactions in said order.

Fixes #20060.
2021-12-07 15:02:46 -08:00
Eeshan Garg 3714a30e63 stream notifications: Add helper for silent user mention syntax.
In many of our stream notification messages, we make use of the
same silent user mention syntax, the template for which was always
hardcoded. This commit adds a helper function that all relevant
callers can call to get the right syntax when mentioning users.

Thanks to Tim Abbott for this suggestion!
2021-12-07 14:53:50 -08:00
Eeshan Garg 8ebe05f644 streams: Add RealmAuditLog entry for message retention updates. 2021-12-07 14:53:50 -08:00
Eeshan Garg d2901892e2 streams: Add notifications for message retention policy updates.
This is a part of #20289.
2021-12-07 14:53:50 -08:00
S-Abhishek 186d1a83e9 narrow_banner: Move empty narrow messages to handlebar templates.
Removed existing empty narrow divs from app/home.html and created
a new javascript module to dynamically load empty narrow messages
using handlebar template.

Fixes #18797
2021-12-07 13:38:48 -08:00
Steve Howell 6381c2e535 tests: Make sure import doesn't corrupt original realm.
The original intention of this was to prevent coding
errors with realm getters that don't, um, filter
on realm.

Unfortunately, you can still write a broken realm getter
that forgets to filter on realm, but which returns a
Set, and the new safeguards won't see any difference.

We could make all the getters return sorted lists
instead, but that's for another day.

This code does serve another purpose, which is to
prevet egregious bugs in the import itself.
2021-12-07 12:27:01 -08:00
Steve Howell fea659eacd tests: Extract get_getters. 2021-12-07 12:27:01 -08:00
Steve Howell 5803057589 tests: Make some helpers class-level.
This is somewhat tactical in nature. I want to
extract a huge chunk of code that minorly depends
on these helpers.
2021-12-07 12:27:01 -08:00
Steve Howell 29bd1e8bd3 tests: Avoid clutter within long list of getters.
The diff here is ugly, but to summarize:

    BEFORE IMPORT:
        define get_user_id
        define get_huddle_hashes

    AFTER IMPORT AND MAKING GETTERS:
        check realm id
        define assert_realm_values
        verify emoji codes
        check huddle hashes
2021-12-07 12:27:01 -08:00
Steve Howell 93761cd237 tests: Add getter decorator for import test. 2021-12-07 12:27:01 -08:00
Steve Howell 5892748c7b tests: Avoid lambdas in import test. 2021-12-07 12:27:01 -08:00
Steve Howell 54a6c82282 tests: Avoid equal flag for huddle hashes.
There's no need to complexify the codepath
for all the normal use cases.
2021-12-07 12:27:01 -08:00
Steve Howell 6d09eab285 export: Export file images for single users.
We don't have automated test coverage on this yet,
but below are the results from manual testing.

Note that we include the realm icon and logo even
though they were not created by Cordelia.

    ./manage.py export_single_user cordelia@zulip.com

    $ (cd /tmp/zulip-export-4v3mo802/ && find .)
    .
    ./emoji
    ./emoji/2
    ./emoji/2/emoji
    ./emoji/2/emoji/images
    ./emoji/2/emoji/images/3.jpg
    ./emoji/records.json
    ./messages-000001.json
    ./realm_icons
    ./realm_icons/2
    ./realm_icons/2/night_logo.original
    ./realm_icons/2/night_logo.png
    ./realm_icons/2/icon.png
    ./realm_icons/2/icon.original
    ./realm_icons/records.json
    ./avatars
    ./avatars/2
    ./avatars/2/c5125af0447f4d66ce34c1b32eac75ac27ebe0e7.original
    ./avatars/2/c5125af0447f4d66ce34c1b32eac75ac27ebe0e7.png
    ./avatars/records.json
    ./uploads
    ./uploads/2
    ./uploads/2/68
    ./uploads/2/68/xyEkC5dTIp8m42_6HJ3kBfdt
    ./uploads/2/68/xyEkC5dTIp8m42_6HJ3kBfdt/denver.jpg
    ./uploads/2/96
    ./uploads/2/96/ol5WE6RTUntvuPDSpJUrYTim
    ./uploads/2/96/ol5WE6RTUntvuPDSpJUrYTim/denver.jpg
    ./uploads/records.json
    ./user.json
2021-12-07 11:16:52 -08:00
Steve Howell b8d9143318 export: Validate emoji paths.
(We lift the RealmEmoji query to be used by
both local and S3 storage helpers.)
2021-12-07 11:16:52 -08:00
Steve Howell ef6d9b10d2 refactor: Extract get_emoji_path. 2021-12-07 11:16:52 -08:00
Steve Howell 5a41904201 export: Add handle_system_bots flag.
We will set this to False for single-user exports.
2021-12-07 11:16:52 -08:00
Steve Howell 0e19deb558 exports: Limit s3 upload exports with path_id checks. 2021-12-07 11:16:52 -08:00
Steve Howell f6cbf931ae refactor: Pass attachments to export_uploads_from_local.
The next commit will use attachments in the s3 path.
2021-12-07 11:16:52 -08:00
Steve Howell 03f40a64d4 refactor: Pass valid_hashes to export_files_from_s3. 2021-12-07 11:16:52 -08:00
Steve Howell 15bc677f35 export: Pass users to export_avatars_from_local. 2021-12-07 11:16:52 -08:00
Eeshan Garg 79e9ba13e2 billing: Add do_change_remote_server_plan_type.
This is a part of the plumbing we need to support billing for
self-hosted customers.

With documentation changes from tabbott.
2021-12-07 10:25:37 -08:00
Eeshan Garg 2cdaae681d actions: Rename do_change_plan_type -> do change_realm_plan_type.
We will soon be adding an equivalent function for RemoteZulipServer,
so it makes sense to rename this function to be more descriptive.
2021-12-06 16:18:53 -08:00
Steve Howell 42ecabe967 export: Add check_metadata flag. 2021-12-06 15:09:37 -08:00
Steve Howell 0166f13d83 s3 exports: Validate user metadata for all assets.
This preps us to download assets for just a single
user.
2021-12-06 15:09:37 -08:00
Steve Howell 946ab22bba refactor: Lift users query to caller.
This preps us to reuse this code for single users
(after a few more subsequent changes).
2021-12-06 15:09:37 -08:00
Steve Howell b0e5c1d3b9 export: Remove paranoid assertion.
There are tactical reasons to remove this assertion.

Basically, the reason it's safe to remove is that it's
been around a long time and we would have seen this
operationally. Also, the check to make sure that the
S3 filename thingy matches the avatar hash is a much
stronger check.

We will soon restore a stronger version of this check
that applies to all of our asset types (emojis/avatars/etc.).
2021-12-06 15:09:37 -08:00
Steve Howell 59951ae52b refactor: Move metadata checks for s3 export.
This technically broadens the check for user_profile_id,
but we write that metadata on every record.
2021-12-06 15:09:37 -08:00
Steve Howell a27a6a4548 refactor: Inline _check_key_metadata.
This was only called in one place.
2021-12-06 15:09:37 -08:00
Steve Howell db39948be5 refactor: Pass in flavor to export_files_from_s3. 2021-12-06 15:09:37 -08:00
Steve Howell f4354c896b refactor: Pass object_key into export_files_from_s3.
This makes it easier to read the calling code and see
the big picture of how the four asset types are
organized.

I also handle uploads first, to be similar to the local
code.

This code is well tested--you can modify any of the callers
to pass in a wrong value of `object_key` and get a failing
test.
2021-12-06 15:09:37 -08:00
Tim Abbott 8aafce5619 test_signup: Fix test failures with week old test database.
The comment explains in more detail, but basically we'd skip
exercising a bit of code in the signup code path if there were no
messages in the last week, resulting in the query count not matching.
2021-12-06 14:08:37 -08:00
Tim Abbott fee05339a2 actions: Extract ONBOARDING_RECENT_TIMEDELTA. 2021-12-06 14:07:54 -08:00
Steve Howell 4088be6017 import/export: Add UserStatus table.
(We support both realm and single-user exports.)
2021-12-06 13:27:25 -08:00
Steve Howell 45addcd506 cosmetic: Sort DATE_FIELDS in export.py source. 2021-12-06 13:27:25 -08:00
Steve Howell f83907d3bb export: Add MutedUser table.
Note that the import was already implemented, but
its test was flawed.
2021-12-06 13:27:25 -08:00
Steve Howell dffe776287 import tests: Have Cordelia react with a realm emoji.
This exercises some rather unusual database-related
manipulations that happen when creating reactions
from either the app or the import process.
2021-12-06 13:27:25 -08:00
Steve Howell 624260dd70 import: Extract re_map_realm_emoji_codes. 2021-12-06 13:27:24 -08:00
Steve Howell 6ec5a5193a tests: Verify emoji codes. 2021-12-06 13:22:47 -08:00
Palash e1796b0c8f onboarding: Manage occurrence of 'help' command.
"help" command occurs in the command list in
initial pms or when bot doesn't understand the message. It doesn't
occur when the bot is respoding to the "help" command itself.
2021-12-06 11:32:00 -08:00
Tim Abbott 3cd543ee98 models: Improve comment description of emoji_code. 2021-12-06 11:17:15 -08:00
Sahil Batra b68ebf5a22 message: Check wildcard mention restrictions while editing message.
This commit adds code to check whether a user is allowed to use
wildcard mention in a large stream or not while editing a message
based on the realm settings.

Previously this was only checked while sending message, thus user
was easily able to use wildcard mention by first sending a normal
message and then using a wildcard mention by editing it.
2021-12-06 10:22:29 -08:00
Tim Abbott a915e1cc26 test_push_notifications: Sort pm_users as integers, not strings.
Fixes the last commit not working as intended.
2021-12-03 17:15:25 -08:00
Tim Abbott eb3ad83560 test_push_notifications: Sort pm_users list.
The production code sorts this list, so this test would fail
nondeterministically if the database returned these elements in
another order.
2021-12-03 16:56:01 -08:00
Aman Agrawal d2d6e1577f events: Explicitly send empty drafts to spectators.
This is purely for readability reasons, we still send empty
drafts without this change since UserProfile is None.

Fixes #20297
2021-12-03 14:34:21 -08:00
Sahil Batra a34dd11b5b test_events: Rename functions to use dark_theme and light_theme.
This commit renames the tests for changing logo in test_events
to use dark_theme and light_theme instead of night_mode and
day_mode.
2021-12-03 14:25:52 -08:00
Tim Abbott 92d717b8a5 accounts: Improve code documentation for confirmation endpoint. 2021-12-02 09:50:41 -08:00