Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/zerver/tests
History
Zixuan Li a0cf624eaa
migrations: Backfill extra_data_json for audit log entries. 
This migration is reasonably complex because of various anomalies in existing
data.

Note that there are cases when extra_data does not contain data that is
proper json with possibly single quotes. Thus we need to use
"ast.literal_eval" to cover that.

There is also a special case for "event_type == USER_FULL_NAME_CHANGED",
where extra_data is a plain str. This event_type is only used for
RealmAuditLog, so the zilencer migration script does not need to handle
it.

The migration does not handle "event_type == REALM_DISCOUNT_CHANGED"
because ast.literal_eval only allow Python literals. We expect the admin
to populate the jsonified extra_data for extra_data_json manually
beforehand.

This chunks the backfilling migration to reduce potential block time.

The migration for zilencer is mostly similar to the one for zerver; except that
the backfill helper is added in a wrapper and unrelated events are
removed.

**Logging and error recovery**

We print out a warning when the extra_data_json field of an entry
would have been overwritten by a value inconsistent with what we derived
from extra_data. Usually this only happens when the extra_data was
corrupted before this migration. This prevents data loss by backing up
possibly corrupted data in extra_data_json with the keys
"inconsistent_old_extra_data" and "inconsistent_old_extra_data_json".
More roundtrips to the database are needed for inconsistent data, which are
expected to be infrequent.

This also outputs messages when there are audit log entries with decimals,
indicating that such entries are not backfilled. Do note that audit log
entries with decimals are not populated with "inconsistent_old_extra_data_*"
in the JSONField, because they are not overwritten.

For such audit log entries with "extra_data_json" marked as inconsistent,
we skip them in the migration.  Because when we have discovered anomalies in a
previous run, there is no need to overwrite them again nesting the extra keys
we added to it.

**Testing**

We create a migration test case utilizing the property of bulk_create
that it doesn't call our modified save method.

We extend ZulipTestCase to support verifying console output at the test
case level. The implementation is crude but the use case should be rare
enough that we don't need it to be too elaborate.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
 		2023-07-15 09:43:23 -07:00
..
fixtures tests: Eliminate narrow.json fixture. 2023-06-29 12:35:55 -07:00
images
…
__init__.py
…
test_alert_words.py tests: Refactor away result.json() calls with helpers. 2022-06-06 23:06:00 -07:00
test_attachments.py upload: Rename delete_message_image to use word "attachment". 2023-03-02 16:36:19 -08:00
test_audit_log.py user_groups: Audit UserGroup group based setting changes. 2023-07-11 08:56:55 -07:00
test_auth_backends.py tests: Remove assert_streaming_content helper in favor of getvalue. 2023-06-15 16:49:27 -07:00
test_bots.py test_classes: Rename and refactor 'tornado_redirected_to_list'. 2023-04-07 09:45:26 -07:00
test_cache.py ruff: Fix N818 exception name should be named with an Error suffix. 2022-11-17 16:52:00 -08:00
test_compatibility.py ruff: Fix RSE102 Unnecessary parentheses on raised exception. 2023-02-04 16:34:55 -08:00
test_create_video_call.py tests: Refactor away result.json() calls with helpers. 2022-06-06 23:06:00 -07:00
test_custom_profile_data.py settings: Add backend code for using user email_address_visibility setting. 2023-02-10 17:35:49 -08:00
test_data_types.py
…
test_decorators.py logging_util: Remove dependence on get_current_request. 2023-07-11 22:23:47 -07:00
test_digest.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
test_docs.py settings: Change "Display settings" to "Preferences". 2023-07-12 07:09:03 -07:00
test_drafts.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_email_change.py settings: Add backend code for using user email_address_visibility setting. 2023-02-10 17:35:49 -08:00
test_email_log.py emails: Add new onboarding email with guide for organization type. 2023-04-10 08:38:09 -07:00
test_email_mirror.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_email_notifications.py missed-message: Merge before calling handle_missedmessage_emails. 2023-07-13 11:50:42 -07:00
test_embedded_bot_system.py ruff: Fix N818 exception name should be named with an Error suffix. 2022-11-17 16:52:00 -08:00
test_event_queue.py mention: Replace 'wildcards' with 'stream_wildcards'. 2023-07-03 22:03:17 -07:00
test_event_system.py event tests: Make sure legacy narrow is passed along. 2023-06-30 11:26:23 -07:00
test_events.py streams: Send stream creation events when subscribing guests. 2023-07-13 14:04:51 -07:00
test_example.py test-example: Revise comment with number of emails generated. 2023-07-11 14:15:52 -07:00
test_external.py name_restrictions: Reject anything with zulip or kandra in it. 2023-06-23 10:45:40 -07:00
test_github.py ruff: Fix N818 exception name should be named with an Error suffix. 2022-11-17 16:52:00 -08:00
test_gitter_importer.py auth: Remove Realm.AUTHENTICATION_FLAGS class attribute. 2023-04-18 09:22:56 -07:00
test_home.py cache: Eliminate get-stream-by-name cache. 2023-07-11 13:45:40 -07:00
test_hotspots.py
…
test_i18n.py i18n: Update translation data from Transifex. 2023-05-02 13:16:25 -07:00
test_import_export.py user_groups: Populate membership audit logs during realm creation. 2023-07-13 11:55:38 -07:00
test_integrations.py
…
test_integrations_dev_panel.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
test_internet.py ruff: Fix B017 `assertRaises(Exception):` should be considered evil. 2022-11-03 12:10:15 -07:00
test_invite.py signup: Extract set_up_streams_for_new_human_user. 2023-07-10 13:41:28 -07:00
test_legacy_subject.py tests: Consistently JSON-encode ‘to’ parameter 2022-09-13 11:05:37 -07:00
test_link_embed.py ruff: Fix ISC003 Explicitly concatenated string. 2023-01-04 16:25:07 -08:00
test_logging_handlers.py error_notify: Drop any remaining browser-side errors in RabbitMQ queue. 2023-04-13 14:59:58 -07:00
test_management_commands.py templates: Improve email confirmation page. 2023-07-02 16:14:41 -07:00
test_markdown.py mention: Determine @topic mention during message rendering. 2023-07-13 11:34:48 -07:00
test_mattermost_importer.py mattermost: Update references to "private message" and "PM". 2023-06-23 11:24:13 -07:00
test_message_dict.py linkifier: Support URL templates for linkifiers. 2023-04-19 12:20:49 -07:00
test_message_edit.py mention: Replace 'wildcards' with 'stream_wildcards'. 2023-07-03 22:03:17 -07:00
test_message_edit_notifications.py mention: Replace 'wildcards' with 'stream_wildcards'. 2023-07-03 22:03:17 -07:00
test_message_fetch.py narrow: Split out narrow_helpers. 2023-06-30 11:26:23 -07:00
test_message_flags.py message-flags: Rename classes for direct message dicts. 2023-06-23 11:24:13 -07:00
test_message_send.py cache: Eliminate get-stream-by-name cache. 2023-07-11 13:45:40 -07:00
test_message_topics.py ruff: Fix N818 exception name should be named with an Error suffix. 2022-11-17 16:52:00 -08:00
test_messages.py presence: Rewrite the backend data model. 2023-04-26 14:26:47 -07:00
test_middleware.py codespell: Correct “requestor” to “requester”. 2023-06-20 16:17:55 -07:00
test_migrations.py migrations: Backfill extra_data_json for audit log entries. 2023-07-15 09:43:23 -07:00
test_mirror_users.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_muted_users.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_new_users.py create-user: Remove notifications sent to admin realm. 2023-03-13 12:28:26 -07:00
test_notification_data.py mention: Replace 'wildcards' with 'stream_wildcards'. 2023-07-03 22:03:17 -07:00
test_onboarding.py
…
test_openapi.py openapi-tests: Add coverage for validating example events. 2023-06-27 11:27:38 -07:00
test_outgoing_http.py test: Replace occurences of `uri` with `url`. 2023-04-08 16:27:55 -07:00
test_outgoing_webhook_interfaces.py
…
test_outgoing_webhook_system.py settings: Add backend code for using user email_address_visibility setting. 2023-02-10 17:35:49 -08:00
test_populate_db.py populate_db: Import timedelta from its canonical module. 2023-03-05 14:46:28 -08:00
test_presence.py presence: Support null values in UserPresence. 2023-04-26 14:26:47 -07:00
test_push_notifications.py test_notifications: Update tests to cover the corner case properly. 2023-07-13 11:34:48 -07:00
test_queue.py
…
test_queue_worker.py queue_processors: Avoid queue worker timeouts in tests. 2023-06-28 11:06:24 -07:00
test_rate_limiter.py test_helpers: Switch add/remove_ratelimit to a contextmanager. 2023-06-12 12:55:27 -07:00
test_reactions.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_read_receipts.py muted users: Make file naming consistent. 2023-02-10 15:39:57 -08:00
test_realm.py welcome-emails: Use followup_day2 for scheduled email tests. 2023-07-11 14:15:52 -07:00
test_realm_domains.py realm_domains: Allow only owners to add, edit or delete domains. 2022-09-16 15:27:52 -07:00
test_realm_emoji.py docs: Add documentation for delete emoji endpoint. 2023-06-28 10:23:47 -07:00
test_realm_export.py tests: Remove assert_streaming_content helper in favor of getvalue. 2023-06-15 16:49:27 -07:00
test_realm_linkifiers.py linkifier: Support URL templates for linkifiers. 2023-04-19 12:20:49 -07:00
test_realm_playgrounds.py rest: Restrict access to json views via basic auth. 2022-11-04 14:44:07 -07:00
test_redis_utils.py typing: Fix missing attribtute of RedisUtilsTest. 2022-06-23 22:05:12 -07:00
test_report.py zerver: Remove now-unused report/ endpoints. 2023-05-09 13:16:28 -07:00
test_retention.py scheduled_messages: Store the final delivered message ID. 2023-05-09 13:48:28 -07:00
test_rocketchat_importer.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_scheduled_messages.py tests: Use time_machine for testing scheduled message delivery. 2023-07-11 17:34:58 -07:00
test_scim.py codespell: Correct “requestor” to “requester”. 2023-06-20 16:17:55 -07:00
test_send_email.py docs: Document how to use SMTP without authentication. 2023-02-27 11:59:48 -08:00
test_server_settings.py settings: Make SHARED_SECRET mandatory. 2022-08-25 12:13:03 -07:00
test_service_bot_system.py ruff: Fix N818 exception name should be named with an Error suffix. 2022-11-17 16:52:00 -08:00
test_sessions.py black: Reformat with Black 23. 2023-02-02 10:40:13 -08:00
test_settings.py test_helpers: Switch add/remove_ratelimit to a contextmanager. 2023-06-12 12:55:27 -07:00
test_signup.py welcome-emails: Separate followup_day1 email from other welcome emails. 2023-07-11 14:15:52 -07:00
test_slack_importer.py user_groups: Populate membership audit logs during realm creation. 2023-07-13 11:55:38 -07:00
test_slack_message_conversion.py
…
test_soft_deactivation.py topic_mentions: Fetch users to be notified of @topic mentions. 2023-07-13 11:34:48 -07:00
test_subdomains.py subdomains: Extend tests for for same-host when STATIC_URL has domain. 2023-02-14 17:17:06 -05:00
test_submessage.py tests: Improve automated tests for submessages. 2023-07-06 16:35:46 -07:00
test_subs.py streams: Send stream creation events when subscribing guests. 2023-07-13 14:04:51 -07:00
test_templates.py widgets: Rename confusing variable name in `tabbed_instructions.ts`. 2023-06-24 07:47:25 -07:00
test_thumbnail.py test_helpers: Switch add/remove_ratelimit to a contextmanager. 2023-06-12 12:55:27 -07:00
test_timeout.py test_timeout: Skip test_timeout_warn on Python 3.11 for coverage issue. 2023-05-18 11:52:22 -07:00
test_timestamp.py ruff: Fix N818 exception name should be named with an Error suffix. 2022-11-17 16:52:00 -08:00
test_timezone.py ruff: Fix DTZ001 `datetime.datetime()` without `tzinfo` argument. 2023-01-04 16:25:07 -08:00
test_tornado.py test_tornado: Avoid deprecated AsyncHTTPTestCase. 2022-07-05 17:54:17 -07:00
test_transfer.py upload: Rename upload_message_file to use word "attachment". 2023-03-02 16:36:19 -08:00
test_tutorial.py settings: Change "Display settings" to "Preferences". 2023-07-12 07:09:03 -07:00
test_typing.py message-type: Add support for "direct" as value for type parameter. 2023-04-18 12:29:33 -07:00
test_upload.py uploads: Improve file not found message. 2023-07-06 09:32:41 -07:00
test_upload_local.py tests: Remove assert_streaming_content helper in favor of getvalue. 2023-06-15 16:49:27 -07:00
test_upload_s3.py codespell: Correct “requestor” to “requester”. 2023-06-20 16:17:55 -07:00
test_urls.py url tests: Mock out html_to_text. 2023-07-02 16:18:24 -07:00
test_user_groups.py user_groups: Remove "@" from name of role-based system groups. 2023-07-11 13:46:02 -07:00
test_user_status.py test_classes: Rename and refactor 'tornado_redirected_to_list'. 2023-04-07 09:45:26 -07:00
test_user_topics.py test_classes: Rename and refactor 'tornado_redirected_to_list'. 2023-04-07 09:45:26 -07:00
test_users.py user_groups: Populate membership audit logs during realm creation. 2023-07-13 11:55:38 -07:00
test_webhooks_common.py backend-tests: Update "private message" or "PM" to "direct message". 2023-06-23 11:24:13 -07:00
test_widgets.py tests: Consistently JSON-encode ‘to’ parameter 2022-09-13 11:05:37 -07:00
test_zcommand.py tests: Refactor away result.json() calls with helpers. 2022-06-06 23:06:00 -07:00
test_zephyr.py zephyr: Check PERSONAL_ZMIRROR_SERVER before updating cache. 2022-08-12 17:08:04 -07:00