mirror of https://github.com/zulip/zulip.git
a0cf624eaa
This migration is reasonably complex because of various anomalies in existing data. Note that there are cases when extra_data does not contain data that is proper json with possibly single quotes. Thus we need to use "ast.literal_eval" to cover that. There is also a special case for "event_type == USER_FULL_NAME_CHANGED", where extra_data is a plain str. This event_type is only used for RealmAuditLog, so the zilencer migration script does not need to handle it. The migration does not handle "event_type == REALM_DISCOUNT_CHANGED" because ast.literal_eval only allow Python literals. We expect the admin to populate the jsonified extra_data for extra_data_json manually beforehand. This chunks the backfilling migration to reduce potential block time. The migration for zilencer is mostly similar to the one for zerver; except that the backfill helper is added in a wrapper and unrelated events are removed. **Logging and error recovery** We print out a warning when the extra_data_json field of an entry would have been overwritten by a value inconsistent with what we derived from extra_data. Usually this only happens when the extra_data was corrupted before this migration. This prevents data loss by backing up possibly corrupted data in extra_data_json with the keys "inconsistent_old_extra_data" and "inconsistent_old_extra_data_json". More roundtrips to the database are needed for inconsistent data, which are expected to be infrequent. This also outputs messages when there are audit log entries with decimals, indicating that such entries are not backfilled. Do note that audit log entries with decimals are not populated with "inconsistent_old_extra_data_*" in the JSONField, because they are not overwritten. For such audit log entries with "extra_data_json" marked as inconsistent, we skip them in the migration. Because when we have discovered anomalies in a previous run, there is no need to overwrite them again nesting the extra keys we added to it. **Testing** We create a migration test case utilizing the property of bulk_create that it doesn't call our modified save method. We extend ZulipTestCase to support verifying console output at the test case level. The implementation is crude but the use case should be rare enough that we don't need it to be too elaborate. Signed-off-by: Zixuan James Li <p359101898@gmail.com> |
||
|---|---|---|
| .. | ||
| 0001_initial.py | ||
| 0002_remote_zulip_server.py | ||
| 0003_add_default_for_remotezulipserver_last_updated_field.py | ||
| 0004_remove_deployment_model.py | ||
| 0005_remotepushdevicetoken_fix_uniqueness.py | ||
| 0006_customer.py | ||
| 0007_remotezulipserver_fix_uniqueness.py | ||
| 0008_customer_billing_user.py | ||
| 0009_plan.py | ||
| 0010_billingprocessor.py | ||
| 0011_customer_has_billing_relationship.py | ||
| 0012_coupon.py | ||
| 0013_remove_customer_billing_user.py | ||
| 0014_cleanup_pushdevicetoken.py | ||
| 0015_delete_billing.py | ||
| 0016_remote_counts.py | ||
| 0017_installationcount_indexes.py | ||
| 0018_remoterealmauditlog.py | ||
| 0019_remotezulipserver_plan_type.py | ||
| 0020_remotezulipserverauditlog.py | ||
| 0021_alter_remotezulipserver_uuid.py | ||
| 0022_remotezulipserver_create_audit_log_backfill.py | ||
| 0023_remotezulipserver_deactivated.py | ||
| 0024_remotepushdevicetoken_user_uuid.py | ||
| 0025_alter_remotepushdevicetoken_user_id_drop_index.py | ||
| 0026_auditlog_models_extra_data_json.py | ||
| 0027_backfill_remote_realmauditlog_extradata_to_json_field.py | ||
| __init__.py | ||