zulip/zerver
Zixuan Li e39e04c3ce
migration: Add `extra_data_json` for audit log models.
Note that we use the DjangoJSONEncoder so that we have builtin support
for parsing Decimal and datetime.

During this intermediate state, the migration that creates
extra_data_json field has been run. We prepare for running the backfilling
migration that populates extra_data_json from extra_data.

This change implements double-write, which is important to keep the
state of extra data consistent. For most extra_data usage, this is
handled by the overriden `save` method on `AbstractRealmAuditLog`, where
we either generates extra_data_json using orjson.loads or
ast.literal_eval.

While backfilling ensures that old realm audit log entries have
extra_data_json populated, double-write ensures that any new entries
generated will also have extra_data_json set. So that we can then safely
rename extra_data_json to extra_data while ensuring the non-nullable
invariant.

For completeness, we additionally set RealmAuditLog.NEW_VALUE for
the USER_FULL_NAME_CHANGED event. This cannot be handled with the
overridden `save`.

This addresses: https://github.com/zulip/zulip/pull/23116#discussion_r1040277795

Note that extra_data_json at this point is not used yet. So the test
cases do not need to switch to testing extra_data_json. This is later
done after we rename extra_data_json to extra_data.

Double-write for the remote server audit logs is special, because we only
get the dumped bytes from an external source. Luckily, none of the
payload carries extra_data that is not generated using orjson.dumps for
audit logs of event types in SYNC_BILLING_EVENTS. This can be verified
by looking at:

`git grep -A 6 -E "event_type=.*(USER_CREATED|USER_ACTIVATED|USER_DEACTIVATED|USER_REACTIVATED|USER_ROLE_CHANGED|REALM_DEACTIVATED|REALM_REACTIVATED)"`

Therefore, we just need to populate extra_data_json doing an
orjson.loads call after a None-check.

Co-authored-by: Zixuan James Li <p359101898@gmail.com>
2023-06-07 12:14:43 -07:00
..
actions migration: Add `extra_data_json` for audit log models. 2023-06-07 12:14:43 -07:00
data_import slack import: Convert threads to nicely named Zulip topics. 2023-05-30 16:35:19 -07:00
integration_fixtures/nagios
lib migration: Add `extra_data_json` for audit log models. 2023-06-07 12:14:43 -07:00
management ruff: Fix PLC0208 Use a sequence type when iterating over values. 2023-06-06 14:58:11 -07:00
migrations migration: Add `extra_data_json` for audit log models. 2023-06-07 12:14:43 -07:00
openapi docs: Fix data type for `submessages` field on `message` type. 2023-06-05 16:21:01 -07:00
tests migration: Add `extra_data_json` for audit log models. 2023-06-07 12:14:43 -07:00
tornado tornado_url: Replace occurrences of `uri` with `url`. 2023-04-26 16:37:16 -07:00
views users: Use a less generic response for unauthorized user creation. 2023-06-05 17:33:26 -07:00
webhooks integrations: Refactor Github Integration to use get_sender_name. 2023-06-05 12:01:27 -07:00
worker realm_export: Handle hard head-of-queue failures. 2023-05-16 14:05:01 -07:00
__init__.py
apps.py requirements: Upgrade Python requirements. 2023-04-03 22:39:21 -07:00
context_processors.py emails: Add corporate_enabled to context for emails. 2023-05-01 10:32:43 -07:00
decorator.py Remove statsd support. 2023-04-25 19:58:16 -07:00
filters.py
forms.py accounts: Allow user to change email visibility during first login. 2023-05-16 13:52:56 -07:00
logging_handlers.py python: Use Python 3.8 typing.{Protocol,TypedDict}. 2022-04-27 12:57:49 -07:00
middleware.py middleware: Redirect non-canonical realm domain names. 2023-05-16 15:13:51 -07:00
models.py migration: Add `extra_data_json` for audit log models. 2023-06-07 12:14:43 -07:00
signals.py requirements: Upgrade to Django 4.0. 2022-07-13 16:07:17 -07:00