Commit Graph

17 Commits

Author SHA1 Message Date
Steve Howell 06b0df5efc export: Remove spurious select_related() call for Client. 2016-08-10 14:16:17 -07:00
Steve Howell cb59a11f0a export: Extract get_primary_ids(). 2016-08-10 14:16:17 -07:00
Steve Howell 90e9083b81 export: Extract filter_by_realm(). 2016-08-10 14:16:17 -07:00
Steve Howell 4b6b1b8ad4 export: Extract filter_by_users(). 2016-08-10 14:16:17 -07:00
Steve Howell db9edfce34 export: Use DATE_FIELDS in fix_datetime_fields().
Now we only call this once per table and use DATE_FIELDS to
look up the data fields.
2016-08-10 14:16:17 -07:00
Steve Howell 35c59fc4d7 export: Clean up export_messages().
This is pretty minor cleanup, but it makes it a little more
explicit what we're writing to the shard file, and it allows
us to use a more specific mypy type when calling
floatify_datetime_fields.
2016-08-10 14:16:17 -07:00
Steve Howell 1d1f36c0b8 export: Always use subprocesses to export UserMessage.
We no longer have an in-process code path to export
UserMessage rows.  We want to only maintain the
subprocess code, which we'll always use in production,
and which will work fine in dev.
2016-08-10 14:16:17 -07:00
Steve Howell 78bbefbf94 export: Create import_attachments. 2016-08-10 14:16:17 -07:00
Steve Howell 7ec6a394fe export: Filter Attachment objects by realm. 2016-08-09 16:47:14 -07:00
Steve Howell cecfaa7761 export: Extract import_message_data(). 2016-08-09 16:47:14 -07:00
Steve Howell 5386ed280e export: Extract update_id_map().
We also use a vanilla dictionary instead of a defaultdict, so
that we explicitly initialize what tables are being re-mapped.
2016-08-09 16:47:14 -07:00
Steve Howell 217ef8a4d2 export: Split fix_foreign_keys() into two functions.
We now have convert_to_id_fields for the simple case, and
re_map_foreign_keys for the more complex case. I also
renamed some parameters and variables.
2016-08-09 16:47:14 -07:00
Steve Howell dd88ffccfd export: Extract make_raw() in lib/export.py. 2016-08-09 15:58:27 -07:00
Steve Howell 09fa343bdd export: Use DATE_FIELDS in floatify_datetime_fields.
This avoids a little bit of code duplication, plus it should
make it a little easier to add new date fields in the future.
2016-08-09 15:58:27 -07:00
Steve Howell c14ab3c91f export: Add annotations to zerver/lib/export.py.
I also fixed some small things like removing unnecessary return
statements, and adding a TODO.

In some cases I explicitly cast stuff at run-time to set() or
str() to appease mypy, as well as make it clear to somebody
reading the code that the callee might not respect ordering
or tolerate unicode.
2016-08-09 15:58:27 -07:00
Steve Howell f18cc4ae3a export: Added export_avatars_local_helper(). 2016-08-09 15:58:27 -07:00
Tim Abbott 6264ff7039 Add new Zulip realm import/export tool.
The previous export tool would only work properly for small realms,
and was missing a number of important features:
* Export of avatars and uploads from S3
* Export of presence data, activity data, etc.
* Faithful export/import of timestamps
* Parallel export of messages
* Not OOM killing for large realms

The new tool runs as a pair of documented management commands, and
solves all of those problems.

Also we add a new management command for exporting the data of an
individual user.
2016-08-08 14:58:18 -07:00