Commit Graph

129 Commits

Author SHA1 Message Date
Steve Howell 30c493ed24 slack import: Generate message_id/reaction_id with NEXT_ID.
This avoids the need to pass tuples of ints around, which
is pretty brittle.
2018-10-29 13:24:50 -07:00
Steve Howell 2f58eb1057 slack import: Extract process_message_files().
This is mostly an extraction, but it does change the
way we calculate `content`.  We append the markdown
links from ALL files to any content that came in the
message itself.

Separating this out also allows us to add more
test coverage for the extracted code.
2018-10-29 13:24:50 -07:00
Steve Howell 00f822a26a conversion: Generate attachment_ids with helpers. 2018-10-29 13:24:50 -07:00
Steve Howell 5cb60f7bea conversions: Use subscriber_map for Slack/Gitter.
We now use subscriber_map for building UserMessage
rows in Slack/Gitter conversions.

This is mostly designed to simplify the code, rather
than having to scan the entire subscribers for each
message.

I am guessing this will improve performance for most
conversions.  We sort small lists on every message,
in order to be deterministic, but the sorting cost
is probably more than offset by avoiding the O(N)
scans across all subscriptions.  Also, it's probably
negligible in the grand scheme of things, compared
to JSON parsing, file I/O, etc.

This commits also fixes some typos with mentioned_users_id ->
mentioned_user_ids and cleans up a test a bit as well.
2018-10-29 13:24:50 -07:00
Steve Howell 5194701787 conversions: Use NEXT_ID for usermessage_id.
This is mostly complicated due to the way that the
Slack import passes around tuples of ids to maintain
four different parallel sequences.
2018-10-29 13:24:50 -07:00
Tim Abbott f9b6eeb488 import: Migrate from json to ujson for better perf.
We expect to get better memory performace from
ujson than json.

We also do a better job of closing file handles.

This likely fixes #10377.
2018-10-17 12:11:08 -07:00
Tim Abbott 78a15dd715 slack import: Fix obscure email address for Slackbot.
Since we know what slackbot is, we don't need to give it a crazy hash
as its email address.
2018-10-16 16:33:41 -07:00
Steve Howell 8accc60ca7 import_util: Support multiple message ids for attachments. 2018-10-13 16:47:44 -07:00
Steve Howell 23d7b3d2cc import: De-dup create_converted_data_files helper. 2018-10-13 16:47:41 -07:00
Rhea Parekh f70b9a3eba import: Move 'build_message' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh 53e9da8e1f import: Build CustomProfileField, CustomProfileFieldValue and RealmEmoji with model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh d98a5925cb import: Build Reaction with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh a5bc701181 import: Move 'build_stream' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh c4f8abbd30 import: Build Message with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh 4ea7302e14 import: Add missing fields in UserProfile object.
The missing fields are checked by `full_clean()` method.
The datetime field errors are ignored as they are fixed
in the `import_realm` script. The field that are
allowed to be null are not included while building
this object.
2018-08-19 22:27:13 -07:00
Rhea Parekh c77763bd8e import: Move 'build_realm' to import_util. 2018-08-19 22:27:13 -07:00
Tim Abbott 8a22838acf slack import: Fix computation of owner email for uploaded files.
The previous code was just always returning the first user in the
organization, due to an incorrect comparison.
2018-08-10 16:20:36 -07:00
Rhea Parekh 3ff339c294 slack import: Add support for uploads in messages through 'files' keyword.
It appears that Slack just changed their export format, and how uses
this `files` list for user-uploaded files.
2018-08-10 16:20:36 -07:00
Rhea Parekh 20bca1409f import: Set emoji records 'last_modified' value in 'import_uploads_s3'.
The 'last_modified' value in emoji records is
needed for uploading the file to the S3 backend.
We set the same in the function 'import_uploads_s3'.

We also have to remove the keyword 'last_modified'
while building the RealmEmoji dict, as it is not
a field which exists in RealmEmoji objects.
2018-08-10 16:20:36 -07:00
Tim Abbott cf8a0ae819 slack import: Set a last_modified timestamp for custom emoji. 2018-08-10 09:27:43 -07:00
Rhea Parekh 18a4904437 import: Move 'build_attachment' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh b6ccc0bc52 import: Move 'build_defaultstream' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh bee3964f14 import: Move 'build_usermessages' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh eefe7cccd2 import: Move 'process_uploads' and 'process_emojis' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh 30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh 87cc1a6280 import: Move 'build_subscription' and 'build_recipient' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh a516f80646 import: Move 'build_avatar' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh 1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh b8e1e8b31d import: Add slack import files in zerver/data_import directory. 2018-08-01 11:52:14 -07:00