Commit Graph

85 Commits

Author SHA1 Message Date
Tim Abbott acd3a364e1 slack import: Refactor handling of dropped messages.
This is a more coherent ordering, because some messages we skip lack a
"text" attribute.
2018-04-18 11:35:59 -07:00
Tim Abbott e759fd9be4 slack import: Improve error handling for invalid messages. 2018-04-18 11:29:40 -07:00
Tim Abbott 1410a1e460 slack import: Remove unnecessary zerver_realm_skeleton.json.
This was stored as a fixture file under zerver/fixtures, which caused
problems, since we don't show that directory under production (as its
part of the test system).

The simplest emergency fix here would be to just move the file, but
when looking at it, it's clear that we don't need or want a fixture
file here; we want a Python object, so we just do that.

A valuable follow-up improvement to this block would be to create an
actual new Realm object (not saved to the database), and dump it the
same code we use in the export tool; that should handle the vast
majority of these correctly.

Fixes #9123.
2018-04-18 10:33:53 -07:00
Rhea Parekh a2070fb7e5 slack importer: Add comment on size information of avatars.
The size information of an avatar is not required during the import.
Check function 'import_uploads_local' and 'import_uploads_s3'
in 'export.py' for this.
2018-04-16 14:44:57 -07:00
Rhea Parekh f6b6aa1e75 slack import: Implement threading as a management command. 2018-04-15 19:53:02 +05:30
Rhea Parekh 7c0c3930a8 slack importer: Thread avatar downloads. 2018-04-15 19:53:01 +05:30
Rhea Parekh ebc2ee28e9 slack importer: Thread emoji downloads. 2018-04-15 19:52:59 +05:30
Rhea Parekh 8a291d0232 slack importer: Thread attachment downloads.
Use Zulip's run_parallel method to run thread downloads.
2018-04-15 19:51:58 +05:30
Rhea Parekh f7398cbb09 slack import: Implement custom profile fields.
Add custom profile fields in the slack converted
data 'realm' file.
Added tests for the custom profile fields.

Fixes #8928
2018-04-10 13:28:53 -07:00
Rhea Parekh 852e8516b4 slack import: Add custom profile fields.
Build CustomProfileField and CustomProfileFieldValue
for every user and process the field type after getting an
entire list of the custom fields.
2018-04-10 13:28:53 -07:00
rht a183186672 slack importer: User session.get to recycle previous connections. 2018-04-09 22:02:01 -07:00
rht 630adb406b Slack importer: Map Slack channel mentions to Zulip stream mentions. 2018-04-09 10:47:39 -07:00
Tim Abbott c41d7ee300 slack import: Write reasonable multi-line JSON.
This is a lot better for debugging.
2018-04-09 10:45:35 -07:00
Rhea Parekh 2baa9bc16e Import: Add subdomain in the import script.
Also remove user input of subdomain in the slack data
conversion script.
2018-04-06 09:12:56 -07:00
Rhea Parekh f4ad464d82 import script: Fix broken links to attachments.
The comments explain this pretty well, but basically because we
rewrite the realm ID during the import process, we need to edit all
the message bodies that link to an attachment to instead link to the
post-processed URL where that file will be hosted on the new server.

Fixes #8926.
2018-04-04 10:05:15 -07:00
Rhea Parekh 5a9cea4134 import script: re map foreign key of UserProfile.last_active_message_id. 2018-04-04 08:53:09 -07:00
Rhea Parekh 1bba6cc4ce slack importer: Support custom emoji reactions. 2018-04-01 23:24:35 -07:00
Rhea Parekh 2c116e46ce slack importer: Get custom emojis. 2018-04-01 23:24:35 -07:00
Rhea Parekh c650b8fa3e slack importer: Add zerver_realmemoji. 2018-04-01 23:24:35 -07:00
Rhea Parekh b133d175a7 slack importer: Change 'get_user_data' function implementation.
Change 'get_user_data' function to a more general function
to get data from the slack api using legacy tokens.

Also, change the error handling as upon invalid token,
the response is 200, but the response has an error
field in it.

For eg. Go to the following link with invalid token:
https://slack.com/api/emoji.list?token=xoxp-249056023425
2018-04-01 23:24:35 -07:00
Rhea Parekh 220ad6a386 slack importer: Map standard reactions.
As mentioned in
https://get.slack.help/hc/en-us/articles/202931348-Use-emoji-and-emoticons,
slack supports the standard emoji codes
(https://www.webpagefx.com/tools/emoji-cheat-sheet/)
and majority of them are already supported in Zulip.
2018-04-01 23:24:35 -07:00
Rhea Parekh 8a028142d8 slack importer: Remove id allocation function and its implementation.
Remove allocation ID function from slack import script. All the IDs
count will start from 0. Hence the ID List returned
by the allocation function is of no use, and we remove its implementation.
(example: get_total_messages_and_attachments function is of no use anymore,
hence we remove it)
2018-04-01 23:10:55 -07:00
Puneeth Chaganti 4ce8f2aaa2 upload: Rename upload_message_image to upload_message_file.
Tweaked by tabbott to also fix a Slack import comment.
2018-03-30 13:38:31 -07:00
Rhea Parekh d0355f52cb slack importer: Show when files are being downloaded. 2018-03-20 14:42:26 -07:00
Rhea Parekh e7291148e8 slack importer: Add '/me' in content for specific subtypes. 2018-03-20 14:42:26 -07:00
Tim Abbott 99bae7fa5d slack import: Improve algorithm and formatting for attachments.
The previous system would crash with some files (because for some
reason the comment count was 1 but there was no "initial comment") and
also the file comment and file name were sorta redundant.
2018-03-16 11:12:58 -07:00
Rhea Parekh 06071166c7 slack importer: Handle case where attachment download link is not from slack. 2018-03-16 11:12:58 -07:00
Rhea Parekh b7d6608ba6 slack importer: Clean 'create_converted_data_files' function.
The 'make_new_dir' bool value was used to create a new directory
every time True is passed. Now that avatars and uploads directory
are being created seperately, we don't need this anymore.
2018-03-16 11:12:58 -07:00
Rhea Parekh 4b66a2d0dc slack importer: Add function to fetch and save uploads. 2018-03-16 11:12:58 -07:00
Rhea Parekh e62945eb86 slack importer: Implement changes in script due to zerver_attachment. 2018-03-16 11:12:58 -07:00
Rhea Parekh 8e2d930644 slack importer: Implement changes in script due to user upload object. 2018-03-16 11:12:58 -07:00
Rhea Parekh b0851eb20b slack importer: Add helper functions to build attachment object. 2018-03-16 11:12:58 -07:00
Rhea Parekh 68af6e4b7a slack importer: Add helper functions to build user uploads object. 2018-03-16 11:12:58 -07:00
Rhea Parekh 6d85f8c9ef slack importer: Randomize stream colors. 2018-03-15 23:50:32 -07:00
Rhea Parekh 90a3ffc5c0 slack importer: Include only slack's purpose field in description. 2018-03-15 23:50:32 -07:00
Rhea Parekh 8a4f307c43 slack importer: Change topic for imported content. 2018-03-15 23:50:32 -07:00
Tim Abbott ec21997bfe slack import: Fix importing of bot users from Slack. 2018-03-15 18:35:40 -07:00
Rhea Parekh a5b0957e5d slack importer: Set domain name in 'do_convert_data'.
The domain name is being set in the helper function
'slack_workspace_to_realm', but it should be set in the main function
'do_convert_data', as we need it in other child functions of
'do_convert_data'.
2018-03-15 18:34:51 -07:00
Rhea Parekh 0d788823c2 slack importer: realm_id cleanup. 2018-03-07 14:07:24 -08:00
Rhea Parekh 7878cc53a3 slack importer: Cleanup build_subscription. 2018-03-07 14:07:24 -08:00
Rhea Parekh f947194e4c slack importer: Cleanup build_avatar. 2018-03-07 14:07:24 -08:00
Rhea Parekh 5efe05d5b6 slack importer: Cleanup build_zerver_usermessage. 2018-03-07 14:07:24 -08:00
Rhea Parekh 6c4eef16e9 slack importer: Sort messages in the order of their dates.
Sort the list of all messages in the order of their timestamp.
2018-03-06 14:07:09 -08:00
Rhea Parekh c2d4b49bf3 slack importer: Use precomputed value in 'channel_message_to_zerver_message'.
Use the List of all messages in this helper function
instead of using the messages channel-wise.
2018-03-06 14:07:09 -08:00
Rhea Parekh 3e4086d1b8 slack importer: Use precomputed value in 'get_total_messages_and_usermessages'.
Use the List of all messages in this helper function
instead of using the messages channel-wise.
2018-03-06 14:07:09 -08:00
Rhea Parekh 10c73ae577 slack importer: Add function to precompute all the messages.
The messages were first being read and passed to the helper
functions channel wise.
This function makes a list of all the messages in the all the channels
beforehand which would be used to pass in the helper functions.
2018-03-06 14:07:09 -08:00
Rhea Parekh 6a07897d3a slack importer: Fetch and save the avatars.
This gets the avatar of size 512 px and saves it in the
user's avatar directory with both the extensions
'.png' and '.original'.
2018-03-01 16:49:37 -08:00
Rhea Parekh 7076a49e04 slack importer: Pass avatar data through the import script. 2018-03-01 16:41:49 -08:00
Rhea Parekh 30b9d35d5e slack importer: Add helper functions to get user avatars.
Here, we create the slack avatar url using the user data and
build the avatar object. Added Tests for the same.
2018-03-01 16:38:55 -08:00
Rhea Parekh 95df8452be slack importer: Remove function 'get_user_avatar_source'.
slack avatar urls have the format:
'https://ca.slack-edge.com/<team_id>-<user_id>-<avatar_hash>-<size>'
For any url of this form, if the user hasn't uploaded an image,
Slack uses default gravatar, but we don't have a way of knowing if Slack
has used the uploaded image or the custom gravatar
eg: https://ca.slack-edge.com/T5YFFM2QY-U6006P1CN-gd41c3c33cbe-512.
Hence, avatar_source should be mapped to 'U'.
2018-03-01 16:38:55 -08:00