Commit Graph

79 Commits

Author SHA1 Message Date
Rhea Parekh 26300110ca import: Fix rendered_content in imported messages.
After the messages have been imported, set the rendered_content of the
messages instead of leaving its value to be 'None'.

This is important to ensure that:
(1) Performance for users is good after completing the import.
(2) The database's full-text indexes have all of the imported messages
(which only happens properly when Message rows have their
rendered_content field edited).

Fixes #9168.
2018-08-09 15:12:53 -07:00
Yago González 6a192ac84c utils: Move random API key generator as generate_api_key.
random_api_key, the function we use to generate random tokens for API
keys, has been moved to zerver/lib/utils.py because it's used in more
parts of the codebase (apart from user creation), and having it in
zerver/lib/create_user.py was prone to cyclic dependencies.

The function has also been renamed to generate_api_key to have an
imperative name, that makes clearer what it does.
2018-08-08 16:45:25 -07:00
Tim Abbott 38dd9e49de import_realm: Add comments for update_model_ids.
This basically explains why in these cases we delay doing
bulk_import_model.
2018-07-26 16:13:14 -07:00
Rhea Parekh fca6bc91aa import: Add 'get_db_table' function.
Implement this function in 'bulk_import_model'
and 'update_model_ids'.

This lets us save on redundant-feeling arguments in these
frequently-called helper functions.
2018-07-26 16:07:31 -07:00
Rhea Parekh 8803ac4af8 import: Fix BotStorageData and BotConfigData import.
The function 'update_model_ids' should be used on
the models BotStorageData and BotConfigData.
It is wrongly added here for UserGroup model.

Also the sequence name for BotStorageData and
BotConfigData is 'zerver_botuserstatedata_id_seq' and
'zerver_botuserconfigdata_id_seq' respectively, which
should be specifically mentioned in the function
'allocate_ids'.

This fixes some nondeterministic test failures.
2018-07-23 11:24:17 -07:00
Tim Abbott 839300d781 import: Fix typo in re_map_foreign_keys_many_to_many.
This was introduced when I rebased together the two implementations of
this function.
2018-07-23 10:19:04 -07:00
Rhea Parekh fe299277f3 import: Import BotStorageData and BotConfigData. 2018-07-23 08:21:00 -07:00
Rhea Parekh 2978e025df import: Import UserGroup. 2018-07-23 08:21:00 -07:00
Rhea Parekh bc2307108d import: Import Service. 2018-07-23 08:20:58 -07:00
Rhea Parekh f444e5b628 import: Import MutedTopic. 2018-07-23 08:20:58 -07:00
Rhea Parekh 091d101e7d import: Import UserHotspot. 2018-07-23 08:20:58 -07:00
Rhea Parekh 6f7b7e143f import: Map user IDs for custom profile field of type 'USER'.
The CustomProfileField object which has the `field_type`
`USER` needs to be updated with the new user IDs.
2018-07-23 08:19:04 -07:00
Rhea Parekh 3879e345d9 import: Add function to re-map foreign keys for ManyToMany fields.
This will be used while for any ManyToMany field which
is being imported.
We add an internal function which takes in the old ID list
of the ManyToMany field and return the new updated ID list.
2018-07-23 08:19:04 -07:00
Anders Kaseorg a0293e8a86 zerver/lib/import_realm.py: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Rhea Parekh e9884916c9 import: Support import of huddles.
For importing huddles we have to have unique huddle hashes.
Huddle hashes are extracted from the list of users participating
in a huddle. So to extract these user ids, we first use huddle
id to getting the matching recipient, and then we use subscription
to get the user ids from the recipient id.

Added tests for the same (tests slightly tweaked by tabbott).
2018-07-12 19:06:52 +05:30
Rhea Parekh 4bbccd8287 import: import RealmAuditLog when 'zerver_realmauditlog` is missing.
* If `zerver_realmauditlog` is present in the exported data,
  `RealmAuditLog` would be imported normally.

* If it is not present, `create_subscription_events`
  function in would create the `subscription_created`
  events for RealmAuditLog. The reason this function
  is in `import_realm` module and not in the individual
  export tool scripts (like Slack) is because this
  function would be common for all export tools.

This fixes #9846 for users who have not already done an import of
their organization from Slack.

Fixes #9846.
2018-07-10 16:00:19 +05:30
Rhea Parekh 70b4794816 import: import RealmAuditLog. 2018-07-10 15:53:15 +05:30
Rhea Parekh d1ba6bae03 import: 'processing_emojis' and 'processing_avatars' should now be True together.
Raise an exception when the fields
'processing_emojis' and 'processing_avatars' are
True at the same time. Also add test for the same.
2018-06-18 23:06:09 +05:30
Rhea Parekh 4d21f7f747 import: 'attachment_path' should be saved with the 's3_path' of the record.
For the S3 backend uploads, 'attachment_path' should be
saved with the 's3_path' of the record, as the original
'path' is changed while exporting files from s3. (See
function 'export_files_from_s3' in export.py for reference.)
2018-06-18 23:06:01 +05:30
Rhea Parekh 0730087111 import: Add elif condition for 'processing_emojis' in 'import_uploads_s3'.
'processing_emojis' should have an 'elif' condition here as
we want the function to work for avatars, emojis or uploads
one at a time.
2018-06-18 23:04:18 +05:30
Rhea Parekh f2b5f5a8f9 import: Fix processing_avatars bug in 'import_uploads_s3'.
All the avatars should be processed later on to run the
'ensure_medium_avatar_image' function. This is similar to
'import_uploads_local'.
2018-06-18 22:37:34 +05:30
Rhea Parekh f66ca9a5c3 import: Pass 'processing_emojis' in 'import_uploads_s3'.
'import_uploads_s3' should be passed with the parameter
'processing_emojis' from 'import_uploads'.
2018-06-18 22:35:36 +05:30
Tim Abbott 328136344a import: Fix typo in zerver_customprofilefieldvalue table name.
Apparently, we were doing this slightly wrong.
2018-05-31 10:47:27 -07:00
Rhea Parekh 7198cc3899 import: Fix RealmEmoji import bug.
RealmEmoji should be imported after UserProfile,
as the new user_profile ids are not allocated
if we import it before.
2018-05-27 21:54:20 -07:00
Rhea Parekh 1b7b9a7164 import: Fix reaction import bug.
In 'zerver_reaction', the emoji_code should be updated
with the RealmEmoji allocated id when the 'reaction_type'
is 'realm_emoji'. Hence we add an extra field 'reaction_field'
in 're_map_foreign_keys', to process the above mentioned
condition.
2018-05-27 21:54:20 -07:00
Rhea Parekh c79d7f1070 Import: Move zerver_reaction from 'messages-000001.json' to 'realm.json'.
Also change the existing slack conversion script structure, to
include 'zerver_realm' in 'realm.json'.
2018-05-27 21:54:20 -07:00
Rhea Parekh c24c249b8c export: Support export of Custom Profile Field. 2018-05-23 09:07:26 -07:00
Aditya Bansal a68376e2ba zerver/lib: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Tim Abbott c4b886d8ae import: Split out import.py into its own module.
This should make it a bit easier to find the code.
2018-04-23 15:21:12 -07:00