Commit Graph

26 Commits

Author SHA1 Message Date
Steve Howell 481488a35e Extract make_subscriber_map().
We extract this function and put it in the shared
library `import_util.py`.

Also, we make it one time higher up in the call
stack, rather than re-building it for every batch
of messages.  I doubt this was super expensive, but
there's no reason to repeatedly execute this.
2018-10-23 17:27:37 -05:00
Steve Howell d1ff903534 refactor: Rename build_user -> build_user_profile.
This makes greps less confusing.
2018-10-23 17:27:37 -05:00
Steve Howell ff61c56f47 hipchat import: Add NotificationMessage support. 2018-10-17 12:11:08 -07:00
Tim Abbott f9b6eeb488 import: Migrate from json to ujson for better perf.
We expect to get better memory performace from
ujson than json.

We also do a better job of closing file handles.

This likely fixes #10377.
2018-10-17 12:11:08 -07:00
Steve Howell 8accc60ca7 import_util: Support multiple message ids for attachments. 2018-10-13 16:47:44 -07:00
Steve Howell 23d7b3d2cc import: De-dup create_converted_data_files helper. 2018-10-13 16:47:41 -07:00
Steve Howell 4b82326376 hipchat import: Support guest users.
We simplify the code for is_realm_admin
and set is_guest as well.

I verified that build_user() is not used
by Slack/Gitter, so the extra argument there
should be fine.

Fixes #10639
2018-10-11 15:28:58 -07:00
Steve Howell 4da664817b hipchat conversion: Add messages. 2018-10-02 16:55:16 -07:00
Steve Howell f296d60dad hipchat conversion: Add emoji support. 2018-10-02 16:55:16 -07:00
Steve Howell 9518b1344a hipchat conversion: Process avatars.
This processes the avatar payloads that we
get in users.json.
2018-10-02 16:55:16 -07:00
Steve Howell c0f15c3860 hipchat conversion: Include deactivated users/streams.
We now include deleted/deactivated data from the old system.
2018-10-02 16:55:16 -07:00
Steve Howell faea26783b Create convert_hipchat_data.
This is a very early version of a tool to convert Hipchat
tar files into data files that can be used by the Zulip
import process.

We include the most fundamental entities--users and
streams.  Customers who don't care about past messages
or customizations could start an instance off of this
and start communicating.

Of course, there are a lot of things missing in the
initial version:

    * messages!
    * file assets -- avatars, emojis, attachments
    * probably lots of other minor things

We currently ignore any incoming dates from Hipchat data
and just use the current time.  This is consistent with
other imports.

We also don't have any docs yet, although the process
will be extremely similar to the "Slack" process:

    https://zulipchat.com/help/import-from-slack

Also, there's a comment at the top of convert_hipchat_data.py
that describes how to test this in dev mode.

I tested this by following the steps in the comment above.
The users just "show up" in /devlogin, so that's nice, and
you can send messages to other users.  To verify the stream
data you have to go into the gear menu and click on "All
Streams", then you can subscribe and send a message.

Production users will need to get new passwords and
re-subscribe to streams.  We will probably auto-subscribe
all users to public streams.
2018-10-02 16:55:16 -07:00
Rhea Parekh 7822ef38c2 import: Change absolute path of downloaded avatars in records.json to relative path. 2018-09-09 09:18:18 -04:00
Rhea Parekh f70b9a3eba import: Move 'build_message' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh a5bc701181 import: Move 'build_stream' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh 66d34b23ef import: Build Attachment with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh 9617b1fbc5 import: Build Recipient and Subscription with model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh c77763bd8e import: Move 'build_realm' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh 18a4904437 import: Move 'build_attachment' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh b6ccc0bc52 import: Move 'build_defaultstream' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh bee3964f14 import: Move 'build_usermessages' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh eefe7cccd2 import: Move 'process_uploads' and 'process_emojis' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh 30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh 87cc1a6280 import: Move 'build_subscription' and 'build_recipient' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh a516f80646 import: Move 'build_avatar' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh 1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00