Commit Graph

148 Commits

Author SHA1 Message Date
Tim Abbott 122e11c678 slack import: Fix handling of messages sent by user U00. 2020-01-25 22:47:49 -08:00
Tim Abbott e052ec58db slack import: Improve error messages around invalid tokens.
This updates our error handling of invalid Slack API tokens (and other
networking error handling) to mostly make sense:
* A token that doesn't start with `xoxp-` gives an extended error early.
* An AssertionError for the codebase is correctly declared as such.
* We check for token shape errors before querying the Slack API.

We could still do useful work to raise custom exception classes here.

Thanks to @stavrospat for raising this issue.
2020-01-22 14:48:32 -08:00
Tlazypanda 6945ced76f slack import: Map Slack guest users to Zulip guests.
Slack's Single-User Guest and Multi-User Guest users should be
imported as Zulip guests during data import.

Fixes #13255.
2019-11-12 12:12:59 -08:00
Tim Abbott aad99ce951 mattermost import: Fix handling of channels with no subscribers.
Previously, we skipped setting the list of subscribers to the channel,
which could result in problems if any messages had been posted there
in the past (e.g. because the channel used to have members, but now
doesn't).  It could be correct to skip importing dead channels
altogether, but probably simpler is to just set an empty subscriber list.
2019-11-04 18:10:37 -08:00
Tim Abbott dc682da47a mattermost: Handle replies to private messages.
Previously, our logic to handle Mattermost's "replies" feature didn't
copy the right fields for private messages, where `channel_members` is
included on the message body rather than a `channel` name.
2019-11-04 18:10:37 -08:00
Vishnu KS 1585ad7bf4 mattermost: Add support for exporting DMs and huddles. 2019-10-10 16:37:03 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Vishnu KS a21856c569 mattermost: Rename user_id to sender_user_id in process_raw_message_batch. 2019-09-25 20:06:47 +05:30
Vishnu KS 23d70bb685 mattermost: Rename get_recipient_id to get_recipient_id_from_receiver_name. 2019-09-25 20:06:04 +05:30
Vishnu Ks c4af0b7bc4 mattermost: Support importing messages without team name.
Mattermost doesn't place private messages within a particular team,
which is what this is needed for.
2019-09-18 11:57:37 -07:00
Vishnu Ks bf5f531e90 import_util: Support huddles in SubscriberHandler. 2019-09-18 11:53:13 -07:00
Vishnu KS d434c0ee88 slack: Remove unnecessary comments.
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
Vishnu KS 99d34fd11d slack: Rename default_channels to slack_default_channels. 2019-08-26 14:10:19 -07:00
Vishnu KS b919514f7f slack: Rename customprofilefield_id to custom_profile_field_id. 2019-08-26 14:10:19 -07:00
Vishnu KS c31355f9c1 slack: Rename custom_field_id_count to custom_profile_field_value_id_count. 2019-08-26 14:10:19 -07:00
Vishnu KS 138c659c97 slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
Vishnu KS 9560736d86 slack: Rename slack_user_id_to_custom_profile_fields.
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
Vishnu KS 01a51c8f4e slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 9d51a1b527 slack: Rename added_users to slack_user_id_to_zulip_user_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 3650f19692 slack: Lookup dir_name key in dict instead of in dict_keys.
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00
Vishnu Ks 1e5c49ad82 slack: Support importing shared channels. 2019-08-26 14:10:19 -07:00
Vishnu Ks e09a29f4d3 slack: Refactor get_slack_api_data to accept multiple query params. 2019-08-26 14:10:19 -07:00
Tim Abbott 69505c30a4 mattermost: Handle users who aren't on any team correctly.
It's not clear to me how this is intended to work in Mattermost's
system in that they don't document this behavior, but some users have
`null` as their list of teams, and presumably are not meant to be
included in any team at all.
2019-08-19 16:06:39 -07:00
Anders Kaseorg e417d3a040 slack_message_conversion: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Tim Abbott 9827801569 slack import: Improve readability of user recipient object allocation.
This loop management tweak makes it a bit more obvious what's
happening in this block of code.
2019-07-30 14:46:14 -07:00
Vishnu KS ff3871fc63 slack_import: Clean up return values of channels_to_zerver_stream.
This commits reduces the number of values returned by
channel_to_zerver_stream function by setting the values
directly in realm dict and returning it instead.
2019-07-30 14:46:14 -07:00
Vishnu Ks 6110f495df slack_import: Support importing pms. 2019-07-30 14:46:14 -07:00
Vishnu Ks 5e6d86c8c4 slack_import: Support importing multiparty IMs. 2019-07-09 15:03:28 -07:00
Vishnu Ks 443439d388 slack_import: Support importing private slack channels. 2019-06-28 11:03:32 -07:00
Vishnu Ks 196388cee3 slack_import: Extract processing channels into a seperate function. 2019-06-28 11:00:59 -07:00
Vishnu Ks 55bf44152a import: Handle hidden_by_limit case for files in slack import.
Fixes #12011
2019-05-30 12:01:09 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Vishnu Ks 02c92e55a2 import: Add tool for importing teams from mattermost. 2019-04-05 17:53:03 -07:00
Vishnu Ks f517f72dd2 import: Make use of is_mirror_dummy in build_user_profile. 2019-04-04 13:51:52 -07:00
Vishnu Ks bd4c3b3ebb import: Move make_user_messages to import_util.py. 2019-04-04 13:51:52 -07:00
Vishnu Ks d921fd25e4 import: Move SubscriberHandler to import_util. 2019-03-20 11:29:51 -07:00
Vishnu Ks 6e3720e0b7 gitter: Fix minor comment typo in build_userprofile. 2019-03-20 10:12:18 -07:00
Ben Muschol d526ff00f2 settings: Rename "user avatar" to "profile picture"
This renames references to user avatars, bot avatars, or organization
icons to profile pictures. The string in the UI are updated,
in addition to the help files, comments, and documentation. Actual
variable/function names, changelog entries, routes, and s3 buckets are
left as-is in order to avoid introducing bugs.

Fixes #11824.
2019-03-15 13:29:56 -07:00
Tim Abbott 412d35900f slack import: Fix handling of tombstone files.
Apparently, the mode attribute is not always present.
2019-03-13 14:39:20 -07:00
Tim Abbott 49680a4503 slack import: Skip processing tombstone files.
The tombstone files undocumented feature of Slack's export format
appears sometimes and has no real data, so we just need to skip these.

Fixes #11619.
2019-03-13 12:43:11 -07:00
Tim Abbott cbc62b8e07 streams: Prevent creation of multi-line stream descriptions.
We do not anticipate our UI for showing stream descriptions looking
reasonable for multi-line descriptions, so we should just ban creating
them.

Given the frontend changes, multi-line descriptions are only likely to
show up from importing content from other tools, in which case
replacing newlines with spaces is cleaner than the alternative.
2019-02-20 12:28:00 -08:00
Rishi Gupta e183c316dd help: Rename help/change-your-avatar to help/set-your-avatar. 2019-02-13 17:50:39 -08:00
Anders Kaseorg 56a675d5ec export: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:25:27 -08:00
Hemanth V. Alluri 73d26c8b28 streams: Render and store the stream description from the backend.
This commit does the following three things:
    1. Update stream model to accomodate rendered description.
    2. Render and save the stream rendered description on update.
    3. Render and save stream descriptions on creation.

Further, the stream's rendered description is also sent whenever the
stream's description is being sent.

This is preparatory work for eliminating the use of the
non-authoritative marked.js markdown parser for stream descriptions.
2019-02-01 22:24:18 -08:00
Tim Abbott 9b25f8789f hipchat: Fix handling of user IDs in Stride import.
We've had this code oscillated a few times; the original comparison
was added as part of Stride import but broke HipChat import.
c34a8f2e69 fixed HipChat import but
regressed Stride.

This change fixes this for both HipChat + Stride.
2019-01-31 12:40:05 -08:00
Pragati Agrawal e1772b3b8f tools: Upgrade Pycodestyle and fix new linter errors.
Here, we are upgrading pycodestyle version from 2.4.0 to 2.5.0.

Fixes: #11396.
2019-01-31 12:21:41 -08:00
Matthew Wegner 370cf1a2cb import: Normalize Slackbot String Comparison.
In very old Slack workspaces, slackbot can appear as "Slackbot", and
the import script only checks for "slackbot" (case sensitive).  This
breaks the import--it throws the assert that immediately follows the
test.  I don't know how common this is, but it definitely affected our
import.

The simple fix is to compare against a lowercased-version of the
user's full name.
2019-01-28 14:59:41 -08:00
Tim Abbott 4c603990d2 hipchat: Use HTML2Text for the content.
While the result is by no means perfect, it's significantly cleaner
than what we had before this.
2019-01-09 16:59:45 -08:00
Tim Abbott 53436766c1 hipchat: Improve import of public room subscribers.
Now, if you pass an api_key, we'll initialize the public room
subscribers to be whatever they were at the time the import happened.

Also, document the situation on the caveats section.
2019-01-09 16:50:00 -08:00