bdc365d0fe
logging: Pass format arguments to logging.
...
https://docs.python.org/3/howto/logging.html#optimization
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
fead14951c
python: Convert assignment type annotations to Python 3.6 style.
...
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.
We can likely further refine the remaining pieces with some testing.
Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:
- invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+ invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(
-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None
- notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
- signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+ notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+ signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
- author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+ author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
- bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+ bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
- default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
- default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+ default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+ default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}
-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}
-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
e03176b272
help: Add doc for setting profile picture back to gravatar.
2020-04-16 20:27:52 -07:00
c734bbd95d
python: Modernize legacy Python 2 syntax with pyupgrade.
...
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
...
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
e257253e64
emoji_codes: Replace JS module with JSON module.
...
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.
Update the backend to use emoji_codes.json too instead of the three
separate JSON files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
df5345705c
import: Support importing team icon from slack.
2020-02-03 14:09:05 -08:00
122e11c678
slack import: Fix handling of messages sent by user U00.
2020-01-25 22:47:49 -08:00
e052ec58db
slack import: Improve error messages around invalid tokens.
...
This updates our error handling of invalid Slack API tokens (and other
networking error handling) to mostly make sense:
* A token that doesn't start with `xoxp-` gives an extended error early.
* An AssertionError for the codebase is correctly declared as such.
* We check for token shape errors before querying the Slack API.
We could still do useful work to raise custom exception classes here.
Thanks to @stavrospat for raising this issue.
2020-01-22 14:48:32 -08:00
6945ced76f
slack import: Map Slack guest users to Zulip guests.
...
Slack's Single-User Guest and Multi-User Guest users should be
imported as Zulip guests during data import.
Fixes #13255 .
2019-11-12 12:12:59 -08:00
e10361a832
models: Replace is_guest and is_realm_admin with UserProfile.role.
...
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
d434c0ee88
slack: Remove unnecessary comments.
...
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
99d34fd11d
slack: Rename default_channels to slack_default_channels.
2019-08-26 14:10:19 -07:00
b919514f7f
slack: Rename customprofilefield_id to custom_profile_field_id.
2019-08-26 14:10:19 -07:00
c31355f9c1
slack: Rename custom_field_id_count to custom_profile_field_value_id_count.
2019-08-26 14:10:19 -07:00
138c659c97
slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
...
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
9560736d86
slack: Rename slack_user_id_to_custom_profile_fields.
...
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
01a51c8f4e
slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id.
2019-08-26 14:10:19 -07:00
9d51a1b527
slack: Rename added_users to slack_user_id_to_zulip_user_id.
2019-08-26 14:10:19 -07:00
3650f19692
slack: Lookup dir_name key in dict instead of in dict_keys.
...
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00
1e5c49ad82
slack: Support importing shared channels.
2019-08-26 14:10:19 -07:00
e09a29f4d3
slack: Refactor get_slack_api_data to accept multiple query params.
2019-08-26 14:10:19 -07:00
9827801569
slack import: Improve readability of user recipient object allocation.
...
This loop management tweak makes it a bit more obvious what's
happening in this block of code.
2019-07-30 14:46:14 -07:00
ff3871fc63
slack_import: Clean up return values of channels_to_zerver_stream.
...
This commits reduces the number of values returned by
channel_to_zerver_stream function by setting the values
directly in realm dict and returning it instead.
2019-07-30 14:46:14 -07:00
6110f495df
slack_import: Support importing pms.
2019-07-30 14:46:14 -07:00
5e6d86c8c4
slack_import: Support importing multiparty IMs.
2019-07-09 15:03:28 -07:00
443439d388
slack_import: Support importing private slack channels.
2019-06-28 11:03:32 -07:00
196388cee3
slack_import: Extract processing channels into a seperate function.
2019-06-28 11:00:59 -07:00
55bf44152a
import: Handle hidden_by_limit case for files in slack import.
...
Fixes #12011
2019-05-30 12:01:09 -07:00
643bd18b9f
lint: Fix code that evaded our lint checks for string % non-tuple.
...
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
d526ff00f2
settings: Rename "user avatar" to "profile picture"
...
This renames references to user avatars, bot avatars, or organization
icons to profile pictures. The string in the UI are updated,
in addition to the help files, comments, and documentation. Actual
variable/function names, changelog entries, routes, and s3 buckets are
left as-is in order to avoid introducing bugs.
Fixes #11824 .
2019-03-15 13:29:56 -07:00
412d35900f
slack import: Fix handling of tombstone files.
...
Apparently, the mode attribute is not always present.
2019-03-13 14:39:20 -07:00
49680a4503
slack import: Skip processing tombstone files.
...
The tombstone files undocumented feature of Slack's export format
appears sometimes and has no real data, so we just need to skip these.
Fixes #11619 .
2019-03-13 12:43:11 -07:00
e183c316dd
help: Rename help/change-your-avatar to help/set-your-avatar.
2019-02-13 17:50:39 -08:00
56a675d5ec
export: Remove unused imports.
...
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:25:27 -08:00
e1772b3b8f
tools: Upgrade Pycodestyle and fix new linter errors.
...
Here, we are upgrading pycodestyle version from 2.4.0 to 2.5.0.
Fixes : #11396 .
2019-01-31 12:21:41 -08:00
370cf1a2cb
import: Normalize Slackbot String Comparison.
...
In very old Slack workspaces, slackbot can appear as "Slackbot", and
the import script only checks for "slackbot" (case sensitive). This
breaks the import--it throws the assert that immediately follows the
test. I don't know how common this is, but it definitely affected our
import.
The simple fix is to compare against a lowercased-version of the
user's full name.
2019-01-28 14:59:41 -08:00
8a90441d2f
slack import: Import long-inactive users as long-term idle.
...
This avoids creating UserMessage rows for long-inactive users in
organizations with many thousands of users.
2018-12-16 18:52:20 -08:00
a6ca95dfc4
slack import: Fix all messages being imported to one channel.
...
This was an ugly variable-escape-from-loop regression introduced in
e59ff6e6db
2018-12-12 17:54:37 -08:00
d6217eb862
slack import: Fix empty values for custom profile fields.
...
The Slack import process would incorrectly issue
CustomProfileFieldValue entries with a value of "" for users who
didn't have a given CustomProfileField (especially common for the
"skype" and "phone" fields). This had no user-visible effect, but
certainly added some clutter in the database.
2018-12-12 12:58:27 -08:00
e59ff6e6db
slack import: Eliminate need to load all messages into memory.
...
This works by yielding messages sorted based on timestamp. Because
the Slack exports are broken into files by date, it's convenient to do
a 2-layer sorting process, where we open all the files for a given
day, and then sort their messages by timestamp before yielding them.
Fixes #10930 .
2018-12-05 12:20:50 -08:00
d86dd165da
gitter/slack/hipchat: Remove "subject" from conversions.
...
We (lexically) remove "subject" from the conversion code. The
`build_message` helper calls `set_topic_name` under the hood,
so things still have "subject" in the JSON.
There was good code coverage on `build_message`.
2018-11-12 15:47:11 -08:00
8b661f2f03
slack import: Correctly detect the commenting user.
...
Fixes #10772 .
2018-11-06 13:14:23 -08:00
30c493ed24
slack import: Generate message_id/reaction_id with NEXT_ID.
...
This avoids the need to pass tuples of ints around, which
is pretty brittle.
2018-10-29 13:24:50 -07:00
2f58eb1057
slack import: Extract process_message_files().
...
This is mostly an extraction, but it does change the
way we calculate `content`. We append the markdown
links from ALL files to any content that came in the
message itself.
Separating this out also allows us to add more
test coverage for the extracted code.
2018-10-29 13:24:50 -07:00
00f822a26a
conversion: Generate attachment_ids with helpers.
2018-10-29 13:24:50 -07:00
5cb60f7bea
conversions: Use subscriber_map for Slack/Gitter.
...
We now use subscriber_map for building UserMessage
rows in Slack/Gitter conversions.
This is mostly designed to simplify the code, rather
than having to scan the entire subscribers for each
message.
I am guessing this will improve performance for most
conversions. We sort small lists on every message,
in order to be deterministic, but the sorting cost
is probably more than offset by avoiding the O(N)
scans across all subscriptions. Also, it's probably
negligible in the grand scheme of things, compared
to JSON parsing, file I/O, etc.
This commits also fixes some typos with mentioned_users_id ->
mentioned_user_ids and cleans up a test a bit as well.
2018-10-29 13:24:50 -07:00
5194701787
conversions: Use NEXT_ID for usermessage_id.
...
This is mostly complicated due to the way that the
Slack import passes around tuples of ids to maintain
four different parallel sequences.
2018-10-29 13:24:50 -07:00
f9b6eeb488
import: Migrate from json to ujson for better perf.
...
We expect to get better memory performace from
ujson than json.
We also do a better job of closing file handles.
This likely fixes #10377 .
2018-10-17 12:11:08 -07:00
78a15dd715
slack import: Fix obscure email address for Slackbot.
...
Since we know what slackbot is, we don't need to give it a crazy hash
as its email address.
2018-10-16 16:33:41 -07:00
8accc60ca7
import_util: Support multiple message ids for attachments.
2018-10-13 16:47:44 -07:00
23d7b3d2cc
import: De-dup create_converted_data_files helper.
2018-10-13 16:47:41 -07:00
f70b9a3eba
import: Move 'build_message' to import_util.
2018-08-19 22:27:13 -07:00
53e9da8e1f
import: Build CustomProfileField, CustomProfileFieldValue and RealmEmoji with model class.
2018-08-19 22:27:13 -07:00
d98a5925cb
import: Build Reaction with the model class.
2018-08-19 22:27:13 -07:00
a5bc701181
import: Move 'build_stream' to import_util.
2018-08-19 22:27:13 -07:00
c4f8abbd30
import: Build Message with the model class.
2018-08-19 22:27:13 -07:00
4ea7302e14
import: Add missing fields in UserProfile object.
...
The missing fields are checked by `full_clean()` method.
The datetime field errors are ignored as they are fixed
in the `import_realm` script. The field that are
allowed to be null are not included while building
this object.
2018-08-19 22:27:13 -07:00
c77763bd8e
import: Move 'build_realm' to import_util.
2018-08-19 22:27:13 -07:00
8a22838acf
slack import: Fix computation of owner email for uploaded files.
...
The previous code was just always returning the first user in the
organization, due to an incorrect comparison.
2018-08-10 16:20:36 -07:00
3ff339c294
slack import: Add support for uploads in messages through 'files' keyword.
...
It appears that Slack just changed their export format, and how uses
this `files` list for user-uploaded files.
2018-08-10 16:20:36 -07:00
20bca1409f
import: Set emoji records 'last_modified' value in 'import_uploads_s3'.
...
The 'last_modified' value in emoji records is
needed for uploading the file to the S3 backend.
We set the same in the function 'import_uploads_s3'.
We also have to remove the keyword 'last_modified'
while building the RealmEmoji dict, as it is not
a field which exists in RealmEmoji objects.
2018-08-10 16:20:36 -07:00
cf8a0ae819
slack import: Set a last_modified timestamp for custom emoji.
2018-08-10 09:27:43 -07:00
18a4904437
import: Move 'build_attachment' to import_util.
2018-08-07 16:45:42 -07:00
b6ccc0bc52
import: Move 'build_defaultstream' to import_util.
2018-08-07 16:45:42 -07:00
bee3964f14
import: Move 'build_usermessages' to import_util.
2018-08-07 16:45:42 -07:00
eefe7cccd2
import: Move 'process_uploads' and 'process_emojis' to import_util.
2018-08-07 16:45:42 -07:00
30cc7354eb
import: Move 'process_avatars' to import_util.
2018-08-07 16:45:40 -07:00
87cc1a6280
import: Move 'build_subscription' and 'build_recipient' to import_util.
2018-08-07 16:35:56 -07:00
a516f80646
import: Move 'build_avatar' to import_util.
2018-08-07 16:35:56 -07:00
1117455a90
import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util.
2018-08-07 16:35:56 -07:00
b8e1e8b31d
import: Add slack import files in zerver/data_import directory.
2018-08-01 11:52:14 -07:00
Anders Kaseorg
Anders Kaseorg
Siddharth Varshney
Stefan Weil
Vishnu KS
Tim Abbott
Tlazypanda
Rishi Gupta
Ben Muschol
Anders Kaseorg
Pragati Agrawal
Matthew Wegner
rht
Steve Howell
Rhea Parekh