Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Decrease memory usage in bulk_create for UserMessage objects. 

By processing UserMessage objects in batches as we go, this avoids
consuming a large amount of memory that is linear in the size of the
messages log.

(imported from commit 0c42d97f0863da9c079836c60bebcbaeec59f849)
This commit is contained in:
Tim Abbott 2012-12-12 19:30:50 -05:00
parent 9811050a4f
commit 7707690231
1 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
zephyr/management/commands/populate_db.py
View File

@ -483,6 +483,7 @@ def restore_saved_messages():
if first_message_id is None:
first_message_id = min(messages_by_id.keys())
tot_user_messages = 0
pending_subs = {}
current_message_id = first_message_id
for old_message in old_messages:
@ -566,8 +567,13 @@ def restore_saved_messages():
message=message)
user_messages_to_create.append(um)
if len(user_messages_to_create) > 100000:
tot_user_messages += len(user_messages_to_create)
batch_bulk_create(UserMessage, user_messages_to_create)
user_messages_to_create = []
print datetime.datetime.now(), "Importing usermessages, part 2..."
tot_user_messages = len(user_messages_to_create)
tot_user_messages += len(user_messages_to_create)
batch_bulk_create(UserMessage, user_messages_to_create)
print datetime.datetime.now(), "Finalizing subscriptions..."