zulip/zerver/management/commands/create_large_indexes.py

97 lines
3.2 KiB
Python
Raw Normal View History

from typing import Any
Add create_large_indexes management command. This management command creates the same indexes as migrations 82, 83, and 95, which are all indexes on the huge UserMessage table. (*) This command quickly no-ops with clear messaging when the indexes already exist, so it's idempotent in that regard. (If somebody somehow creates an index by the same name incorrectly, they can always drop it in dbshell and re-run this command.) If any of the migrations have not been run, which we detect simply by the existence of the indexes, then we create them using a `CREATE INDEX CONCURRENTLY` command. This functionality in postgres allows you to create indexes against large tables without disrupting queries against those tables. The tradeoff here is that creating indexes concurrently takes significantly longer than doing them non-concurrently. Since most tables are small, we typically just use regular Django migrations and run them during a brief interval while the app is down. For indexes on big tables, we will want to run this command as part of the upgrade process, and we will want to run it while the app is still up, otherwise it's pointless. All the code in create_indexes() is literally copy/pasted from the relevant migrations, and that scheme should work going forward. (It uses a different implementation of create_index_if_not_exist than the migrations use, but the code is identical lexically in the function.) If we ever do major restructuring of our large tables, such as UserMessage, and we end up droppping some of these indexes, then we will need to make this command migrations-aware. For now it's safe to assume that indexes are generally additive in nature, and the sooner we create them during the upgrade process, the better. (*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
from django.db import connection
from zerver.lib.management import ZulipBaseCommand
def create_indexes() -> None:
Add create_large_indexes management command. This management command creates the same indexes as migrations 82, 83, and 95, which are all indexes on the huge UserMessage table. (*) This command quickly no-ops with clear messaging when the indexes already exist, so it's idempotent in that regard. (If somebody somehow creates an index by the same name incorrectly, they can always drop it in dbshell and re-run this command.) If any of the migrations have not been run, which we detect simply by the existence of the indexes, then we create them using a `CREATE INDEX CONCURRENTLY` command. This functionality in postgres allows you to create indexes against large tables without disrupting queries against those tables. The tradeoff here is that creating indexes concurrently takes significantly longer than doing them non-concurrently. Since most tables are small, we typically just use regular Django migrations and run them during a brief interval while the app is down. For indexes on big tables, we will want to run this command as part of the upgrade process, and we will want to run it while the app is still up, otherwise it's pointless. All the code in create_indexes() is literally copy/pasted from the relevant migrations, and that scheme should work going forward. (It uses a different implementation of create_index_if_not_exist than the migrations use, but the code is identical lexically in the function.) If we ever do major restructuring of our large tables, such as UserMessage, and we end up droppping some of these indexes, then we will need to make this command migrations-aware. For now it's safe to assume that indexes are generally additive in nature, and the sooner we create them during the upgrade process, the better. (*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
# Creating concurrent indexes is kind of a pain with current versions
# of Django/PostgreSQL, because you will get this error with seemingly
Add create_large_indexes management command. This management command creates the same indexes as migrations 82, 83, and 95, which are all indexes on the huge UserMessage table. (*) This command quickly no-ops with clear messaging when the indexes already exist, so it's idempotent in that regard. (If somebody somehow creates an index by the same name incorrectly, they can always drop it in dbshell and re-run this command.) If any of the migrations have not been run, which we detect simply by the existence of the indexes, then we create them using a `CREATE INDEX CONCURRENTLY` command. This functionality in postgres allows you to create indexes against large tables without disrupting queries against those tables. The tradeoff here is that creating indexes concurrently takes significantly longer than doing them non-concurrently. Since most tables are small, we typically just use regular Django migrations and run them during a brief interval while the app is down. For indexes on big tables, we will want to run this command as part of the upgrade process, and we will want to run it while the app is still up, otherwise it's pointless. All the code in create_indexes() is literally copy/pasted from the relevant migrations, and that scheme should work going forward. (It uses a different implementation of create_index_if_not_exist than the migrations use, but the code is identical lexically in the function.) If we ever do major restructuring of our large tables, such as UserMessage, and we end up droppping some of these indexes, then we will need to make this command migrations-aware. For now it's safe to assume that indexes are generally additive in nature, and the sooner we create them during the upgrade process, the better. (*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
# reasonable code:
#
# CREATE INDEX CONCURRENTLY cannot be executed from a function or multi-command string
#
# For a lot more detail on this process, refer to the commit message
# that added this file to the repo.
with connection.cursor() as cursor:
# copied from 0082
print("Creating index zerver_usermessage_starred_message_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_starred_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 2) != 0;
"""
)
# copied from 0083
print("Creating index zerver_usermessage_mentioned_message_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_mentioned_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 8) != 0;
"""
)
# copied from 0095
print("Creating index zerver_usermessage_unread_message_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_unread_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 1) = 0;
"""
)
# copied from 0098
print("Creating index zerver_usermessage_has_alert_word_message_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_has_alert_word_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 512) != 0;
"""
)
# copied from 0099
print("Creating index zerver_usermessage_wildcard_mentioned_message_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_wildcard_mentioned_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 8) != 0 OR (flags & 16) != 0;
"""
)
# copied from 0177
print("Creating index zerver_usermessage_is_private_message_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_is_private_message_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 2048) != 0;
"""
)
# copied from 0180
print("Creating index zerver_usermessage_active_mobile_push_notification_id.")
cursor.execute(
"""
CREATE INDEX IF NOT EXISTS zerver_usermessage_active_mobile_push_notification_id
ON zerver_usermessage (user_profile_id, message_id)
WHERE (flags & 4096) != 0;
"""
)
print("Finished.")
Add create_large_indexes management command. This management command creates the same indexes as migrations 82, 83, and 95, which are all indexes on the huge UserMessage table. (*) This command quickly no-ops with clear messaging when the indexes already exist, so it's idempotent in that regard. (If somebody somehow creates an index by the same name incorrectly, they can always drop it in dbshell and re-run this command.) If any of the migrations have not been run, which we detect simply by the existence of the indexes, then we create them using a `CREATE INDEX CONCURRENTLY` command. This functionality in postgres allows you to create indexes against large tables without disrupting queries against those tables. The tradeoff here is that creating indexes concurrently takes significantly longer than doing them non-concurrently. Since most tables are small, we typically just use regular Django migrations and run them during a brief interval while the app is down. For indexes on big tables, we will want to run this command as part of the upgrade process, and we will want to run it while the app is still up, otherwise it's pointless. All the code in create_indexes() is literally copy/pasted from the relevant migrations, and that scheme should work going forward. (It uses a different implementation of create_index_if_not_exist than the migrations use, but the code is identical lexically in the function.) If we ever do major restructuring of our large tables, such as UserMessage, and we end up droppping some of these indexes, then we will need to make this command migrations-aware. For now it's safe to assume that indexes are generally additive in nature, and the sooner we create them during the upgrade process, the better. (*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
class Command(ZulipBaseCommand):
help = """Create concurrent indexes for large tables."""
def handle(self, *args: Any, **options: str) -> None:
Add create_large_indexes management command. This management command creates the same indexes as migrations 82, 83, and 95, which are all indexes on the huge UserMessage table. (*) This command quickly no-ops with clear messaging when the indexes already exist, so it's idempotent in that regard. (If somebody somehow creates an index by the same name incorrectly, they can always drop it in dbshell and re-run this command.) If any of the migrations have not been run, which we detect simply by the existence of the indexes, then we create them using a `CREATE INDEX CONCURRENTLY` command. This functionality in postgres allows you to create indexes against large tables without disrupting queries against those tables. The tradeoff here is that creating indexes concurrently takes significantly longer than doing them non-concurrently. Since most tables are small, we typically just use regular Django migrations and run them during a brief interval while the app is down. For indexes on big tables, we will want to run this command as part of the upgrade process, and we will want to run it while the app is still up, otherwise it's pointless. All the code in create_indexes() is literally copy/pasted from the relevant migrations, and that scheme should work going forward. (It uses a different implementation of create_index_if_not_exist than the migrations use, but the code is identical lexically in the function.) If we ever do major restructuring of our large tables, such as UserMessage, and we end up droppping some of these indexes, then we will need to make this command migrations-aware. For now it's safe to assume that indexes are generally additive in nature, and the sooner we create them during the upgrade process, the better. (*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
create_indexes()