2019-02-02 23:53:29 +01:00
|
|
|
from typing import Any
|
Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)
This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)
If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command. This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables. The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.
Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.
For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.
All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward. (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)
If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware. For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.
(*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
|
|
|
|
|
|
|
from django.db import connection
|
|
|
|
|
|
|
|
from zerver.lib.management import ZulipBaseCommand
|
|
|
|
|
2020-01-14 21:59:46 +01:00
|
|
|
|
2019-06-26 08:53:36 +02:00
|
|
|
def create_indexes() -> None:
|
Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)
This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)
If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command. This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables. The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.
Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.
For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.
All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward. (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)
If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware. For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.
(*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
|
|
|
# Creating concurrent indexes is kind of a pain with current versions
|
2020-10-26 22:27:53 +01:00
|
|
|
# of Django/PostgreSQL, because you will get this error with seemingly
|
Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)
This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)
If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command. This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables. The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.
Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.
For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.
All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward. (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)
If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware. For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.
(*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
|
|
|
# reasonable code:
|
|
|
|
#
|
|
|
|
# CREATE INDEX CONCURRENTLY cannot be executed from a function or multi-command string
|
|
|
|
#
|
|
|
|
# For a lot more detail on this process, refer to the commit message
|
|
|
|
# that added this file to the repo.
|
|
|
|
|
|
|
|
with connection.cursor() as cursor:
|
2019-06-26 08:53:36 +02:00
|
|
|
# copied from 0082
|
|
|
|
print("Creating index zerver_usermessage_starred_message_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_starred_message_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 2) != 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
# copied from 0083
|
|
|
|
print("Creating index zerver_usermessage_mentioned_message_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_mentioned_message_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 8) != 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
# copied from 0095
|
|
|
|
print("Creating index zerver_usermessage_unread_message_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_unread_message_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 1) = 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
# copied from 0098
|
|
|
|
print("Creating index zerver_usermessage_has_alert_word_message_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_has_alert_word_message_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 512) != 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
# copied from 0099
|
|
|
|
print("Creating index zerver_usermessage_wildcard_mentioned_message_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_wildcard_mentioned_message_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 8) != 0 OR (flags & 16) != 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
# copied from 0177
|
|
|
|
print("Creating index zerver_usermessage_is_private_message_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_is_private_message_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 2048) != 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
# copied from 0180
|
|
|
|
print("Creating index zerver_usermessage_active_mobile_push_notification_id.")
|
2021-02-12 08:19:30 +01:00
|
|
|
cursor.execute(
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2019-06-26 08:53:36 +02:00
|
|
|
CREATE INDEX IF NOT EXISTS zerver_usermessage_active_mobile_push_notification_id
|
|
|
|
ON zerver_usermessage (user_profile_id, message_id)
|
|
|
|
WHERE (flags & 4096) != 0;
|
2021-02-12 08:20:45 +01:00
|
|
|
"""
|
2021-02-12 08:19:30 +01:00
|
|
|
)
|
2019-06-26 08:53:36 +02:00
|
|
|
|
|
|
|
print("Finished.")
|
2018-08-02 01:06:14 +02:00
|
|
|
|
2021-02-12 08:19:30 +01:00
|
|
|
|
Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)
This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)
If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command. This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables. The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.
Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.
For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.
All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward. (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)
If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware. For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.
(*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
|
|
|
class Command(ZulipBaseCommand):
|
|
|
|
help = """Create concurrent indexes for large tables."""
|
|
|
|
|
2017-10-26 11:35:57 +02:00
|
|
|
def handle(self, *args: Any, **options: str) -> None:
|
Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)
This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)
If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command. This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables. The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.
Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.
For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.
All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward. (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)
If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware. For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.
(*) UserMessage is huge for large installations, of course.
2017-08-16 17:42:25 +02:00
|
|
|
create_indexes()
|