Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update schema changes content and formatting. 

(imported from commit 46f3d43bfe62883d490ea870b2d29f7a45397c08)
This commit is contained in:
Yoyo Zhou 2015-08-19 17:05:00 -07:00
parent 4a6e132a44
commit f78abd2dcc
1 changed files with 84 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

164
docs/schema-changes.rst
View File

@ -1,5 +1,5 @@
============== ==============
Schema Changes Schema changes
============== ==============
If you are making a change that requires a database schema upgrade, If you are making a change that requires a database schema upgrade,
@ -8,68 +8,65 @@ there are a few extra things you need to keep in mind.
Using South for migrations Using South for migrations
-------------------------- --------------------------
#. Discuss the change informally with other devs. 1. Discuss the change informally with your team.
#. Edit zerver/models.py for your particular class. #. Edit ``zerver/models.py`` for your particular class.
#. See notes below about keep\_default. * See notes below about keep\_default.
#. Run ``./manage.py schemamigration zerver --auto`` #. Run ``./manage.py schemamigration zerver --auto``
#. This will create the ``000#_***.py`` schema migration file in * This will create the ``000#_***.py`` schema migration file in
``zerver/migrations``. ``zerver/migrations``.
#. Read Notes section below, as you may need to edit the migration. A #. Read `Notes and Cautions`_ section below, as you may need to edit
common step here is setting keep\_default to True. the migration. A common step here is setting keep\_default to True.
#. Do "git add" with your new migration. #. Do ``git add`` with your new migration.
#. Run ./manage.py migrate zerver. #. Run ``./manage.py migrate zerver``.
#. Write supporting code or otherwise validate the DB change locally. #. Write supporting code or otherwise validate the DB change locally.
#. Email devel@ about your proposed changes to get formal sign off. TODO: Advice on testing schema changes?
#. Commit your changes #. Commit your changes:
#. The migration must be in the same commit as the models.py changes. a. The migration must be in the same commit as the models.py changes.
#. Include [schema] in the commit message. #. Include [schema] in the commit message.
#. Include [manual] in the commit message if additional steps are #. Include [manual] in the commit message if additional steps are
required. required.
#. Before deploying your code fix, read the notes on "Deploying to #. Before deploying your code fix, read the notes on `Deploying to
staging". staging`_.
Deploying to staging Deploying to staging
-------------------- --------------------
Always follow this process. Always follow this process.
(1) Schedule the migration for after hours. 1. Schedule the migration for after hours.
(2) For long-running migrations, double check that you use appropriate #. For long-running migrations, double check that you use appropriate
library helpers in migrate.py to ensure that changes happen in small library helpers in ``migrate.py`` to ensure that changes happen in small
batches that are committed frequently. batches that are committed frequently.
(3) Announce that you are doing the migration to the team, to avoid #. Announce that you are doing the migration to your team, to avoid
simultaneous migrations and other outcomes of poor communication. simultaneous migrations and other outcomes of poor communication.
(4) Do any administrative steps, such as increasing #. Do any administrative steps, such as increasing
checkpoint\_segments. checkpoint\_segments.
(5) Apply the migration in advance from staging, using commands similar #. Apply the migration in advance from staging, using commands similar
to the following, where "[your commit]" is the commit that has your to the following, where ``[your commit]`` is the commit that has your
migration: migration::
:: cd ~/zulip
git fetch
cd /tmp
git clone ~/zulip
cd zulip
git checkout [your commit]
./manage.py migrate zerver
cd /tmp
rm -Rf zulip
ssh zulip@staging.zulip.net #. Undo any temporary administrative changes, such as increasing
cd zulip checkpoint\_segments.
git fetch
cd /tmp
git clone ~/zulip
cd zulip
git checkout [your commit]
./manage.py migrate zerver
cd /tmp
rm -Rf zulip
(6) Undo any temporary administrative changes, such as increasing
checkpoint segments.
Because staging and prod share a database, for most migrations, nothing Because staging and prod share a database, for most migrations, nothing
special needs to be done when deploying to prod since the shared special needs to be done when deploying to prod since the shared
@ -80,48 +77,55 @@ Migrating to a new schema
------------------------- -------------------------
When doing a git pull and noticing a [schema] commit, you must manually When doing a git pull and noticing a [schema] commit, you must manually
perform a schema upgrade: ``./manage.py migrate zephyr``. You used to perform a schema upgrade: ``./manage.py migrate zerver``.
need to run ``generate-fixtures --force`` to update the test data set, ``generate-fixtures`` should automatically detect whether
but these days ``generate-fixtures`` should automatically detect whether
the schema has changed and update things accordingly. the schema has changed and update things accordingly.
Notes and Cautions Notes and Cautions
------------------ ------------------
- **Large tables**: For large tables like Message and UserMessage, you **Large tables**
want to take precautions when adding columns to the table, performing For large tables like Message and UserMessage, you
data backfills, or building indexes. We have a migrate.py library to want to take precautions when adding columns to the table, performing
help with adding columns and backfilling data. For building indexes, data backfills, or building indexes. We have a ``migrate.py`` library to
we should do this outside of South using postgres's CONCURRENTLY help with adding columns and backfilling data. For building indexes,
keyword. we should do this outside of South using postgres's CONCURRENTLY
- **Numbering conflicts across branches**: If you've done your schema keyword.
change in a branch, and meanwhile another schema change has taken
place, South will now have two migrations with the same number. To **Numbering conflicts across branches**
fix this, delete the migration file that South generated, and re-run If you've done your schema change in a branch, and meanwhile another
``./manage.py schemamigration zephyr --auto`` schema change has taken place, South will now have two migrations with
- **Avoid nullables**: You generally no longer need a Nullable column the same number. To fix this, delete the migration file that South
to avoid problems with staging and prod not having the same models. generated, and re-run ``./manage.py schemamigration zerver --auto``.
See the next point about setting ``keep_default=True``.
- **Use keep\_default**: When adding a new column to an existing table, **Avoid nullables**
you almost always will want to set ``keep_default=True`` in the South You generally no longer need a Nullable column
migration ``db.add_column`` call. If you don't, everything will to avoid problems with staging and prod not having the same models.
appear to work fine in testing and on staging, but once the schema See the next point about setting ``keep_default=True``.
migration is done, the pre-migration code running on prod will be
unable to save new rows for that table (so e.g. if you were adding a **Use keep\_default**
new field to UserProfile, we'd be unable to create new users). The When adding a new column to an existing table,
exception to this rule is when your field default is not a constant you almost always will want to set ``keep_default=True`` in the South
value. In this case, you'll need to do something special to either migration ``db.add_column`` call. If you don't, everything will
set a database-level default or use a Nullable field and a multi-step appear to work fine in testing and on staging, but once the schema
schema deploy process. migration is done, the pre-migration code running on prod will be
- **Rebase pain**: If you ever need to rebase a schema change past unable to save new rows for that table (so e.g. if you were adding a
other schema changes made on other branches, in addition to new field to UserProfile, we'd be unable to create new users). The
renumbering your schema change, youalso need to be sure to regenerate exception to this rule is when your field default is not a constant
at least the bottom part of your migration (which shows the current value. In this case, you'll need to do something special to either
state of all the models) after rebasing; if you don't, then the next set a database-level default or use a Nullable field and a multi-step
migration made after your migration is merged will incorrectly schema deploy process.
attempt to re-apply all the schema changes made in the migration you
skipped. This can be potentially dangerous. **Rebase pain**
- **Enterprise Version**: Keep in mind that any schema migrations we If you ever need to rebase a schema change past
write will also be run on Zulip Enterprise installations. So don't do other schema changes made on other branches, in addition to
anything to set particular values appropriate for our site which may renumbering your schema change, youalso need to be sure to regenerate
not be appropriate for Zulip Enterprise. at least the bottom part of your migration (which shows the current
state of all the models) after rebasing; if you don't, then the next
migration made after your migration is merged will incorrectly
attempt to re-apply all the schema changes made in the migration you
skipped. This can be potentially dangerous.
**Upstreaming**
We recommend upstreaming schema changes as soon as possible to
avoid schema numbering conflicts (see above).