# Renumbering Migrations When you rebase your development branch off of a newer copy of master, and your branch contains new database migrations, you can occasionally get thrown off by conflicting migrations that are new in master. To help you understand how to deal with these conflicts, I am about to narrate an exercise where I bring my development branch called showell-topic up to date with master. **Note:** You can also use the command `./tools/renumber-migrations` to automatically perform migration renumbering. In this example, there is a migration on master called `0024_realm_allow_message_editing.py`, and that was the most recent migration at the time I started working on my branch. In my branch I created migrations 0025 and 0026, but then meanwhile on master somebody else created their own migration 0025. Anyway, on with the details... First, I go to showell-topic and run tests to make sure that I'm starting with a clean, albeit out-of-date, dev branch: ``` showell@Steves-MBP ~/zulip (showell-approximate) $ git checkout showell-topic Switched to branch 'showell-topic' showell@Steves-MBP ~/zulip (showell-topic) $ git status # On branch showell-topic nothing to commit, working directory clean (zulip-py3-venv) vagrant@ubuntu-bionic:/srv/zulip$ ./tools/test-backend DONE! ``` Next, I fetch changes from upstream: ``` showell@Steves-MBP ~/zulip (showell-topic) $ git checkout master Switched to branch 'master' showell@Steves-MBP ~/zulip (master) $ git fetch upstream showell@Steves-MBP ~/zulip (master) $ git merge upstream/master Updating 2967341..09754c9 Fast-forward ``` Then I go back to showell-topic: ``` showell@Steves-MBP ~/zulip (master) $ git checkout showell-topic Switched to branch 'showell-topic' ``` You may want to make note of your HEAD commit on your branch before you start rebasing, in case you need to start over, or do like I do and rely on being able to find it via github. I'm not showing the details of that, since people have different styles for managing botched rebases. Anyway, I rebase to master as follows: ``` showell@Steves-MBP ~/zulip (showell-topic) $ git rebase -i master Successfully rebased and updated refs/heads/showell-topic. ``` Note that my rebase was conflict-free from git's point of view, but I still need to run the tests to make sure there weren't any semantic conflicts with the new changes from master: ``` (zulip-py3-venv) vagrant@ubuntu-bionic:/srv/zulip$ ./tools/test-backend File "/srv/zulip-venv-cache/ad3a375e95a56d911510d7edba7e17280d227bc7/zulip-venv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 105, in handle "'./manage.py makemigrations --merge'" % name_str django.core.management.base.CommandError: Conflicting migrations detected (0026_topics_backfill, 0025_realm_message_content_edit_limit in zerver). To fix them run './manage.py makemigrations --merge' File "/srv/zulip/zerver/lib/db.py", line 33, in execute return wrapper_execute(self, super().execute, query, vars) File "/srv/zulip/zerver/lib/db.py", line 20, in wrapper_execute return action(sql, params) django.db.utils.ProgrammingError: relation "zerver_realmfilter" does not exist LINE 1: ...n", "zerver_realmfilter"."url_format_string" FROM "zerver_re... ``` The above traceback is fairly noisy, but it's pretty apparent that I have migrations that are out of order. More precisely, my 0025 migration points to 0024 as its dependency, where it really should point to the other 0025 as its dependency, and I need to renumber my migrations to 0026 and 0027. Let's take a peek at the migrations directory: ``` showell@Steves-MBP ~/zulip (showell-topic) $ ls -r zerver/migrations/*.py | head -6 zerver/migrations/__init__.py zerver/migrations/0026_topics_backfill.py zerver/migrations/0025_realm_message_content_edit_limit.py zerver/migrations/0025_add_topic_table.py zerver/migrations/0024_realm_allow_message_editing.py zerver/migrations/0023_userprofile_default_language.py ``` We have two different 0025 migrations that both depend on 0024: ``` showell@Steves-MBP ~/zulip (showell-topic) $ grep -B 1 0024 zerver/migrations/*.py zerver/migrations/0025_add_topic_table.py- dependencies = [ zerver/migrations/0025_add_topic_table.py: ('zerver', '0024_realm_allow_message_editing'), -- zerver/migrations/0025_realm_message_content_edit_limit.py- dependencies = [ zerver/migrations/0025_realm_message_content_edit_limit.py: ('zerver', '0024_realm_allow_message_editing'), ``` I will now start the process of renaming `0025_add_topic_table.py` to be `0026_add_topic_table.py` and having it depend on `0025_realm_message_content_edit_limit`. Before I start, I want to know which of my commits created my 0025 migration: ``` showell@Steves-MBP ~/zulip (showell-topic) $ git log --pretty=oneline zerver/migrations/0025_add_topic_table.py d859e6ffc165e822cec39152a5814ca7ce94d172 Add Topic and Message.topic to models ``` Here is the transcript, and hopefully what I did inside of vim is apparent from the diff: ``` showell@Steves-MBP ~/zulip (showell-topic) $ cd zerver/migrations/ showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git mv 0025_add_topic_table.py 0026_add_topic_table.py showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic +) $ vim 0026_add_topic_table.py showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git diff diff --git a/zerver/migrations/0026_add_topic_table.py b/zerver/migrations/0026_add_topic_table.py index 2c8c07a..43351eb 100644 --- a/zerver/migrations/0026_add_topic_table.py +++ b/zerver/migrations/0026_add_topic_table.py @@ -8,7 +8,7 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('zerver', '0024_realm_allow_message_editing'), + ('zerver', '0025_realm_message_content_edit_limit'), ] operations = [ showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git commit -am 'temp rename migration' [showell-topic 45cf5e9] temp rename migration 1 file changed, 1 insertion(+), 1 deletion(-) rename zerver/migrations/{0025_add_topic_table.py => 0026_add_topic_table.py} (93%) showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git status # On branch showell-topic nothing to commit, working directory clean ``` Next, I want to rewrite the history of my branch. When I'm in the interactive rebase (not shown), I need to make the temp commit be a "fix" for the original commit that I noted above: ``` showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git rebase -i master [detached HEAD c1f2e69] Add Topic and Message.topic to models 2 files changed, 38 insertions(+) create mode 100644 zerver/migrations/0026_add_topic_table.py Successfully rebased and updated refs/heads/showell-topic. ``` I did this to verify that I rebased correctly: ``` showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git log 0026_add_topic_table.py commit c1f2e69e716beb4031c628fe2189b49f04770d03 Author: Steve Howell Date: Thu Jul 14 13:26:15 2016 -0700 Add Topic and Message.topic to models showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git show c1f2e69e716beb4031c628fe2189b49f04770d03 ``` Next, I follow a very similar process for my second migration-related commit on my branch: ``` showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git log --pretty=oneline 0026_topics_backfill.py ba43e1ffb072f4e6a66ffb5c4030ff3a17d53792 (unfinished) stub commit for topic backfill showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git mv 0026_topics_backfill.py 0027_topics_backfill.py showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic +) $ vim 0027_topics_backfill.py showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git diff diff --git a/zerver/migrations/0027_topics_backfill.py b/zerver/migrations/0027_topics_backfill.py index 766b075..05ea2bb 100644 --- a/zerver/migrations/0027_topics_backfill.py +++ b/zerver/migrations/0027_topics_backfill.py @@ -8,7 +8,7 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('zerver', '0025_add_topic_table'), + ('zerver', '0026_add_topic_table'), ] operations = [ showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git commit -am 'temp rename migration' [showell-topic 02ef15b] temp rename migration 1 file changed, 1 insertion(+), 1 deletion(-) rename zerver/migrations/{0026_topics_backfill.py => 0027_topics_backfill.py} (90%) showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git rebase -i master [detached HEAD 8022839] (unfinished) stub commit for topic backfill 1 file changed, 20 insertions(+) create mode 100644 zerver/migrations/0027_topics_backfill.py Successfully rebased and updated refs/heads/showell-topic. ``` My rebase looked something like this after I edited the commits: ``` 1 pick eee291d Add subject_topic_awareness() test helper. 2 pick c1f2e69 Add Topic and Message.topic to models 3 pick 46c04b5 Call new update_topic() in pre_save_message(). 4 pick df20dc8 Write to Topic table for topic edits. 5 pick ba43e1f (unfinished) stub commit for topic backfill 6 f 02ef15b temp rename migration 7 pick b8e93d2 Add CATCH_TOPIC_MIGRATION_BUGS. 8 pick 644ccae Have get_context_for_message use topic_id. 9 pick 6303f5b Have update_message_flags user topic_id. 10 pick 9f9da5a Have narrowing searches use topic id. 11 pick 0b37ef7 Assert that new_topic=True obliterates message.subject 12 pick dff8eee Use new_topics=True in test_bulk_message_fetching(). 13 pick bc377a0 Use topic_id when propagating message edits. 14 pick 11fde9c Have message cache use Topic table (and more...). 15 pick 518649b Use topic_name() in to_log_dict(). 16 17 # Rebase 09754c9..02ef15b onto 09754c9 18 # 19 # Commands: 20 # p, pick = use commit 21 # r, reword = use commit, but edit the commit message 22 # e, edit = use commit, but stop for amending 23 # s, squash = use commit, but meld into previous commit 24 # f, fixup = like "squash", but discard this commit's log message 25 # x, exec = run command (the rest of the line) using shell ``` I double check that everything went fine: ``` showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git log 0027_topics_backfill.py commit 8022839f9168e643ae08365bfb40f1de2e64d426 Author: Steve Howell Date: Thu Jul 14 15:51:06 2016 -0700 (unfinished) stub commit for topic backfill showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git show 8022839f9168e643ae08365bfb40f1de2e64d426 ``` And then I run the tests and cross my fingers!!!: ``` (zulip-py3-venv) vagrant@ubuntu-bionic:/srv/zulip$ ./tools/test-backend Applying zerver.0023_userprofile_default_language... OK Applying zerver.0024_realm_allow_message_editing... OK Applying zerver.0025_realm_message_content_edit_limit... OK Applying zerver.0026_add_topic_table... OK Applying zerver.0027_topics_backfill... OK Applying zilencer.0001_initial... OK Successfully populated test database. DROP DATABASE CREATE DATABASE Running zerver.tests.test_auth_backends.AuthBackendTest.test_devauth_backend Running zerver.tests.test_auth_backends.AuthBackendTest.test_dummy_backend Running zerver.tests.test_auth_backends.AuthBackendTest.test_email_auth_backend FAILED! ``` Ugh, my tests still fail due to some non-migration-related changes on master. The good news, however, is that my migrations are cleaned up. So I've shown you the excruciating details of fixing up migrations in a complicated branch, but let's step back and look at the big picture. You need to get these things right: - Rename the migrations on your branch. - Fix their dependencies. - Rewrite your git history so that it appears like you never branched off an old copy of master. The hardest part of the process will probably be cleaning up your git history.