upgrade-postgresql: Do not remove other supervisor configs.

We previously used `zulip-puppet-apply` with a custom config file,
with an updated PostgreSQL version but more limited set of
`puppet_classes`, to pre-create the basic settings for the new cluster
before running `pg_upgradecluster`.

Unfortunately, the supervisor config uses `purge => true` to remove
all SUPERVISOR configuration files that are not included in the puppet
configuration; this leads to it removing all other supervisor
processes during the upgrade, only to add them back and start them
during the second `zulip-puppet-apply`.

It also leads to `process-fts-updates` not being started after the
upgrade completes; this is the one supervisor config file which was
not removed and re-added, and thus the one that is not re-started due
to having been re-added.  This was not detected in CI because CI added
a `start-server` command which was not in the upgrade documentation.

Set a custom facter fact that prevents the `purge` behaviour of the
supervisor configuration.  We want to preserve that behaviour in
general, and using `zulip-puppet-apply` continues to be the best way
to pre-set-up the PostgreSQL configuration -- but we wish to avoid
that behaviour when we know we are applying a subset of the puppet
classes.

Since supervisor configs are no longer removed and re-added, this
requires an explicit start-server step in the instructions after the
upgrades complete.  This brings the documentation into alignment with
what CI is testing.

docs/production/upgrade-or-modify.md

@@ -480,22 +480,34 @@ To upgrade the version of PostgreSQL on the Zulip server:
 
 1. Upgrade your server to the latest Zulip release (at least 3.0).
 
-2. Stop the server and take a backup:
+1. Stop the server, as the `zulip` user:
+
+    ```bash
+    # On Zulip before 4.0, use `supervisor stop all` instead
+    /home/zulip/deployments/current/scripts/stop-server
+    ```
+
+1. Take a backup, in case of any problems:
 
     ```bash
-    supervisorctl stop all
     /home/zulip/deployments/current/manage.py backup --output=/home/zulip/postgresql-upgrade.backup.tar.gz
     ```
 
-3. As root, run the database upgrade tool:
+1. As root, run the database upgrade tool:
 
     ```bash
     /home/zulip/deployments/current/scripts/setup/upgrade-postgresql
     ```
 
-`upgrade-postgresql` will have finished by restarting your Zulip server;
+1. As the `zulip` user, start the server again:
-you should now be able to navigate to its URL and confirm everything
+
-is working correctly.
+    ```bash
+    # On Zulip before 4.0, use `restart-server` instead of `start-server` instead
+    /home/zulip/deployments/current/scripts/start-server
+    ```
+
+You should now be able to navigate to the Zulip server's URL and
+confirm everything is working correctly.
 
 
 ## Modifying Zulip

puppet/zulip/manifests/supervisor.pp

@@ -12,12 +12,15 @@ class zulip::supervisor {
   }
 
   $conf_dir = $zulip::common::supervisor_conf_dir
+  # lint:ignore:quoted_booleans
+  $should_purge = $facts['leave_supervisor'] != 'true'
+  # lint:endignore
   file { $conf_dir:
     ensure  => 'directory',
     require => Package['supervisor'],
     owner   => 'root',
     group   => 'root',
-    purge   => true,
+    purge   => $should_purge,
     recurse => true,
     notify  => Service[$supervisor_service],
   }

scripts/setup/upgrade-postgresql

@@ -34,7 +34,7 @@ fi
     crudini --set "$ZULIP_CONF" machine puppet_classes zulip::profile::base,zulip::postgresql_base
     touch "/usr/share/postgresql/$UPGRADE_TO/pgroonga_setup.sql.applied"
 
-    "$ZULIP_PATH"/scripts/zulip-puppet-apply -f --config "$ZULIP_CONF"
+    FACTER_LEAVE_SUPERVISOR=true "$ZULIP_PATH"/scripts/zulip-puppet-apply -f --config "$ZULIP_CONF"
     rm -rf "$TEMP_CONF_DIR"
 )