upgrade-zulip-from-git: Stop mirroring the remote.

The local `/srv/zulip.git` directory has been cloned with `--mirror`
since it was first created as a local cache in dc4b89fb08.  This
made some sense at the time, since it was purely a cache of the
remote, and not a home to local branches of its own.

That changed in 3f83b843c2, when we began using `git worktree`,
which caused the `deployment-...` branches to begin being stored in
`/src/zulip.git`.  This caused intermixing of local and remote
branches.

When 02582c6956 landed, the addition of `--prune` caused all but the
most recent deployment branch to be deleted upon every fetch --
leaving previous deployments with non-existent branches checked out:

```
zulip@example-prod-host:~/deployments/last$ git status
On branch deployment-2022-04-15-23-07-55

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   .browserslistrc
	new file:   .codecov.yml
	new file:   .codespellignore
	new file:   .editorconfig
[...snip list of every file in repo...]
```

Switch `/srv/zulip.git` to no longer be a `--mirror` cache of the
origin.  We reconfigure the remote to drop `remote.origin.mirror`, and
delete all refs under `refs/pulls/` and `refs/heads/`, while
preserving any checked-out branches.  `refs/pulls/`, if the remote is
the canonical upstream, contains _tens of thousands_ of refs, so
pruning those refs trims off 20% of the repository size.

Those savings require a `git gc --prune=now`, otherwise the dangling
objects are ejected from the packfiles, which would balloon the
repository up to more than three times its previous size.  Repacking
the repository is reasonable, in general, after removing such a large
number of refs -- and the `--prune=now` is safe and will not lose
data, as the `--mirror` was good at ensuring that the repository could
not be used for any local state.

The refname in the upgrade process was previously resolved from the
union of local and remote refs, since they were in the same namespace.
We instead now only resolve arguments as tags, then origin branches;
this means that stale local branches will be skipped.  Users who want
to deploy from local branches can use `--remote-url=.`.

Because the `scripts/lib/upgrade-zulip-from-git` file is "stage 1" and
run from the old version's code, this will take two invocations of
`upgrade-zulip-from-git` to take effect.

Fixes #21901.

scripts/lib/upgrade-zulip-from-git

@@ -62,9 +62,10 @@ try:
     if not os.path.exists(LOCAL_GIT_CACHE_DIR):
         logging.info("Cloning the repository")
         subprocess.check_call(
-            ["git", "clone", "-q", remote_url, "--mirror", LOCAL_GIT_CACHE_DIR],
+            ["git", "clone", "-q", remote_url, LOCAL_GIT_CACHE_DIR],
             stdout=subprocess.DEVNULL,
         )
+
     if os.stat(LOCAL_GIT_CACHE_DIR).st_uid == 0:
         subprocess.check_call(["chown", "-R", "zulip:zulip", LOCAL_GIT_CACHE_DIR])
 
@@ -73,6 +74,57 @@ try:
         ["git", "remote", "set-url", "origin", remote_url], preexec_fn=su_to_zulip
     )
 
+    # Check to see if it's the old "mirror" configuration
+    is_mirror = subprocess.check_output(
+        ["git", "config", "--bool", "--default=false", "remote.origin.mirror"],
+        preexec_fn=su_to_zulip,
+        text=True,
+    )
+    if is_mirror.strip() == "true":
+        subprocess.check_call(
+            ["git", "config", "--unset", "remote.origin.mirror"],
+            preexec_fn=su_to_zulip,
+        )
+        subprocess.check_call(
+            ["git", "config", "remote.origin.fetch", "+refs/heads/*:refs/remotes/origin/*"],
+            preexec_fn=su_to_zulip,
+        )
+        matching_refs = subprocess.check_output(
+            ["git", "for-each-ref", "--format=%(refname)", "refs/pull/", "refs/heads/"],
+            preexec_fn=su_to_zulip,
+            text=True,
+        ).splitlines()
+
+        # We can't use `git worktree list --porcelain -z` here because
+        # Ubuntu 20.04 Focal only has git 2.25.1, and -z was
+        # introduced in 2.36
+        worktree_data = subprocess.check_output(
+            ["git", "worktree", "list", "--porcelain"],
+            preexec_fn=su_to_zulip,
+            text=True,
+        ).splitlines()
+        keep_refs = set()
+        for worktree_line in worktree_data:
+            if worktree_line.startswith("branch "):
+                keep_refs.add(worktree_line[len("branch ") :])
+
+        delete_input = "".join(
+            f"delete {refname}\n" for refname in matching_refs if refname not in keep_refs
+        )
+        subprocess.run(
+            ["git", "update-ref", "--stdin"],
+            check=True,
+            preexec_fn=su_to_zulip,
+            input=delete_input,
+            text=True,
+        )
+
+        logging.info("Repacking repository after pruning unnecessary refs...")
+        subprocess.check_call(
+            ["git", "gc", "--prune=now"],
+            preexec_fn=su_to_zulip,
+        )
+
     # Ensure upstream remote is configured; we need this to make `git describe` accurate.
     remotes = subprocess.check_output(["git", "remote"], preexec_fn=su_to_zulip).split(b"\n")
     if b"upstream" not in remotes:
@@ -90,11 +142,25 @@ try:
     )
 
     # Generate the deployment directory via git worktree from our local repository.
-    commit_hash = subprocess.check_output(
+    try:
-        ["git", "rev-parse", refname],
+        fullref = f"refs/tags/{refname}"
-        preexec_fn=su_to_zulip,
+        commit_hash = subprocess.check_output(
-        text=True,
+            ["git", "rev-parse", "--verify", fullref],
-    ).strip()
+            preexec_fn=su_to_zulip,
+            text=True,
+            stderr=subprocess.DEVNULL,
+        ).strip()
+    except subprocess.CalledProcessError as e:
+        if e.returncode == 128:
+            # Try in the origin namespace
+            fullref = f"refs/remotes/origin/{refname}"
+            commit_hash = subprocess.check_output(
+                ["git", "rev-parse", "--verify", fullref],
+                preexec_fn=su_to_zulip,
+                text=True,
+                stderr=subprocess.DEVNULL,
+            ).strip()
+    refname = fullref
     logging.info("Upgrading to %s, in %s", commit_hash, deploy_path)
     subprocess.check_call(
         ["git", "worktree", "add", "--detach", deploy_path, refname],
@@ -102,8 +168,19 @@ try:
         preexec_fn=su_to_zulip,
     )
     os.chdir(deploy_path)
+    extra_flags = []
+    if not refname.startswith("refs/tags/"):
+        extra_flags = ["-t"]
     subprocess.check_call(
-        ["git", "checkout", "-qtb", "deployment-" + os.path.basename(deploy_path), refname],
+        [
+            "git",
+            "checkout",
+            "-q",
+            *extra_flags,
+            "-b",
+            "deployment-" + os.path.basename(deploy_path),
+            refname,
+        ],
         preexec_fn=su_to_zulip,
     )