Docs-site rollback

When to use this

A commit to main in uvic-rcs/kestrel-docs has shipped a bad docs build — broken page, wrong copy, or a build regression that only surfaced in production. The bad image is live at kestrel.arbutus.cloud. This page is the emergency-button procedure to revert.

Rollback in one revert

image.tag in ubernetes-applications is managed by argocd-image-updater, not by MRs: the updater tracks the mutable main tag of git.computecanada.ca:4567/uvic-rcs/kestrel-docs with the digest strategy and commits new digests directly to main (commits titled build: automatic update of docs-kestrel). Do not revert one of those commits — the updater re-writes the bad digest on its next cycle.

Rollback is a single revert on the docs repo instead:

1. In uvic-rcs/kestrel-docs, find the bad commit on main and revert it (Revert button on the commit in the GitLab UI, or git revert <sha> and push).
2. CI rebuilds and publishes a fresh image under the main tag.
3. argocd-image-updater sees the new digest and commits the updated image.tag to environments/arbutus/kestrel/kestrel-prod/docs/kestrel/values.yaml on ubernetes-applications.
4. ArgoCD picks up the change within its ~3-minute poll interval (click Sync on the docs-kestrel Application for an immediate reconcile) and rolls the Deployment.
5. Verify at https://kestrel.arbutus.cloud that the bad content is gone.

End-to-end: most of the wait is the docs CI build plus one updater cycle — budget ~10 minutes.

Fallback: pin the previous digest

If the revert path is unavailable (docs CI is broken, or you need the site back before a rebuild can finish), pin the previous digest by hand — but disable the updater in the same change, or it overwrites your pin on its next cycle:

1. Find the previous digest in the git history of environments/arbutus/kestrel/kestrel-prod/docs/kestrel/values.yaml — each build: automatic update of docs-kestrel commit shows the digest it replaced.
2. In one MR against that file, set imageUpdater.enabled: false and set image.tag to the previous main@sha256:<digest>.
3. Merge; ArgoCD deploys the pinned image.
4. Once the docs repo is fixed, remove both overrides in a follow-up MR to hand image.tag back to the updater.

Why this works

Every docs-site deploy is a specific image digest (main@sha256:<digest>) pinned in environments/arbutus/kestrel/kestrel-prod/docs/kestrel/values.yaml, written there by argocd-image-updater. Because each digest is immutable, any previously deployed image normally still exists in the GitLab Container Registry (unless a registry cleanup policy has removed it). The docs-repo revert works because it publishes a fresh image whose digest the updater rolls forward to; the pin fallback works because the digest addresses the exact prior build directly.

Verification

Once the rollback lands in ubernetes-applications (the updater’s write-back commit, or your pin MR), the docs-kestrel Application in ArgoCD goes from Synced → OutOfSync → Syncing → Synced. The pod re-rolls with a brief Terminating on the old pod. Open https://kestrel.arbutus.cloud in a browser (hard refresh to bypass CDN/browser cache) and confirm the previous content is live.

If the site still shows the bad content after 5 minutes, check:

• kubectl -n docs-kestrel get deployment kestrel-docs-site -o yaml | grep image: — should show the rolled-back digest.
• kubectl -n docs-kestrel get pods — should show Running 1/1 with the reverted pod age.
• kubectl -n argo-cd describe application docs-kestrel — check for sync errors.