Upgrades

Scope

This runbook covers three upgrade paths:

1. Kubernetes version upgrades — moving the cluster control plane and nodes to a new minor release.
2. Addon/chart upgrades — bumping Helm chart versions for cluster addons (ArgoCD, cert-manager, Capsule, etc.).
3. Tooling upgrades — updating CLI tools and CI pipelines (kubectl, helm, argocd CLI).

Each follows the same pre-flight → upgrade → verify → rollback-if-needed pattern.

---

Pre-flight checklist

Before any upgrade:

• Read the upstream changelog / release notes for breaking changes.
• Confirm the target version is tested in the staging cluster (if available).
• Notify tenants of the maintenance window via the RCS ops channel.
• Take an etcd snapshot: etcdctl snapshot save /tmp/etcd-backup-$(date +%Y%m%d).db.
• Verify current cluster health: kubectl get nodes, kubectl get pods -A | grep -v Running.
• Confirm ArgoCD shows all Applications as Synced and Healthy.

---

Kubernetes version upgrade

Control plane

1. Update the Kubernetes version in the kubespray inventory/playbook variables for the cluster.
2. Run the kubespray upgrade playbook — the control plane components (api-server, controller-manager, scheduler) roll one at a time.
3. Verify: kubectl version should show the new server version. kubectl get --raw /healthz should return ok (prefer this over the deprecated kubectl get cs).

Worker nodes

1. Cordon and drain nodes one at a time: kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.
2. Update the node OS image or kubelet version (depends on provisioning method).
3. Uncordon the node: kubectl uncordon <node>.
4. Verify the node rejoins as Ready with the new kubelet version: kubectl get node <node> -o wide.
5. Repeat for each node. Wait for all workloads to reschedule before moving to the next node.

---

Addon / chart upgrades

1. Identify the addon to upgrade and its current chart version in ubernetes-applications.
2. Check the chart’s changelog for breaking changes (especially CRD updates — these often require manual steps).
3. Update the chart version in the appropriate values file.
4. If the chart includes CRD changes: apply CRDs manually first (kubectl apply -f crds/) — Helm does not upgrade CRDs automatically.
5. Commit, merge, and let ArgoCD sync.
6. Verify the Application returns to Synced + Healthy. Check pod logs for startup errors.

Common addon-specific notes

Addon	Watch for
ArgoCD	RBAC policy changes, UI breaking changes, ApplicationSet CRD updates
cert-manager	CRD version jumps, webhook certificate rotation
Capsule	Tenant CRD spec changes, admission webhook compatibility
Traefik	IngressRoute/middleware CRD changes, static/dynamic config key renames

---

Rollback procedure

Kubernetes version

Kubernetes does not support in-place downgrade of the control plane. If a version upgrade fails:

1. Restore the etcd snapshot taken during pre-flight.
2. Re-provision control plane components at the previous version.
3. Worker nodes: drain, re-image to the previous version, uncordon.

This is a last-resort procedure — prefer fixing forward when possible.

Addon / chart

1. Revert the version-bump MR in ubernetes-applications.
2. If CRDs were updated: check whether the old chart version is compatible with the new CRDs. If not, manually restore the old CRDs from the previous chart tarball.
3. ArgoCD syncs the reverted values within ~60 seconds.
4. Verify the addon pods restart with the previous version.

---

Post-upgrade verification

After any upgrade:

1. kubectl get nodes -o wide — all nodes Ready, correct version.
2. kubectl get pods -A | grep -v Running — no unexpected non-running pods.
3. ArgoCD dashboard — all Applications Synced and Healthy.
4. Spot-check tenant workloads — ask one or two active tenants to confirm their services are operating normally.
5. Update the lastVerified date on this page and any affected platform-admin docs.