Skip to content
Kestrel Docs

Upgrades

Scope

Section titled “Scope”

This runbook covers three upgrade paths:

  1. Kubernetes version upgrades — moving the cluster control plane and nodes to a new minor release.
  2. Addon/chart upgrades — bumping Helm chart versions for cluster addons (ArgoCD, cert-manager, Capsule, etc.).
  3. Tooling upgrades — updating CLI tools and CI pipelines (kubectl, helm, argocd CLI).

Each follows the same pre-flight → upgrade → verify → rollback-if-needed pattern.

Pre-flight checklist

Section titled “Pre-flight checklist”

Before any upgrade:

  • Read the upstream changelog / release notes for breaking changes.

  • Confirm the target version is tested in the staging cluster (if available).

  • Notify tenants of the maintenance window via the RCS ops channel.

  • Take an etcd snapshot on a control-plane node — etcdctl needs the etcd client certs (see documentation/Etcd.md in the ubernetes repo):

    Terminal window
    sudo /usr/local/bin/etcdctl --endpoints=https://localhost:2379 \
      --cacert /etc/kubernetes/ssl/etcd/ca.crt \
      --cert /etc/kubernetes/ssl/etcd/peer.crt \
      --key /etc/kubernetes/ssl/etcd/peer.key \
      snapshot save /tmp/etcd-backup-$(date +%Y%m%d).db

    Then copy the snapshot off the node (e.g. scp it to the control host) — a backup left in /tmp on the node can be lost during the very upgrade it protects against.

  • Verify current cluster health: kubectl get nodes, kubectl get pods -A | grep -v Running.

  • Confirm ArgoCD shows all Applications as Synced and Healthy.

Kubernetes version upgrade

Section titled “Kubernetes version upgrade”

Control plane

Section titled “Control plane”
  1. Update the Kubernetes version in the kubespray inventory/playbook variables for the cluster.
  2. Run the kubespray upgrade playbook — the control plane components (api-server, controller-manager, scheduler) roll one at a time.
  3. Verify: kubectl version should show the new server version. kubectl get --raw /healthz should return ok (prefer this over the deprecated kubectl get cs).

Worker nodes

Section titled “Worker nodes”

On Kestrel, worker kubelets are upgraded by the same kubespray run — upgrade-cluster.yml cordons, drains, upgrades, and uncordons nodes one at a time after the control plane. Run it from the cluster’s inventory directory in the ubernetes repo (e.g. kestrel-prod/); see documentation/ClusterOperations.md there for the invocation and the node-based --limit pattern.

  1. Run the kubespray upgrade playbook (as in the control-plane section) and let it work through the workers serially.
  2. Verify each node rejoins as Ready with the new kubelet version: kubectl get nodes -o wide.
  3. Wait for all workloads to reschedule and settle before ending the maintenance window.

If a node must be handled outside kubespray (e.g. an OS re-image), the generic sequence is: kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets --delete-emptydir-data, upgrade or re-image, then kubectl uncordon <node> — one node at a time.

Addon / chart upgrades

Section titled “Addon / chart upgrades”
  1. Identify the addon to upgrade and its current chart version in ubernetes-applications.
  2. Check the chart’s changelog for breaking changes (especially CRD updates — these often require manual steps).
  3. Update addonChartVersion in the addon’s ApplicationSet manifest under addons/ in ubernetes-applications (e.g. addons/cert-manager-appset.yaml) — it is consumed as the Helm source targetRevision. Values files carry chart configuration, not chart versions.
  4. If the chart includes CRD changes: pull and extract the new chart version (helm pull <repo>/<chart> --version <new-version> --untar) and apply its crds/ directory (kubectl apply -f <chart>/crds/) before merging the version bump — Helm does not upgrade CRDs automatically.
  5. Commit, merge, and let ArgoCD sync.
  6. Verify the Application returns to Synced + Healthy. Check pod logs for startup errors.

Common addon-specific notes

Section titled “Common addon-specific notes”
AddonWatch for
ArgoCDRBAC policy changes, UI breaking changes, ApplicationSet CRD updates
cert-managerCRD version jumps, webhook certificate rotation
CapsuleTenant CRD spec changes, admission webhook compatibility
TraefikIngressRoute/middleware CRD changes, static/dynamic config key renames

Rollback procedure

Section titled “Rollback procedure”

Kubernetes version

Section titled “Kubernetes version”

Kubernetes does not support in-place downgrade of the control plane. If a version upgrade fails:

  1. Restore the etcd snapshot taken during pre-flight.
  2. Re-provision control plane components at the previous version.
  3. Worker nodes: drain, re-image to the previous version, uncordon.

This is a last-resort procedure — prefer fixing forward when possible.

Addon / chart

Section titled “Addon / chart”
  1. Revert the version-bump MR in ubernetes-applications.
  2. If CRDs were updated: check whether the old chart version is compatible with the new CRDs. If not, manually restore the old CRDs from the previous chart tarball.
  3. ArgoCD picks up the revert on its next Git poll (~3 minutes) and syncs the reverted values.
  4. Verify the addon pods restart with the previous version.

Post-upgrade verification

Section titled “Post-upgrade verification”

After any upgrade:

  1. kubectl get nodes -o wide — all nodes Ready, correct version.
  2. kubectl get pods -A | grep -v Running — no unexpected non-running pods.
  3. ArgoCD dashboard — all Applications Synced and Healthy.
  4. Spot-check tenant workloads — ask one or two active tenants to confirm their services are operating normally.
  5. Update the lastVerified date on this page and any affected platform-admin docs.