Troubleshooting index

A flat search index of error messages and symptoms you might hit on Kestrel, organized for grep. Each heading is a verbatim error string or recognizable symptom so you can search this page with Ctrl+F (or Cmd+F) from your terminal output.

If you are in the first 60 seconds of debugging and want a guided flow instead, start with the Triage page — it has four branches that narrow the problem quickly. This page is the deeper reference for when you already know which error you are looking at.

For question-oriented answers (“why does X happen?”), see the FAQ.

---

kubelogin and identity

error: You must be logged in to the server (Unauthorized)

Cause: your OIDC token is missing, expired, or was issued by the wrong identity provider.

Fix: clear the token cache and trigger a fresh login:

kubectl oidc-login clean
kubectl get ns

If the fresh login still fails, confirm you are using the int128/kubelogin plugin (not the Microsoft Azure AD one) and that your kubeconfig has the correct oidc-issuer-url and oidc-client-id. See Install kubelogin for the full setup.

State does not match

Cause: stale OIDC state in the local cache, usually from an aborted login or a duplicate browser tab that followed an old callback link.

Fix: clear the cache and retry in a fresh browser window:

kubectl oidc-login clean
kubectl get ns

If it persists, try an incognito/private browser window so the Keycloak session cookie is clean. See kubelogin troubleshooting — state does not match.

kubectl get ns hangs — no browser opens

Cause: you are running inside WSL2, tmux, or a remote SSH session with no local GUI browser for kubelogin to open.

Fix: add --skip-open-browser to your kubeconfig exec.args, copy the printed URL to a local browser, and forward the callback port if the session is remote:

ssh -L 8000:localhost:8000 <user>@<remote-host>

Full steps at kubelogin troubleshooting — browser does not open.

kubectl get ns returns zero namespaces (empty result, no error)

Cause (most likely — not an error): your tenant is freshly provisioned. Kestrel does not pre-create any namespaces for you, so an empty list is the normal state until you create your first namespace.

Cause (access problem): your OIDC token is valid, but its groups claim does not contain a tenant group that maps to a Capsule Tenant. The most common reason is that you have not been added to the tenant’s Keycloak group yet — membership is currently managed by RCS staff, and sponsorship in CCDB alone does not grant access.

Fix: tell the two cases apart first:

kubectl auth whoami

If the Groups row contains oidc:<your-tenant>, login is working and the empty list is normal — create your first namespace by following Your first deployment. Only if the oidc: tenant group is missing is this an access problem: ask your PI to confirm the add-user request to RCS was made and completed (see Managing membership), then clear the cache and retry:

kubectl oidc-login clean
kubectl auth whoami

See Requesting access for the end-to-end flow and kubelogin troubleshooting — missing tenant group.

listen tcp 127.0.0.1:8000: bind: address already in use

Cause: another process already bound localhost:8000 — a dev server, a previous kubelogin session, or a Docker port publish.

Fix: add --listen-address=localhost:18000 to your existing kubeconfig exec.args — keep the other lines exactly as you set them up in Install kubelogin:

args:
  - oidc-login
  - get-token
  - --oidc-issuer-url=https://keycloak.arbutus.alliancecan.ca/realms/atmosphere
  - --oidc-client-id=kubernetes
  - --token-cache-storage=keyring
  - --listen-address=localhost:18000  # the only new line

See kubelogin troubleshooting — port 8000 already in use.

---

ArgoCD sync errors

admission webhook "validate.kyverno.svc-fail" denied the request

Cause: your Pod spec violates the Kyverno-enforced Pod Security Standards (restricted profile). All four of runAsNonRoot, seccompProfile.type, capabilities.drop, and allowPrivilegeEscalation are required — a spec missing any one of them is denied.

Fix: add the required security context to each container — capabilities and allowPrivilegeEscalation are only valid at the container level (containers[].securityContext), not in the pod-level spec.securityContext:

containers:
  - name: app
    # ...
    securityContext:
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop: [ALL]

See Known limitations — Pod Security restricted for the full checklist.

The namespace doesn't match the tenant prefix, expected prefix "<tenant>"

Cause: the namespace name does not start with your tenant prefix (<tenant>-*). Capsule’s forceTenantPrefix admission webhook rejects namespace names that do not match, wrapping this message in an admission webhook ... denied the request error.

Fix: rename the namespace to include your tenant prefix. For example, if your tenant is def-profname, the namespace must be def-profname-<suffix>. See Known limitations — namespace prefix and Namespaces.

Pod Priority Class "<name>" is forbidden for the current Tenant

Cause: the priority class you specified is not in the Capsule allowlist for your tenant. The create is denied by an admission webhook with this message.

Fix: use one of the three allowlisted priority classes: uber-user-significant, uber-user-preempt-medium, or uber-user-preempt-high. See Priority classes for what each one means and Known limitations — priority class allowlist for the rationale.

Application shows Unknown or ComparisonError

Cause: ArgoCD cannot parse or diff the manifests. This usually means invalid YAML, a missing CRD on the cluster, or a Helm template rendering error.

Fix: validate your manifests locally with kubectl apply --dry-run=client -f <manifest>.yaml. If using Helm, run helm template locally and check the output. See ArgoCD on Kestrel for the recommended workflow.

---

Ingress and TLS

Ingress returns 404 with a valid TLS certificate

Cause: the TLS handshake succeeded (cert-manager issued the cert), but Traefik cannot route to your backend. The most common cause is a Service selector that does not match Pod labels.

Fix: compare spec.selector on your Service with spec.template.metadata.labels on your Deployment — they must match exactly. Also confirm the Service port matches the container port, and that spec.ingressClassName: traefik is set on the Ingress — a wrong or misspelled class name means Traefik never picks up the Ingress and the edge answers with a 404. See Ingress on Kestrel for the full recipe.

Ingress returns 404 — no TLS certificate yet

Cause: cert-manager has not finished issuing the certificate. The first issue takes about 30 seconds; if it takes longer, the ACME challenge may be failing — on Kestrel the most common reason is that the hostname’s DNS record does not exist yet.

Fix: check the DNS record first. Automated DNS for tenant hostnames is not set up yet — creating an Ingress does not create the matching DNS record, and the certificate stays READY: False until the record exists. If you have not requested the hostname, open a ticket with RCS; once the record exists, cert-manager retries and issues the certificate without any action on your side. Then check certificate status:

kubectl get certificate -n <your-tenant>-<namespace>
kubectl describe certificate -n <your-tenant>-<namespace> <cert-name>

Look at the Events section for ACME challenge errors — a challenge that cannot be resolved or reached points back at the missing DNS record. See TLS on Kestrel for the full flow.

connection refused when visiting the Ingress hostname

Cause: the hostname resolved to an address where nothing is listening. Common reasons: a wrong or stale DNS record, a local hosts-file override, or testing an address that bypasses the Traefik edge. (A missing or wrong ingressClassName produces a 404 from the edge, not connection refused — see the 404 entries above.)

Fix: check what the hostname actually resolves to and where the connection stops:

dig +short <hostname>
curl -v https://<hostname>/

If the hostname does not resolve at all, the symptom is could not resolve host rather than connection refused — automated DNS is not live yet, so DNS records for tenant hostnames are created manually by RCS. Open a ticket with the hostname you need. See Ingress on Kestrel and Triage — Ingress 404.

LoadBalancer service types are forbidden for the tenant

Cause: Capsule restricts Service types on Kestrel. LoadBalancer is not available — the create is denied by an admission webhook with this message.

Fix: use ClusterIP with an Ingress instead. See Service types and Known limitations — LoadBalancer blocked.

---

Storage and quotas

PVC stuck in Pending

Diagnose first — the PVC’s events name the specific reason:

kubectl describe pvc -n <your-tenant>-<namespace> <pvc-name>

Then match what the events say to one of these causes:

Cause (quota): the tenant’s ResourcePool storage quota is exhausted. Check with kubectl describe resourcepool (it is cluster-scoped — the -n flag is ignored).

Cause (wrong access mode): you requested ReadWriteMany (RWX) on a Cinder storage class. Kestrel provides two OpenStack Cinder storage classes: csi-cinder-sc-delete (the default; reclaim policy Delete) and csi-cinder-sc-retain (reclaim policy Retain). Both are block volumes and support ReadWriteOnce only. ReadWriteMany (shared) volumes are not offered through a storage class — open a ticket with RCS to have a per-tenant NFS volume provisioned.

Cause (unknown class): the storageClassName in your PVC does not match any available class on the cluster. The PVC is accepted anyway — there is no admission check on class names — and stays Pending with no provisioner reacting to it (often with no events at all).

See Storage classes and Persistent volumes.

Insufficient cpu or Insufficient memory — Pod fails to schedule

Cause: the tenant’s ResourcePool quota for CPU or memory is exhausted.

Fix: check current usage:

kubectl describe resourcepool

Common culprits: completed Jobs still holding resources, pending Pods, orphaned PVCs, overprovisioned requests vs actual usage. See Viewing your allocation for diagnosis and Requesting quota changes if you need a larger tier.

exceeded quota in ArgoCD sync

Cause: the resource request in your manifest exceeds the remaining ResourcePool quota for your tenant.

Fix: either reduce the resource requests in your manifest or request a tier upgrade. See Resource pools and quotas for the tier numbers and Requesting quota changes for the upgrade process.