Break and fix

How to use this page

Each entry follows a symptom → cause → fix format. Check the symptom heading, confirm the cause matches, then apply the fix. If the issue is not listed here, escalate to RCS via the channels on the Support and contact page.

---

Pod stuck in Pending

Symptom: A tenant reports pods staying in Pending for more than 5 minutes.

Likely cause: The namespace has hit its ResourceQuota ceiling, or the cluster is out of schedulable capacity.

Fix:

1. kubectl describe pod <pod> -n <ns> — check the Events section for FailedScheduling.
2. If quota-related: kubectl get resourcequota -n <ns> — compare used vs hard.
3. If cluster-capacity-related: the FailedScheduling event message states the shortfall (Insufficient cpu / Insufficient memory). Confirm with kubectl describe nodes — check the Allocated resources section for nodes near 100% of allocatable requests. (kubectl top nodes shows live usage, not requests, so it cannot confirm schedulability.)
4. For quota issues, the tenant admin can request a quota increase via the process in the tenant-admin docs. For capacity issues, escalate to the RCS infrastructure team.

---

Namespace creation denied

Symptom: A tenant owner gets namespaces "foo" is forbidden when creating a namespace, or an admission webhook denial: The namespace doesn't match the tenant prefix, expected prefix "<tenant>".

Likely cause: The namespace name does not start with the tenant prefix (every tenant has forceTenantPrefix: true — this is the most common cause), the Capsule namespace quota is exhausted, or the user’s Keycloak group mapping is stale.

Fix:

1. Check the name: it must start with the tenant prefix (e.g. <tenant>-myproject). A bare name like foo is rejected by the Capsule webhook.
2. kubectl get tenant <tenant-name> -o jsonpath='{.status.size}' — compare with spec.namespaceOptions.quota. Note that namespaceOptions is {} in the chart defaults, so this field is only present for tenants with an explicit namespace-quota override — most tenants have no namespace quota.
3. If at quota: the tenant needs to delete unused namespaces or request a quota increase.
4. If neither applies: check that the user’s identity resolves the right groups — have them run kubectl auth whoami and confirm the Groups row contains oidc:<tenant>. For the raw token payload, kubectl oidc-login with --v=6.

---

ArgoCD Application stuck OutOfSync

Symptom: An Application shows OutOfSync indefinitely after a merge.

Likely cause: A schema validation error in the rendered manifests, or a webhook is rejecting the resource.

Fix:

1. kubectl -n argo-cd describe application <app-name> — read the sync error message.
2. If schema error: check the diff in the ArgoCD UI — usually a CRD version mismatch or a removed field.
3. If webhook rejection: kubectl get events -n <target-ns> --sort-by=.lastTimestamp — look for admission webhook errors.
4. Fix the manifest in ubernetes-applications, merge, and wait for the next sync cycle.

---

Certificate not issuing

Symptom: The Certificate for a new Ingress shows READY: False (kubectl get certificate -n <ns>), or the browser shows a TLS warning.

Likely cause: The hostname’s DNS record does not exist yet, or cert-manager cannot complete the ACME challenge (rate-limit hit).

DNS records are currently created manually

Automated DNS for tenant hostnames (external-dns) is not live yet. Creating an Ingress does not create the matching DNS record — tenant hostname records are created manually by RCS via ticket, and the certificate stays READY: False until the record exists. A missing record never “propagates” — waiting does not fix it.

Fix:

1. dig <hostname> — if the record does not resolve, it has not been created yet. Create it via the manual RCS DNS process (or confirm the tenant’s hostname ticket has been actioned). cert-manager retries and issues the certificate once the record exists — no further action needed.
2. kubectl get certificate -n <ns> — check the READY column.
3. kubectl describe certificaterequest -n <ns> — look for ACME solver errors.
4. kubectl logs -n cert-manager deploy/cert-manager -c cert-manager --tail=50 — check for rate-limit or authorization failures.
5. If rate-limited: check Let’s Encrypt rate limit status and wait for the window to reset.

---

Node NotReady

Symptom: kubectl get nodes shows one or more nodes as NotReady.

Likely cause: The kubelet has stopped reporting, usually due to resource exhaustion (disk pressure, memory pressure) or a VM-level issue on the OpenStack host.

Fix:

1. kubectl describe node <node> — check Conditions for DiskPressure, MemoryPressure, or NetworkUnavailable.
2. If disk pressure: identify and clean up large files (container images, logs). crictl rmi --prune removes unused images.
3. If the node is unreachable: check the VM status in the OpenStack dashboard. Reboot the VM if it is in an error state.
4. If the node does not recover within 15 minutes, cordon and drain it: kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.