Tenant provisioning

Overview

A “tenant” on Kestrel is a Capsule Tenant backed by a Keycloak group. You do not hand-author a Tenant CR. Instead, you add a per-tenant values file under the environments hierarchy and the rcs-tenants ApplicationSet renders the Capsule Tenant, TenantOwner, and ResourcePool (plus any optional extraObjects) from the shared tenant Helm chart. The default network policies and limit ranges are not part of the chart — they are applied cluster-wide to all tenants by GlobalTenantResources in the capsule addon config.

Provisioning a new tenant means:

1. Ensuring the tenant’s Keycloak group exists (authN — see the note below).
2. Adding environments/arbutus/kestrel/<cluster>/tenants/<rap-group>/values.yaml (often empty) and merging it (authZ + namespace limits, applied by the chart).

Pre-flight

Before starting, confirm:

• You have cluster-admin access to the target Kestrel cluster.
• The tenant name is the exact Cloud RAP POSIX group for the project. RAP/Cloud-RAP group names may contain uppercase characters — use the group as-is (real example: CCInternal-RAID).
• The requesting PI or lead has been added to the RCS intake tracker.

Note

The tenant directory basename is the tenant identity. The Capsule Tenant name is the basename lowercased (via the chart’s name helper / basenameNormalized), while the owner OIDC group is oidc:<basename> with the basename’s case preserved. For example, a directory CCInternal-RAID/ produces a Tenant named ccinternal-raid whose owner group is oidc:CCInternal-RAID.

Step 1 — Ensure the Keycloak group exists

The tenant’s owner group is the Keycloak group whose name matches the tenant basename. Membership is managed in Keycloak.

Caution

The membership / group lifecycle for tenants (creating and disabling Keycloak groups) is part of the identity layer, which is currently being reworked. Verify the current group-management procedure with the RCS identity owner before relying on the steps below.

1. Log into the Keycloak admin console for the Kestrel realm.
2. Confirm a group exists matching the tenant name (e.g. CCInternal-RAID), or create it.
3. Add the initial members (at minimum the requesting PI or lead).

Step 2 — Add the tenant values file

Create the per-tenant values file the rcs-tenants ApplicationSet expects:

environments/arbutus/kestrel/<cluster>/tenants/<rap-group>/values.yaml

For example, for the production cluster:

environments/arbutus/kestrel/kestrel-prod/tenants/CCInternal-RAID/values.yaml

For most tenants this file is empty (or near-empty) — the platform defaults already cover everything a standard tenant needs:

---

Those defaults come from three different layers — edit the right one if a change is needed:

Set in the cluster-group tenant values (environments/arbutus/default/tenants/values.yaml, rendered by the tenant chart):

• Priority classes — uber-user-significant, uber-user-preempt-medium, uber-user-preempt-high.
• Resource pool — cluster-scoped, resourcePool.tier: standard. The pool is shared across all of the tenant’s namespaces.

Applied cluster-wide by the capsule addon’s GlobalTenantResources (environments/arbutus/kestrel/default/addons/capsule/ — not the tenant chart):

• Network policies — inbound default-deny with pre-approved ingress allows: the traefik namespace, scrape from the monitoring namespace, intra-namespace, and intra-tenant traffic. Outbound traffic is not restricted by the default policy.
• Limit range defaults — limit cpu 1 / memory 1Gi; request cpu 100m / memory 128Mi.

Supplied by cluster addons (independent of the tenant spec, which restricts neither):

• Storage classes — csi-cinder-sc-delete (default) and csi-cinder-sc-retain, from the openstack-cinder-csi chart installed at cluster build (the ubernetes kubespray repo).
• Ingress class — traefik (the cluster default), from the Traefik addon.

Add per-tenant overrides to this file only when the tenant needs something beyond the defaults (for example, a static NFS PersistentVolume for ReadWriteMany storage, as the mlp-explorer tenant does).

Commit and merge — the rcs-tenants ApplicationSet renders an Application named tenant-<rap-group-lowercased> (via basenameNormalized, e.g. CCInternal-RAID/ → tenant-ccinternal-raid) and ArgoCD picks up the merge on its next Git poll (~3 minutes) and syncs the Tenant.

Post-provisioning checks

After the ArgoCD sync completes:

1. kubectl get tenant <tenant-name> — status should be Active. (<tenant-name> is the lowercased basename, e.g. ccinternal-raid.)
2. kubectl auth can-i create namespace --as=test-user --as-group=oidc:<rap-group> — should return yes. kubectl rejects --as-group without --as, so a user must be impersonated too — any placeholder username works. The oidc: prefix is required and the basename case is preserved (e.g. --as=test-user --as-group=oidc:CCInternal-RAID).
3. Ask the tenant lead to log in via kubectl oidc-login and create a test namespace — this validates the full Keycloak → Capsule → RBAC chain.
4. Verify the namespaces the tenant owns: kubectl get tenant <tenant-name> -o jsonpath='{.status.namespaces}' returns the array of namespace names; kubectl get tenant <tenant-name> -o jsonpath='{.status.size}' returns the namespace count.
5. Confirm the resource pool: kubectl describe resourcepool <tenant-name> (the resource pool is cluster-scoped, so do not pass -n — it is ignored). For per-namespace usage, use kubectl describe resourcequota -n <ns>.

Deprovisioning

To remove a tenant:

1. Delete all namespaces owned by the tenant (Capsule cascades resource cleanup).
2. Remove the tenant’s tenants/<rap-group>/ directory from ubernetes-applications and merge. This deletes the generated Application, but it does not prune the rendered resources — the rcs-tenants ApplicationSet sets syncPolicy.preserveResourcesOnDeletion: true, so the Tenant, ResourcePool, and TenantOwner CRs are orphaned, not deleted.
3. Remove the orphaned CRs manually. Every Tenant is rendered with preventDeletion: true, which blocks deletion until cleared:
   • kubectl patch tenant <tenant-name> --type merge -p '{"spec":{"preventDeletion":false}}'
   • kubectl delete tenant <tenant-name>
   • kubectl delete resourcepool <tenant-name>
   • kubectl delete tenantowner -l projectcapsule.dev/tenant=<tenant-name> (TenantOwner names carry a hash suffix, so delete by label).
4. Verify with kubectl get tenant — the tenant should no longer be listed.
5. Disable (do not delete) the Keycloak group — keeps the audit trail intact. (See the identity-rework caution in Step 1.)