Tenant provisioning

Overview

A “tenant” on Kestrel is a Capsule Tenant backed by a Keycloak group. You do not hand-author a Tenant CR. Instead, you add a per-tenant values file under the environments hierarchy and the rcs-tenants ApplicationSet renders the Capsule Tenant (plus its quotas, network policies, limit ranges, and resource pool) from the shared tenant Helm chart.

Provisioning a new tenant means:

1. Ensuring the tenant’s Keycloak group exists (authN — see the note below).
2. Adding environments/arbutus/kestrel/<cluster>/tenants/<rap-group>/values.yaml (often empty) and merging it (authZ + namespace limits, applied by the chart).

Pre-flight

Before starting, confirm:

• You have cluster-admin access to the target Kestrel cluster.
• The tenant name is the exact Cloud RAP POSIX group for the project. RAP/Cloud-RAP group names may contain uppercase characters — use the group as-is (real example: CCInternal-RAID).
• The requesting PI or lead has been added to the RCS intake tracker.

Note

The tenant directory basename is the tenant identity. The Capsule Tenant name is the basename lowercased (via the chart’s name helper / basenameNormalized), while the owner OIDC group is oidc:<basename> with the basename’s case preserved. For example, a directory CCInternal-RAID/ produces a Tenant named ccinternal-raid whose owner group is oidc:CCInternal-RAID.

Step 1 — Ensure the Keycloak group exists

The tenant’s owner group is the Keycloak group whose name matches the tenant basename. Membership is managed in Keycloak.

1. Log into the Keycloak admin console for the Kestrel realm.
2. Confirm a group exists matching the tenant name (e.g. CCInternal-RAID), or create it.
3. Add the initial members (at minimum the requesting PI or lead).

Caution

The membership / group lifecycle for tenants (creating and disabling Keycloak groups) is part of the identity layer, which is currently being reworked. Verify the current group-management procedure with the RCS identity owner before relying on the steps above.

Step 2 — Add the tenant values file

Create the per-tenant values file the rcs-tenants ApplicationSet expects:

environments/arbutus/kestrel/<cluster>/tenants/<rap-group>/values.yaml

For example, for the production cluster:

environments/arbutus/kestrel/kestrel-prod/tenants/CCInternal-RAID/values.yaml

For most tenants this file is empty (or near-empty) — the cluster-group defaults already supply the storage classes, ingress class, priority classes, resource pool, network policies, and limit ranges:

---

The defaults inherited from the cluster-group values include:

• Resource pool — cluster-scoped, resourcePool.tier: standard. The pool is shared across all of the tenant’s namespaces.
• Network policies — default-deny, allowing ingress from the traefik namespace, scrape from the monitoring namespace, intra-tenant traffic, and DNS. There is no general egress allow.
• Limit range defaults — limit cpu 1 / memory 1Gi; request cpu 100m / memory 128Mi.
• Storage classes — csi-cinder-sc-delete (default) and csi-cinder-sc-retain.
• Priority classes — uber-user-significant, uber-user-preempt-medium, uber-user-preempt-high.

Add per-tenant overrides to this file only when the tenant needs something beyond the defaults (for example, a static NFS PersistentVolume for ReadWriteMany storage, as the mlp-explorer tenant does).

Commit and merge — the rcs-tenants ApplicationSet renders tenant-<rap-group> and ArgoCD syncs the Tenant within ~60 seconds.

Post-provisioning checks

After the ArgoCD sync completes:

1. kubectl get tenant <tenant-name> — status should be Active. (<tenant-name> is the lowercased basename, e.g. ccinternal-raid.)
2. kubectl auth can-i create namespace --as-group=oidc:<rap-group> — should return yes. The oidc: prefix is required and the basename case is preserved (e.g. --as-group=oidc:CCInternal-RAID).
3. Ask the tenant lead to log in via kubectl oidc-login and create a test namespace — this validates the full Keycloak → Capsule → RBAC chain.
4. Verify the namespaces the tenant owns: kubectl get tenant <tenant-name> -o jsonpath='{.status.namespaces}' returns the array of namespace names; kubectl get tenant <tenant-name> -o jsonpath='{.status.size}' returns the namespace count.
5. Confirm the resource pool: kubectl describe resourcepool <tenant-name> (the resource pool is cluster-scoped, so do not pass -n — it is ignored). For per-namespace usage, use kubectl describe resourcequota -n <ns>.

Deprovisioning

To remove a tenant:

1. Delete all namespaces owned by the tenant (Capsule cascades resource cleanup).
2. Remove the tenant’s tenants/<rap-group>/ directory from ubernetes-applications and merge — the ApplicationSet prunes the rendered Tenant.
3. Disable (do not delete) the Keycloak group — keeps the audit trail intact. (See the identity-rework caution in Step 1.)