Monitoring on Kestrel

Kestrel runs kube-prometheus-stack in the monitoring namespace. This gives tenants two things:

1. Metrics exposure — commit a ServiceMonitor into your tenant namespace and Prometheus scrapes it automatically.
2. Grafana dashboards — browse your metrics at the platform Grafana instance.

Exposing metrics with ServiceMonitor

Your workload must expose a Prometheus-compatible /metrics endpoint. Kestrel’s Prometheus instance is configured to scrape every ServiceMonitor cluster-wide (serviceMonitorSelectorNilUsesHelmValues: false), so a ServiceMonitor in your tenant namespace is picked up automatically — no RCS intervention required.

Step 1 — Instrument your workload

Your application must serve Prometheus text-format metrics on a named port. Use the official Prometheus client libraries for your language. Kestrel does not provide instrumentation — your code owns the /metrics endpoint.

Step 2 — Expose the metrics port in your Service

Make sure your Service selects the Pods and exposes the metrics port by name:

apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: <your-tenant>-prod
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  ports:
    - name: metrics
      port: 9090
      targetPort: 9090

Step 3 — Commit a ServiceMonitor

Add a ServiceMonitor manifest to your tenant repository. Prometheus picks it up on the next reconciliation cycle.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-metrics
  namespace: <your-tenant>-prod
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

The ServiceMonitor lives in your tenant namespace. Prometheus (in the monitoring namespace) scrapes it cross-namespace — the platform NetworkPolicy already permits this.

Annotation-based scraping does not work

Adding prometheus.io/scrape: "true" annotations to your Pods has no effect on Kestrel. The Prometheus instance is configured to discover targets exclusively through ServiceMonitor and PodMonitor resources. If your metrics are not being scraped, check that you have a ServiceMonitor — not annotations.

PodMonitor alternative

If your workload exposes metrics but does not need a Service for any other reason, you can use a PodMonitor instead. The shape is similar — PodMonitor selects Pods directly by label rather than going through a Service. See the Prometheus Operator PodMonitor docs for the spec.

Grafana

Grafana is at:

https://kestrel.arbutus.cloud/grafana

Logging in

Grafana authenticates through Keycloak — the same Alliance LDAP-backed identity used by ArgoCD and kubelogin. Your group membership carries over.

Click Sign in with Keycloak and authenticate with your Alliance CCDB credentials.

Non-UVic-Staff access gap

On current Kestrel, only users in the UVic Staff COmanage group receive a functional Grafana role at login. Non-UVic-Staff researchers (including most Alliance collaborators) land with a None role — no datasources, no dashboards, no way to query metrics.

If this affects you: open an RCS support ticket requesting Grafana role provisioning for your tenant. RCS can grant access on a per-user basis.

This does not affect metrics collection. Your ServiceMonitor still works and Prometheus still scrapes your workloads regardless of your Grafana access.

Finding your dashboards

Once you have a functional Grafana role, navigate to Dashboards in the left sidebar. Tenant-scoped dashboards are organized by namespace. If you do not see a dashboard for your tenant, confirm that your ServiceMonitor is deployed and that Prometheus is scraping it (check the Targets page under Status → Targets in Grafana).

Alerting

Alertmanager is deployed on Kestrel, but tenant-authored alert rules are not part of v1. If you need alerting on a specific metric, contact RCS to discuss options.

Next steps

• Your repo, your workloads — how to structure your tenant repository for GitOps deployment.
• Workload recipes — Deployment, Job, and dev-pod recipes that pair with ServiceMonitor.
• Priority classes — picking the right priorityClassName for your workloads.