Skip to content

Observability

Everything is under Observability in the site, per controller and time range.

Reconciles, errors, reconcile time and queue depth, plus the controller’s CPU, memory and network. Scraped every 15 seconds from where your controller serves them (:8080/metrics unless you set it) and kept 15 days.

Lines on the charts mark when a controller changed, so you can match a shift in its metrics to a new version. Point at a line, or move to it with the arrow keys, to see what changed, for example image ghcr.io/acme/op:v1 → ghcr.io/acme/op:v2.

  • Solid lines are changes to the controller: deploying it, or a new image, CPU or memory limit, args, env or probes. Args and env show that they changed (env by name), never their values.
  • Dotted lines are restarts, stops and starts, and rollouts by the platform itself, such as a new tunnel or a renewed certificate.

A line marks when the change was made. Metrics follow a little later: the new pod has to start, and rates are averaged over a few minutes. Changes only show from when ctrlplane started recording them, and an image pushed again under the same tag shows as a restart, not a new version.

The same records are in your logs store, under the stream {ctrlplane_source="deploys"}.

Your controllers’ output, with a live tail and search in LogsQL, for example "failed to sync".

What your controllers record about objects, newest first; warnings also show on the overview. Your API server drops events after an hour, but ctrlplane keeps them as long as logs, under the stream {ctrlplane_source="events"}.

Your API server serves this data, pinned to your control plane:

Data source URL
Prometheus https://<id>.api.ctrlplane.run/platform/metrics
VictoriaLogs https://<id>.api.ctrlplane.run/platform/logs
  1. Create a token for a service account (it may only read metrics and logs):

    Terminal window
    kubectl create serviceaccount grafana
    kubectl create token grafana --duration=8760h
  2. In each data source, add the header Authorization: Bearer <token>.

  3. Under TLS, add your control plane’s CA from your kubeconfig:

    Terminal window
    kubectl config view --raw --minify \
    -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d