Skip to content

Monitoring and query history

The observatory runs with the scheduler and serves the cluster dashboard. It keeps the cluster's query history and the host metrics (CPU and memory usage) that the monitoring component sends from every node.

Query history

The observatory stores profiling data and query history in an SQLite database file of several tens of MB. SQLite does not allow concurrent access, so a database serves only one cluster at a time.

By default the database is in an emptyDir. observatory.persistentVolumeClaim keeps it on a persistent volume claim instead, across scheduler pod restarts and reinstalls of the chart. The chart creates the claim with the ReadWriteOnce access mode and keeps it on helm uninstall, so a reinstall under the same release name and namespace picks it up again. On a cluster without a default StorageClass, set observatory.persistentVolumeClaim.storageClassName.

observatory:
  persistentVolumeClaim:
    enabled: true

Enable the observatory on the scheduler node, the only node it can run on, and the monitoring component on every node. Without them, no profiling data is collected. observatory.database_path sets the database file, or, with a trailing /, a directory to create it in. The observatory adds the cluster_id to the file name, and the directory must exist. Restart the cluster with the same path to keep its query history.

[scheduler]
enabled = true

[observatory]
enabled = true
max_metrics_bytes_total = 104857600
database_path = "/opt/db/observatory.db"

[monitoring]
enabled = true

See Network addresses and static leader.

Host metrics

The observatory stores host metrics in a buffer in memory, allocated up front, at about 50 bytes per second per node. It divides the buffer over the node count plus one: the size of a fixed-size Helm chart cluster, the pool maximum of an autoscaled one, or 500 when neither is known. At 500 nodes, 1 hour of history needs about 90 MB (50 * 3600 * 501 bytes). The Helm chart defaults to 104857600 bytes.

observatory.maxMetricsBytesTotal sets the buffer size.

observatory.max_metrics_bytes_total sets the buffer size and is required.

OpenTelemetry and logs

Polars Nanook sends OTLP traces to a collector you configure. Logs are written to standard output and standard error in JSON format.

telemetry.otlpEndpoint sets the OTLP collector endpoint, and logLevel the log level: off, error, warn, info (the default), debug or trace.

The OTLP_ENDPOINT environment variable sets the OTLP collector endpoint, and PLC_LOG_LEVEL the log level. Both are listed on the Environment variables page.