Worker resources and memory
A cluster you run has one scheduler and a pool of workers. Scale a query covers the size of the pool and autoscaling.
Worker resources and placement
The Polars engine uses all available CPU and consumes memory as needed. Polars Nanook performs best on dedicated nodes, with one worker per node.
Size the worker requests to the resources a node has available, which is less than its nominal size because Kubernetes reserves some for system daemons. For example, a node with 4 vCPUs and 16 GiB of memory may have 3770m CPU and 14.31 GiB available. Keep other workloads off the worker nodes with node selectors, taints and tolerations.
Worker resources are under worker.deployment.runtimeContainer.resources, and placement under
worker.deployment.nodeSelector, tolerations and affinity. The scheduler pod takes the same
settings under scheduler.deployment.
worker:
deployment:
runtimeContainer:
resources:
requests:
cpu: 3770m
memory: 14.31Gi
limits:
cpu: 3770m
memory: 14.31Gi
nodeSelector:
dedicated: polars
tolerations:
- key: "dedicated"
operator: "Equal"
value: "polars"
effect: "NoSchedule"
If possible, assign physical cores to the polars-nanook-compute service to avoid contention with
other processes. Memory is limited per node with memory_limit (see
Out-of-memory behavior).
Out-of-memory behavior
A worker consists of a main process and an executor process. The executor does most of the
computation. If it dies, it loses all progress of the stage it was working on, and the scheduler
reschedules that stage. The main process keeps the progress of previous stages, the
shuffle data. The worker sets oom_score_adj on its executor, so that under memory
pressure the kernel kills the executor first.
Set the worker memory limit in worker.deployment.runtimeContainer.resources.limits.memory.
Workers spill data to disk from 80% of that limit by default
(worker.outOfCore.memoryBudgetFraction), into the temporaryData volume. A query whose spill
files exceed worker.outOfCore.diskBudgetMB (default 64000) is aborted. The temporaryData
volume is an emptyDir by default. To size it for the data the largest queries spill, use an
ephemeral volume claim:
temporaryData:
ephemeralVolumeClaim:
enabled: true
storageClassName: <STORAGE_CLASS>
size: <SIZE>
A worker aborts a task once its memory use exceeds worker.outOfCore.oomKillThresholdMB, by
default about 95% of the container memory limit, or of the node's allocatable memory without a
limit. The query then fails with an out-of-memory error, and the worker stays up.
cgroup v2 Kubernetes clusters kill the whole worker on OOM
On cgroup v2 nodes, the kubelet of Kubernetes 1.28 and later sets memory.oom.group=1 on
each container: exceeding the memory limit kills every process in the worker, not only the
executor. Store shuffle data on a shared filesystem or object storage, so the
scheduler reschedules the work on another worker with no data loss.
When other system-critical processes run on the node, delegate a cgroup to the
polars-nanook-compute process and set memory_limit, in bytes as a plain integer, in the
configuration file. The service then creates subgroups and
chooses a memory limit for each of its components.
cluster_id = "polars-cluster"
instance_id = "node-0"
memory_limit = 10737418240 # 10 GiB
# ...