Slurm

A bare-metal cluster can run on a Slurm Workload Manager cluster as an ordinary batch job. One sbatch job starts the polars-nanook-compute scheduler on one node and a worker on each other node via srun. The same job then submits Polars queries with the polars_nanook client.

Docker Compose setup

This page was tested with the containerized setup from this repo, which is representative of a bare, minimal Slurm setup. The only addition is a shared filesystem to exchange files between nodes, represented here by a volume mounted on all containers.

A bare-metal cluster only supports a static leader, so the name and address of the scheduler must be known to all worker nodes at startup. In a real Slurm cluster each node has static characteristics, notably its hostname and IP. These values are usually known cluster-wide through a mapping stored on the filesystem, and a comparison with the SLURM_JOB_NODELIST environment variable of a running job is enough to infer the IP of the service. The sinfo command gives a similar result.

This example assumes neither such a mapping nor unrestricted access to sinfo. Instead, the information about the node hosting the scheduler is fetched and stored on a shared volume, in the file /opt/polars/scheduler.txt. The same volume stores the license.json file and the query results. A cluster with a license file only starts once the EULA is accepted, so the scheduler sets POLARS_EULA_ACCEPTED=1. The uv command needed to set up the Python virtual environment with the polars-nanook-compute package is assumed to be available system-wide.

This page is not an introduction to Slurm. It focuses on a functional minimal setup: two nodes (#SBATCH --nodes=2) and a single batch job running two services (#SBATCH --ntasks=2), each with the cores set by #SBATCH --cpus-per-task (without it, each service gets one core). The complete scripts are at the bottom of the page.

The versions below are an example. Take the polars version of your cluster release from the Releases page. Each polars-nanook-compute and polars-nanook release is pinned to one polars release, so pinning polars installs a matching release of both.

Cluster startup inside a Slurm job

Scheduler node

The scheduler node writes its hostname and IP to /opt/polars/scheduler.txt on the shared volume so worker nodes and the client can find it. The script then generates the configuration (the same single-node configuration as on the Get started on bare-metal page, with the scheduler's IP as public address), creates the Python virtual environment and starts the service.

POLARS_VERSION="2.1.0"
PYTHON_VERSION="3.12"

# Fetch and store scheduler node information
SCHEDULER_ID=$(hostname)
SCHEDULER_IP=$(hostname -I | awk '{print$1}')
echo -e "SCHEDULER_ID=$SCHEDULER_ID\nSCHEDULER_IP=$SCHEDULER_IP" > /opt/polars/scheduler.txt.tmp
mv /opt/polars/scheduler.txt.tmp /opt/polars/scheduler.txt

mkdir -p /tmp/polars/{observatory,shuffle-data-path} /opt/polars/results-data

# Generate the configuration for a scheduler + worker node
cat << EOF > /tmp/polars/config.toml
cluster_id = "polars-cluster"
instance_id = "$SCHEDULER_ID"
license.on_prem_enterprise.license_path = "/opt/polars/license.json"

[scheduler]
enabled = true
allow_local_sinks = true
allow_local_scans = true
anonymous_result_location.local.path = "/opt/polars/results-data"

[worker]
enabled = true
shuffle_location.local.path = "/tmp/polars/shuffle-data-path"
task_service.public_addr = "$SCHEDULER_IP"
shuffle_service.public_addr = "$SCHEDULER_IP"

[observatory]
enabled = true
max_metrics_bytes_total = 8000000
database_path = "/tmp/polars/observatory/"

[monitoring]
enabled = true

[static_leader]
leader_instance_id = "$SCHEDULER_ID"
observatory_service.public_addr = "$SCHEDULER_IP"
scheduler_service.public_addr = "$SCHEDULER_IP"
EOF

# Create and activate the Python virtual environment
uv venv --allow-existing --managed-python --python=$PYTHON_VERSION
source .venv/bin/activate
uv pip install polars[cloudpickle]==$POLARS_VERSION polars-nanook-compute

# Start the service
POLARS_EULA_ACCEPTED=1 polars-nanook-compute service --config-path=/tmp/polars/config.toml

The scheduler node now runs the scheduler, a worker, the observatory and the monitoring components as the job step, and /opt/polars/scheduler.txt holds its hostname and IP for the other nodes.

Observatory & query profiler

If the Slurm cluster allows exposing a frontend, the cluster's observatory and query profiler are reachable in a browser. The HTTP address is the scheduler node, and the port is 3001 (the default, see Network addresses). In the setup described here it remains available only for the duration of the Slurm job.

Worker nodes

The steps for a worker are nearly identical, since the scheduler node information is available from /opt/polars/scheduler.txt. The generated configuration skips the sections a worker does not need.

POLARS_VERSION="2.1.0"
PYTHON_VERSION="3.12"

# Wait for the scheduler node information
until [ -f /opt/polars/scheduler.txt ]; do sleep 1; done
source /opt/polars/scheduler.txt
export SCHEDULER_ID
export SCHEDULER_IP

WORKER_IP=$(hostname -I | awk '{print $1}')
mkdir -p /tmp/polars/{observatory,results-data,shuffle-data-path}

# Generate the configuration for a worker node
cat << EOF > /tmp/polars/config.toml
cluster_id = "polars-cluster"
instance_id = "$(hostname)"
license.on_prem_enterprise.license_path = "/opt/polars/license.json"

[worker]
enabled = true
shuffle_location.local.path = "/tmp/polars/shuffle-data-path"
task_service.public_addr = "$WORKER_IP"
shuffle_service.public_addr = "$WORKER_IP"

[monitoring]
enabled = true

[static_leader]
leader_instance_id = "$SCHEDULER_ID"
observatory_service.public_addr = "$SCHEDULER_IP"
scheduler_service.public_addr = "$SCHEDULER_IP"
EOF

# Create and activate the Python virtual environment
uv venv --allow-existing --managed-python --python=$PYTHON_VERSION
source .venv/bin/activate
uv pip install polars[cloudpickle]==$POLARS_VERSION polars-nanook-compute

# Start the service
polars-nanook-compute service --config-path=/tmp/polars/config.toml

Query submission from the Slurm job

Once the cluster is up, Polars queries can be submitted to the scheduler. The client also needs the scheduler IP, read from the same /opt/polars/scheduler.txt through the same environment variable mechanism as above.

POLARS_VERSION="2.1.0"
PYTHON_VERSION="3.12"

# Make the scheduler node information known
until [ -f /opt/polars/scheduler.txt ]; do sleep 1; done
source /opt/polars/scheduler.txt
export SCHEDULER_IP

# Create and activate the Python virtual environment
uv venv --allow-existing --managed-python --python=$PYTHON_VERSION
source .venv/bin/activate
uv pip install polars[cloudpickle]==$POLARS_VERSION polars-nanook

# Submit query
python query.py

To submit several queries to the same cluster:

POLARS_VERSION="2.1.0"
PYTHON_VERSION="3.12"

# Make the scheduler node information known
until [ -f /opt/polars/scheduler.txt ]; do sleep 1; done
source /opt/polars/scheduler.txt
export SCHEDULER_IP

# Create and activate the Python virtual environment
uv venv --allow-existing --managed-python --python=$PYTHON_VERSION
source .venv/bin/activate
uv pip install polars[cloudpickle]==$POLARS_VERSION polars-nanook

# Submit queries
PID_QUERIES=()

python query1.py & PID_QUERIES+=($!)  # Store query PID
python query2.py & PID_QUERIES+=($!)
python query3.py & PID_QUERIES+=($!)

# Wait for all queries to finish, set job exit code to 1 if any failed
EXIT_CODE=0
for pid in "${PID_QUERIES[@]}"; do
  wait $pid || EXIT_CODE=1  # Store query exit code
done
exit $EXIT_CODE

A query script has the following structure. The ClusterContext API is described under Run a remote query.

import os
import datetime

import polars as pl
import polars_nanook as pn

ctx = pn.ClusterContext(uri="http://"+os.environ["SCHEDULER_IP"])

(
    pl.scan_parquet(...)
    .filter(pl.col("date_column") <= datetime.date(2025, 9, 2))
    .group_by("column_a", "column_b")
    .agg(pl.col("column_c").sum())
    .remote(ctx)
    .sink_parquet(...)
    .await_result()
)

Complete `submit.sh` and `setup-service.sh`

Much of the code above overlaps and can be factored into a script or a set of functions.

The full example below is split in two scripts: the main job, submit.sh, submitted with sbatch, and a helper, setup-service.sh, that srun runs once on the scheduler node and once on every worker node. Put both scripts and the query scripts in one directory on the shared volume and run sbatch submit.sh from there. The shared virtual environment keeps its Python on the shared volume as well (UV_PYTHON_INSTALL_DIR), because uv otherwise installs it in the home directory of the first node.

If the Python virtual environment is not shared between Slurm nodes and is set up in the setup-service.sh helper instead, a sleep after each srun call may be required to give uv time to finish before the services start.

submit.sh: cluster setup and query submission

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=8

INFO_FILE="/opt/polars/scheduler.txt"
VENV_PATH="/opt/polars/venv"

POLARS_VERSION="2.1.0"
PYTHON_VERSION="3.12"

# Create a shared Python virtual environment
export UV_PYTHON_INSTALL_DIR=/opt/polars/python
uv venv --allow-existing --managed-python --python=$PYTHON_VERSION $VENV_PATH
source $VENV_PATH/bin/activate
uv pip install polars[cloudpickle]==$POLARS_VERSION polars-nanook-compute polars-nanook

# Remove the scheduler node information of an earlier job
rm -f "$INFO_FILE"

# Start scheduler + worker on the first node
SERVICE=scheduler srun --nodes=1 --ntasks=1 --relative=0 bash setup-service.sh &

# Start a worker on every other node
WORKERS=$((SLURM_NNODES - 1))
SERVICE=worker srun --nodes=$WORKERS --ntasks=$WORKERS --relative=1 bash setup-service.sh &

# Wait for the scheduler node information
until [ -f "$INFO_FILE" ]; do sleep 1; done
source "$INFO_FILE"
export SCHEDULER_IP

# Submit queries
PID_QUERIES=()

python query1.py & PID_QUERIES+=($!)
python query2.py & PID_QUERIES+=($!)

EXIT_CODE=0
for pid in "${PID_QUERIES[@]}"; do
  wait $pid || EXIT_CODE=1
done
exit $EXIT_CODE

setup-service.sh: service setup

#!/bin/bash

INFO_FILE="/opt/polars/scheduler.txt"
VENV_PATH="/opt/polars/venv"

mkdir -p /tmp/polars/{observatory,shuffle-data-path} /opt/polars/results-data

if [ "$SERVICE" = "scheduler" ]; then

  # Fetch and store scheduler node information
  SCHEDULER_ID=$(hostname)
  SCHEDULER_IP=$(hostname -I | awk '{print$1}')
  echo -e "SCHEDULER_ID=$SCHEDULER_ID\nSCHEDULER_IP=$SCHEDULER_IP" > "$INFO_FILE.tmp"
  mv "$INFO_FILE.tmp" "$INFO_FILE"

  # Generate the configuration for a scheduler + worker node
  cat << EOF > /tmp/polars/config.toml
cluster_id = "polars-cluster"
instance_id = "$SCHEDULER_ID"
license.on_prem_enterprise.license_path = "/opt/polars/license.json"

[scheduler]
enabled = true
allow_local_sinks = true
allow_local_scans = true
anonymous_result_location.local.path = "/opt/polars/results-data"

[worker]
enabled = true
shuffle_location.local.path = "/tmp/polars/shuffle-data-path"
task_service.public_addr = "$SCHEDULER_IP"
shuffle_service.public_addr = "$SCHEDULER_IP"

[observatory]
enabled = true
max_metrics_bytes_total = 8000000
database_path = "/tmp/polars/observatory/"

[monitoring]
enabled = true

[static_leader]
leader_instance_id = "$SCHEDULER_ID"
observatory_service.public_addr = "$SCHEDULER_IP"
scheduler_service.public_addr = "$SCHEDULER_IP"
EOF

elif [ "$SERVICE" = "worker" ]; then

  # Wait for the scheduler node information
  until [ -f "$INFO_FILE" ]; do sleep 1; done
  source "$INFO_FILE"
  WORKER_IP=$(hostname -I | awk '{print$1}')

  # Generate the configuration for a worker node
  cat << EOF > /tmp/polars/config.toml
cluster_id = "polars-cluster"
instance_id = "$(hostname)"
license.on_prem_enterprise.license_path = "/opt/polars/license.json"

[worker]
enabled = true
shuffle_location.local.path = "/tmp/polars/shuffle-data-path"
task_service.public_addr = "$WORKER_IP"
shuffle_service.public_addr = "$WORKER_IP"

[monitoring]
enabled = true

[static_leader]
leader_instance_id = "$SCHEDULER_ID"
observatory_service.public_addr = "$SCHEDULER_IP"
scheduler_service.public_addr = "$SCHEDULER_IP"
EOF

fi

# Activate the shared Python virtual environment
source $VENV_PATH/bin/activate

# Start the service
POLARS_EULA_ACCEPTED=1 polars-nanook-compute service --config-path=/tmp/polars/config.toml

Summary

Leader election: none. Bare-metal deployments use [static_leader], so the scheduler's instance_id and IP must be known to every worker at startup.
Scheduler address: the scheduler node writes SCHEDULER_ID and SCHEDULER_IP to a file on a shared volume (/opt/polars/scheduler.txt) that workers and the client source.
Node count: two (#SBATCH --nodes=2 --ntasks=2), one scheduler+worker node and one worker-only node, each service with the cores of #SBATCH --cpus-per-task.
Observatory dashboard: on the scheduler node, port 3001 by default, and only for the lifetime of the Slurm job.
Requirements: uv with access to PyPI and to the Python downloads (or mirrors of both), and the license.json file.
Job end: once the queries finish, submit.sh exits and Slurm stops the service steps, so sacct lists them as CANCELLED and srun prints Terminated. The job itself completes.