Skip to content

Ray configuration reference

A Ray cluster is configured with the Python classes of the polars_nanook_ray.config module, rooted at PolarsRayClusterConfig. Each actor renders them into the configuration file of a bare-metal node, which describes what each key does. Keys that no field sets can be set with a NANOOK__<SECTION>__<KEY> environment variable on the Ray nodes (see Environment variables). Durations take an ISO 8601 duration such as PT5S or a jiff friendly duration such as 5 secs. Sizes are in bytes.

Cluster

PolarsRayClusterConfig configures the whole cluster. PolarsRayCluster and RayClusterContext take it as their only argument.

Field Default Description
license required One of the license classes.
scheduler required A PolarsSchedulerConfig.
worker required A PolarsWorkerConfig, shared by all workers.
cluster_id "polars-nanook" ID of the cluster, and the Ray namespace of its actors. Letters, digits, dots, dashes and underscores, starting with a letter or digit.
num_workers 0 Number of workers to start.
min_workers 0 Minimum size of the worker pool with autoscaling.
max_workers unbounded Maximum size of the worker pool with autoscaling.
runtime PolarsSubprocessConfig() How the actors run polars-nanook-compute, see Runtimes.
monitoring enabled A PolarsMonitoringConfig: enabled and disable_host_metrics. See Host metrics.
lineage disabled A PolarsLineageConfig, with http=LineageHttpConfig(endpoint=...) or stdio=LineageStdioConfig(output_stream="stdout"). See Emit OpenLineage events.
accept_eula False Accepts the EULA, required with a license file. Sets POLARS_EULA_ACCEPTED.
log_level "info" Log level: info, debug or trace. Sets PLC_LOG_LEVEL.
rust_backtrace "full" Backtraces of a crash: 0, 1 or full. Sets RUST_BACKTRACE.
startup_timeout 30 Seconds start() waits for the scheduler and the workers to be ready.
actor_response_timeout 30 Seconds a call into an actor may take, such as an address lookup or a scaling request.
worker_max_restarts 3 Number of times Ray restarts a worker actor that died, -1 for no limit.

Scheduler

PolarsSchedulerConfig configures the scheduler actor, which runs the scheduler and the observatory on the head node.

Field Default Description
cpus_hint 1 CPUs Ray reserves for the actor. See Worker resources and memory.
memory_hint 1 GiB Memory Ray reserves for the actor.
cpu_reserved unset cpu_reserved of the node, a scheduling hint.
memory_limit unset memory_limit of the node, enforced in a delegated cgroup.
client_port 5051 Port clients submit queries to.
worker_registration_port 5050 Port workers register on.
anonymous_result LocalStorageConfig(path="/tmp/polars/anonymous-results") Where stored results go, a storage location.
allow_local_scans True Whether queries may read the workers' local disk.
allow_local_sinks True Whether queries may write to the workers' local disk.
deny_anonymous_users False Rejects queries submitted without a username.
default_workers_per_query num_workers, else min_workers, else 1 with autoscaling Workers a query uses when it does not request a count. See Scale a query.
max_workers_per_query max_workers, else num_workers Maximum number of workers a query may use; unbounded when all three are unset.
default_partitions_per_worker unset Partitions per worker when the query does not set them.
checkpoint disabled A PolarsCheckpointConfig(period=...) that enables checkpointing at the period given.
scaling disabled A PolarsScalingConfig, see Autoscaling.
observatory enabled A PolarsObservatoryConfig, see Observatory.
connection unset A PolarsTlsConnectionConfig(certificate_key_file=..., private_key_file=...) that requires TLS on the client port.
auth unset A PolarsJwksAuthConfig(url=..., issuers=..., audience=...) that requires a JWT, validated against a JWKS endpoint, on the client port.

Observatory

PolarsObservatoryConfig configures the observatory, the cluster dashboard. See Monitoring and query history.

Field Default Description
enabled True Runs the observatory.
database_path "/tmp/polars/observatory/data.db" SQLite database of the query history. Its directory must exist.
max_metrics_bytes_total 100 MiB Size of the host metrics buffer.
database_export unset Path the metrics are exported to periodically.
rest_port 3001 Port of the dashboard and its REST API.
otlp_port 5049 Port of the OpenTelemetry receiver.
rest_connection unset A PolarsTlsConnectionConfig that requires TLS on the dashboard port.
rest_auth unset A PolarsJwksAuthConfig that requires a JWT, validated against a JWKS endpoint, on the dashboard.

Autoscaling

PolarsScalingConfig(enabled=True) starts an autoscaling actor on the head node. The scheduler sends its scaling requests to that actor on port (4001 by default), and the actor adds and removes worker actors between min_workers and max_workers. The port only listens on the loopback interface, or on the Docker network gateway under the Docker runtime. See Autoscaling.

Worker

PolarsWorkerConfig configures every worker actor.

Field Default Description
cpus_hint 1 CPUs Ray reserves for each worker. See Worker resources and memory.
memory_hint 2 GiB Memory Ray reserves for each worker.
cpu_reserved unset cpu_reserved of the node, a scheduling hint.
memory_limit unset memory_limit of the node, enforced in a delegated cgroup.
task_port 5052 Port the scheduler sends tasks to.
shuffle_port 5053 Port the other workers fetch shuffle data from.
shuffle_data LocalStorageConfig(path="/tmp/polars/shuffle-data") Where shuffle data goes, a storage location.
checkpoint_location unset Where checkpoints go, a shared filesystem or object storage location. Enables checkpointing with scheduler.checkpoint.
temporary_data_dir "/tmp/polars/temporary-data" Directory the worker spills to. Sets POLARS_TEMP_DIR.
env_vars {} Environment variables for Polars and user code in the queries, such as {"POLARS_MAX_THREADS": "8"}.
heartbeat_period_secs 5 Seconds between two heartbeats of a worker to the scheduler.

Licenses

The license field takes one of these classes. See Connect to your workspace and Install and license.

Class Fields License
PolarsServiceAccountLicenseConfig workspace_id, client_id, client_secret, cert_dir ("/tmp/polars/license") A service account of a connected cluster.
PolarsEnterpriseLicenseConfig license_path An Enterprise license file, on every node. Needs accept_eula=True.
PolarsLicenseServerConfig uri An offline license server, such as license_server.get_bind_addr().

Storage locations

scheduler.anonymous_result, worker.shuffle_data and worker.checkpoint_location take one of these classes. options takes the same keys as the storage_options of scan_parquet(), such as aws_access_key_id; see Object storage.

Class Fields Location
LocalStorageConfig path A directory on the local disk of each node.
SharedFilesystemStorageConfig path A shared filesystem such as NFS or CephFS, mounted at the same path on each node.
S3LocationConfig url, options, external_endpoint, presign_duration, allow_deletes S3 or a store with the S3 API, such as MinIO with external_endpoint.
GcsLocationConfig url, options, presign_duration, allow_deletes Google Cloud Storage.
AbsLocationConfig url, options, presign_duration, allow_deletes Azure Blob Storage.

Runtimes

The runtime field takes one of these classes. See Runtimes.

Class Field Default Description
PolarsSubprocessConfig binary_path the package Executable to run. Unset, the polars-nanook-compute command of the node's Python environment.
single_host_cluster False Offsets the ports of each worker, so several workers run on one node. See Actor placement.
PolarsDockerConfig image required Image that runs polars-nanook-compute as its entrypoint.
network "polars-nanook-ray" Docker network the containers join, created if missing.
extra_volume_mounts [] Host paths mounted into the containers, for local paths that queries scan or sink and the configuration does not name.

License server

PolarsRayLicenseServer (polars_nanook_ray.license_server) runs an offline license server as an actor on the head node, which any number of air-gapped Ray clusters validate their license against. It is not part of a cluster: start it once, in its own namespace, before the clusters that use it. Like a cluster, it reconnects to the running actor when started again from another process.

import ray

from polars_nanook_ray.config import PolarsRayLicenseServerConfig, PolarsSubprocessConfig
from polars_nanook_ray.license_server import PolarsRayLicenseServer

ray.init(address="auto", namespace="polars-nanook-license-server")

license_server = PolarsRayLicenseServer(
    PolarsRayLicenseServerConfig(
        runtime=PolarsSubprocessConfig(binary_path="/opt/polars/polars-nanook-license-server"),
        report_dir="/var/lib/polars/license-server",
        license_path="/etc/polars/license.json",
        tls_bundle_path="/etc/polars/tls-bundle.pem",
    )
)
license_server.start()

print(license_server.get_bind_addr())

The license server is a binary, so the subprocess runtime needs its binary_path. See Ray package for where to get it. Clusters point their license at PolarsLicenseServerConfig(uri=...) with the address get_bind_addr() returns. Under the Docker runtime, clusters reach the license server by container name, so they run in containers on the same Docker network.

Field Default Description
report_dir required Directory of the state database and of the signed daily usage reports.
license_path required Enterprise license file.
tls_bundle_path required PEM bundle with the TLS material of the license server.
runtime PolarsSubprocessConfig(binary_path="polars-nanook-license-server") How the actor runs the binary, see Runtimes.
grpc_port 50051 TLS port the clusters register with, the address get_bind_addr() returns.
http_port 8081 Port of /healthz, /readyz and the Prometheus /metrics of get_metrics_addr().
db_disable_file_locking False Opens the state database without file locking, only safe with a single writer.
cpus_hint 1 CPUs Ray reserves for the actor.
memory_hint 50 MiB Memory Ray reserves for the actor.
startup_timeout 30 Seconds start() waits for the license server to be ready.
actor_response_timeout 30 Seconds a call into the actor may take.