Ray configuration reference
A Ray cluster is configured with the Python classes of the polars_nanook_ray.config module, rooted
at PolarsRayClusterConfig. Each actor renders them into the configuration file of a
bare-metal node, which describes what each key does. Keys that no
field sets can be set with a NANOOK__<SECTION>__<KEY> environment variable on the Ray nodes (see
Environment variables). Durations take an ISO 8601 duration such
as PT5S or a jiff friendly duration such as
5 secs. Sizes are in bytes.
Cluster
PolarsRayClusterConfig configures the whole cluster. PolarsRayCluster and RayClusterContext
take it as their only argument.
| Field | Default | Description |
|---|---|---|
license |
required | One of the license classes. |
scheduler |
required | A PolarsSchedulerConfig. |
worker |
required | A PolarsWorkerConfig, shared by all workers. |
cluster_id |
"polars-nanook" |
ID of the cluster, and the Ray namespace of its actors. Letters, digits, dots, dashes and underscores, starting with a letter or digit. |
num_workers |
0 |
Number of workers to start. |
min_workers |
0 |
Minimum size of the worker pool with autoscaling. |
max_workers |
unbounded | Maximum size of the worker pool with autoscaling. |
runtime |
PolarsSubprocessConfig() |
How the actors run polars-nanook-compute, see Runtimes. |
monitoring |
enabled | A PolarsMonitoringConfig: enabled and disable_host_metrics. See Host metrics. |
lineage |
disabled | A PolarsLineageConfig, with http=LineageHttpConfig(endpoint=...) or stdio=LineageStdioConfig(output_stream="stdout"). See Emit OpenLineage events. |
accept_eula |
False |
Accepts the EULA, required with a license file. Sets POLARS_EULA_ACCEPTED. |
log_level |
"info" |
Log level: info, debug or trace. Sets PLC_LOG_LEVEL. |
rust_backtrace |
"full" |
Backtraces of a crash: 0, 1 or full. Sets RUST_BACKTRACE. |
startup_timeout |
30 |
Seconds start() waits for the scheduler and the workers to be ready. |
actor_response_timeout |
30 |
Seconds a call into an actor may take, such as an address lookup or a scaling request. |
worker_max_restarts |
3 |
Number of times Ray restarts a worker actor that died, -1 for no limit. |
Scheduler
PolarsSchedulerConfig configures the scheduler actor, which runs the scheduler and the observatory
on the head node.
| Field | Default | Description |
|---|---|---|
cpus_hint |
1 |
CPUs Ray reserves for the actor. See Worker resources and memory. |
memory_hint |
1 GiB | Memory Ray reserves for the actor. |
cpu_reserved |
unset | cpu_reserved of the node, a scheduling hint. |
memory_limit |
unset | memory_limit of the node, enforced in a delegated cgroup. |
client_port |
5051 |
Port clients submit queries to. |
worker_registration_port |
5050 |
Port workers register on. |
anonymous_result |
LocalStorageConfig(path="/tmp/polars/anonymous-results") |
Where stored results go, a storage location. |
allow_local_scans |
True |
Whether queries may read the workers' local disk. |
allow_local_sinks |
True |
Whether queries may write to the workers' local disk. |
deny_anonymous_users |
False |
Rejects queries submitted without a username. |
default_workers_per_query |
num_workers, else min_workers, else 1 with autoscaling |
Workers a query uses when it does not request a count. See Scale a query. |
max_workers_per_query |
max_workers, else num_workers |
Maximum number of workers a query may use; unbounded when all three are unset. |
default_partitions_per_worker |
unset | Partitions per worker when the query does not set them. |
checkpoint |
disabled | A PolarsCheckpointConfig(period=...) that enables checkpointing at the period given. |
scaling |
disabled | A PolarsScalingConfig, see Autoscaling. |
observatory |
enabled | A PolarsObservatoryConfig, see Observatory. |
connection |
unset | A PolarsTlsConnectionConfig(certificate_key_file=..., private_key_file=...) that requires TLS on the client port. |
auth |
unset | A PolarsJwksAuthConfig(url=..., issuers=..., audience=...) that requires a JWT, validated against a JWKS endpoint, on the client port. |
Observatory
PolarsObservatoryConfig configures the observatory, the cluster dashboard. See
Monitoring and query history.
| Field | Default | Description |
|---|---|---|
enabled |
True |
Runs the observatory. |
database_path |
"/tmp/polars/observatory/data.db" |
SQLite database of the query history. Its directory must exist. |
max_metrics_bytes_total |
100 MiB | Size of the host metrics buffer. |
database_export |
unset | Path the metrics are exported to periodically. |
rest_port |
3001 |
Port of the dashboard and its REST API. |
otlp_port |
5049 |
Port of the OpenTelemetry receiver. |
rest_connection |
unset | A PolarsTlsConnectionConfig that requires TLS on the dashboard port. |
rest_auth |
unset | A PolarsJwksAuthConfig that requires a JWT, validated against a JWKS endpoint, on the dashboard. |
Autoscaling
PolarsScalingConfig(enabled=True) starts an autoscaling actor on the head node. The scheduler
sends its scaling requests to that actor on port (4001 by default), and the actor adds and
removes worker actors between min_workers and max_workers. The port only listens on the loopback
interface, or on the Docker network gateway under the Docker runtime. See
Autoscaling.
Worker
PolarsWorkerConfig configures every worker actor.
| Field | Default | Description |
|---|---|---|
cpus_hint |
1 |
CPUs Ray reserves for each worker. See Worker resources and memory. |
memory_hint |
2 GiB | Memory Ray reserves for each worker. |
cpu_reserved |
unset | cpu_reserved of the node, a scheduling hint. |
memory_limit |
unset | memory_limit of the node, enforced in a delegated cgroup. |
task_port |
5052 |
Port the scheduler sends tasks to. |
shuffle_port |
5053 |
Port the other workers fetch shuffle data from. |
shuffle_data |
LocalStorageConfig(path="/tmp/polars/shuffle-data") |
Where shuffle data goes, a storage location. |
checkpoint_location |
unset | Where checkpoints go, a shared filesystem or object storage location. Enables checkpointing with scheduler.checkpoint. |
temporary_data_dir |
"/tmp/polars/temporary-data" |
Directory the worker spills to. Sets POLARS_TEMP_DIR. |
env_vars |
{} |
Environment variables for Polars and user code in the queries, such as {"POLARS_MAX_THREADS": "8"}. |
heartbeat_period_secs |
5 |
Seconds between two heartbeats of a worker to the scheduler. |
Licenses
The license field takes one of these classes. See
Connect to your workspace and
Install and license.
| Class | Fields | License |
|---|---|---|
PolarsServiceAccountLicenseConfig |
workspace_id, client_id, client_secret, cert_dir ("/tmp/polars/license") |
A service account of a connected cluster. |
PolarsEnterpriseLicenseConfig |
license_path |
An Enterprise license file, on every node. Needs accept_eula=True. |
PolarsLicenseServerConfig |
uri |
An offline license server, such as license_server.get_bind_addr(). |
Storage locations
scheduler.anonymous_result, worker.shuffle_data and worker.checkpoint_location take one of
these classes. options takes the same keys as the storage_options of
scan_parquet(),
such as aws_access_key_id; see Object storage.
| Class | Fields | Location |
|---|---|---|
LocalStorageConfig |
path |
A directory on the local disk of each node. |
SharedFilesystemStorageConfig |
path |
A shared filesystem such as NFS or CephFS, mounted at the same path on each node. |
S3LocationConfig |
url, options, external_endpoint, presign_duration, allow_deletes |
S3 or a store with the S3 API, such as MinIO with external_endpoint. |
GcsLocationConfig |
url, options, presign_duration, allow_deletes |
Google Cloud Storage. |
AbsLocationConfig |
url, options, presign_duration, allow_deletes |
Azure Blob Storage. |
Runtimes
The runtime field takes one of these classes. See Runtimes.
| Class | Field | Default | Description |
|---|---|---|---|
PolarsSubprocessConfig |
binary_path |
the package | Executable to run. Unset, the polars-nanook-compute command of the node's Python environment. |
single_host_cluster |
False |
Offsets the ports of each worker, so several workers run on one node. See Actor placement. | |
PolarsDockerConfig |
image |
required | Image that runs polars-nanook-compute as its entrypoint. |
network |
"polars-nanook-ray" |
Docker network the containers join, created if missing. | |
extra_volume_mounts |
[] |
Host paths mounted into the containers, for local paths that queries scan or sink and the configuration does not name. |
License server
PolarsRayLicenseServer (polars_nanook_ray.license_server) runs an offline license server as an
actor on the head node, which any number of air-gapped Ray clusters
validate their license against. It is not part of a cluster: start it once, in its own namespace,
before the clusters that use it. Like a cluster, it reconnects to the running actor when started
again from another process.
import ray
from polars_nanook_ray.config import PolarsRayLicenseServerConfig, PolarsSubprocessConfig
from polars_nanook_ray.license_server import PolarsRayLicenseServer
ray.init(address="auto", namespace="polars-nanook-license-server")
license_server = PolarsRayLicenseServer(
PolarsRayLicenseServerConfig(
runtime=PolarsSubprocessConfig(binary_path="/opt/polars/polars-nanook-license-server"),
report_dir="/var/lib/polars/license-server",
license_path="/etc/polars/license.json",
tls_bundle_path="/etc/polars/tls-bundle.pem",
)
)
license_server.start()
print(license_server.get_bind_addr())
The license server is a binary, so the subprocess runtime needs its binary_path. See
Ray package for where to get it. Clusters point their license at
PolarsLicenseServerConfig(uri=...) with the address get_bind_addr() returns. Under the Docker
runtime, clusters reach the license server by container name, so they run in containers on the same
Docker network.
| Field | Default | Description |
|---|---|---|
report_dir |
required | Directory of the state database and of the signed daily usage reports. |
license_path |
required | Enterprise license file. |
tls_bundle_path |
required | PEM bundle with the TLS material of the license server. |
runtime |
PolarsSubprocessConfig(binary_path="polars-nanook-license-server") |
How the actor runs the binary, see Runtimes. |
grpc_port |
50051 |
TLS port the clusters register with, the address get_bind_addr() returns. |
http_port |
8081 |
Port of /healthz, /readyz and the Prometheus /metrics of get_metrics_addr(). |
db_disable_file_locking |
False |
Opens the state database without file locking, only safe with a single writer. |
cpus_hint |
1 |
CPUs Ray reserves for the actor. |
memory_hint |
50 MiB | Memory Ray reserves for the actor. |
startup_timeout |
30 |
Seconds start() waits for the license server to be ready. |
actor_response_timeout |
30 |
Seconds a call into the actor may take. |