Get started on Ray
On Ray, the Polars Nanook scheduler and workers run as actors on a Ray cluster you already operate,
started from Python with the polars-nanook-ray package. Start a single-machine cluster, connect it
to your workspace and run a first remote query. Before you start,
install the client and sign up.
You need a Linux machine (x86-64 with AVX2, or ARM64) with at least 16 GiB of memory, and
uv.
Step 1: Create a service account
The cluster registers with your workspace through a service account: a client ID and secret for the workspace.
Open Integrations > Infrastructure in the portal sidebar and click Connect on the Ray card. The Create service account step generates a service account. Copy the client ID and secret: they are shown only once. The portal fills them in, together with your workspace ID, in the script of step 4.
nanook service-account create \
--workspace-name "My workspace" \
--name polars-ray
The command prints the API client ID, the API secret and the Workspace ID you need in step 4. Store the secret: it is shown only once.
Step 2: Install the package
Install the polars-nanook-ray package in a virtual environment. It brings ray, the
polars-nanook client and the polars-nanook-compute package the actors run:
uv venv --python 3.12 && source .venv/bin/activate
uv pip install polars-nanook-ray
mkdir --parents /tmp/polars/{anonymous-results,license,observatory,shuffle-data,temporary-data}
The directories are the default locations of the stored results, the license, the query history, the shuffle data and the temporary data.
Step 3: Start Ray
Start a Ray head node in the same shell:
ray start --head --disable-usage-stats --resources='{"head":1}'
The custom head resource marks the node the scheduler runs on. See
Actor placement.
Step 4: Run a test query
This script starts a scheduler and two workers as Ray actors, runs a query on them and stops them
again. single_host_cluster=True lets the two workers share the machine.
import polars as pl
import ray
from polars_nanook_ray.config import (
PolarsRayClusterConfig,
PolarsSchedulerConfig,
PolarsServiceAccountLicenseConfig,
PolarsSubprocessConfig,
PolarsWorkerConfig,
)
from polars_nanook_ray.context import RayClusterContext
config = PolarsRayClusterConfig(
runtime=PolarsSubprocessConfig(single_host_cluster=True),
cluster_id="polars-cluster",
num_workers=2,
license=PolarsServiceAccountLicenseConfig(
workspace_id="<WORKSPACE_ID>",
client_id="<CLIENT_ID>",
client_secret="<CLIENT_SECRET>",
),
scheduler=PolarsSchedulerConfig(
cpus_hint=1,
memory_hint=500 * 1024**2,
),
worker=PolarsWorkerConfig(
cpus_hint=2,
memory_hint=4 * 1024**3,
),
)
ray.init(address="auto", namespace=config.cluster_id)
with RayClusterContext(config) as ctx:
print(
pl.LazyFrame({"a": [1, 2, 3], "b": [4, 4, 5]})
.with_columns(pl.col("a").max().over("b").alias("c"))
.remote(ctx)
.collect()
)
ray.shutdown()
Save it as first_query.py and run it in the virtual environment of step 2:
python first_query.py
shape: (3, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 4 ┆ 2 │
│ 2 ┆ 4 ┆ 2 │
│ 3 ┆ 5 ┆ 3 │
└─────┴─────┴─────┘
RayClusterContext registers the cluster with your workspace, and is the
cluster context the query runs on. The query's profile shows up under
Queries in the portal and in the cluster's observatory on port 3001 of the machine. If a step
fails, see Error messages.
Remove the cluster
Leaving the with block stops the scheduler and the workers. Stop Ray with:
ray stop
The service account stays valid until you delete it in the workspace settings.
Next steps
- Production install: Install on Ray
- Every configuration class and field: Ray configuration reference
- Run without a connection to the Polars control plane: Air-gapped clusters