Skip to content

Get started on Ray

On Ray, the Polars Nanook scheduler and workers run as actors on a Ray cluster you already operate, started from Python with the polars-nanook-ray package. Start a single-machine cluster, connect it to your workspace and run a first remote query. Before you start, install the client and sign up.

You need a Linux machine (x86-64 with AVX2, or ARM64) with at least 16 GiB of memory, and uv.

Step 1: Create a service account

The cluster registers with your workspace through a service account: a client ID and secret for the workspace.

Open Integrations > Infrastructure in the portal sidebar and click Connect on the Ray card. The Create service account step generates a service account. Copy the client ID and secret: they are shown only once. The portal fills them in, together with your workspace ID, in the script of step 4.

nanook service-account create \
  --workspace-name "My workspace" \
  --name polars-ray

The command prints the API client ID, the API secret and the Workspace ID you need in step 4. Store the secret: it is shown only once.

Step 2: Install the package

Install the polars-nanook-ray package in a virtual environment. It brings ray, the polars-nanook client and the polars-nanook-compute package the actors run:

uv venv --python 3.12 && source .venv/bin/activate
uv pip install polars-nanook-ray
mkdir --parents /tmp/polars/{anonymous-results,license,observatory,shuffle-data,temporary-data}

The directories are the default locations of the stored results, the license, the query history, the shuffle data and the temporary data.

Step 3: Start Ray

Start a Ray head node in the same shell:

ray start --head --disable-usage-stats --resources='{"head":1}'

The custom head resource marks the node the scheduler runs on. See Actor placement.

Step 4: Run a test query

This script starts a scheduler and two workers as Ray actors, runs a query on them and stops them again. single_host_cluster=True lets the two workers share the machine.

import polars as pl
import ray

from polars_nanook_ray.config import (
    PolarsRayClusterConfig,
    PolarsSchedulerConfig,
    PolarsServiceAccountLicenseConfig,
    PolarsSubprocessConfig,
    PolarsWorkerConfig,
)
from polars_nanook_ray.context import RayClusterContext

config = PolarsRayClusterConfig(
    runtime=PolarsSubprocessConfig(single_host_cluster=True),
    cluster_id="polars-cluster",
    num_workers=2,
    license=PolarsServiceAccountLicenseConfig(
        workspace_id="<WORKSPACE_ID>",
        client_id="<CLIENT_ID>",
        client_secret="<CLIENT_SECRET>",
    ),
    scheduler=PolarsSchedulerConfig(
        cpus_hint=1,
        memory_hint=500 * 1024**2,
    ),
    worker=PolarsWorkerConfig(
        cpus_hint=2,
        memory_hint=4 * 1024**3,
    ),
)

ray.init(address="auto", namespace=config.cluster_id)

with RayClusterContext(config) as ctx:
    print(
        pl.LazyFrame({"a": [1, 2, 3], "b": [4, 4, 5]})
        .with_columns(pl.col("a").max().over("b").alias("c"))
        .remote(ctx)
        .collect()
    )

ray.shutdown()

Save it as first_query.py and run it in the virtual environment of step 2:

python first_query.py
Output
shape: (3, 3)
┌─────┬─────┬─────┐
│ a   ┆ b   ┆ c   │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1   ┆ 4   ┆ 2   │
│ 2   ┆ 4   ┆ 2   │
│ 3   ┆ 5   ┆ 3   │
└─────┴─────┴─────┘

RayClusterContext registers the cluster with your workspace, and is the cluster context the query runs on. The query's profile shows up under Queries in the portal and in the cluster's observatory on port 3001 of the machine. If a step fails, see Error messages.

Remove the cluster

Leaving the with block stops the scheduler and the workers. Stop Ray with:

ray stop

The service account stays valid until you delete it in the workspace settings.

Next steps