Polars Nanook

Polars Nanook scales Polars from your laptop to a cluster. A query you write and test locally runs unchanged on one large machine or distributed across many, in your own infrastructure. Your data stays there, and an air-gapped cluster sends nothing. See Security and data boundaries for what reaches Polars.

What Polars Nanook adds

• The API is the same at any scale. A query that runs on your laptop with .collect() runs on a cluster with .remote(ctx).collect(), with no other changes.
• The distributed engine, which the open-source library does not include, spreads one query over many workers.
• Fault tolerance lets queries complete even if a machine fails or is interrupted.
• With compute on demand on AWS, instances start for a query and stop when they idle.
• Query profiling records the runtime and I/O of every operator and the resource usage per node, for local and remote queries. You see it in the portal and in the observatory of a cluster you run.
• You can run queries from a notebook, an Airflow, Dagster or Prefect task, an AWS Lambda function or a server (see Orchestration).

Get started

Run a remote query

Install the open-source Polars library together with polars-nanook, the Polars Nanook client:

pip install polars polars-nanook

Write the query with Polars as usual and call .remote() on it. This example runs an aggregation on the PDS-H lineitem table at scale factor 10 from the public datasets. ctx is the context of your deployment, which each quickstart creates: a ComputeContext on AWS or a ClusterContext for a cluster you run.

LazyFrameRemote

import polars as pl
import polars_nanook as pn

ctx: pn.ComputeContext | pn.ClusterContext

lineitem = pl.scan_parquet(
    "s3://polars-cloud-samples-us-east-2-prd/pdsh/sf10/lineitem.parquet",
    storage_options={"request_payer": "true", "aws_region": "us-east-2"},
)

query = lineitem.group_by("l_returnflag", "l_linestatus").agg(
    avg_price=pl.mean("l_extendedprice"),
    avg_disc=pl.mean("l_discount"),
    count_order=pl.len(),
)

result = query.remote(context=ctx).collect()
print(result)

Plans and billing describes the Developer, Team and Enterprise plans, and the FAQ answers common questions.