Skip to content

Run queries

Polars Nanook runs the lazy Polars queries you already write, on compute in your infrastructure. The query stays the same. You add .remote(ctx) and call .collect(), where ctx is a context that says where the query runs:

  • A ComputeContext starts compute in your AWS account.
  • A ClusterContext connects to a cluster you run, with the Helm chart or on bare-metal.

Connect and run

  • Log in

    Authenticate the client, interactively or with a service account.

    Log in

  • Run a remote query

    Run a query remotely and stream the result back or write it to storage.

    Run a remote query

  • Connect to AWS compute

    Size a ComputeContext, keep it running with the idle timeout and manifests, and reconnect to it.

    Connect to AWS compute

  • Connect to a cluster you run

    Point a ClusterContext at the scheduler of a Helm chart or bare-metal cluster.

    Connect to a cluster you run

  • Use Python packages and plugins

    Expression plugins and Python UDFs on the workers.

    Use Python packages and plugins

Data

Performance

  • How the distributed engine works

    From a LazyFrame to stages, partitions and shuffles across workers.

    How the distributed engine works

  • Profile a query

    See where a query spends its time: plans, operators, stages and shuffles.

    Profile a query

  • Scale a query

    Single node or distributed, and the workers a query uses.

    Scale a query

  • Run queries on GPUs

    Run distributed queries on NVIDIA GPUs with cudf-polars.

    Run queries on GPUs

Orchestrate

  • Orchestrate queries

    Schedule queries from Airflow, Dagster, Prefect or AWS Lambda.

    Orchestrate queries