Skip to content

Orchestrate queries

You can schedule Polars Nanook queries from Airflow, Dagster, Prefect or AWS Lambda. The orchestrator only schedules and monitors the task, and Polars Nanook runs the query. Each task creates a context and calls .remote(ctx) on a lazy Polars query.

Guides

  • Apache Airflow Airflow

    Credentials in an Airflow Connection, and clusters stopped or shared across tasks.

    Airflow

  • AWS Lambda

    Submit a query on an EventBridge schedule. The query keeps running after the function exits.

    AWS Lambda

  • Dagster

    One resource logs in with a service account, another starts compute sized per asset.

    Dagster

  • Prefect Prefect

    Credentials in Prefect Secret blocks, one login per flow and one context per machine size.

    Prefect

The guides use a compute context, which starts compute in your AWS account. On a cluster you run, they work the same with a cluster context that points at the cluster's scheduler. The cluster is already running, so the steps that start and stop compute do not apply.

Authentication of an orchestrated task

A task that uses a ComputeContext cannot open a browser, so it logs in with a service account set in the task environment, as in Log in from a job. Each guide above shows how its tool injects the credentials. A task that uses a ClusterContext needs no login.