AWS Lambda

An AWS Lambda function can submit a Polars Nanook query on a schedule set in Amazon EventBridge. The function submits the query and exits, so no dedicated orchestration infrastructure is required.

Query submission from a Lambda function

The first step packages the polars and polars-nanook Python packages into the Lambda deployment. They are larger than the 250 MB limit of a .zip deployment package, so deploy the function as a container image. The code of the Lambda function reduces to the following (pseudo-code):

import boto3
import polars as pl
import polars_nanook as pn


def handler(event, context):
    client = boto3.client("secretsmanager")

    # authenticate to Polars Nanook with the service-account secrets
    # (client ID and client secret are stored as two separate secrets)
    pn.authenticate(
        client_id=client.get_secret_value(SecretId="<CLIENT_ID_SECRET_NAME>").get("SecretString"),
        client_secret=client.get_secret_value(SecretId="<CLIENT_SECRET_SECRET_NAME>").get("SecretString"),
    )

    # define the compute context
    cc = pn.ComputeContext(workspace="My workspace", cpus=2, memory=4)

    # submit the query
    pl.scan_csv(...).remote(cc).sink_parquet(...)

Store the service-account client ID and client secret as two secrets in AWS Secrets Manager beforehand (see service accounts). Submitting the query starts the compute and waits until it is ready, so raise the function timeout (3 seconds by default, at most 15 minutes) above the cluster start time. Once the query is submitted the Lambda exits gracefully and leaves the rest of the handling to Polars Nanook. The query keeps running on the compute and its status is shown in the portal.

Scheduling with EventBridge

Without dedicated orchestrator infrastructure (such as Airflow), triggering rules in AWS EventBridge schedule the function. Rules are defined in the AWS Console or with the AWS CLI, as documented here. A CRON rule triggers the query at a given interval.