Skip to content

Emit OpenLineage events

A Polars Nanook cluster is an OpenLineage producer. When it is configured with a lineage transport, each remote query tagged with with_lineage() emits OpenLineage RunEvents over HTTP(S) to a collector such as Marquez.

Set up

lineage:
  enabled: true
  transport:
    http:
      endpoint: "http://<MARQUEZ_HOST>:5000"

See the lineage section of the Helm chart documentation.

[lineage]
enabled = true
transport.http.endpoint = "http://<MARQUEZ_HOST>:5000"

See the Bare-metal config file reference.

Tag a query

with_lineage() requires job_namespace and job_name. The cluster uses the query ID as the runId of each run.

The example concatenates two CSV inputs, writes Parquet and tags the run as job demo.job1 in namespace prod:

import polars as pl
import polars_nanook as pn

ctx: pn.ClusterContext

# src1, src2: paths of the CSV inputs; dst: a directory for the Parquet output
query = (
    pl.concat(
        [
            pl.scan_csv(src1),
            pl.scan_csv(src2),
        ],
        how="horizontal",
    )
    .remote(context=ctx)
    .with_lineage(
        job_namespace="prod",
        job_name="demo.job1"
    )
    .sink_parquet(dst)
    .await_result()
)

To add more metadata, such as Parent Run metadata, see with_lineage() in the API reference.

Supported facets

Polars Nanook supports these facets:

  • Run facets: Error Message, Parent Run
  • Dataset facets: Schema, Output Statistics