Public datasets

Polars Nanook hosts three public Parquet datasets for testing queries at different data sizes, in a Requester Pays S3 bucket.

• PDS-H, derived from the TPC-H benchmark, 10 GB to 1 TB, for joins, aggregations and filters. The queries are in the Polars benchmark repository.
• PDS-DS, derived from the TPC-DS decision support benchmark, 1 GB to 300 GB.
• NYC Taxi, the yellow taxi NYC trip records of 2023 and 2024, one file per month.

How do I read a public dataset in a Polars Nanook query?

Scan any dataset from your Polars code and execute the query remotely. This template scans one table with the Requester Pays option and runs a query on it:

import polars as pl
import polars_nanook as pn

ctx: pn.ComputeContext | pn.ClusterContext

data = pl.scan_parquet(
    "s3://polars-cloud-samples-us-east-2-prd/pdsh/sf10/lineitem.parquet",
    storage_options={"request_payer": "true", "aws_region": "us-east-2"}
)
df = data.select(pl.len()).remote(ctx).collect()

Replace the path with any path listed below. aws_region names the bucket's region, so the scan also works where another AWS region is configured, such as AWS_REGION on EKS.

Where is each dataset and how large is it?

All datasets are hosted in AWS region us-east-2. Run compute in us-east-2 for the lowest latency.

PDS-H (TPC-H derived)

Example:

data = pl.scan_parquet(
    "s3://polars-cloud-samples-us-east-2-prd/pdsh/sf100/lineitem/*.parquet",
    storage_options={"request_payer": "true", "aws_region": "us-east-2"}
)

PDS-DS (TPC-DS derived)

Example:

data = pl.scan_parquet(
    "s3://polars-cloud-samples-us-east-2-prd/pdsds/sf10/store_sales.parquet",
    storage_options={"request_payer": "true", "aws_region": "us-east-2"}
)

NYC Taxi

Example:

data = pl.scan_parquet(
    "s3://polars-cloud-samples-us-east-2-prd/taxi/2024/"
    "yellow_tripdata_2024-01.parquet",
    storage_options={"request_payer": "true", "aws_region": "us-east-2"}
)