Keep results on the cluster

A remote query can return its result in three ways. .collect() streams it back to the client, and a sink writes it to storage you name. The methods on this page instead store it at a location that belongs to the cluster, called the results location, so the client can fetch it later or build the next query on it without downloading it. That is useful for a query you do not want to wait for, a preview of the first rows with .show(), and a result you keep querying with .await_and_scan(). The examples run the query from Run a remote query.

Set up

These methods only work once the cluster has a results location, which the configuration calls the anonymous results location. On AWS the stack creates it, and on a cluster you run, you choose it. The workers write each stored result there, and the client reads it through a presigned URL, or from the same path on a shared filesystem. A client that cannot reach the location still receives query status updates, but no results. A cluster you run never deletes stored results, so set a retention policy on the location.

anonymousResults:
  s3:
    enabled: true
    endpoint: "s3://<BUCKET>/<PREFIX>"
    options:
      - name: aws_access_key_id
        valueFrom:
          secretKeyRef:
            name: <SECRET_NAME>
            key: accessKeyId
      - name: aws_secret_access_key
        valueFrom:
          secretKeyRef:
            name: <SECRET_NAME>
            key: secretAccessKey

[scheduler]
enabled = true
anonymous_result_location.s3.url = "s3://<BUCKET>/<PREFIX>"
anonymous_result_location.s3.aws_access_key_id = "<AWS_ACCESS_KEY_ID>"
anonymous_result_location.s3.aws_secret_access_key = "<AWS_SECRET_ACCESS_KEY>"

Store the result

.execute() runs the query and returns a QueryResult, with the first rows in head and the paths of the result in location. .execute(blocking=False) instead returns right away with a query in progress. Call .await_result() on it to wait for the query to finish.

Preview the result

.show() runs the query limited to its first 10 rows and returns them:

import polars_nanook as pn

ctx: pn.ComputeContext | pn.ClusterContext

print(pdsh_q3(customer_sf10, lineitem_sf10, orders_sf10).remote(context=ctx).show())

shape: (10, 4)
┌────────────┬─────────────┬─────────────┬────────────────┐
│ l_orderkey ┆ revenue     ┆ o_orderdate ┆ o_shippriority │
│ ---        ┆ ---         ┆ ---         ┆ ---            │
│ i64        ┆ f64         ┆ date        ┆ i64            │
╞════════════╪═════════════╪═════════════╪════════════════╡
│ 4791171    ┆ 440715.2185 ┆ 1995-02-23  ┆ 0              │
│ 46678469   ┆ 439855.325  ┆ 1995-01-27  ┆ 0              │
│ 23906758   ┆ 432728.5737 ┆ 1995-03-14  ┆ 0              │
│ 23861382   ┆ 428739.1368 ┆ 1995-03-09  ┆ 0              │
│ 59393639   ┆ 426036.0662 ┆ 1995-02-12  ┆ 0              │
│ 3355202    ┆ 425100.6657 ┆ 1995-03-04  ┆ 0              │
│ 9806272    ┆ 425088.0568 ┆ 1995-03-13  ┆ 0              │
│ 22810436   ┆ 423231.969  ┆ 1995-01-02  ┆ 0              │
│ 16384100   ┆ 421478.7294 ┆ 1995-03-02  ┆ 0              │
│ 52974151   ┆ 415367.1195 ┆ 1995-02-05  ┆ 0              │
└────────────┴─────────────┴─────────────┴────────────────┘

Keep chaining

.await_and_scan() waits for the query and returns a LazyFrame over the stored result, so you can continue the query from there. For a result that must persist, use a sink.

import polars_nanook as pn

ctx: pn.ComputeContext | pn.ClusterContext

result = (
    pdsh_q3(customer_sf10, lineitem_sf10, orders_sf10)
    .remote(context=ctx)
    .await_and_scan()
)

print(result.collect())