Connect to a cluster you run

A ClusterContext points remote queries at a cluster you run, with the Helm chart or on bare-metal. The client talks to the cluster's scheduler directly, bypassing the Polars control plane, so it needs no login and no service account.

Set up

Install the client release listed with the cluster release under Cluster software releases. Both pin the same polars version, and ctx.show_versions() prints the cluster's. The scheduler reads each query plan with its own Polars version and rejects a plan from an incompatible one.

Run the client on the cluster's Python release, down to the patch version (for example 3.12.16 on both). The query plan carries Python objects, such as the default cloud storage credentials and Python UDFs, and the cluster rejects them from another Python release. A bare-metal cluster runs the Python of the environment it is installed in, and the Helm chart's runtime runs Python 3.13.9 by default.

Clients reach the cluster through two services on the scheduler node: the scheduler on port 5051, which takes queries and returns their status, and the observatory REST API on port 3001, which serves the dashboard with cluster status, submitted queries and query profiles.

Helm chartBare-metal

Both Services are of type ClusterIP by default, so they are only reachable from inside the Kubernetes cluster. The Helm chart quickstart port-forwards them for a test. For production, expose them through a LoadBalancer with TLS so the connection to the cluster is encrypted.

scheduler.services.scheduler and scheduler.services.observatory set the type (ClusterIP, LoadBalancer or NodePort) and annotations of the two Services. The annotations configure the controller that sets up the load balancer or TLS termination. See the chart values.

Clients connect to the scheduler node's address. Network addresses and static leader describes the bind and public addresses.

Create the context

The only required argument is the scheduler URI. Without a port, the client uses the default ports.

import polars_nanook as pn

ctx = pn.ClusterContext(uri="https://<SCHEDULER_HOST>")

For a certificate from a private CA, pass tls_options=pn.TLSOptions(ca_cert=...). The API reference lists the other options.

Creating the context tries to open a gRPC connection to the scheduler right away, so an unreachable uri fails at construction time, before the first query.

Pass the context to .remote(). From there, the query and its results work as described in Run a remote query.

User names

By default, queries can be submitted to the scheduler without a user name. The observatory lists queries submitted anonymously as Anonymous User. To attach a user name to a query, set the NANOOK_USER_NAME environment variable or call pn.Config.set_user_name():

import polars_nanook as pn
pn.Config.set_user_name("[email protected]")