Network addresses and static leader

A bare-metal cluster exposes several network services for communication between nodes. You can bind each service to a specific address, for example to use one network interface of a node or to avoid port collisions. Most addresses have defaults.

Static leader

Every node in a bare-metal cluster needs to know whether it is the leader (the node that runs the scheduler and observatory services) or, if not, where to find the leader. The section below can be the same on every node. It names the instance ID of the leader node and the public addresses of the observatory and scheduler services.

[static_leader]
leader_instance_id = "node-0"
observatory_service.public_addr = "192.168.1.2"
scheduler_service.public_addr = "192.168.1.2"

If the current node's instance ID matches the leader_instance_id, the node runs the observatory and scheduler services (if those components are enabled).

Scheduler

The scheduler has two main services: the client service and the worker service. The Python client uses the client service to submit queries and receive query status updates. The workers use the worker service to register and send heartbeats. By default, they bind to 0.0.0.0:5051 and 0.0.0.0:5050, respectively.

Observatory

The observatory has two components. The observatory service receives OpenTelemetry profiling traces from all cluster nodes. The observatory REST API serves the dashboard and exposes the query plans, profiling data and host metrics.

The observatory service (profiling) binds to 0.0.0.0:5049 and the REST API to 0.0.0.0:3001 by default. To send traces to the observatory, enable the monitoring section in the configuration of each worker node, and set the observatory service public address in the static leader section on the worker nodes.

instance_id = "node-0"
# ...

[scheduler]
enabled = true
# ...

[observatory]
enabled = true
max_metrics_bytes_total = 104857600

[static_leader]
leader_instance_id = "node-0"
observatory_service.public_addr = "192.168.1.2"
# ...

instance_id = "node-1"
# ...

[worker]
enabled = true
# ...

[monitoring]
enabled = true

[static_leader]
leader_instance_id = "node-0"
observatory_service.public_addr = "192.168.1.2"
# ...

Workers

The worker has two services, the task service and the shuffle service. The scheduler uses the task service to send tasks to the worker, and the workers use the shuffle service to serve shuffles to each other. By default, they bind to 0.0.0.0:5052 and 0.0.0.0:5053, respectively. The shuffle service is only bound when local shuffle storage is used.

The worker services also need a public address, which the scheduler uses to reach them. It is required when the bind address is 0.0.0.0. Also set it when the ports go through a NAT or load balancer. A public address without a port uses the bind port.

The following example changes the bind and public addresses of the worker services:

[worker]
enabled = true
task_service.bind_addr.port = 4005
task_service.public_addr = "192.168.1.1"
shuffle_service.bind_addr.port = 4006
shuffle_service.public_addr = "192.168.1.1"