apache_beam.runners.interactive.dataproc.types module
- class apache_beam.runners.interactive.dataproc.types.ClusterMetadata(project_id: str | None = None, region: str | None = 'us-central1', cluster_name: str | None = <factory>, subnetwork: str | None = None, num_workers: int | None = None, machine_type: int | None = None, master_url: str | None = None, dashboard: str | None = None)[source]
Bases:
objectMetadata of a provisioned worker cluster that executes Beam pipelines.
Apache Beam supports running Beam pipelines on different runners provisioned in different setups based on the runner and pipeline options associated with each pipeline. To provide similar portability features, Interactive Beam automatically extracts such ClusterMetadata information from pipeline options of a pipeline in the REPL context and provision suitable clusters to execute the pipeline. The lifecyle of the clusters is managed by Interactive Beam and the user doesn not need to interact with it.
It’s not recommended to build this ClusterMetadata from raw values nor use it to interact with the cluster management logic directly.
Interactive Beam now supports:
1. Runner: FlinkRunner; Setup: on Google Cloud with Flink on Dataproc.