apache_beam.transforms.resources module

A module for defining resource requirements for execution of transforms.

Pipeline authors can use resource hints to provide additional information to runners about the desired aspects of the execution environment.

Resource hints can be specified on a transform level for parts of the pipeline, or globally via –resource_hint pipeline option.

See also: PTransforms.with_resource_hints().

class apache_beam.transforms.resources.ResourceHint

Bases: object

A superclass to define resource hints.

urn: str | None = None

classmethod parse(value: str) → Dict[str, bytes]

Describes how to parse the hint. Override to specify a custom parsing logic.

classmethod get_merged_value(outer_value: bytes, inner_value: bytes) → bytes

Reconciles values of a hint when the hint specified on a transform is also defined in an outer context, for example on a composite transform, or specified in the transform’s execution environment. Override to specify a custom merging logic.

static get_by_urn(urn)

static get_by_name(name)

static is_registered(name)

static register_resource_hint(hint_name: str, hint_class: type) → None

class apache_beam.transforms.resources.AcceleratorHint

Bases: ResourceHint

Describes desired hardware accelerators in execution environment.

urn: str | None = 'beam:resources:accelerator:v1'

class apache_beam.transforms.resources.MinRamHint

Bases: ResourceHint

Describes min RAM requirements for transform’s execution environment.

urn: str | None = 'beam:resources:min_ram_bytes:v1'

classmethod parse(value: str) → Dict[str, bytes]

classmethod get_merged_value(outer_value: bytes, inner_value: bytes) → bytes

class apache_beam.transforms.resources.CpuCountHint

Bases: ResourceHint

Describes number of CPUs available in transform’s execution environment.

urn: str | None = 'beam:resources:cpu_count:v1'

classmethod get_merged_value(outer_value: bytes, inner_value: bytes) → bytes

apache_beam.transforms.resources.merge_resource_hints(outer_hints: Mapping[str, bytes], inner_hints: Mapping[str, bytes]) → Dict[str, bytes]

apache_beam.transforms.resources.parse_resource_hints(hints: Dict[Any, Any]) → Dict[str, bytes]

apache_beam.transforms.resources.resource_hints_from_options(options: PipelineOptions | None) → Dict[str, bytes]