apache_beam.yaml.yaml_mapping module

This module defines the basic MapToFields operation.

apache_beam.yaml.yaml_mapping.normalize_mapping(spec)

Normalizes various fields for mapping transforms.

apache_beam.yaml.yaml_mapping.is_literal(expr: str) → bool

apache_beam.yaml.yaml_mapping.validate_generic_expression(expr_dict: dict, input_fields: Collection[str], allow_cmp: bool, error_field: str) → None

apache_beam.yaml.yaml_mapping.validate_generic_expressions(base_type, config, input_pcolls) → None

apache_beam.yaml.yaml_mapping.py_value_to_js_dict(py_value)

class apache_beam.yaml.yaml_mapping.Validate(schema: dict[str, Any], error_handling: Mapping[str, Any] | None = None)

Bases: PTransform

Validates each element of a PCollection against a json schema.

Parameters:

• schema – A json schema against which to validate each element.

• error_handling – Whether and how to handle errors during iteration. If this is not set, invalid elements will fail the pipeline, otherwise invalid elements will be passed to the specified error output along with information about how the schema was invalidated.

expand(pcoll)

with_exception_handling(**kwargs)

apache_beam.yaml.yaml_mapping.is_expr(v)

apache_beam.yaml.yaml_mapping.normalize_fields(pcoll, fields, drop=(), append=False, language='generic')

class apache_beam.yaml.yaml_mapping.PaneInfoTuple(is_first, is_last, timing, index, nonspeculative_index)

Bases: NamedTuple

Create new instance of PaneInfoTuple(is_first, is_last, timing, index, nonspeculative_index)

is_first: bool

Alias for field number 0

is_last: bool

Alias for field number 1

timing: str

Alias for field number 2

index: int

Alias for field number 3

nonspeculative_index: int

Alias for field number 4

classmethod from_pane_info(pane_info)

apache_beam.yaml.yaml_mapping.create_mapping_providers()