apache_beam.runners.dataflow.ptransform_overrides module

Ptransform overrides for DataflowRunner.

class apache_beam.runners.dataflow.ptransform_overrides.StreamingPubSubWriteDoFnOverride

Bases: PTransformOverride

Override ParDo(_PubSubWriteDoFn) for streaming mode in DataflowRunner.

This override specifically targets the final ParDo step in WriteToPubSub and replaces it with Write(sink) for streaming optimization.

matches(applied_ptransform)

get_replacement_transform_for_applied_ptransform(applied_ptransform)

apache_beam.runners.dataflow.ptransform_overrides.get_dataflow_transform_overrides(pipeline_options)

Returns DataflowRunner-specific transform overrides.

Parameters:

pipeline_options – Pipeline options to determine which overrides to apply.

Returns:

List of PTransformOverride objects for DataflowRunner.

class apache_beam.runners.dataflow.ptransform_overrides.NativeReadPTransformOverride

Bases: PTransformOverride

A PTransformOverride for Read using native sources.

The DataflowRunner expects that the Read PTransform using native sources act as a primitive. So this override replaces the Read with a primitive.

matches(applied_ptransform)

get_replacement_transform(ptransform)