apache_beam.runners.direct.watermark_manager module

Manages watermarks of PCollections and AppliedPTransforms.

class apache_beam.runners.direct.watermark_manager.WatermarkManager(clock, root_transforms, value_to_consumers, transform_keyed_states)[source]

Bases: object

For internal use only; no backwards-compatibility guarantees.

Tracks and updates watermarks for all AppliedPTransforms.

WATERMARK_POS_INF = Timestamp(9223372036854.775000)
WATERMARK_NEG_INF = Timestamp(-9223372036854.775000)
get_watermarks(applied_ptransform: AppliedPTransform) _TransformWatermarks[source]

Gets the input and output watermarks for an AppliedPTransform.

If the applied_ptransform has not processed any elements, return a watermark with minimum value.

Parameters:

applied_ptransform – AppliedPTransform to get the watermarks for.

Returns:

A snapshot (TransformWatermarks) of the input watermark and output watermark for the provided transform.

update_watermarks(completed_committed_bundle: _Bundle, applied_ptransform: AppliedPTransform, completed_timers, outputs, unprocessed_bundles, keyed_earliest_holds, side_inputs_container)[source]
extract_all_timers() Tuple[List[Tuple[AppliedPTransform, List[TimerFiring]]], bool][source]

Extracts fired timers for all transforms and reports if there are any timers set.