apache_beam.transforms.periodicsequence module

class apache_beam.transforms.periodicsequence.ImpulseSeqGenRestrictionProvider

Bases: RestrictionProvider

initial_restriction(element)

create_tracker(restriction)

restriction_size(element, restriction)

truncate(unused_element, unused_restriction)

class apache_beam.transforms.periodicsequence.ImpulseSeqGenDoFn(*unused_args, **unused_kwargs)

Bases: DoFn

ImpulseSeqGenDoFn fn receives tuple elements with three parts:

• first_timestamp = first timestamp to output element for.

• last_timestamp = last timestamp/time to output element for.

• fire_interval = how often to fire an element.

For each input element received, ImpulseSeqGenDoFn fn will start generating output elements in following pattern:

• if element timestamp is less than current runtime then output element.

• if element timestamp is greater than current runtime, wait until next element timestamp.

ImpulseSeqGenDoFn can’t guarantee that each element is output at exact time. ImpulseSeqGenDoFn guarantees that elements would not be output prior to given runtime timestamp.

process(element, restriction_tracker=RestrictionParam(ImpulseSeqGenRestrictionProvider), watermark_estimator=WatermarkEstimatorProvider)

Parameters:

• element – (start_timestamp, end_timestamp, interval)

• restriction_tracker

Returns:

yields elements at processing real-time intervals with value of target output timestamp for the element.

class apache_beam.transforms.periodicsequence.PeriodicSequence

Bases: PTransform

PeriodicSequence transform receives tuple elements with three parts:

• first_timestamp = first timestamp to output element for.

• last_timestamp = last timestamp/time to output element for.

• fire_interval = how often to fire an element.

For each input element received, PeriodicSequence transform will start generating output elements in following pattern:

• if element timestamp is less than current runtime then output element.

• if element timestamp is greater than current runtime, wait until next element timestamp.

PeriodicSequence can’t guarantee that each element is output at exact time. PeriodicSequence guarantees that elements would not be output prior to given runtime timestamp. The PCollection generated by PeriodicSequence is unbounded.

expand(pcoll)

class apache_beam.transforms.periodicsequence.PeriodicImpulse(start_timestamp=Timestamp(1728881126.105190), stop_timestamp=Timestamp(9223372036854.775000), fire_interval=360.0, apply_windowing=False)

Bases: PTransform

PeriodicImpulse transform generates an infinite sequence of elements with given runtime interval.

PeriodicImpulse transform behaves same as {@link PeriodicSequence} transform, but can be used as first transform in pipeline. The PCollection generated by PeriodicImpulse is unbounded.

Parameters:

• start_timestamp – Timestamp for first element.

• stop_timestamp – Timestamp after which no elements will be output.

• fire_interval – Interval at which to output elements.

• apply_windowing – Whether each element should be assigned to individual window. If false, all elements will reside in global window.

expand(pbegin)