Flatten
JavadocMerges multiple PCollection objects into a single logical PCollection.
By default, the coder for the output PCollection is the same as the coder
for the first PCollection in the input PCollectionList. However, the
input PCollection objects can each use different coders, as long as
they all contain the same data type in your chosen language.
When using Flatten to merge PCollection objects that have a windowing
strategy applied, all of the PCollection objects you want to merge must
use a compatible windowing strategy and window sizing. For example, all
the collections you’re merging must all use (hypothetically) identical
5-minute fixed windows or 4-minute sliding windows starting every 30 seconds.
If your pipeline attempts to use Flatten to merge PCollection objects
with incompatible windows, Beam generates an IllegalStateException error
when your pipeline is constructed
See more information in the Beam Programming Guide.
Examples
Example: Apply a Flatten transform to merge multiple PCollection objects
The resulting collection now has all the elements: “Hello”, “World”, “Beam”, “Is”, and “Fun”.
Related transforms
Last updated on 2024/11/14
Have you found everything you were looking for?
Was it all useful and clear? Is there anything that you would like to change? Let us know!