CombinePerKey

Pydoc Pydoc




Combines all elements for each key in a collection.

See more information in the Beam Programming Guide.

Examples

In the following examples, we create a pipeline with a PCollection of produce. Then, we apply CombinePerKey in multiple ways to combine all the elements in the PCollection.

CombinePerKey accepts a function that takes a list of values as an input, and combines them for each key.

Example 1: Combining with a predefined function

We use the function sum which takes an iterable of numbers and adds them together.

Example 2: Combining with a function

We define a function saturated_sum which takes an iterable of numbers and adds them together, up to a predefined maximum number.

Example 3: Combining with a lambda function

We can also use lambda functions to simplify Example 2.

Example 4: Combining with multiple arguments

You can pass functions with multiple arguments to CombinePerKey. They are passed as additional positional arguments or keyword arguments to the function.

In this example, the lambda function takes values and max_value as arguments.

Example 5: Combining with a CombineFn

The more general way to combine elements, and the most flexible, is with a class that inherits from CombineFn.

You can use the following combiner transforms:

See also GroupBy which allows you to combine more than one field at once.

Pydoc Pydoc