Common pipeline patterns

Pipeline patterns demonstrate common Beam use cases. Pipeline patterns are based on real-world Beam deployments. Each pattern has a description, examples, and a solution or psuedocode.

File processing patterns - Patterns for reading from and writing to files

• Processing files as they arrive
• Accessing filenames

Side input patterns - Patterns for processing supplementary data

• Slowly updating global window side inputs

Pipeline option patterns - Patterns for configuring pipelines

• Retroactively logging runtime parameters

Custom I/O patterns - Patterns for pipeline I/O

• Choosing between built-in and custom connectors

Custom window patterns - Patterns for windowing functions

• Using data to dynamically set session window gaps

BigQuery patterns - Patterns for BigQueryIO

• Google BigQuery patterns

AI Platform integration patterns - Patterns for using Google Cloud AI Platform transforms

• Analysing the structure and meaning of text
• Getting predictions

Schema patterns - Patterns for using Schemas

• Using Joins

BQML integration patterns - Patterns on integrating BigQuery ML into your Beam pipeline

• Model export and on-worker serving using BQML and TFX_BSL

Cross-language patterns - Patterns for creating cross-language pipelines

• Cross-language patterns

State & timers patterns - Patterns for using state & timers

• Grouping elements for efficient external service calls
• Dynamically grouping elements

Cache with a shared object - Patterns for using a shared object as a cache using the Python SDK

• Create a cache on a batch pipeline
• Create a cache and update it regularly on a streaming pipeline

Contributing a pattern

To contribute a new pipeline pattern, create a feature request and add details to the issue description. See Get started contributing for more information.

What’s next

• Try an end-to-end example
• Execute your pipeline on a runner