The 2.40 release is one of Beam Go’s biggest yet, and we wanted to highlight some of the biggest changes coming with this important release!

Native Streaming Support

2.40 marks the release of one of our most anticipated feature sets yet: native streaming Go pipelines. This includes adding support for:

• Self Checkpointing
• Watermark Estimation
• Pipeline Drain/Truncation
• Bundle Finalization (added in 2.39)

With all of these features, it is now possible to write your own streaming pipeline source DoFns in Go without relying on cross-language transforms from Java or Python. We encourage you to try out all of these new features in your streaming pipelines! The programming guide has additional information on getting started with native Go streaming DoFns.

Generic Registration (Make Your Pipelines 3x Faster)

The release of Go Generics in Go 1.18 unlocked significant performance improvements for Beam Go. With generics, we were able to add simple registration functions that can massively reduce your pipeline’s runtime and resource consumption. For example, registering the ParDo’s in our load tests which are designed to simulate a basic pipeline reduced execution time from around 25 minutes to around 7 minutes on average

• an over 70% reduction!

To get started with registering your own DoFns and unlocking these performance gains, check out the registration doc page.

What’s Next?

Moving forward, we remain focused on improving the streaming experience and leveraging generics to improve the SDK. Specific improvements we are considering include adding State & Timers support, introducing a Go expansion service so that Go DoFns can be used in other languages, and wrapping more Java and Python IOs so that they can be easily used in Go. As always, please let us know what changes you would like to see by filing an issue, emailing the dev list, or starting a slack thread!