We are happy to present the new 2.36.0 release of Apache Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.36.0, check out the detailed release notes.

I/Os

• Support for stopReadTime on KafkaIO SDF (Java).(BEAM-13171).

New Features / Improvements

• 💻 Support for ARM64 / Mac M1 out of the box. (BEAM-11703).
• Added support for cloudpickle as a pickling library for Python SDK (BEAM-8123). To use cloudpickle, set pipeline option: –pickle_library=cloudpickle
• Added option to specify triggering frequency when streaming to BigQuery (Python) (BEAM-12865).
• Added option to enable caching uploaded artifacts across job runs for Python Dataflow jobs (BEAM-13459). To enable, set pipeline option: –enable_artifact_caching, this will be enabled by default in a future release.

Breaking Changes

• Updated the jedis from 3.x to 4.x to Java RedisIO. If you are using RedisIO and using jedis directly, please refer to this page to update it. (BEAM-12092).
• Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public BEAM-13638.
• Properly check output timestamps on elements output from DoFns, timers, and onWindowExpiration in Java BEAM-12931.
• Fixed a bug with DeferredDataFrame.xs when used with a non-tuple key (BEAM-13421).
• Beam Python now requires google-cloud-pubsub>=2.1.0. The API surface for apache_beam.io.gcp.pubsub has not changed, but code that uses the PubSub client directly may need to be updated.

Known Issues

• Users may encounter an unexpected java.lang.ArithmeticException when outputting a timestamp for an element further than allowedSkew from an allowed DoFN skew set to a value more than Integer.MAX_VALUE.
• S3 object metadata retrieval broken in Python SDK (BEAM-13980)
• See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.36.0 release. Thank you to all contributors!

Ada Wong Ahmet Altay Alexander Alexander Dahl Alexandr Zhuravlev Alexey Romanenko AlikRodriguez Anand Inguva Andrew Pilloud Andy Ye Arkadiusz Gasiński Artur Khanin Arun Pandian Aydar Farrakhov Aydar Zainutdinov AydarZaynutdinov Benjamin Gonzalez Brian Hulette Chamikara Jayalath Daniel Collins Daniel Oliveira Daniel Thevessen Daniela Martín David Hinkes David Huntsperger Emily Ye Etienne Chauchot Evan Galpin Heejong Lee Ilya Ilya Kozyrev In-Ho Yi Jack McCluskey Janek Bevendorff Jarek Potiuk Ke Wu KevinGG Kyle Hersey Kyle Weaver Luís Bianchin Luke Cwik Masato Nakamura Matthias Baetens Mehdi Drissi Melissa Pashniak Michel Davit Miguel Hernandez MiguelAnzoWizeline Milena Bukal Moritz Mack Mostafa Aghajani Nathan J Mehl Niel Markwick Ning Kang Pablo Estrada Pavel Avilov Quentin Sommer Reuben van Ammers Reuven Lax Ritesh Ghorse Robert Bradshaw Robert Burke Ryan Thompson Sam Whittle Sayat Sergei Lebedev Sergey Kalinin Steve Niemitz Talat Uyarer Thiago Nunes Tianyang Hu Tim Robertson Valentyn Tymofieiev Vitaly Ivanov Yichi Zhang Yiru Tang Yu Feng Yu ISHIKAWA Zachary Houfek blais daria-malkova daria.malkova darshan-sj dpcollins-google emily ewianda johnjcasey kileys lam206 laraschmidt mosche msbukal@google.com tvalentyn