Connector-related efforts that will benefit multiple SDKs.

Splittable DoFn

Splittable DoFn is the next generation sources framework for Beam that will replace current frameworks for developing bounded and unbounded sources. Splittable DoFn is being developed along side current Beam portability efforts. See Beam portability framework roadmap for more details.

Cross-language transforms

Last updated on May 2020.

As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits.

Completed and Ongoing Efforts

Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below.

Cross-language transforms API and expansion service

Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services.

Work related to making cross-language transforms available for Flink runner.

Support for Dataflow runner

Work related to making cross-language transforms available for Dataflow runner.

Support for Direct runner

Work related to making cross-language transforms available on Direct runner

Connector/transform support

Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework.

Portable Beam schema

Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms.

Integration/Performance testing

Documentation

Work related to adding documenting on cross-language transforms to Beam Website.