Apache Beam Roadmap
Apache Beam is not governed or steered by any one commercial entity, but by its Project Management Committee (PMC), so we do not have a roadmap in the sense of a plan with a specific timeline. Instead, we share our vision for the future and major initiatives receiving and attention and contribution that users can look forward to.
The major components of Beam each have their own roadmap which you can find via the menu. Below are some highlights for the project as a whole.
Portability is the primary Beam vision: running pipelines authored with any SDK on any runner. This is a cross-cutting effort across Java, Python, and Go, and every Beam runner. Portability is currently supported on the Flink and Spark runners.
See the details on the Portability Roadmap
As a benefit of the portability effort, we are able to utilize Beam transforms across SDKs. Examples include using Java connectors and Beam SQL from Python or Go pipelines or Beam TFX transforms from Java and Go. For details see Roadmap for multi-SDK efforts.
The Go SDK is the newest SDK, and is the first SDK built entirely on the portability framework. See the Go SDK’s Roadmap if this piques your interest.
Python 3 support
See details on the Python SDK’s Roadmap.
Java 11 support
The Java SDK is eager to add support for Java’s first new LTS (Long Term Support) version. See details on the Java SDK’s Roadmap.
Beam’s SQL module is rapidly maturing to allow users to author batch and streaming pipelines using only SQL, but also to allow Beam Java developers to use SQL in components of their pipeline for added efficiency. See the Beam SQL Roadmap
Schemas allow SDKs and runners to understand the structure of user data and unlock relational optimization possibilities. Portable schemas enable compatibility between rows in Python and Java. A particularly interesting use case is the combination of SQL (implemented in Java) with the Python SDK via Beam’s cross-language support. Learn more about portable schemas from this presentation.
Euphoria is Beam’s newest API, offering a high-level, fluent style for Beam Java developers. See the Euphoria API Roadmap.