What happened?

Pipeline I ran failed with an error:

Pipeline construction environment and pipeline runtime environment are not compatible. If you use a custom container image, check that the Python interpreter minor version and the Apache Beam version in your image match the versions used at pipeline construction time. Submission environment: beam:version:sdk_base:apache/beam_python3.11_sdk:2.50.0rc1. Runtime environment: beam:version:sdk_base:apache/beam_python3.11_sdk:2.50.0. Worker ID: beamapp-valentyn-08220117-08211817-m76c-harness-v38w

The rootcause is: starting from 2.50.0, we no longer stage Beam SDK. Starting from several releases back we also check that submission and runtime versions match. However Python Docker containers we build for RCs don't install the SDK RC version of Beam SDK tarball.

docker run --rm -it --entrypoint=/bin/bash apache/beam_python3.10_sdk:2.50.0.rc1

cat /usr/local/lib/python3.10/site-packages/apache_beam/version.py
...
"""Apache Beam SDK version information and utilities."""

__version__ = '2.50.0'

This issue blocks further validation of RC1 for Python Dataflow pipelines.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner