What happened?

When installing Apache Beam with Poetry, if any other package requires some version of protobuf, for example 3.20.0, then the dependency resolution fails.

This is because Poetry uses the PyPI definition for its dependency resolution, this definition can be found at https://pypi.org/pypi/apache-beam/2.44.0/json for the current version.

The issue appeared in Beam 2.43 (it worked well with 2.42), and in particular this block of code:

We can see, in this JSON, that the requirement is the tight bound for the quite particular case of Python 3.10 + macOS, rather than the wide general requirement <4. Since I don't think the PyPI repository definition supports dependency setting per environment, it would make more sense to have the wide requirement than the narrow requirement in PyPI, and let the setup script take care of failing installation if the proper protobuf is not installed.

See discussion in #23982 @tvalentyn @AnandInguva

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner