We install these libraries in an isolated environment outside of user code. In this environment, we only run Airflow code. At this point, Airflow is not a framework, but an application. We do not need an extension point here, i.e. change versions. Version pinning is a good solution here to maintain stability.

I am not sure if this is possible for example in Composer environment. But I like it.

The Cloud Composer team must think of a way to access pip if they want this operator to work. This team is following changes in MLEngine and is aware of new requirements.

They can also prepare this virtual environment in a Docker image and provide it with the installation. In the next step, they can modify this call to use a pre-baked environment.

Dataflow does not print anything on the screen by default. Good practice says to configure the logger to be able to track the progress. This code is run in a separate process, so it's safe.