R: @jrmccluskey

Run Python 3.8 PostCommit

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

Codecov Report

Merging #28571 (0c37a96) into master (18ce60c) will decrease coverage by 0.03%.
Report is 26 commits behind head on master.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #28571      +/-   ##
==========================================
- Coverage   72.22%   72.20%   -0.03%     
==========================================
  Files         684      684              
  Lines      101070   101131      +61     
==========================================
+ Hits        72999    73018      +19     
- Misses      26492    26534      +42     
  Partials     1579     1579              

Flags with carried forward coverage won't be shown. Click here to find out more.

see 9 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Was it running twice? And does this mean you're going to update all of the other IT tests that double-up on markers?

Run Python 3.8 PostCommit

Yes, it's running twice, see: https://ci-beam.apache.org/job/beam_PostCommit_Python38/4614/consoleText

failing run uses a wrong requirements.txt file, see: https://pantheon.corp.google.com/dataflow/jobs/2023-09-20_21_04_05-17827361344221303215?project=apache-beam-testing&e=13802955&mods=dataflow_dev

there is definitely something to clean up. we can't apply the same change i do here to some of the other postcommits (e.g. pytorch) without more changes, but we should restructure these configs to create less entropy going forward. i can take a closer look later, this is a targeted fix to unblock broken postcommit.

filed: #28578

there is definitely something to clean up. we can't apply the same change i do here to some of the other postcommits (e.g. pytorch) without more changes, but we should restructure these configs to create less entropy going forward. i can take a closer look later, this is a targeted fix to unblock broken postcommit.

+1 - I think this is done wrong all over the place. I know personally I did not understand how these tags worked correctly, so I'm sure I've contributed :(

looks like beam release cut has affected the tests, rebasing.

Run Python 3.8 PostCommit

       RuntimeError: Pipeline construction environment and pipeline runtime environment are not compatible. If you use a custom container image, check that the Python interpreter minor version and the Apache Beam version in your image match the versions used at pipeline construction time. Submission environment: beam:version:sdk_base:apache/beam_python3.8_sdk:2.52.0.dev. Runtime environment: beam:version:sdk_base:apache/beam_python3.8_sdk:2.51.0.dev.

New error:

I wonder if we somehow build the from a different commit than we test?

       RuntimeError: Pipeline construction environment and pipeline runtime environment are not compatible. If you use a custom container image, check that the Python interpreter minor version and the Apache Beam version in your image match the versions used at pipeline construction time. Submission environment: beam:version:sdk_base:apache/beam_python3.8_sdk:2.52.0.dev. Runtime environment: beam:version:sdk_base:apache/beam_python3.8_sdk:2.51.0.dev.

New error:

I wonder if we somehow build the from a different commit than we test?

I think that might mean that we haven't published the new snapshot containers?

I'm pretty sure we're not building/testing from different commits since its all just one big gradle task

The error could happen if tests don't specify --sdk_location.

Or this flag is somehow not respected.

I do see --sdk_location /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python38_PR/src/sdks/python/build/apache_beam-2.52.0.dev0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in Dataflow job page: https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-09-21_13_26_42-12639981593393828172

will look further in #28653

Run Python 3.8 PostCommit

Run Python PreCommit