Because of the new image build system, whenever we create a new Jenkins test, it will fail in all in-flight pull requests once merged. Examples can be seen in #15472 and #15469.

This is happening because Jenkins attempts to rebase the code before executing tests. Since images are built externally, Jenkins rebases the code for the tests only. The image build process doesn't rebase before building. We therefore run new tests with an agent or operator that doesn't support the tested feature yet.

We have at least three ways to fix this:

• We extend the image build process to attempt a rebase similarly to Jenkins.
• We tell everyone to ignore new, failing tests if they haven't rebased yet and hope we don't introduce too many new Jenkins tests.
• We disable rebasing in Jenkins. That may end up being worse though.