In our CI, roach{prod,test} is built from HEAD and run against all our major release branches. It's also run from HEAD when testing out our alpha releases. #51535 (comment) is an example of what can go wrong with this approach.

The roachtest code running here is from master, and landed alongside changes that introduced this membership column (#50329). In this SHA, that we're attempting to cut a release from, we didn't pick up the CRDB changes, but we are running it using roachtest code on master, which is expecting to find this column.

This "temporary mismatch" between roachprod code and CRDB code, where roachprod expects to find certain commits as part of the 20.2 release and does not find them because of when the alpha was cut doesn't seem like something roachprod should be taught to handle. If anything, we should start considering running roachprod from the SHA being tested. Asking around, we've been running it from a version built off of HEAD because roach{prod,test} used to change quite a lot, which I don't think is any longer the case. In fact, making roachprod compatible across multiple versions is increasingly cumbersome and fragile, not to mention largely untested.

If we don't run our unit tests from HEAD, we probably shouldn't do the same for our integration tests. Admittedly doing so forces us to think about compatibility issues, but I'm not sure that's enough of a reason to do such a thing.