raising this to continue the discussion in https://tracker.ceph.com/issues/55853

jenkins test api

jenkins test make check

Running some octopus-x/parallel and pacific-x/parallel tests here with your branch: http://pulpito.front.sepia.ceph.com/lflores-2022-08-05_18:10:59-upgrade-main-distro-default-smithi/ ./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c main -S 4bc2f657124bc7b731b0ce4c7516e9f09e21c5a3 -s upgrade --suite-repo https://github.com/cbodley/ceph.git --suite-branch wip-55853 --filter-all "parallel" --filter-out "stress-split" -p 75

Seems to have an issue with how the task is defined:
http://pulpito.front.sepia.ceph.com/lflores-2022-08-05_18:10:59-upgrade-main-distro-default-smithi/6959779/
Invalid task definition: {'branch': 'octopus', 'clients': {'client.0': ['cls']}, 'workunit': None}

Rhel failures are unrelated.

thanks for testing @ljflores, i think i messed up some indents

i also added some more info to https://tracker.ceph.com/issues/55853, and none of those upgrade failures look like they're due to missing backports. i'll try to get someone to dig a little deeper into the failures

@cbodley retesting here: http://pulpito.front.sepia.ceph.com/lflores-2022-08-09_14:35:32-upgrade-main-distro-default-smithi/

./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c main -S 4bc2f657124bc7b731b0ce4c7516e9f09e21c5a3 --suite-repo https://github.com/cbodley/ceph.git --suite-branch wip-55853 -r lflores-2022-08-05_18:10:59-upgrade-main-distro-default-smithi -p 75

jenkins retest this please

@cbodley I meant to give this another review. Looks like the most recent tests I ran still were experiencing issues.

I did a fresh run here. We'll probably get the same results, but this is just to make sure we're looking at the most recent results:
http://pulpito.front.sepia.ceph.com:80/lflores-2022-08-23_21:30:37-upgrade-main-distro-default-smithi/

Let me know if you want to retest again. As far as I'm concerned, this PR seems like a viable solution. If the changes aren't easily backportable, it makes sense to run cls tests before and after the upgrade. If we can get at least one more +1 from someone familiar with upgrade tests (@neha-ojha @jdurgin ?) I'd be okay with merging this. (Of course when runs are green).

@ljflores my understanding from https://tracker.ceph.com/issues/55853#note-24 is that the relevant fixes have all been backported, so i can't tell why these are failing. i'm hoping we won't need to change these upgrade tests, but it needs more investigation on our end

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved