⚠️ This is based on another PR (#13886) that is currently under review and should be merged first. It is sufficient to review starting with commit 551b34e

drive-by comment: I do that quite often too and I find that selecting the previous PR's branch as the merge target for the second PR is quite handy to keep things clear.

Thanks @kaeluka! Just to double check my understanding: this design does mean that during normal operation (i.e. not regression testing) we need to implement a filter in the script that will skip the candidates with an ai-{foo} provenance, correct?

In practice I imagine that the majority of candidates will not have an AI-provenance and so that skipping will be the exception (and so won't impact performance) but ooi have we also considered a 2 queries design whereby we would add a AutomodelApplicationModeExtractCandidatesRegression.ql query?

Just to double check my understanding: this design does mean that during normal operation (i.e. not regression testing) we need to implement a filter in the automodel.py script that will skip the candidates with an ai-{foo} provenance, correct?

Yes! 👍

In practice I imagine that the majority of candidates will not have an AI-provenance and so that skipping will be the exception (and so won't impact performance) but ooi have we also considered a 2 queries design whereby we would add a AutomodelApplicationModeExtractCandidatesRegression.ql query?

I have, and have discussed that in our standup on Monday, remember? Two queries will mean increased maintenance load, even though the two queries would be almost exactly the same.

drive-by comment: I do that quite often too and I find that selecting the previous PR's branch as the merge target for the second PR is quite handy to keep things clear.

Thanks! I considered that but have never done it before because I wasn't sure how the UI would present that! Will do that next time ;)

Edit: now that the 'base' branch was merged, I could update this PR and it only shows the new commits as intended.

Shall we have similar changes for framework mode in the same PR?

Two queries will mean increased maintenance load, even though the two queries would be almost exactly the same.

It feels a little weird to always extract all candidates instead of just the ones we need (seems like the wrong away around) but we can easily modify that later if need be.

In the meantime we should probably avoiding merging this until we have the filter implemented in the script.