I would like to believe that the number of consumers is indeed small, but experience tells that these assumptions are usually wrong when it comes to Prometheus.

If it's not an excessive amount of effort, I would like to have both ways in parallel for a while so that users can migrate gradually. Once the new way has proven itself, we can deprecate the old way and then drop the old way a minor release later or so (and then have the simplification we want after all).

(Note: I still need to look at the code in detail, which might change my opinion above. ;)

Sorry, this is still on my list. I'm just too busy with company-internal things at the moment. I hope I get to this before the week ends.

Thanks @beorn7 for the review.

One thing, though: I think we should (or need to) relax the requirement of "no annotations" for a simple eval command. That's a change of behavior, but it won't let any test fail that passes now (because it relaxes passing criteria). However, we clearly have to state that to keep the old strict behavior, you must add a expect no_warn and expect no_info now.

Since this is requested, we can relax the "no annotations" for a simple eval cmd. So for this do we need to add both expect no_warn and expect no_info to each line that are simple eval statements?

More general TODOs:
Update documentation (in promql/promqltest/README.md) with the new stuff.
Mark the old stuff as deprecated in the documentation.
Ideally, provide conversion scripts (old → new) in the documentation (sed line or something) or as a file (if it's an actual program of some kind in a language of choice – back in my days, I used Perl for that, but I guess that would look weird today… ;).

Thanks. I will update the documentation for these TODOs.

As discussed separately, we do not want to add expect no_warn and expect no_info to each and every test that formerly used a simple eval. Instead, we want to add those explicit expectation of no annotations only to those tests that have the intention of testing for the lack of annotations. A typical example would be if there is one test testing how an annotation triggers, and then there is another test just before or after testing something similar but with a change to prevent the annotation to show up.

About the documentation update:

What we need in this PR is an update of the documentation (in promql/promqltest/README.md) with the new stuff, and a deprecation note for the old stuff. Also, very important, the note that a simple eval, while formerly asserting that there are no annotations, will now tolerate annotations. So in the (probably rare) case that the tester intends to assert that there are no annotations, they now have to add the no_warn/no_info lines.

We don't necessarily need the conversion scripts in this PR, but we should give written instructions how the old and the now way looks like. This could go along with the stuff above (like discussing when to add no_warn/no_info).

As discussed with @NeerajGartia21 separately, we'll have a separate PR to convert our own tests (those in the prometheus/prometheus repo). We can then also discuss where we want the explicit no_warn/no_info lines. And we might want to add tests for specific annotation content at the same time.

BTW, given that adding the no_warn/no_info lines and adding tests for specific annotation content needs additional information and also best judgement, maybe we should not strive for very sophisticated auto-conversion. Just do the minimal thing to convert old syntax into valid new syntax, but without no_warn/no_info lines. @NeerajGartia21 while working on the conversion, you could explore what's the best way to do so in a scripted fashion.

@beorn7 I have added the suggestions and updated the ReadMe with the new stuffs. Please have another look.

@charleskorn do you have any comments or concerns left?

Sorry, I've had a couple of other things on my plate, I'll try to review this today or early next week if not.

Tiny documentation improvements, but otherwise looks good.

I have updated the documentation based on your suggestions.

Thank you. Let's wait a bit for @charleskorn .

I've updated the PR. Please have a look, @charleskorn, @beorn7.

@NeerajGartia21 thank you very much for following up.

@charleskorn are all your concerns addressed?

KubeCon week is starting, so I'll have limited availability during the coming week, but it would be great if we could wrap this up soon.

In fact, if we get this merged before Wednesday, we could give the test conversions (including maybe a script) to attendees of the Prometheus contrib fest as one of the possible tasks. See https://kccnceu2025.sched.com/event/1tcx4/contribfest-contribution-guide-and-workshop-help-us-improve-the-prometheus-ecosystem

Thanks @charleskorn @beorn7 for the reviews!

Great. I'll give this a final look and then merge.
And we'll propose the test conversion as a ContribFest task tomorrow. (@NeerajGartia21 no need for you to work on it right now. Let's first see what kind of contribution we get from the ContribFest.)

Created #16367 for the conversion.

And we'll propose the test conversion as a ContribFest task tomorrow. (@NeerajGartia21 no need for you to work on it right now. Let's first see what kind of contribution we get from the ContribFest.)

That's great! Meanwhile, I'll work on other issues.