Isn't it weird that tests for errors, which don't generate a reference, have to specify a target?

I think there should be a distinction between target in the sense of std.target (i.e., paged vs. HTML) and output format, which in the case of tests would be PNG, HTML, or PDF tags (but in the future could include SVG as well). A test should always have a target (or maybe even allow multiple targets to test) because Typst code may depend on the target. However, the output format list should be allowed to be empty, for example in the case of tests that test for errors or warnings.

This distinction is similar to the way the query CLI command allows specifying an export target through --target, but not a specific output format.

That seems sensible. The export format should imply a target, so that we don't have to specify things twice.

Isn't it weird that tests for errors, which don't generate a reference, have to specify a target?

I think there should be a distinction between target in the sense of std.target (i.e., paged vs. HTML) and output format, which in the case of tests would be PNG, HTML, or PDF tags (but in the future could include SVG as well). A test should always have a target (or maybe even allow multiple targets to test) because Typst code may depend on the target. However, the output format list should be allowed to be empty, for example in the case of tests that test for errors or warnings.

Good point, there will be. Also to avoid compiling the document multiple times with the paged target. But I'm not sure how that's gonna look like. There could be the paged attribute, but does that imply svg, pdf and render outputs, or does it just run the paged compile step? If it also runs the output, paged should be mutually exclusive with the specific output types.

The main idea was to introduce these changes as I start working on the test suite to minimize conflicts. I think moving the render tests is quite uncontroversial, but now I'm unsure about the render attribute for tests that error.

I'd have assumed that svg, pdf, and render imply paged, but not the other way around. If you'd write just paged, you'd get no reference output at all. However with TYPST_TESTS_EXTENDED, we could still check whether any of those targets crashes.

In this setup, however, the purpose of paged would be to check for errors, which is not particularly clear. Perhaps, we should instead introduce a diagnostic target that expects at least one annotated diagnostic and requires paged or html in addition? This would make the intent clearer.

There could also be two distinct targets for diagnostics: diagnostic-paged and diagnostic-html. Because paged only makes sense for diagnostics, and html for diagnostics may become more general than the html output format in the future, with things like EPUB.

could be, but I think having two might be more elegant and compose better if we ever add another thing like diagnostic

So I gave this all a bit of thought while working on the test suite: main...test-suite
Maybe we can land something close to this before continuing.

I'd have assumed that svg, pdf, and render imply paged, but not the other way around. If you'd write just paged, you'd get no reference output at all. However with TYPST_TESTS_EXTENDED, we could still check whether any of those targets crashes.

This is how I've implemented it now.

In this setup, however, the purpose of paged would be to check for errors, which is not particularly clear. Perhaps, we should instead introduce a diagnostic target that expects at least one annotated diagnostic and requires paged or html in addition? This would make the intent clearer.

I've also implemented this, but I think diagnostic should work more like flag (not a full target/output). The diagnostic attribute requires at least one annotated diagnostic note and won't save any reference output. The seen flag is now a bitflags field that tracks where each error has been seen. We could assert that the errors must be present in all specified targets (paged or html) and outputs (render, pdf, pdftags, or svg). Although here probably only pdf makes sense.

Regarding PDF, I think it makes sense to have some way of specifying the standard, and get rid of the nopdfua attribute.
I was thinking of introducing a new attribute syntax similar to function calls: pdf-standard(ua-1), or allow specyfing the standard in the test body similar to how notes are currently parsed: // PDF-standard: UA-1. But I think I'd prefer having this in the attributes.
In the function call syntax it could also be an argument to pdf or pdftags. But it could be somewhat confusing to know to which of these the standard applies to, and it would also allow specifying different standards which is probably not that useful. I would rather like to add a way (maybe through a feature flag) to generate both the PDF document and the formatted tag tree in one run.

There could also be two distinct targets for diagnostics: diagnostic-paged and diagnostic-html. Because paged only makes sense for diagnostics ...

I didn't think of it before, but the paged target also makes sense for tests that only assert certain values using the test(a, b) function.

... and html for diagnostics may become more general than the html output format in the future, with things like EPUB.

This sounds reasonable, but I think we can deal with that once this becomes an issue, for now it would only complicate things and would require us to come up with different names for the html target and output.

Closing in favor of #7278