Skip to content

Test runner changes#7278

Merged
laurmaedje merged 8 commits intomainfrom
test-suite
Nov 17, 2025
Merged

Test runner changes#7278
laurmaedje merged 8 commits intomainfrom
test-suite

Conversation

@saecki
Copy link
Member

@saecki saecki commented Oct 31, 2025

Continuing from #7239. This is the first step in adding support for PDF and SVG tests. These tests will only store a hash of the reference output to reduce the storage needed for tests. I think the PDFs generated should be fully deterministic, but maybe @LaurenzV knows more?

The plan is to add a test tool later on that can build the test suite at another git revision on test failure to compare the previous and current reference output. Then static HTML is generated that allows conveniently inspecting the changes/diffs. The tool will be usable both in PRs and locally.

This PR changes a few things in the test suite:

  • Move the render references to a sub directory
  • Replace render with the paged attribute which always runs render, pdf, and svg output.
  • Replace nopdfua with pdfstandard which also applies to the pdf output
  • Require at least one of paged, html, or pdftags on a test, there is no default target anymore.
  • The pdf output always runs both the standard and the PDF/UA-1 export
  • pdftags can either be used in conjunction with paged or alone. If used alone only a tag tree reference is stored.
  • pdftags are now generated from a finished PDF document using hayro
    • This allows reading a few extra attributes:
      • /Lang, Alt, ActualText, and E of marked content sequences
      • /Contents and /URI of link annotations.
    • There are also some other changes to the tag tree
      • The structure element types and attribute names now match the PDF spec exactly
      • BBoxes are formatted a little more compact
  • Add a --stages flag to only run specific which test targets/outputs
    • This is useful to update only one test output
  • Fix running only parser tests with --no-default-features

Currently I've disabled pdf and svg references since there currently is no easy way to get the previous reference output, if there is a hash mismatch.

I had to update krilla because empty table cell headers were previously omitted from PDFs, and it now requires rust 1.89.

@laurmaedje laurmaedje added refactor A refactoring. internals About internal changes to the codebase. labels Nov 7, 2025
@saecki saecki force-pushed the test-suite branch 8 times, most recently from 3463656 to 228d1c0 Compare November 11, 2025 13:44
@saecki saecki mentioned this pull request Nov 12, 2025
Copy link
Member

@laurmaedje laurmaedje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking very nice! I left a few comments. One general remark is that it would be nice if you could add brief doc comments to at least all structs and ideally also methods.

Regarding merge: I'd like to keep the commits in principle. However, I think in the way it's currently set up, not all intermediate commits pass the tests. (Most importantly because of the last "Disable hashed references for now" commit.) Do you think it would be feasible to rebase in a way that lets each commit pass in isolation? Otherwise, I'm also fine with squashing. Let me know what you prefer.

To get the fuzzer tests to pass, I guess we also need to update the nightly toolchain used for fuzzing? I'd need to find a good nightly and sync the nightly versions across here and the web app. Perhaps, it would be worthwhile to decouple the whole krilla update from this PR. So basically: Update nightly toolchain for tests -> Update krilla -> Test runner changes. I can take care of the first PR.

@saecki saecki mentioned this pull request Nov 12, 2025
@saecki saecki force-pushed the test-suite branch 4 times, most recently from 86c9f5f to 8a2c869 Compare November 13, 2025 11:27
@saecki
Copy link
Member Author

saecki commented Nov 13, 2025

One general remark is that it would be nice if you could add brief doc comments to at least all structs and ideally also methods.

Done

Regarding merge: I'd like to keep the commits in principle. However, I think in the way it's currently set up, not all intermediate commits pass the tests. (Most importantly because of the last "Disable hashed references for now" commit.) Do you think it would be feasible to rebase in a way that lets each commit pass in isolation? Otherwise, I'm also fine with squashing. Let me know what you prefer.

The commits should all pass CI now

To get the fuzzer tests to pass, I guess we also need to update the nightly toolchain used for fuzzing? I'd need to find a good nightly and sync the nightly versions across here and the web app. Perhaps, it would be worthwhile to decouple the whole krilla update from this PR. So basically: Update nightly toolchain for tests -> Update krilla -> Test runner changes. I can take care of the first PR.

I've opened a separate PR to Update krilla

@saecki saecki force-pushed the test-suite branch 4 times, most recently from 16af406 to aa5e565 Compare November 13, 2025 22:48
@saecki
Copy link
Member Author

saecki commented Nov 14, 2025

The test suite might be a little bit slower now that svg and pdf are also tested and due to #7365.
Running cargo testit --stages=render,html,pdftags should be just as fast as before.

@saecki saecki force-pushed the test-suite branch 2 times, most recently from b5ecbf5 to bc3d0e5 Compare November 14, 2025 13:41
@saecki saecki marked this pull request as ready for review November 14, 2025 15:31
@laurmaedje
Copy link
Member

laurmaedje commented Nov 14, 2025

This LGTM apart from one thing: The VS Code test helper extension has some issues now. This line needs adjustment with the parameters

const re = /^--- ([\d\w-]+)(( [\d\w-]+)*) ---$/;

And the preview also does not work correctly for tests that are have both paged and html. The extension does some light parsing of the attributes which needs to be adjusted.

@saecki
Copy link
Member Author

saecki commented Nov 17, 2025

I think that should fix it

@laurmaedje laurmaedje merged commit 970b0d5 into main Nov 17, 2025
16 checks passed
@laurmaedje laurmaedje deleted the test-suite branch November 17, 2025 15:05
YDX-2147483647 added a commit to YDX-2147483647/typst that referenced this pull request Nov 19, 2025
YDX-2147483647 added a commit to YDX-2147483647/typst that referenced this pull request Nov 19, 2025
YDX-2147483647 added a commit to YDX-2147483647/typst that referenced this pull request Nov 19, 2025
RungeCC added a commit to RungeCC/typst that referenced this pull request Nov 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internals About internal changes to the codebase. refactor A refactoring.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants