Skip to content

feat(cli): show parse times in testing output#3939

Merged
amaanq merged 4 commits intotree-sitter:masterfrom
WillLillis:test_timers
Dec 23, 2024
Merged

feat(cli): show parse times in testing output#3939
amaanq merged 4 commits intotree-sitter:masterfrom
WillLillis:test_timers

Conversation

@WillLillis
Copy link
Member

@WillLillis WillLillis commented Nov 13, 2024

This adds timing information in two forms for the cli's test subcommand. By default, the CLI will now print out some timing stats at the end of a run. For example, running tree-sitter test on tree-sitter-c now looks like:

ambiguities:
      1. ✓ pointer declarations vs expressions
      2. ✓ casts vs multiplications
      3. ✓ function-like type macros vs function calls
     ...
     83. ✓ Attributes
  types:
     84. ✓ Primitive types
     85. ✓ Type modifiers

Total parses: 85; successful parses: 85; failed parses: 0; success percentage: 100.00%; average speed: 2058 bytes/ms

syntax highlighting:
    ✓ keywords.c (3 assertions)
    ✓ names.c (20 assertions)

In addition, running tree-sitter test --show-times will display the parsing time for each individual test case. The use case outlined in #3929 describes finding tests with slow parse rates. Simply printing the parse times isn't useful in this regard (tests vary a lot from run to run even on the same hardware). Providing the parse rate is more useful, but still leaves the user with a lot of guess work and manual checking. A solution I arrived at is to check for statistically significant slow parse rates. The process is basically as follows:

  1. When tree-sitter test is run, parse all relevant test cases and record their adjusted (logarithmic) parse rates.
  • Log parse rates are recorded in order to mitigate against the effect of outliers, which leads to a large standard deviation, making our statistical tests useless. This is just a heuristic.
  1. Calculate the mean and standard deviation of all parse log parse rates.
  2. Run all the tests normally. When each test is run, if its adjusted parsing rate is more than 3 standard deviations below the mean parsing rate (the "Empirical Rule"), a warning is printed alongside the test name.

By default the --stat outliers-and-total option is passed, meaning that running tree-sitter test will display a warning if any tests have a slow parse rate. If --stat all is passed, the parse rates for every test will be displayed as well as any outliers. Finally, passing --stat total-only will only display the total parsing statistics at the end.

This could look like the following for tree-sitter-c, which has one test that occasionally triggers this check:

❯ ts_dev test
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.08s
  ambiguities:
      1. ✓ pointer declarations vs expressions
      2. ✓ casts vs multiplications
      3. ✓ function-like type macros vs function calls
     ...
     57. ✓ Top Level Empty Expression Statement -- Warning: Slow parse rate (333.333bytes/ms)
     ...
     83. ✓ Attributes
  types:
     84. ✓ Primitive types
     85. ✓ Type modifiers

Total parses: 85; successful parses: 85; failed parses: 0; success percentage: 100.00%; average speed: 2143 bytes/ms

syntax highlighting:
    ✓ keywords.c (3 assertions)
    ✓ names.c (20 assertions)

And here's a screenshot of the --stat all flag being passed for good measure:

image

Closes #3929

Copy link
Member

@amaanq amaanq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks cool, but maybe we can restructure this to have multiple "options" for showing stats, like I mentioned offline (enum for all stats, outliers + total (default), and just total).

@WillLillis

This comment was marked as resolved.

@WillLillis
Copy link
Member Author

WillLillis commented Dec 20, 2024

(In case anyone is curious)

Performance comparison against master for the original approach in this PR:

image

After 251d7ed (only runs the test corpus once):

image

Copy link
Member

@amaanq amaanq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome work on the perf improvement, looks much better!

@amaanq amaanq merged commit 5d9870e into tree-sitter:master Dec 23, 2024
@WillLillis WillLillis deleted the test_timers branch December 23, 2024 06:24
@WillLillis WillLillis mentioned this pull request Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Corpus test: time elapsed

2 participants