Test Refactor by WillLillis · Pull Request #4908 · tree-sitter/tree-sitter

WillLillis · 2025-09-29T03:44:57Z

This PR aims to refactor a good chunk of the CLI tool's internal testing logic. The only user-facing change should be the addition of the --json-summary flag, which outputs all test results in a JSON format. Internally, there are a few changes:

cleanup the filtering logic a bit for parse tests. Also, only update a test case's expected output if the --update flag has been passed.
Collect all test results as structured data, rather than printing results as we go. Once all tests have been run (or, if an unrecoverable error occurrs), we then display the results all at once. All of the mutable state which previously lived in opts now belongs to test_summary.
Greatly declutter the logic displaying parse rates alongside individual test case's pass/fail status. Parse rates are now logically associated along with their associated test case, rather than "zipped" up at the end in the display logic.
Generalize the display of test diffs a bit. This was mostly so the diff key and test case diffs could be used inside of TestSummary's Display implementation.

This PR also deprecates the --json flags in favor of --json-summary.

tree-sitter-c json test report

{
  "parse_results": [
    {
      "name": "ambiguities",
      "children": [
        {
          "name": "pointer declarations vs expressions",
          "outcome": "Passed",
          "parse_rate": 2.772817603707442,
          "test_num": 1
        },
        {
          "name": "casts vs multiplications",
          "outcome": "Passed",
          "parse_rate": 1706.0606562536232,
          "test_num": 2
        },
        {
          "name": "function-like type macros vs function calls",
          "outcome": "Passed",
          "parse_rate": 3052.5877618981544,
          "test_num": 3
        },
        {
          "name": "function calls vs parenthesized declarators vs macro types",
          "outcome": "Passed",
          "parse_rate": 254.71817733995675,
          "test_num": 4
        },
        {
          "name": "Call expressions vs empty declarations w/ macros as types",
          "outcome": "Passed",
          "parse_rate": 1783.2222049930222,
          "test_num": 5
        },
        {
          "name": "Comments after for loops with ambiguities",
          "outcome": "Passed",
          "parse_rate": 33.21285186802364,
          "test_num": 6
        },
        {
          "name": "Top-level macro invocations",
          "outcome": "Passed",
          "parse_rate": 1488.5549782333942,
          "test_num": 7
        }
      ]
    },
    {
      "name": "crlf",
      "children": [
        {
          "name": "Line comments with escaped CRLF line endings",
          "outcome": "Passed",
          "parse_rate": 10115.157173980702,
          "test_num": 8
        }
      ]
    },
    {
      "name": "declarations",
      "children": [
        {
          "name": "Struct declarations",
          "outcome": "Passed",
          "parse_rate": 286.3490496173784,
          "test_num": 9
        },
        {
          "name": "Union declarations",
          "outcome": "Passed",
          "parse_rate": 4849.137931034483,
          "test_num": 10
        },
        {
          "name": "Enum declarations",
          "outcome": "Passed",
          "parse_rate": 195.72193575949055,
          "test_num": 11
        },
        {
          "name": "Struct declarations containing preprocessor directives",
          "outcome": "Passed",
          "parse_rate": 1431.2575421659699,
          "test_num": 12
        },
        {
          "name": "Primitive-typed variable declarations",
          "outcome": "Passed",
          "parse_rate": 2258.0822823153603,
          "test_num": 13
        },
        {
          "name": "Variable storage classes",
          "outcome": "Passed",
          "parse_rate": 281.08975496068837,
          "test_num": 14
        },
        {
          "name": "Composite-typed variable declarations",
          "outcome": "Passed",
          "parse_rate": 3107.5596144089295,
          "test_num": 15
        },
        {
          "name": "Pointer variable declarations",
          "outcome": "Passed",
          "parse_rate": 5190.796857463524,
          "test_num": 16
        },
        {
          "name": "Typedefs",
          "outcome": "Passed",
          "parse_rate": 1822.489865593738,
          "test_num": 17
        },
        {
          "name": "Function declarations",
          "outcome": "Passed",
          "parse_rate": 3670.833139723991,
          "test_num": 18
        },
        {
          "name": "Function definitions",
          "outcome": "Passed",
          "parse_rate": 353.91850043420555,
          "test_num": 19
        },
        {
          "name": "Function specifiers after types",
          "outcome": "Passed",
          "parse_rate": 4266.280286565242,
          "test_num": 20
        },
        {
          "name": "Function definitions with macro attributes",
          "outcome": "Passed",
          "parse_rate": 5470.980019029495,
          "test_num": 21
        },
        {
          "name": "Linkage specifications",
          "outcome": "Passed",
          "parse_rate": 176.14161786076005,
          "test_num": 22
        },
        {
          "name": "Type qualifiers",
          "outcome": "Passed",
          "parse_rate": 49.66529907147485,
          "test_num": 23
        },
        {
          "name": "Local array declarations",
          "outcome": "Passed",
          "parse_rate": 4574.042309891366,
          "test_num": 24
        },
        {
          "name": "Attributes",
          "outcome": "Passed",
          "parse_rate": 167.5651170348062,
          "test_num": 25
        },
        {
          "name": "More Assembly",
          "outcome": "Passed",
          "parse_rate": 5007.6306753147655,
          "test_num": 26
        },
        {
          "name": "Static in Array Declarations",
          "outcome": "Passed",
          "parse_rate": 3418.601909701757,
          "test_num": 27
        }
      ]
    },
    {
      "name": "expressions",
      "children": [
        {
          "name": "Number literals",
          "outcome": "Passed",
          "parse_rate": 523.2388769313878,
          "test_num": 28
        },
        {
          "name": "Identifiers",
          "outcome": "Passed",
          "parse_rate": 2472.646349755826,
          "test_num": 29
        },
        {
          "name": "Unicode Identifiers",
          "outcome": "Passed",
          "parse_rate": 4602.312957281095,
          "test_num": 30
        },
        {
          "name": "Common constants",
          "outcome": "Passed",
          "parse_rate": 9379.430690369723,
          "test_num": 31
        },
        {
          "name": "Function calls",
          "outcome": "Passed",
          "parse_rate": 5015.7904514211405,
          "test_num": 32
        },
        {
          "name": "GNU inline assembly",
          "outcome": "Passed",
          "parse_rate": 6939.357723890666,
          "test_num": 33
        },
        {
          "name": "Function call with compound statement",
          "outcome": "Passed",
          "parse_rate": 7468.259895444362,
          "test_num": 34
        },
        {
          "name": "String literals",
          "outcome": "Passed",
          "parse_rate": 6150.472012968437,
          "test_num": 35
        },
        {
          "name": "Character literals",
          "outcome": "Passed",
          "parse_rate": 5703.537406710758,
          "test_num": 36
        },
        {
          "name": "Field access",
          "outcome": "Passed",
          "parse_rate": 4598.123965422108,
          "test_num": 37
        },
        {
          "name": "Boolean operators",
          "outcome": "Passed",
          "parse_rate": 3390.8754623921086,
          "test_num": 38
        },
        {
          "name": "Math operators",
          "outcome": "Passed",
          "parse_rate": 3746.7629070472203,
          "test_num": 39
        },
        {
          "name": "The comma operator",
          "outcome": "Passed",
          "parse_rate": 3851.2117227127565,
          "test_num": 40
        },
        {
          "name": "Assignments",
          "outcome": "Passed",
          "parse_rate": 4041.3006914595403,
          "test_num": 41
        },
        {
          "name": "Pointer operations",
          "outcome": "Passed",
          "parse_rate": 4009.284659210804,
          "test_num": 42
        },
        {
          "name": "Type-casts",
          "outcome": "Passed",
          "parse_rate": 4886.339494370087,
          "test_num": 43
        },
        {
          "name": "Sizeof expressions",
          "outcome": "Passed",
          "parse_rate": 5101.769676356486,
          "test_num": 44
        },
        {
          "name": "Alignof expressions",
          "outcome": "Passed",
          "parse_rate": 6841.439576865586,
          "test_num": 45
        },
        {
          "name": "Offsetof expressions",
          "outcome": "Passed",
          "parse_rate": 4267.354736267553,
          "test_num": 46
        },
        {
          "name": "Compound literals",
          "outcome": "Passed",
          "parse_rate": 4680.808684393985,
          "test_num": 47
        },
        {
          "name": "Compound literals with trailing commas",
          "outcome": "Passed",
          "parse_rate": 5364.142768722922,
          "test_num": 48
        },
        {
          "name": "Comments with escaped newlines",
          "outcome": "Passed",
          "parse_rate": 7575.757575757575,
          "test_num": 49
        },
        {
          "name": "Comments with escaped chars and newlines",
          "outcome": "Passed",
          "parse_rate": 16106.110847939364,
          "test_num": 50
        },
        {
          "name": "Generic Expressions",
          "outcome": "Passed",
          "parse_rate": 6460.116934641888,
          "test_num": 51
        },
        {
          "name": "Noreturn Type Qualifier",
          "outcome": "Passed",
          "parse_rate": 4731.370229720475,
          "test_num": 52
        },
        {
          "name": "Restrict Type Qualifier",
          "outcome": "Passed",
          "parse_rate": 4606.733961113746,
          "test_num": 53
        },
        {
          "name": "Ternary",
          "outcome": "Passed",
          "parse_rate": 3980.6161301488405,
          "test_num": 54
        },
        {
          "name": "Concatenated strings",
          "outcome": "Passed",
          "parse_rate": 4658.491470850175,
          "test_num": 55
        },
        {
          "name": "Nonnull Type Qualifier",
          "outcome": "Passed",
          "parse_rate": 4346.158762623035,
          "test_num": 56
        },
        {
          "name": "Top Level Empty Expression Statement",
          "outcome": "Passed",
          "parse_rate": 1179.245283018868,
          "test_num": 57
        },
        {
          "name": "Sized Type Specifier With Type Qualifiers",
          "outcome": "Passed",
          "parse_rate": 2974.7149231531976,
          "test_num": 58
        }
      ]
    },
    {
      "name": "microsoft",
      "children": [
        {
          "name": "declaration specs",
          "outcome": "Passed",
          "parse_rate": 7153.356197942185,
          "test_num": 59
        },
        {
          "name": "pointers",
          "outcome": "Passed",
          "parse_rate": 4615.279579436063,
          "test_num": 60
        },
        {
          "name": "call modifiers",
          "outcome": "Passed",
          "parse_rate": 5407.513323800858,
          "test_num": 61
        },
        {
          "name": "SEH exception handling",
          "outcome": "Passed",
          "parse_rate": 7301.859461377006,
          "test_num": 62
        }
      ]
    },
    {
      "name": "preprocessor",
      "children": [
        {
          "name": "Include directives",
          "outcome": "Passed",
          "parse_rate": 8254.831504321646,
          "test_num": 63
        },
        {
          "name": "Object-like macro definitions",
          "outcome": "Passed",
          "parse_rate": 18574.42562554367,
          "test_num": 64
        },
        {
          "name": "Function-like macro definitions",
          "outcome": "Passed",
          "parse_rate": 8989.764129951045,
          "test_num": 65
        },
        {
          "name": "Ifdefs",
          "outcome": "Passed",
          "parse_rate": 19.740301631808933,
          "test_num": 66
        },
        {
          "name": "Elifdefs",
          "outcome": "Passed",
          "parse_rate": 2466.184812095688,
          "test_num": 67
        },
        {
          "name": "Mixing #elif and #elifdef",
          "outcome": "Passed",
          "parse_rate": 5503.372367277692,
          "test_num": 68
        },
        {
          "name": "General if blocks",
          "outcome": "Passed",
          "parse_rate": 2717.291968757235,
          "test_num": 69
        },
        {
          "name": "Preprocessor conditionals in functions",
          "outcome": "Passed",
          "parse_rate": 40.32549236957283,
          "test_num": 70
        },
        {
          "name": "Preprocessor conditionals in struct/union bodies",
          "outcome": "Passed",
          "parse_rate": 2594.4857496902105,
          "test_num": 71
        },
        {
          "name": "Unknown preprocessor directives",
          "outcome": "Passed",
          "parse_rate": 58.88241182358829,
          "test_num": 72
        },
        {
          "name": "Preprocessor expressions",
          "outcome": "Passed",
          "parse_rate": 2225.1891410769913,
          "test_num": 73
        }
      ]
    },
    {
      "name": "statements",
      "children": [
        {
          "name": "If statements",
          "outcome": "Passed",
          "parse_rate": 1209.8511717325732,
          "test_num": 74
        },
        {
          "name": "For loops",
          "outcome": "Passed",
          "parse_rate": 3147.4820143884895,
          "test_num": 75
        },
        {
          "name": "While loops",
          "outcome": "Passed",
          "parse_rate": 3574.2652899126288,
          "test_num": 76
        },
        {
          "name": "Labeled statements",
          "outcome": "Passed",
          "parse_rate": 75.50909648646736,
          "test_num": 77
        },
        {
          "name": "Switch statements",
          "outcome": "Passed",
          "parse_rate": 111.43406353598856,
          "test_num": 78
        },
        {
          "name": "Case statements separate from switch statements",
          "outcome": "Passed",
          "parse_rate": 3812.5979463429467,
          "test_num": 79
        },
        {
          "name": "Return statements",
          "outcome": "Passed",
          "parse_rate": 4375.631100639515,
          "test_num": 80
        },
        {
          "name": "Comments with asterisks",
          "outcome": "Passed",
          "parse_rate": 19662.921348314605,
          "test_num": 81
        },
        {
          "name": "Comment with multiple backslashes",
          "outcome": "Passed",
          "parse_rate": 5696.326851306227,
          "test_num": 82
        },
        {
          "name": "Attributes",
          "outcome": "Passed",
          "parse_rate": 746.3034656467187,
          "test_num": 83
        }
      ]
    },
    {
      "name": "types",
      "children": [
        {
          "name": "Primitive types",
          "outcome": "Passed",
          "parse_rate": 5210.089184467805,
          "test_num": 84
        },
        {
          "name": "Type modifiers",
          "outcome": "Passed",
          "parse_rate": 2609.2020966802565,
          "test_num": 85
        }
      ]
    }
  ],
  "parse_failures": [],
  "parse_stats": {
    "successful_parses": 85,
    "total_parses": 85,
    "total_bytes": 12658,
    "total_duration": {
      "secs": 0,
      "nanos": 101925265
    }
  },
  "highlight_results": [
    {
      "name": "names.c",
      "outcome": {
        "AssertionPassed": {
          "assertion_count": 20
        }
      },
      "test_num": 1
    },
    {
      "name": "keywords.c",
      "outcome": {
        "AssertionPassed": {
          "assertion_count": 3
        }
      },
      "test_num": 2
    }
  ],
  "tag_results": [],
  "query_results": []
}

Closes Generate pretty test report #3891

ObserverOfTime · 2025-09-29T07:12:50Z

Can the results actually be grouped instead of adding groups in between tests?
If nesting the tests inside groups is too hard we can just add the group name to the test object.

ObserverOfTime · 2025-09-29T17:56:40Z

This PR also deprecates the --json flags in favor of --json-summary.

Why not just keep the original name?

maxbrunsfeld

The JSON structure is a bit weird.

{
 "name": "Type modifiers",
 "info": {
   "ParseTest": {
     "outcome": "Passed",
     "parse_rate": [
       4094.764551038315,
       8.31746449794023
     ],
     "test_num": 85
   }
 }

}

I can't tell what the two parse_rate numbers are supposed to represent. Why is there test_num instead of just listing the tests in order in the array? Also, it's a bit weird that the info is nested under both info and ParseTest. Seems like those fields should just be top-level, alongside name.

crates/cli/src/main.rs

WillLillis · 2025-09-29T21:07:08Z

The JSON structure is a bit weird.

{
 "name": "Type modifiers",
 "info": {
   "ParseTest": {
     "outcome": "Passed",
     "parse_rate": [
       4094.764551038315,
       8.31746449794023
     ],
     "test_num": 85
   }
 }

Taking another look at this, agreed. I'll spend some time re-working how this information is serialized before we present it to the user.

I can't tell what the two parse_rate numbers are supposed to represent.

The first is the true parse rate, and the second is an adjusted parse rate used to detect statistically significant slow parse rates (#3939). The second number is definitely an internal implementation detail, and can be removed from the output.

Why is there test_num instead of just listing the tests in order in the array?

Tests can be filtered out (with --include and --exclude), meaning that not all test cases will appear in the output (json or regular) if one of these flags is passed. One of the reasons we have these canonical test numbers is for ts parse -n, which can be useful for debugging.

Also, it's a bit weird that the info is nested under both info and ParseTest. Seems like those fields should just be top-level, alongside name.

Agreed yeah, those can be flattened those out.

Re what @ObserverOfTime mentioned with nesting the test cases underneath their containing groups, I'll take a stab at that later tonight along w/ the other changes you mentioned.

WillLillis · 2025-09-29T21:09:34Z

This PR also deprecates the --json flags in favor of --json-summary.

Why not just keep the original name?

@amaanq and I were looking over the other json-related flags in the CLI when adding this one for the test command. We decided json was a bit too vague for the generate and parse commands, and decided to unify all of them to use json-summary instead.

ObserverOfTime · 2025-09-30T06:12:11Z

Since the parse command also accepts --xml and --cst, maybe we should consolidate them under --format.

ObserverOfTime · 2025-09-30T06:13:07Z

Also, is --report-states-for-rule supported in the JSON format?

WillLillis · 2025-10-01T04:59:39Z

Since the parse command also accepts --xml and --cst, maybe we should consolidate them under --format.

The json flag for parse outputs a json summary of the parsing, it doesn't output the tree in a json format like --xml and --cst do:

-> tree-sitter parse --json-summary
int x = 1;

{
  "parse_summaries": [
    {
      ...
  ],
  "cumulative_stats": {
   ...
    }
  },
  "source_count": 0
}

Also, is --report-states-for-rule supported in the JSON format?

No, but adding that should be fairly doable if desired.

WillLillis · 2025-10-01T08:48:35Z

After fighting with the borrow checker for awhile, I finally settled on creating TestResultHierarchy to aggregate our arbitrarily nested test groups and cases.

Here's what the JSON looks like for tree-sitter-c now:

{
  "parse_results": [
    {
      "name": "ambiguities",
      "children": [
        {
          "name": "pointer declarations vs expressions",
          "outcome": "Passed",
          "parse_rate": 2.772817603707442,
          "test_num": 1
        },
        {
          "name": "casts vs multiplications",
          "outcome": "Passed",
          "parse_rate": 1706.0606562536232,
          "test_num": 2
        },
        {
          "name": "function-like type macros vs function calls",
          "outcome": "Passed",
          "parse_rate": 3052.5877618981544,
          "test_num": 3
        },
        {
          "name": "function calls vs parenthesized declarators vs macro types",
          "outcome": "Passed",
          "parse_rate": 254.71817733995675,
          "test_num": 4
        },
        {
          "name": "Call expressions vs empty declarations w/ macros as types",
          "outcome": "Passed",
          "parse_rate": 1783.2222049930222,
          "test_num": 5
        },
        {
          "name": "Comments after for loops with ambiguities",
          "outcome": "Passed",
          "parse_rate": 33.21285186802364,
          "test_num": 6
        },
        {
          "name": "Top-level macro invocations",
          "outcome": "Passed",
          "parse_rate": 1488.5549782333942,
          "test_num": 7
        }
      ]
    },
    {
      "name": "crlf",
      "children": [
        {
          "name": "Line comments with escaped CRLF line endings",
          "outcome": "Passed",
          "parse_rate": 10115.157173980702,
          "test_num": 8
        }
      ]
    },
    {
      "name": "declarations",
      "children": [
        {
          "name": "Struct declarations",
          "outcome": "Passed",
          "parse_rate": 286.3490496173784,
          "test_num": 9
        },
        {
          "name": "Union declarations",
          "outcome": "Passed",
          "parse_rate": 4849.137931034483,
          "test_num": 10
        },
        {
          "name": "Enum declarations",
          "outcome": "Passed",
          "parse_rate": 195.72193575949055,
          "test_num": 11
        },
        {
          "name": "Struct declarations containing preprocessor directives",
          "outcome": "Passed",
          "parse_rate": 1431.2575421659699,
          "test_num": 12
        },
        {
          "name": "Primitive-typed variable declarations",
          "outcome": "Passed",
          "parse_rate": 2258.0822823153603,
          "test_num": 13
        },
        {
          "name": "Variable storage classes",
          "outcome": "Passed",
          "parse_rate": 281.08975496068837,
          "test_num": 14
        },
        {
          "name": "Composite-typed variable declarations",
          "outcome": "Passed",
          "parse_rate": 3107.5596144089295,
          "test_num": 15
        },
        {
          "name": "Pointer variable declarations",
          "outcome": "Passed",
          "parse_rate": 5190.796857463524,
          "test_num": 16
        },
        {
          "name": "Typedefs",
          "outcome": "Passed",
          "parse_rate": 1822.489865593738,
          "test_num": 17
        },
        {
          "name": "Function declarations",
          "outcome": "Passed",
          "parse_rate": 3670.833139723991,
          "test_num": 18
        },
        {
          "name": "Function definitions",
          "outcome": "Passed",
          "parse_rate": 353.91850043420555,
          "test_num": 19
        },
        {
          "name": "Function specifiers after types",
          "outcome": "Passed",
          "parse_rate": 4266.280286565242,
          "test_num": 20
        },
        {
          "name": "Function definitions with macro attributes",
          "outcome": "Passed",
          "parse_rate": 5470.980019029495,
          "test_num": 21
        },
        {
          "name": "Linkage specifications",
          "outcome": "Passed",
          "parse_rate": 176.14161786076005,
          "test_num": 22
        },
        {
          "name": "Type qualifiers",
          "outcome": "Passed",
          "parse_rate": 49.66529907147485,
          "test_num": 23
        },
        {
          "name": "Local array declarations",
          "outcome": "Passed",
          "parse_rate": 4574.042309891366,
          "test_num": 24
        },
        {
          "name": "Attributes",
          "outcome": "Passed",
          "parse_rate": 167.5651170348062,
          "test_num": 25
        },
        {
          "name": "More Assembly",
          "outcome": "Passed",
          "parse_rate": 5007.6306753147655,
          "test_num": 26
        },
        {
          "name": "Static in Array Declarations",
          "outcome": "Passed",
          "parse_rate": 3418.601909701757,
          "test_num": 27
        }
      ]
    },
    {
      "name": "expressions",
      "children": [
        {
          "name": "Number literals",
          "outcome": "Passed",
          "parse_rate": 523.2388769313878,
          "test_num": 28
        },
        {
          "name": "Identifiers",
          "outcome": "Passed",
          "parse_rate": 2472.646349755826,
          "test_num": 29
        },
        {
          "name": "Unicode Identifiers",
          "outcome": "Passed",
          "parse_rate": 4602.312957281095,
          "test_num": 30
        },
        {
          "name": "Common constants",
          "outcome": "Passed",
          "parse_rate": 9379.430690369723,
          "test_num": 31
        },
        {
          "name": "Function calls",
          "outcome": "Passed",
          "parse_rate": 5015.7904514211405,
          "test_num": 32
        },
        {
          "name": "GNU inline assembly",
          "outcome": "Passed",
          "parse_rate": 6939.357723890666,
          "test_num": 33
        },
        {
          "name": "Function call with compound statement",
          "outcome": "Passed",
          "parse_rate": 7468.259895444362,
          "test_num": 34
        },
        {
          "name": "String literals",
          "outcome": "Passed",
          "parse_rate": 6150.472012968437,
          "test_num": 35
        },
        {
          "name": "Character literals",
          "outcome": "Passed",
          "parse_rate": 5703.537406710758,
          "test_num": 36
        },
        {
          "name": "Field access",
          "outcome": "Passed",
          "parse_rate": 4598.123965422108,
          "test_num": 37
        },
        {
          "name": "Boolean operators",
          "outcome": "Passed",
          "parse_rate": 3390.8754623921086,
          "test_num": 38
        },
        {
          "name": "Math operators",
          "outcome": "Passed",
          "parse_rate": 3746.7629070472203,
          "test_num": 39
        },
        {
          "name": "The comma operator",
          "outcome": "Passed",
          "parse_rate": 3851.2117227127565,
          "test_num": 40
        },
        {
          "name": "Assignments",
          "outcome": "Passed",
          "parse_rate": 4041.3006914595403,
          "test_num": 41
        },
        {
          "name": "Pointer operations",
          "outcome": "Passed",
          "parse_rate": 4009.284659210804,
          "test_num": 42
        },
        {
          "name": "Type-casts",
          "outcome": "Passed",
          "parse_rate": 4886.339494370087,
          "test_num": 43
        },
        {
          "name": "Sizeof expressions",
          "outcome": "Passed",
          "parse_rate": 5101.769676356486,
          "test_num": 44
        },
        {
          "name": "Alignof expressions",
          "outcome": "Passed",
          "parse_rate": 6841.439576865586,
          "test_num": 45
        },
        {
          "name": "Offsetof expressions",
          "outcome": "Passed",
          "parse_rate": 4267.354736267553,
          "test_num": 46
        },
        {
          "name": "Compound literals",
          "outcome": "Passed",
          "parse_rate": 4680.808684393985,
          "test_num": 47
        },
        {
          "name": "Compound literals with trailing commas",
          "outcome": "Passed",
          "parse_rate": 5364.142768722922,
          "test_num": 48
        },
        {
          "name": "Comments with escaped newlines",
          "outcome": "Passed",
          "parse_rate": 7575.757575757575,
          "test_num": 49
        },
        {
          "name": "Comments with escaped chars and newlines",
          "outcome": "Passed",
          "parse_rate": 16106.110847939364,
          "test_num": 50
        },
        {
          "name": "Generic Expressions",
          "outcome": "Passed",
          "parse_rate": 6460.116934641888,
          "test_num": 51
        },
        {
          "name": "Noreturn Type Qualifier",
          "outcome": "Passed",
          "parse_rate": 4731.370229720475,
          "test_num": 52
        },
        {
          "name": "Restrict Type Qualifier",
          "outcome": "Passed",
          "parse_rate": 4606.733961113746,
          "test_num": 53
        },
        {
          "name": "Ternary",
          "outcome": "Passed",
          "parse_rate": 3980.6161301488405,
          "test_num": 54
        },
        {
          "name": "Concatenated strings",
          "outcome": "Passed",
          "parse_rate": 4658.491470850175,
          "test_num": 55
        },
        {
          "name": "Nonnull Type Qualifier",
          "outcome": "Passed",
          "parse_rate": 4346.158762623035,
          "test_num": 56
        },
        {
          "name": "Top Level Empty Expression Statement",
          "outcome": "Passed",
          "parse_rate": 1179.245283018868,
          "test_num": 57
        },
        {
          "name": "Sized Type Specifier With Type Qualifiers",
          "outcome": "Passed",
          "parse_rate": 2974.7149231531976,
          "test_num": 58
        }
      ]
    },
    {
      "name": "microsoft",
      "children": [
        {
          "name": "declaration specs",
          "outcome": "Passed",
          "parse_rate": 7153.356197942185,
          "test_num": 59
        },
        {
          "name": "pointers",
          "outcome": "Passed",
          "parse_rate": 4615.279579436063,
          "test_num": 60
        },
        {
          "name": "call modifiers",
          "outcome": "Passed",
          "parse_rate": 5407.513323800858,
          "test_num": 61
        },
        {
          "name": "SEH exception handling",
          "outcome": "Passed",
          "parse_rate": 7301.859461377006,
          "test_num": 62
        }
      ]
    },
    {
      "name": "preprocessor",
      "children": [
        {
          "name": "Include directives",
          "outcome": "Passed",
          "parse_rate": 8254.831504321646,
          "test_num": 63
        },
        {
          "name": "Object-like macro definitions",
          "outcome": "Passed",
          "parse_rate": 18574.42562554367,
          "test_num": 64
        },
        {
          "name": "Function-like macro definitions",
          "outcome": "Passed",
          "parse_rate": 8989.764129951045,
          "test_num": 65
        },
        {
          "name": "Ifdefs",
          "outcome": "Passed",
          "parse_rate": 19.740301631808933,
          "test_num": 66
        },
        {
          "name": "Elifdefs",
          "outcome": "Passed",
          "parse_rate": 2466.184812095688,
          "test_num": 67
        },
        {
          "name": "Mixing #elif and #elifdef",
          "outcome": "Passed",
          "parse_rate": 5503.372367277692,
          "test_num": 68
        },
        {
          "name": "General if blocks",
          "outcome": "Passed",
          "parse_rate": 2717.291968757235,
          "test_num": 69
        },
        {
          "name": "Preprocessor conditionals in functions",
          "outcome": "Passed",
          "parse_rate": 40.32549236957283,
          "test_num": 70
        },
        {
          "name": "Preprocessor conditionals in struct/union bodies",
          "outcome": "Passed",
          "parse_rate": 2594.4857496902105,
          "test_num": 71
        },
        {
          "name": "Unknown preprocessor directives",
          "outcome": "Passed",
          "parse_rate": 58.88241182358829,
          "test_num": 72
        },
        {
          "name": "Preprocessor expressions",
          "outcome": "Passed",
          "parse_rate": 2225.1891410769913,
          "test_num": 73
        }
      ]
    },
    {
      "name": "statements",
      "children": [
        {
          "name": "If statements",
          "outcome": "Passed",
          "parse_rate": 1209.8511717325732,
          "test_num": 74
        },
        {
          "name": "For loops",
          "outcome": "Passed",
          "parse_rate": 3147.4820143884895,
          "test_num": 75
        },
        {
          "name": "While loops",
          "outcome": "Passed",
          "parse_rate": 3574.2652899126288,
          "test_num": 76
        },
        {
          "name": "Labeled statements",
          "outcome": "Passed",
          "parse_rate": 75.50909648646736,
          "test_num": 77
        },
        {
          "name": "Switch statements",
          "outcome": "Passed",
          "parse_rate": 111.43406353598856,
          "test_num": 78
        },
        {
          "name": "Case statements separate from switch statements",
          "outcome": "Passed",
          "parse_rate": 3812.5979463429467,
          "test_num": 79
        },
        {
          "name": "Return statements",
          "outcome": "Passed",
          "parse_rate": 4375.631100639515,
          "test_num": 80
        },
        {
          "name": "Comments with asterisks",
          "outcome": "Passed",
          "parse_rate": 19662.921348314605,
          "test_num": 81
        },
        {
          "name": "Comment with multiple backslashes",
          "outcome": "Passed",
          "parse_rate": 5696.326851306227,
          "test_num": 82
        },
        {
          "name": "Attributes",
          "outcome": "Passed",
          "parse_rate": 746.3034656467187,
          "test_num": 83
        }
      ]
    },
    {
      "name": "types",
      "children": [
        {
          "name": "Primitive types",
          "outcome": "Passed",
          "parse_rate": 5210.089184467805,
          "test_num": 84
        },
        {
          "name": "Type modifiers",
          "outcome": "Passed",
          "parse_rate": 2609.2020966802565,
          "test_num": 85
        }
      ]
    }
  ],
  "parse_failures": [],
  "parse_stats": {
    "successful_parses": 85,
    "total_parses": 85,
    "total_bytes": 12658,
    "total_duration": {
      "secs": 0,
      "nanos": 101925265
    }
  },
  "highlight_results": [
    {
      "name": "names.c",
      "outcome": {
        "AssertionPassed": {
          "assertion_count": 20
        }
      },
      "test_num": 1
    },
    {
      "name": "keywords.c",
      "outcome": {
        "AssertionPassed": {
          "assertion_count": 3
        }
      },
      "test_num": 2
    }
  ],
  "tag_results": [],
  "query_results": []
}

Next step: Investigate why the new tests are passing locally and failing in CI.

ObserverOfTime · 2025-10-01T08:51:03Z

Bikeshedding, but cases or tests are better names than children.

WillLillis · 2025-10-01T09:00:56Z

Bikeshedding, but cases or tests are better names than children.

I'd argue against cases or tests since groups themselves can contain additional test groups. I'm not strongly tied to children though if we can settle on a different alternate name.

WillLillis · 2025-10-12T03:15:42Z

Also, is --report-states-for-rule supported in the JSON format?

CC @ObserverOfTime this turned out to be fairly messy. We need to reproduce the traversal logic that currently lives in here (and a few other places):

tree-sitter/crates/generate/src/build_tables/item.rs

Lines 308 to 328 in ae54350

    
           impl fmt::Display for ParseItemSetDisplay<'_> { 
        
               fn fmt(&self, f: &mut fmt::Formatter) -> Result<(), fmt::Error> { 
        
                   for entry in &self.0.entries { 
        
                       write!( 
        
                           f, 
        
                           "{}\t{}", 
        
                           ParseItemDisplay(&entry.item, self.1, self.2), 
        
                           TokenSetDisplay(&entry.lookaheads, self.1, self.2), 
        
                       )?; 
        
                       if entry.following_reserved_word_set != ReservedWordSetId::default() { 
        
                           write!( 
        
                               f, 
        
                               "\treserved word set: {}", 
        
                               entry.following_reserved_word_set 
        
                           )?; 
        
                       } 
        
                       writeln!(f)?; 
        
                   } 
        
                   Ok(()) 
        
               } 
        
           }

to translate the displayed data into new data structures. I should have some more time to spend on it soon, but I'm not sure if it'll be worth the added line count.

ObserverOfTime · 2025-10-12T05:15:00Z

All right, just make the flags conflicting.

clason · 2025-10-31T10:24:53Z

Would it make sense to provide a schema for this json?

WillLillis · 2025-10-31T21:47:06Z

Would it make sense to provide a schema for this json?

I don't see any reason not to as long as we can generate the schema without too much trouble. I'll look into that later tonight/ this weekend.

Edit: Actually, that was really easy. This needs some discussion regarding where the schema lives/how we make it accessible to end users.

clason · 2025-10-31T22:25:01Z

I think there's already some schemas in this repo? Just put it next to the rest of the fire :)

WillLillis · 2025-10-31T22:30:13Z

Hmmmm, it looks like the schemars-generated schema may be ignoring the serialize_as_array serde trick we used to give a nice JSON representation (traversal_idxs is exposed). Will take another look at this later.

GREsau/schemars#15

clason · 2025-11-01T09:48:03Z

This was mostly triggered by the initial format in the PR description; the actual format you ended up with is much simpler and self-explanatory enough, so I don't think a schema is needed. (It can still be useful of course, but certainly not a blocker. Let's punt this so we can merge the breaking changes quickly?)

WillLillis · 2025-11-01T09:59:32Z

This was mostly triggered by the initial format in the PR description; the actual format you ended up with is much simpler and self-explanatory enough, so I don't think a schema is needed. (It can still be useful of course, but certainly not a blocker. Let's punt this so we can merge the breaking changes quickly?)

The schema in this PR should be fine now, I commented a bit prematurely 😆. Unless you see any issues with it now of course.

clason · 2025-11-01T10:01:25Z

Nope, not at all. Well done!

Also, only update the expected output of a case when it is skipped if the `update` flag has been passed

for `generate` command as conflicting

WillLillis added cli test Related to tests labels Sep 29, 2025

WillLillis force-pushed the test_sweep branch 2 times, most recently from dd85d63 to 7132c1c Compare September 29, 2025 07:04

amaanq approved these changes Sep 29, 2025

View reviewed changes

maxbrunsfeld reviewed Sep 29, 2025

View reviewed changes

crates/cli/src/main.rs Show resolved Hide resolved

WillLillis force-pushed the test_sweep branch 2 times, most recently from def1951 to 02521e8 Compare October 1, 2025 08:37

WillLillis force-pushed the test_sweep branch 3 times, most recently from 475c11f to d31483b Compare October 2, 2025 05:46

ObserverOfTime mentioned this pull request Oct 3, 2025

Add "containing range" APIs to QueryCursor #4919

Closed

5 tasks

WillLillis force-pushed the test_sweep branch from d31483b to 3152029 Compare October 12, 2025 03:09

clason mentioned this pull request Oct 31, 2025

Tree-sitter 0.26.x release checklist #5013

Closed

9 tasks

clason added this to the 0.26 milestone Oct 31, 2025

WillLillis force-pushed the test_sweep branch 2 times, most recently from 1bd0f3d to 3b9bb53 Compare November 1, 2025 00:52

WillLillis mentioned this pull request Nov 1, 2025

fix(docs): update cli docs to reflect changes to various subcommand arguments #5018

Merged

WillLillis and others added 8 commits November 2, 2025 01:14

refactor(test): clean up test filtering logic

196742d

Also, only update the expected output of a case when it is skipped if the `update` flag has been passed

refactor(test): generalize printing of test diff keys and diffs

133b162

feat(test): display test results in JSON format

197f1be

refactor(cli)!: deprecate json flags in favor of json-summary

342b345

test: add coverage for new test aggregation method

69d8efe

fix(cli): correct behavior of parse --stat and --json-summary flags

4c299f5

fix(cli): mark report_states_for_rule and json/json_summary flags

9725dc9

for `generate` command as conflicting

feat(xtask): generate JSON schema for cli TestSummary

2ec3a01

WillLillis force-pushed the test_sweep branch from 3b9bb53 to 2ec3a01 Compare November 2, 2025 05:14

WillLillis merged commit c7b5f89 into tree-sitter:master Nov 3, 2025
19 checks passed

WillLillis deleted the test_sweep branch November 3, 2025 02:09

skewb1k mentioned this pull request Jan 4, 2026

fix(cli): restore test summary output for tree-sitter test #5197

Merged

Uh oh!

Conversation

WillLillis commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ObserverOfTime commented Sep 29, 2025

Uh oh!

ObserverOfTime commented Sep 29, 2025

Uh oh!

maxbrunsfeld left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

WillLillis commented Sep 29, 2025

Uh oh!

WillLillis commented Sep 29, 2025

Uh oh!

ObserverOfTime commented Sep 30, 2025

Uh oh!

ObserverOfTime commented Sep 30, 2025

Uh oh!

WillLillis commented Oct 1, 2025

Uh oh!

WillLillis commented Oct 1, 2025

Uh oh!

ObserverOfTime commented Oct 1, 2025

Uh oh!

WillLillis commented Oct 1, 2025

Uh oh!

WillLillis commented Oct 12, 2025

Uh oh!

ObserverOfTime commented Oct 12, 2025

Uh oh!

clason commented Oct 31, 2025

Uh oh!

WillLillis commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clason commented Oct 31, 2025

Uh oh!

WillLillis commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clason commented Nov 1, 2025

Uh oh!

WillLillis commented Nov 1, 2025

Uh oh!

clason commented Nov 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

WillLillis commented Sep 29, 2025 •

edited

Loading

WillLillis commented Oct 31, 2025 •

edited

Loading

WillLillis commented Oct 31, 2025 •

edited

Loading