^{Built mostly by Gemini 3 Pro using opencode.}

A CLI tool to make debugging VSCode syntax highlighting easier by simulating the full TextMate + Semantic Token pipeline, generating visual previews, and providing snapshot-based regression testing with diffing capabilities.

• Hybrid Engine: Simulates VS Code's highlighting pipeline by merging TextMate scopes (using vscode-textmate + vscode-oniguruma) with LSP Semantic Tokens.
• Theme Support: Resolves colors using VS Code compatible JSON themes.
• Visualization: Generates HTML previews with tooltips to inspect the full scope stack and token provenance (TextMate vs Semantic).

• Regression Testing: Snapshot-based testing to verify that changes to the semantic tokens, grammar, or theme do not introduce unexpected differences.
• Diffing: When verifying snapshots, generates a diff of any mismatches to quickly identify issues.

bun install

Run the tool with a configuration file:

bun run src/index.ts config.json

The configuration file controls which grammar, LSP, and files are tested. Paths in the config are generally relative to the location of the config.json file itself.

{
  "grammar": "./path/to/grammar.tmLanguage.json",
  "scopeName": "source.swift",
  "lsp": {
    "command": ["/path/to/language-server", "--arg1", "--arg2"],
    "rootUri": "file:///path/to/project"
  },
  "theme": "Dark Modern", 
  "files": ["./path/to/source1", "./path/to/source2"],
  "outDir": "./results",
  "snapshotDir": "./snapshots"
}

To verify that the current output matches previously saved snapshots:

bun run src/index.ts config.json --verify

To update snapshots (accept current output as correct):

bun run src/index.ts config.json --update

When running in --verify mode, if there are any mismatches between the generated tokens and the snapshot, a visual diff report will be generated for each file, showing side-by-side comparisons of the expected vs actual tokens.

You can create multiple versions of snapshots for the same file (e.g., to test different configurations or scenarios) using the --snapshot-name option.

To create a named snapshot:

bun run src/index.ts config.json --update --snapshot-name v2
# Creates: snapshots/filename.v2.tokens.json

To verify against a specific named snapshot:

bun run src/index.ts config.json --verify --snapshot-name v2

You can also customize the label for the "generated" side of the diff using --generated-name:

bun run src/index.ts config.json --verify --generated-name "Proposed Changes"

You can manually generate a visual diff HTML report between any two token JSON files (e.g., a snapshot and a newly generated output) using the diff command.

bun run src/index.ts diff <snapshot.json> <generated.json> [output.html]

• snapshot.json: The "expected" token file (left side of diff).
• generated.json: The "actual" token file (right side of diff).
• output.html: (Optional) The output path for the HTML report. Defaults to diff.html.

This generates an interactive side-by-side view where you can verify discrepancies in color, style, or scopes.

1. Set up a config pointing to your grammar, LSP, and test files.
2. Run the tool with --update to generate a baseline snapshot.
3. Make changes to your grammar or language server.
4. Run the tool with --verify to get a report of any differences from the baseline.

For each input file, the tool generates:

1. filename.html: An interactive preview of the highlighting. Hover over tokens to see scope details.
2. filename.tokens.json: A raw JSON dump of the resolved tokens (used for snapshots).

MIT License. See LICENSE for details.