feat(tui): syntax highlighting via syntect with theme picker#11447
feat(tui): syntax highlighting via syntect with theme picker#11447etraut-openai merged 37 commits intoopenai:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds syntect-based syntax highlighting to the TUI (markdown fenced code blocks + unified diffs) and introduces a /theme picker with live preview and persisted theme selection, wiring theme state into config + runtime rendering.
Changes:
- Replace the prior tree-sitter-based highlighting with a syntect/two-face engine, including theme management, guardrails, and tests.
- Add
/themecommand and a theme picker UI with side-by-side/stacked previews and cancel/restore behavior. - Update diff and markdown renderers to consume the new highlighter and add snapshot/unit test coverage.
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| codex-rs/tui/src/theme_picker.rs | New /theme picker dialog with live preview + responsive wide/narrow preview renderables. |
| codex-rs/tui/src/render/highlight.rs | New syntect/two-face highlighting engine, global theme lifecycle, guardrails, and unit tests. |
| codex-rs/tui/src/markdown_render.rs | Buffer-and-highlight fenced code blocks (language-aware) with info-string token parsing. |
| codex-rs/tui/src/markdown_render_tests.rs | Update/add tests to validate highlighting behavior, fallbacks, and trailing blank-line preservation. |
| codex-rs/tui/src/diff_render.rs | Add syntax-aware diff rendering and styled wrapping that preserves span styles across wraps. |
| codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__syntax_highlighted_insert_wraps_text.snap | Snapshot for syntax-highlighted wrapping text output. |
| codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__syntax_highlighted_insert_wraps.snap | Snapshot for syntax-highlighted wrapping rendering output. |
| codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__diff_gallery_94x35.snap | Snapshot gallery updated for syntax-highlighted diff rendering at 94x35. |
| codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__diff_gallery_80x24.snap | Snapshot gallery updated for syntax-highlighted diff rendering at 80x24. |
| codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__diff_gallery_120x40.snap | Snapshot gallery updated for syntax-highlighted diff rendering at 120x40. |
| codex-rs/tui/src/slash_command.rs | Add /theme to the slash command enum + description/availability. |
| codex-rs/tui/src/lib.rs | Initialize theme override from the final resolved config and surface startup warnings. |
| codex-rs/tui/src/chatwidget.rs | Wire /theme to open the picker; add setter to keep widget config copy in sync. |
| codex-rs/tui/src/bottom_pane/mod.rs | Re-export SideContentWidth for use by the theme picker. |
| codex-rs/tui/src/bottom_pane/list_selection_view.rs | Add side-content panel support + selection/cancel callbacks for live preview and cancel-restore. |
| codex-rs/tui/src/app_event.rs | Add AppEvent::SyntaxThemeSelected. |
| codex-rs/tui/src/app.rs | Persist theme selection to config, apply runtime theme, and handle errors/restore behavior. |
| codex-rs/tui/Cargo.toml | Add syntect and two-face; remove tree-sitter highlight deps from this crate. |
| codex-rs/tui/src/chatwidget/snapshots/codex_tui__chatwidget__tests__exec_approval_modal_exec.snap | Snapshot updated due to new syntax highlighting output styles. |
| codex-rs/core/src/config/types.rs | Add [tui] theme field to config TOML types. |
| codex-rs/core/src/config/mod.rs | Add Config.tui_theme plumbing + tests for deserialization/defaulting. |
| codex-rs/core/src/config/edit.rs | Add syntax_theme_edit() helper for persisting theme selection. |
| codex-rs/core/config.schema.json | Extend JSON schema with [tui] theme. |
| codex-rs/Cargo.toml | Remove tree-sitter-highlight dependency at the workspace root. |
| codex-rs/Cargo.lock | Lockfile updates for syntect/two-face and transitive deps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
3418ea6 to
c087974
Compare
|
@fcoury, I was interested in understanding how much this would increase the binary size. It adds a little over 3M, or about 1%, to the total binary size. I think that's acceptable. |
c087974 to
dd654ca
Compare
Replace tree-sitter-bash and tree-sitter-highlight with syntect for code highlighting. Syntect's HighlightLines API resolves overlapping spans internally, significantly simplifying the implementation. - Add syntect workspace dependency, remove tree-sitter-bash/highlight from tui - Rewrite highlight.rs with syntect singletons (SyntaxSet + base16-ocean.dark theme) - Normalize language aliases (js→javascript, py→python, rs→rust, etc.) - Multi-strategy syntax lookup: token, name, case-insensitive name, extension - Add guardrails: skip highlighting for inputs >512KB or >10K lines - Comprehensive tests for style conversion, language resolution, content preservation
Integrate syntax highlighting into diff rendering and markdown code blocks, with proper line wrapping that preserves styled spans. - Add detect_lang_for_path() for extension-based language detection - Syntax-aware diff rendering for Add/Delete/Update file changes - wrap_styled_spans() splits long highlighted lines across display rows - Buffer fenced code blocks in markdown_render for batch highlighting - Add snapshot tests for highlighted diff wrapping - Replace code_block_unhighlighted test with 3 syntax color assertions
The exec approval modal's command line now gets syntax-colored spans from syntect (base16-ocean.dark) instead of tree-sitter-bash.
No crate references it after the TUI switched to syntect.
… theme Replace syntect's default ~40 language set with two-face's bat-sourced ~250 language set, adding support for TypeScript, TSX, Kotlin, Swift, and Zig. Auto-detect terminal background to pick CatppuccinMocha (dark) or CatppuccinLatte (light). Suppress italic modifier by default since many terminals render it poorly.
two-face's ~250 language syntax set resolves almost all names and extensions on its own. Remove the redundant normalize_lang layer and keep only 3 patched aliases (golang, python3, shell) that two-face cannot resolve. This enables highlighting for any language in bat's syntax set (elixir, haskell, scala, dart, php, html, css, etc.) without needing explicit entries.
Add `[tui] theme = "..."` config option that lets users override the auto-detected syntax highlighting theme with any of the 32 bundled two-face themes by kebab-case name. Invalid names log a warning and fall back to adaptive detection. Also suppress FontStyle::UNDERLINE in convert_style — themes like Dracula mark type scopes with underline which produces distracting underlines on type/module names in terminal output.
trim_end_matches('\n') stripped all trailing newlines from the code
block buffer before highlighting, discarding intentional blank lines
authored inside fences. pulldown-cmark appends exactly one trailing
'\n' to the text content, and LinesWithEndings handles it correctly
as a line terminator, so no trimming is needed. Remove the trim to
preserve user-authored blank lines faithfully.
Two fixes:
1. highlight.rs: The fallback path used split('\n') which produces a
phantom empty trailing element for inputs ending in '\n' (as
pulldown-cmark emits for code blocks). Switch to lines() which
handles trailing newlines correctly.
2. diff_render.rs: wrap_styled_spans only flushed the current line
when col >= max_cols AND the current span had remaining content.
This missed the case where one span ends exactly at max_cols and
the next span has content — the next span's first character was
appended to the already-full line. Remove the !remaining.is_empty()
guard so the line flushes at span boundaries too.
Extend theme resolution so `theme = "foo"` checks ~/.codex/themes/foo.tmTheme before falling back to auto-detection. Surface unresolved theme names as a ⚠ startup warning in the TUI via the existing startup_warnings pipeline.
Two fixes from code review:
1. `trim_end_matches('\n')` left a stray `\r` on CRLF inputs, which
propagated into rendered code blocks and diffs. Now strips both
`\r` and `\n`.
2. Unified diff highlighting called `highlight_code_to_styled_spans`
per line, bypassing the global size guardrails (max bytes/lines).
On large patches this triggered thousands of parser initializations.
Added a pre-check on aggregate patch size that skips highlighting
when limits are exceeded.
- Tabs in syntax-highlighted diff spans now count as 4 columns instead of zero, so tab-indented code wraps correctly in the diff view. - Move theme warning injection to after the last possible config reload (session resume/fork) so the ⚠ banner is not silently discarded.
`set_theme_override` was called with `initial_config.tui_theme` before onboarding/resume/fork could reload config. Because the theme state lives in a `OnceLock`, it was permanently locked to the initial value. Move the call to after the last possible config reload so the theme and its warning always reflect the final session config.
Add a `/theme` slash command that opens an interactive picker listing all bundled and custom (.tmTheme) syntax themes. The picker shows a live-preview code snippet below the list that re-highlights as the user navigates, and restores the original theme on cancel. Key changes: - ListSelectionView gains `footer_content`, `on_selection_changed`, and `on_cancel` to support rich preview and lifecycle callbacks - highlight.rs: swap global Theme from OnceLock to RwLock for live swapping; add resolve/list/set/current helpers for theme management - New theme_picker module with ThemePreviewRenderable and picker builder - diff_render: make DiffLineType and helpers pub(crate); fix rename highlighting to use destination extension - markdown_render: fix CRLF double-newlines and info-string metadata parsing for fenced code blocks
Highlight each hunk as a single block instead of per-line so that multiline constructs (e.g. strings spanning multiple lines) get correct syntax coloring. Adds a regression test.
Extract selected_actual_idx() helper and guard move_up/move_down so the callback is skipped when wrapping doesn't change the selected item (e.g. single-item lists). Adds a test for the single-item case.
set_theme_override now updates the runtime theme immediately instead of only setting the OnceLock, enabling live switching from /theme. Replace hardcoded ~/.codex/themes references with resolved $CODEX_HOME/themes paths in warnings, schema docs, and picker subtitle.
Add `RUSTSEC-2024-0320` and `RUSTSEC-2025-0141` to the `ignore` list in `codex-rs/deny.toml`. Both advisories are pulled in via `syntect v5.3.0` used by `codex-tui` syntax highlighting and currently have no fixed release. This keeps `cargo deny` focused on actionable findings while the exception is tracked with a TODO.
Regenerate `codex-rs/core/config.schema.json` with `just write-config-schema` so the fixture matches the current `ConfigToml` schema output. This fixes `config_schema_matches_fixture` in CI after the syntax-highlighting `theme` description changed to mention `kebab-case` naming.
609ef91 to
f293043
Compare
Summary
Adds syntax highlighting to the TUI for fenced code blocks in markdown responses and file diffs, plus a
/themecommand with live preview and persistent theme selection. Uses syntect (~250 grammars, 32 bundled themes, ~1 MB binary cost) — the same engine behindbat,delta, andxi-editor. Includes guardrails for large inputs, graceful fallback to plain text, and SSH-aware clipboard integration for the/copycommand.Problem
Code blocks in the TUI (markdown responses and file diffs) render without syntax highlighting, making it hard to scan code at a glance. Users also have no way to pick a color theme that matches their terminal aesthetic.
Mental model
The highlighting system has three layers:
Syntax engine (
render::highlight) -- a thin wrapper around syntect + two-face. It owns a process-globalSyntaxSet(~250 grammars) and aRwLock<Theme>that can be swapped at runtime. All public entry points accept(code, lang)and return ratatuiSpan/Linevectors orNonewhen the language is unrecognized or the input exceeds safety guardrails.Rendering consumers --
markdown_renderfeeds fenced code blocks through the engine;diff_renderhighlights Add/Delete content as a whole file and Update hunks per-hunk (preserving parser state across hunk lines). Both callers fall back to plain unstyled text when the engine returnsNone.Theme lifecycle -- at startup the config's
tui.themeis resolved to a syntectThemeviaset_theme_override. At runtime the/themepicker callsset_syntax_themeto swap themes live; on cancel it restores the snapshot taken at open. On confirm it persists[tui] theme = "..."to config.toml.Non-goals
.tmThemefiles.Tradeoffs
bat,delta,xi-editor). tree-sitter would add ~12 MB for 20-30 languages or ~35 MB for full coverage.RwLock<Theme>/themepreview without threading Theme through every call siteArchitecture
Key files:
tui/src/render/highlight.rs-- engine, theme management, guardrailstui/src/diff_render.rs-- syntax-aware diff line wrappingtui/src/theme_picker.rs--/themecommand buildertui/src/bottom_pane/list_selection_view.rs-- side content panel, callbackscore/src/config/types.rs--Tui::themefieldcore/src/config/edit.rs--syntax_theme_edit()helperObservability
tracing::warnwhen a configured theme name cannot be resolved.Config::startup_warningssurfaces the same message as a TUI banner.tracing::errorwhen persisting theme selection fails.Tests
highlight.rs: language coverage, fallback behavior, CRLF stripping, style conversion, guardrail enforcement, theme name mapping exhaustiveness.diff_render.rs: snapshot gallery at multiple terminal sizes (80x24, 94x35, 120x40), syntax-highlighted wrapping, large-diff guardrail, rename-to-different-extension highlighting, parser state preservation across hunk lines.theme_picker.rs: preview rendering (wide + narrow), dim overlay on deletions, subtitle truncation, cancel-restore, fallback for unavailable configured theme.list_selection_view.rs: side layout geometry, stacked fallback, buffer clearing, cancel/selection-changed callbacks.lib.rs: theme warning uses the final (post-resume) config.Cargo Deny: Unmaintained Dependency Exceptions
This PR adds two
cargo denyadvisory exceptions for transitive dependencies pulled in bysyntect v5.3.0:yaml-rustbincodeWhy this is safe in our usage:
bincodeis used by syntect to deserialize pre-compiled syntax sets. Again, these are static vendored artifacts baked into the binary at build time. No user-supplied bincode data is ever deserialized. - Attack surface is zero for both crates; exploitation would require a supply-chain compromise of our own build artifacts.yaml-rust2and dropsbincode, or when alternative crates are available upstream.