Skip to content

feat(tui): syntax highlighting via syntect with theme picker#11447

Merged
etraut-openai merged 37 commits intoopenai:mainfrom
fcoury:feat/syntax-highlighting
Feb 22, 2026
Merged

feat(tui): syntax highlighting via syntect with theme picker#11447
etraut-openai merged 37 commits intoopenai:mainfrom
fcoury:feat/syntax-highlighting

Conversation

@fcoury
Copy link
Copy Markdown
Contributor

@fcoury fcoury commented Feb 11, 2026

Summary

Adds syntax highlighting to the TUI for fenced code blocks in markdown responses and file diffs, plus a /theme command with live preview and persistent theme selection. Uses syntect (~250 grammars, 32 bundled themes, ~1 MB binary cost) — the same engine behind bat, delta, and xi-editor. Includes guardrails for large inputs, graceful fallback to plain text, and SSH-aware clipboard integration for the /copy command.

image image

Problem

Code blocks in the TUI (markdown responses and file diffs) render without syntax highlighting, making it hard to scan code at a glance. Users also have no way to pick a color theme that matches their terminal aesthetic.

Mental model

The highlighting system has three layers:

  1. Syntax engine (render::highlight) -- a thin wrapper around syntect + two-face. It owns a process-global SyntaxSet (~250 grammars) and a RwLock<Theme> that can be swapped at runtime. All public entry points accept (code, lang) and return ratatui Span/Line vectors or None when the language is unrecognized or the input exceeds safety guardrails.

  2. Rendering consumers -- markdown_render feeds fenced code blocks through the engine; diff_render highlights Add/Delete content as a whole file and Update hunks per-hunk (preserving parser state across hunk lines). Both callers fall back to plain unstyled text when the engine returns None.

  3. Theme lifecycle -- at startup the config's tui.theme is resolved to a syntect Theme via set_theme_override. At runtime the /theme picker calls set_syntax_theme to swap themes live; on cancel it restores the snapshot taken at open. On confirm it persists [tui] theme = "..." to config.toml.

Non-goals

  • Inline diff highlighting (word-level change detection within a line).
  • Semantic / LSP-backed highlighting.
  • Theme authoring tooling; users supply standard .tmTheme files.

Tradeoffs

Decision Upside Downside
syntect over tree-sitter / arborium ~1 MB binary increase for ~250 grammars + 32 themes; battle-tested crate powering widely-used tools (bat, delta, xi-editor). tree-sitter would add ~12 MB for 20-30 languages or ~35 MB for full coverage. Regex-based; less structurally accurate than tree-sitter for some languages (e.g. language injections like JS-in-HTML).
Global RwLock<Theme> Enables live /theme preview without threading Theme through every call site Lock contention risk (mitigated: reads vastly outnumber writes, single UI thread)
Skip background / italic / underline from themes Terminal BG preserved, avoids ugly rendering on some themes Themes that rely on these properties lose fidelity
Guardrails: 512 KB / 10k lines Prevents pathological stalls on huge diffs or pastes Very large files render without color

Architecture

config.toml  ─[tui.theme]─>  set_theme_override()  ─>  THEME (RwLock)
                                                              │
                  ┌───────────────────────────────────────────┘
                  │
  markdown_render ─── highlight_code_to_lines(code, lang) ─> Vec<Line>
  diff_render     ─── highlight_code_to_styled_spans(code, lang) ─> Option<Vec<Vec<Span>>>
                  │
                  │   (None ⇒ plain text fallback)
                  │
  /theme picker   ─── set_syntax_theme(theme)    // live preview swap
                  ─── current_syntax_theme()      // snapshot for cancel
                  ─── resolve_theme_by_name(name) // lookup by kebab-case

Key files:

  • tui/src/render/highlight.rs -- engine, theme management, guardrails
  • tui/src/diff_render.rs -- syntax-aware diff line wrapping
  • tui/src/theme_picker.rs -- /theme command builder
  • tui/src/bottom_pane/list_selection_view.rs -- side content panel, callbacks
  • core/src/config/types.rs -- Tui::theme field
  • core/src/config/edit.rs -- syntax_theme_edit() helper

Observability

  • tracing::warn when a configured theme name cannot be resolved.
  • Config::startup_warnings surfaces the same message as a TUI banner.
  • tracing::error when persisting theme selection fails.

Tests

  • Unit tests in highlight.rs: language coverage, fallback behavior, CRLF stripping, style conversion, guardrail enforcement, theme name mapping exhaustiveness.
  • Unit tests in diff_render.rs: snapshot gallery at multiple terminal sizes (80x24, 94x35, 120x40), syntax-highlighted wrapping, large-diff guardrail, rename-to-different-extension highlighting, parser state preservation across hunk lines.
  • Unit tests in theme_picker.rs: preview rendering (wide + narrow), dim overlay on deletions, subtitle truncation, cancel-restore, fallback for unavailable configured theme.
  • Unit tests in list_selection_view.rs: side layout geometry, stacked fallback, buffer clearing, cancel/selection-changed callbacks.
  • Integration test in lib.rs: theme warning uses the final (post-resume) config.

Cargo Deny: Unmaintained Dependency Exceptions

This PR adds two cargo deny advisory exceptions for transitive dependencies pulled in by syntect v5.3.0:

Advisory Crate Status
RUSTSEC-2024-0320 yaml-rust Unmaintained (maintainer unreachable)
RUSTSEC-2025-0141 bincode Unmaintained (development ceased; v1.3.3 considered complete)

Why this is safe in our usage:

  • Neither advisory describes a known security vulnerability. Both are "unmaintained" notices only.
  • bincode is used by syntect to deserialize pre-compiled syntax sets. Again, these are static vendored artifacts baked into the binary at build time. No user-supplied bincode data is ever deserialized. - Attack surface is zero for both crates; exploitation would require a supply-chain compromise of our own build artifacts.
  • These exceptions can be removed when syntect migrates to yaml-rust2 and drops bincode, or when alternative crates are available upstream.

Copilot AI review requested due to automatic review settings February 11, 2026 13:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds syntect-based syntax highlighting to the TUI (markdown fenced code blocks + unified diffs) and introduces a /theme picker with live preview and persisted theme selection, wiring theme state into config + runtime rendering.

Changes:

  • Replace the prior tree-sitter-based highlighting with a syntect/two-face engine, including theme management, guardrails, and tests.
  • Add /theme command and a theme picker UI with side-by-side/stacked previews and cancel/restore behavior.
  • Update diff and markdown renderers to consume the new highlighter and add snapshot/unit test coverage.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
codex-rs/tui/src/theme_picker.rs New /theme picker dialog with live preview + responsive wide/narrow preview renderables.
codex-rs/tui/src/render/highlight.rs New syntect/two-face highlighting engine, global theme lifecycle, guardrails, and unit tests.
codex-rs/tui/src/markdown_render.rs Buffer-and-highlight fenced code blocks (language-aware) with info-string token parsing.
codex-rs/tui/src/markdown_render_tests.rs Update/add tests to validate highlighting behavior, fallbacks, and trailing blank-line preservation.
codex-rs/tui/src/diff_render.rs Add syntax-aware diff rendering and styled wrapping that preserves span styles across wraps.
codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__syntax_highlighted_insert_wraps_text.snap Snapshot for syntax-highlighted wrapping text output.
codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__syntax_highlighted_insert_wraps.snap Snapshot for syntax-highlighted wrapping rendering output.
codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__diff_gallery_94x35.snap Snapshot gallery updated for syntax-highlighted diff rendering at 94x35.
codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__diff_gallery_80x24.snap Snapshot gallery updated for syntax-highlighted diff rendering at 80x24.
codex-rs/tui/src/snapshots/codex_tui__diff_render__tests__diff_gallery_120x40.snap Snapshot gallery updated for syntax-highlighted diff rendering at 120x40.
codex-rs/tui/src/slash_command.rs Add /theme to the slash command enum + description/availability.
codex-rs/tui/src/lib.rs Initialize theme override from the final resolved config and surface startup warnings.
codex-rs/tui/src/chatwidget.rs Wire /theme to open the picker; add setter to keep widget config copy in sync.
codex-rs/tui/src/bottom_pane/mod.rs Re-export SideContentWidth for use by the theme picker.
codex-rs/tui/src/bottom_pane/list_selection_view.rs Add side-content panel support + selection/cancel callbacks for live preview and cancel-restore.
codex-rs/tui/src/app_event.rs Add AppEvent::SyntaxThemeSelected.
codex-rs/tui/src/app.rs Persist theme selection to config, apply runtime theme, and handle errors/restore behavior.
codex-rs/tui/Cargo.toml Add syntect and two-face; remove tree-sitter highlight deps from this crate.
codex-rs/tui/src/chatwidget/snapshots/codex_tui__chatwidget__tests__exec_approval_modal_exec.snap Snapshot updated due to new syntax highlighting output styles.
codex-rs/core/src/config/types.rs Add [tui] theme field to config TOML types.
codex-rs/core/src/config/mod.rs Add Config.tui_theme plumbing + tests for deserialization/defaulting.
codex-rs/core/src/config/edit.rs Add syntax_theme_edit() helper for persisting theme selection.
codex-rs/core/config.schema.json Extend JSON schema with [tui] theme.
codex-rs/Cargo.toml Remove tree-sitter-highlight dependency at the workspace root.
codex-rs/Cargo.lock Lockfile updates for syntect/two-face and transitive deps.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@etraut-openai
Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector
Copy link
Copy Markdown
Contributor

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@fcoury fcoury force-pushed the feat/syntax-highlighting branch from 3418ea6 to c087974 Compare February 11, 2026 18:17
@etraut-openai etraut-openai added the oai PRs contributed by OpenAI employees label Feb 11, 2026
@etraut-openai
Copy link
Copy Markdown
Collaborator

@fcoury, I was interested in understanding how much this would increase the binary size. It adds a little over 3M, or about 1%, to the total binary size. I think that's acceptable.

@fcoury fcoury force-pushed the feat/syntax-highlighting branch from c087974 to dd654ca Compare February 21, 2026 22:28
Replace tree-sitter-bash and tree-sitter-highlight with syntect for
code highlighting. Syntect's HighlightLines API resolves overlapping
spans internally, significantly simplifying the implementation.

- Add syntect workspace dependency, remove tree-sitter-bash/highlight from tui
- Rewrite highlight.rs with syntect singletons (SyntaxSet + base16-ocean.dark theme)
- Normalize language aliases (js→javascript, py→python, rs→rust, etc.)
- Multi-strategy syntax lookup: token, name, case-insensitive name, extension
- Add guardrails: skip highlighting for inputs >512KB or >10K lines
- Comprehensive tests for style conversion, language resolution, content preservation
Integrate syntax highlighting into diff rendering and markdown code
blocks, with proper line wrapping that preserves styled spans.

- Add detect_lang_for_path() for extension-based language detection
- Syntax-aware diff rendering for Add/Delete/Update file changes
- wrap_styled_spans() splits long highlighted lines across display rows
- Buffer fenced code blocks in markdown_render for batch highlighting
- Add snapshot tests for highlighted diff wrapping
- Replace code_block_unhighlighted test with 3 syntax color assertions
The exec approval modal's command line now gets syntax-colored spans
from syntect (base16-ocean.dark) instead of tree-sitter-bash.
No crate references it after the TUI switched to syntect.
… theme

Replace syntect's default ~40 language set with two-face's bat-sourced
~250 language set, adding support for TypeScript, TSX, Kotlin, Swift,
and Zig. Auto-detect terminal background to pick CatppuccinMocha (dark)
or CatppuccinLatte (light). Suppress italic modifier by default since
many terminals render it poorly.
two-face's ~250 language syntax set resolves almost all names and
extensions on its own. Remove the redundant normalize_lang layer and
keep only 3 patched aliases (golang, python3, shell) that two-face
cannot resolve. This enables highlighting for any language in bat's
syntax set (elixir, haskell, scala, dart, php, html, css, etc.)
without needing explicit entries.
Add `[tui] theme = "..."` config option that lets users override the
auto-detected syntax highlighting theme with any of the 32 bundled
two-face themes by kebab-case name. Invalid names log a warning and
fall back to adaptive detection.

Also suppress FontStyle::UNDERLINE in convert_style — themes like
Dracula mark type scopes with underline which produces distracting
underlines on type/module names in terminal output.
trim_end_matches('\n') stripped all trailing newlines from the code
block buffer before highlighting, discarding intentional blank lines
authored inside fences. pulldown-cmark appends exactly one trailing
'\n' to the text content, and LinesWithEndings handles it correctly
as a line terminator, so no trimming is needed. Remove the trim to
preserve user-authored blank lines faithfully.
Two fixes:

1. highlight.rs: The fallback path used split('\n') which produces a
   phantom empty trailing element for inputs ending in '\n' (as
   pulldown-cmark emits for code blocks). Switch to lines() which
   handles trailing newlines correctly.

2. diff_render.rs: wrap_styled_spans only flushed the current line
   when col >= max_cols AND the current span had remaining content.
   This missed the case where one span ends exactly at max_cols and
   the next span has content — the next span's first character was
   appended to the already-full line. Remove the !remaining.is_empty()
   guard so the line flushes at span boundaries too.
Extend theme resolution so `theme = "foo"` checks
~/.codex/themes/foo.tmTheme before falling back to auto-detection.
Surface unresolved theme names as a ⚠ startup warning in the TUI
via the existing startup_warnings pipeline.
Two fixes from code review:

1. `trim_end_matches('\n')` left a stray `\r` on CRLF inputs, which
   propagated into rendered code blocks and diffs. Now strips both
   `\r` and `\n`.

2. Unified diff highlighting called `highlight_code_to_styled_spans`
   per line, bypassing the global size guardrails (max bytes/lines).
   On large patches this triggered thousands of parser initializations.
   Added a pre-check on aggregate patch size that skips highlighting
   when limits are exceeded.
- Tabs in syntax-highlighted diff spans now count as 4 columns instead
  of zero, so tab-indented code wraps correctly in the diff view.

- Move theme warning injection to after the last possible config reload
  (session resume/fork) so the ⚠ banner is not silently discarded.
`set_theme_override` was called with `initial_config.tui_theme` before
onboarding/resume/fork could reload config. Because the theme state
lives in a `OnceLock`, it was permanently locked to the initial value.

Move the call to after the last possible config reload so the theme
and its warning always reflect the final session config.
Add a `/theme` slash command that opens an interactive picker listing
all bundled and custom (.tmTheme) syntax themes. The picker shows a
live-preview code snippet below the list that re-highlights as the
user navigates, and restores the original theme on cancel.

Key changes:
- ListSelectionView gains `footer_content`, `on_selection_changed`,
  and `on_cancel` to support rich preview and lifecycle callbacks
- highlight.rs: swap global Theme from OnceLock to RwLock for live
  swapping; add resolve/list/set/current helpers for theme management
- New theme_picker module with ThemePreviewRenderable and picker builder
- diff_render: make DiffLineType and helpers pub(crate); fix rename
  highlighting to use destination extension
- markdown_render: fix CRLF double-newlines and info-string metadata
  parsing for fenced code blocks
Highlight each hunk as a single block instead of per-line so that
multiline constructs (e.g. strings spanning multiple lines) get
correct syntax coloring. Adds a regression test.
Extract selected_actual_idx() helper and guard move_up/move_down so
the callback is skipped when wrapping doesn't change the selected
item (e.g. single-item lists). Adds a test for the single-item case.
set_theme_override now updates the runtime theme immediately instead
of only setting the OnceLock, enabling live switching from /theme.
Replace hardcoded ~/.codex/themes references with resolved
$CODEX_HOME/themes paths in warnings, schema docs, and picker subtitle.
Add `RUSTSEC-2024-0320` and `RUSTSEC-2025-0141` to the `ignore`
list in `codex-rs/deny.toml`.

Both advisories are pulled in via `syntect v5.3.0` used by
`codex-tui` syntax highlighting and currently have no fixed release.
This keeps `cargo deny` focused on actionable findings while the
exception is tracked with a TODO.
Regenerate `codex-rs/core/config.schema.json` with `just
write-config-schema` so the fixture matches the current `ConfigToml`
schema output.

This fixes `config_schema_matches_fixture` in CI after the
syntax-highlighting `theme` description changed to mention
`kebab-case` naming.
@fcoury fcoury force-pushed the feat/syntax-highlighting branch from 609ef91 to f293043 Compare February 21, 2026 23:33
@etraut-openai etraut-openai merged commit c4f1af7 into openai:main Feb 22, 2026
48 of 55 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Feb 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

oai PRs contributed by OpenAI employees

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants