Add pyphen-based hyphenation abstraction layer#20145
Merged
Merged
Conversation
Introduces textUtils.hyphenation with getHyphenPositions() — a locale-aware API over the pyphen library. Includes py2exe hook to freeze pyphen dictionaries outside library.zip for frozen builds. Part of nvaccess#17010
5 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds hyphenation support to NVDA via the pyphen library, including a new utility module, a py2exe build hook to package pyphen dictionaries alongside the frozen executable, and unit tests.
Changes:
- New
textUtils.hyphenationmodule exposinggetHyphenPositionsbacked by aLocaleDataMapcache ofPypheninstances. - A py2exe hook that copies
hyph_*.dicfiles next to the executable and AST-rewrites pyphen'sdictionariesassignment so it resolves at runtime in the frozen build. - Adds
pyphento project dependencies and unit tests for the new module.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| pyproject.toml | Adds pyphen runtime dependency. |
| source/setup.py | Adds _PyphenTransformer and _hook_pyphen py2exe hook to bundle and relocate pyphen dictionaries. |
| source/textUtils/hyphenation.py | New module providing getHyphenPositions with locale-aware caching via LocaleDataMap. |
| tests/unit/test_hyphenation.py | Unit tests covering known and unknown locale behaviour. |
12 tasks
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
seanbudd
reviewed
May 18, 2026
- Add type hints to `_PyphenTransformer.visit_Try` - Call `generic_visit` on non-matching Try nodes so nested Try nodes are traversed - Inject `from pathlib import Path` into the rewritten pyphen module Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
seanbudd
pushed a commit
that referenced
this pull request
May 20, 2026
Part of #17010. Split out from #19916 at reviewer request — this is the second of three PRs. The pyphen abstraction layer was shipped in #20145; the syllable-aware wrap mode follows in a separate PR. Summary of the issue: The braille word-wrap setting is a single boolean that gives users no control over how words are broken at the display edge. When a word is cut mid-way there is also no visual indication that the word continues on the next row. Description of user facing changes: The boolean "word wrap" checkbox in the braille settings has been replaced with a Text wrap combo box with three choices: Off — Wrap at the raw edge of the display, cutting words in the middle if necessary. No visual indication that a word was cut. Show mark when words are cut — Wrap at the raw edge, but whenever a word is cut mid-way, replace the last cell of the row with a continuation mark (braille dots 7-8) so the reader knows the word continues on the next row. At word boundaries — Prefer breaking at spaces. If no space fits on the row, fall back to cutting the word and showing the continuation mark. Existing config profiles with the old setting are automatically upgraded. Description of developer facing changes: BrailleTextWrapFlag feature flag enum added to config.featureFlagEnums with members DEFAULT, NONE, MARK_WORD_CUTS, AT_WORD_BOUNDARIES. Config schema bumped v22 → v23; old wordWrap boolean is deprecated and bridged bidirectionally to textWrap via _linkDeprecatedValues, so add-ons reading or writing the old key keep working (with a deprecation warning). CONTINUATION_SHAPE = 0xC0 (dots 7-8) constant added to braille. _WindowRowPositions frozen dataclass added to braille to hold the start/end buffer positions and continuation-mark flag for each row of the braille window, replacing the previous anonymous tuple. Description of development approach: The continuation mark is unified: it consistently means "a word was cut here" regardless of mode, so readers get a predictable signal. BrailleBuffer._calculateWindowRowBufferOffsets is extended to implement all three modes. Each entry in _windowRowBufferOffsets is a _WindowRowPositions instance whose showContinuationMark field records whether that row needs a continuation mark. BrailleBuffer._get_windowBrailleCells reads that flag to insert the mark. BrailleBuffer._set_windowEndPos short-circuits space-seeking for NONE and MARK_WORD_CUTS modes (backwards scroll alignment).
5 tasks
seanbudd
pushed a commit
that referenced
this pull request
May 25, 2026
) Closes #17010 Follow-up for #20146 and #20145. This is the last of three PRs replacing #19916. Summary of the issue: Word wrap is sometimes pretty aggressive, especially on shorter braille displays. The previous two PRs added the text wrap infrastructure and continuation marks; this PR adds the final mode that splits long words at syllable boundaries using hyphenation dictionaries. Description of user facing changes: A fourth option, At word or syllable boundaries, is added to the Text wrap combo box in braille settings. Like "At word boundaries", it avoids splitting words mid-way, but when a word is too long to fit on the display it additionally tries to split at a syllable boundary (using hyphenation dictionaries from the pyphen library) so less of the word spills onto the next row. NVDA marks the split with the continuation mark (braille dots 7-8). For locales without a pyphen dictionary, the mode falls back cleanly to word-boundary behaviour without any error. Description of developer facing changes: BrailleTextWrapFlag.AT_WORD_OR_SYLLABLE_BOUNDARIES member added to config.featureFlagEnums. Region._languageIndexes (dict[int, str]) tracks language-span boundaries within a braille region. Populated during _addFieldText and _addTextWithFields when format fields carry a language attribute or when field text is in a different language than the surrounding content. Region._getLanguageAtPos(pos) looks up the language at a raw-text offset using a bisect on the (always-ascending) keys of _languageIndexes. BrailleBuffer._getLanguageAtBufferPos(pos) delegates to the region that owns that braille cell. louisHelper.getTableLanguage(table) queries louis.getTableInfo for the "language" key and normalises the result, providing the default language for a region when no format-field language is known. Description of development approach: When AT_WORD_OR_SYLLABLE_BOUNDARIES is selected and a word straddles a row boundary, _calculateWindowRowBufferOffsets already finds the last space before the display edge. This PR adds a second pass: it looks up the full word (from that space to the next space), retrieves the language at the word's braille position, and calls textUtils.hyphenation.getHyphenPositions (introduced in #20145) to obtain candidate hyphen offsets. It then iterates the candidates from the end (closest to the display edge) and picks the first that falls within the current row, updating end accordingly and setting showContinuationMark. Language tracking in Region ensures that the correct pyphen dictionary is selected even when a braille region contains multilingual content (e.g. a paragraph with inline foreign phrases).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to issue number:
Part of #17010. Split out from #19916 at reviewer request — this is the first of three PRs. The braille text-wrap refactor and syllable-aware wrap mode follow in separate PRs.
Summary of the issue:
NVDA lacks a locale-aware hyphenation API. One is needed to implement syllable-boundary braille text wrapping, so that long words can be broken at linguistically correct positions rather than always at the raw display edge.
Description of user facing changes:
None. This PR adds internal infrastructure only; no behaviour changes for users.
Description of developer facing changes:
New public function in
textUtils.hyphenation:Returns the character offsets within
textat which a hyphen may be inserted for the given locale. Returns an empty tuple for locales without a pyphen dictionary, logging a debug message once per locale per map lifetime, so callers can fall back cleanly without raising.A py2exe hook in
source/setup.pybundles pyphen'shyph_*.dicfiles intodist/pyphenDictionaries/and rewrites pyphen's dictionary lookup path at freeze time so the dictionaries are accessible in frozen builds.Description of development approach:
LocaleDataMap(already used in NVDA for locale-aware character processing) handles locale fallback and caching. The_pyphenFactoryfunction deliberately rejects region-subtag fallbacks — e.g. it will not silently serveendictionaries for anen_USlookup — delegating that fallback logic toLocaleDataMapso region matching stays consistent with the rest of NVDA's locale handling.Testing strategy:
Unit tests cover
getHyphenPositionsfor a known locale (en_US— returns a non-empty tuple of valid positions) and an unknown locale (returns()without raising, idempotent on repeated calls).Manual testing: confirmed
scons.bat distproducesdist/pyphenDictionaries/containing onlyhyph_*.dicfiles.Known issues with pull request:
None.
Code Review Checklist:
tests/unit/test_hyphenation.py)