feat: v0.10.7 - UTF-8 fixes + 100% stdlib API compatibility #86
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fixed '.' to match UTF-8 codepoints instead of bytes - Implemented compileUTF8Any() for proper 1-4 byte UTF-8 sequences - Fixed NumSubexp() to match stdlib behavior (exclude group 0) - Added comprehensive stdlib compatibility tests (1800+ lines) - Added regression tests for UTF-8 dot matching - Added fuzz tests comparing against stdlib - Updated STDLIB_COMPATIBILITY.md documentation Fixes #85
Benchmark ComparisonComparing Summary:
|
- Replace incorrect single-byte matching with proper UTF-8 sequences
- Build UTF-8 automata for each Unicode range (1/2/3/4-byte)
- Handle surrogate gap (U+D800-U+DFFF) correctly
- Optimize: use efficient 'any UTF-8' for simple negated ASCII classes
- Add comprehensive tests for \P{Han}, \P{Latin}, etc.
Before: \P{Han} on 中 returned 3 matches (one per byte)
After: \P{Han} on 中 returns 0 matches (correct)
- Update current version to v0.10.7 - Add v0.10.6 and v0.10.7 to release history - Mark UTF-8 Automata Optimization as partial (OPT-014) - Update release strategy flow
- Rename 'Fully Compatible' to 'Implemented Functions' - Add 'Not Yet Implemented' section with planned methods - Update compatibility level to '90%+' to be accurate - Reader-based methods, Expand, SubexpIndex, serialization not yet impl
Implement all remaining stdlib regexp methods: - CompilePOSIX, MustCompilePOSIX (POSIX ERE semantics) - Package-level Match, MatchString, MatchReader - SubexpIndex(name) for named capture groups - LiteralPrefix() with anchor handling - Expand, ExpandString template substitution - Copy() (deprecated since Go 1.12) - MarshalText, UnmarshalText (encoding.TextMarshaler) - MatchReader, FindReaderIndex, FindReaderSubmatchIndex Add comprehensive stdlib compatibility tests. Update documentation to reflect 100% API coverage.
db7a91a to
b617111
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
v0.10.7 delivers UTF-8/Unicode fixes and complete stdlib API compatibility.
UTF-8/Unicode Fixes
[^\S\s]) no longer match empty strings\P{Han}) use proper UTF-8 automata100% stdlib API Compatibility
All stdlib regexp methods implemented:
CompilePOSIX,MustCompilePOSIX- POSIX ERE semanticsMatch,MatchString,MatchReader- package-level functionsSubexpIndex(name)- named capture group lookupLiteralPrefix()- literal prefix extraction with anchor handlingExpand,ExpandString- template substitution ($0, $1, etc.)Copy()- regex duplication (deprecated since Go 1.12)MarshalText,UnmarshalText- encoding.TextMarshaler interfaceMatchReader,FindReaderIndex,FindReaderSubmatchIndex- io.RuneReader methodsChanges
nfa/compile.go: UTF-8 range compilation for Unicode classesregex.go: +288 lines - all missing stdlib API methodsregex_stdlib_compat_test.go: New - 16 compatibility testsdocs/STDLIB_COMPATIBILITY.md: Updated to 100% API coverageCHANGELOG.md,ROADMAP.md: v0.10.7 documentationTest plan
go test ./...)Fixes
Closes #85, #87, #88, #90, #91