fix(migration): handle non-UTF-8 files in OpenClaw migration#9145
Open
Tranquil-Flow wants to merge 2 commits into
Open
fix(migration): handle non-UTF-8 files in OpenClaw migration#9145Tranquil-Flow wants to merge 2 commits into
Tranquil-Flow wants to merge 2 commits into
Conversation
Collaborator
|
Likely duplicate of #8898 — both fix UnicodeDecodeError in OpenClaw migration for non-UTF-8 locales, targeting the same file (openclaw_to_hermes.py) with the same approach (errors='replace'). |
d571979 to
b828baa
Compare
On Windows with non-UTF-8 locales (e.g. GBK/cp936), the migration script crashed with UnicodeDecodeError when encountering binary files or system-encoded text. All file-reading functions now use errors="replace" and per-file error recovery so the migration completes instead of aborting. Closes NousResearch#8901
b828baa to
e00c158
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixed
UnicodeDecodeErrorcrash duringhermes claw migrateon Windows systems with non-UTF-8 locales (e.g. GBK/cp936).On Chinese/Japanese/Korean Windows, the default encoding is not UTF-8. The migration script crashed when encountering binary files (images, SQLite) or system-encoded text files, making it impossible to migrate from OpenClaw. This PR adds
errors="replace"decoding (invalid bytes become U+FFFD instead of crashing) plus per-file try/except so a single bad file no longer aborts the whole migration.Related Issue
Closes #8901
Type of Change
Changes Made
Five changes in
openclaw_to_hermes.py, all consistent in approach:read_text()— addederrors="replace"so invalid bytes become U+FFFD instead of crashingload_yaml_file()— addederrors="replace"+UnicodeDecodeErrorin except clauseparse_env_file()— addederrors="replace"load_openclaw_config()— addederrors="replace"+UnicodeDecodeErrorin except clausemigrate_daily_memory()— added per-file try/except so one bad file doesn't abort the entire migrationHow to Test
pytest tests/skills/test_openclaw_migration.py -v— all 37 tests pass (5 new + 32 existing)Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/AScreenshots / Logs