Skip to content

Replace ruamel.yaml with yamltrip for YAML round-tripping#1937

Open
nathanjmcdougall wants to merge 29 commits into
mainfrom
refactor/replace-ruamel-yaml-with-yamltrip
Open

Replace ruamel.yaml with yamltrip for YAML round-tripping#1937
nathanjmcdougall wants to merge 29 commits into
mainfrom
refactor/replace-ruamel-yaml-with-yamltrip

Conversation

@nathanjmcdougall

@nathanjmcdougall nathanjmcdougall commented May 15, 2026

Copy link
Copy Markdown
Collaborator

ruamel.yaml has been showing signs of potential abandonment from PyPI (see #1190). This commit replaces it with yamltrip, a Rust-backed (tree-sitter-yaml) round-tripping YAML library that provides the same comment- and formatting-preservation guarantees with better performance and a smaller dependency footprint.

Key changes:

Dependency swap:

  • Remove ruamel-yaml from project dependencies
  • Add yamltrip>=0.5.0

Core YAML I/O rewrite (src/usethis/file/yaml/io.py):

  • Rewrite YAMLFileManager to use yamltrip.Document internally
  • Introduce YAMLDocument wrapper (replacing the raw ruamel CommentedMap/CommentedSeq)
  • Add _upsert_safe() for robust scalar upserts handling empty documents, non-mapping routes, and exists_ok semantics
  • Add _upsert_complex() for list/dict values using a document-rebuild approach (yamltrip only supports scalar upsert/replace natively)
  • Add _serialize_yaml_value/_serialize_yaml_scalar helpers for producing valid block-style YAML from Python objects
  • Handle flow-sequence fallback in extend_list when yamltrip cannot append to flow sequences (e.g. 'repos: []')

Pre-commit integration (src/usethis/_integrations/pre_commit/):

  • Rewrite PreCommitConfigYAMLManager to use YAMLDocument
  • Refactor remove_hook() to use surgical remove_from_list for whole- repo removal (preserves inline comments on sibling repos) and only falls back to full rebuild when hooks within a repo are modified
  • Remove _pre_commit_fancy_dump (no longer needed; serialization is handled by the YAML I/O layer)

Deleted modules:

  • src/usethis/file/yaml/typing.py (ruamel type aliases)
  • src/usethis/_file/yaml/update.py (ruamel update helpers)
  • tests/usethis/_file/yaml/test_update.py

Test migration:

  • Rewrite all YAML I/O tests to use yamltrip assertions
  • Update pre-commit tests (test_hooks.py, test_yaml.py) to use yamltrip.loads() and YAMLDocument semantics
  • Relax exact-format assertions where the new serializer produces semantically equivalent but stylistically different output

Documentation:

  • Update FAQ and pipeweld docs to reference yamltrip
  • Regenerate functions.txt, module-tree.txt, AGENTS.md via hooks

Resolves #1190

ruamel.yaml has been showing signs of potential abandonment from PyPI
(see #1190). This commit replaces it with yamltrip, a Rust-backed
(tree-sitter-yaml) round-tripping YAML library that provides the same
comment- and formatting-preservation guarantees with better performance
and a smaller dependency footprint.

Key changes:

Dependency swap:
- Remove ruamel-yaml from project dependencies
- Add yamltrip>=0.2.0

Core YAML I/O rewrite (src/usethis/_file/yaml/io_.py):
- Rewrite YAMLFileManager to use yamltrip.Document internally
- Introduce YAMLDocument wrapper (replacing the raw ruamel
  CommentedMap/CommentedSeq)
- Add _upsert_safe() for robust scalar upserts handling empty
  documents, non-mapping routes, and exists_ok semantics
- Add _upsert_complex() for list/dict values using a document-rebuild
  approach (yamltrip only supports scalar upsert/replace natively)
- Add _serialize_yaml_value/_serialize_yaml_scalar helpers for
  producing valid block-style YAML from Python objects
- Handle flow-sequence fallback in extend_list when yamltrip cannot
  append to flow sequences (e.g. 'repos: []')

Pre-commit integration (src/usethis/_integrations/pre_commit/):
- Rewrite PreCommitConfigYAMLManager to use YAMLDocument
- Refactor remove_hook() to use surgical remove_from_list for whole-
  repo removal (preserves inline comments on sibling repos) and only
  falls back to full rebuild when hooks within a repo are modified
- Remove _pre_commit_fancy_dump (no longer needed; serialization is
  handled by the YAML I/O layer)

Deleted modules:
- src/usethis/_file/yaml/typing_.py (ruamel type aliases)
- src/usethis/_file/yaml/update.py (ruamel update helpers)
- tests/usethis/_file/yaml/test_update.py

Test migration:
- Rewrite all YAML I/O tests to use yamltrip assertions
- Update pre-commit tests (test_hooks.py, test_yaml.py) to use
  yamltrip.loads() and YAMLDocument semantics
- Relax exact-format assertions where the new serializer produces
  semantically equivalent but stylistically different output

Documentation:
- Update FAQ and pipeweld docs to reference yamltrip
- Regenerate functions.txt, module-tree.txt, AGENTS.md via hooks

Resolves #1190
Add explicit type arguments to dict and list parameters to satisfy
pyright reportMissingTypeArgument checks.
- Suppress unreachable code warning in fancy_model_dump else branch
- Add pyright: ignore for _FillValue type narrowing in zip_longest loop
- Fix dict type annotations in test references
- Remove unnecessary pyright ignore comments
@codspeed-hq

codspeed-hq Bot commented May 15, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 5.69%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 1 untouched benchmark

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation test_several_tools_add_and_remove 1.2 s 1.1 s +5.71%
Simulation test_several_tools_add_and_remove_no_backend 467.2 ms 442.1 ms +5.68%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing refactor/replace-ruamel-yaml-with-yamltrip (5424968) with main (9c9ae10)

Open in CodSpeed

@codecov

codecov Bot commented May 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

Demonstrates that removing one hook from a multi-hook repo currently
loses inline comments due to the document-rebuild workaround. This test
will pass once yamltrip supports complex values in upsert/replace.

Tracks: usethis-python/yamltrip#18
The check-comment-keywords hook bans 'pragma:' in comments.
…ative complex values

yamltrip 0.3.0 adds native complex value support in upsert/replace and a
.root property. This eliminates ~130 lines of custom serialization code:
- Remove _upsert_complex, _deep_set, and all _serialize_yaml_* functions
- Replace doc[()] with doc.root throughout
- Add placeholder-then-replace pattern in _upsert_safe to force block-style
  output for new complex value keys
- Update test assertions for yamltrip's compact list indentation style
- Update xfail reason for partial hook removal test
- Add 13 new tests covering edge cases: scalar/empty root docs,
  regex keys, flow sequence fallback, complex values on empty docs
- Merge duplicate QueryError/PatchError handlers in __delitem__
  (prune_remove never raises QueryError with yamltrip)
- io_.py coverage: 87% -> 99% (1 defensive line remaining)
@nathanjmcdougall nathanjmcdougall force-pushed the refactor/replace-ruamel-yaml-with-yamltrip branch from e1a5f61 to fc5c913 Compare May 18, 2026 02:30
@nathanjmcdougall nathanjmcdougall marked this pull request as ready for review May 18, 2026 06:00
@nathanjmcdougall nathanjmcdougall linked an issue May 18, 2026 that may be closed by this pull request
@nathanjmcdougall nathanjmcdougall force-pushed the refactor/replace-ruamel-yaml-with-yamltrip branch from ecdbedc to e0c4ba0 Compare May 21, 2026 05:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request migrates usethis’s YAML round-tripping backend from ruamel.yaml to yamltrip, updating the core YAML file manager, the pre-commit YAML integration, and the surrounding tests/docs to match the new YAML editing model.

Changes:

  • Replace ruamel-yaml dependency with yamltrip (lockfile + project metadata updated).
  • Rewrite YAML read/modify/write logic to use yamltrip.Document and new upsert/sync behavior.
  • Update pre-commit/YAML-related tests and documentation to reflect new round-tripping semantics and formatting.

Reviewed changes

Copilot reviewed 20 out of 31 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
uv.lock Removes ruamel-yaml(+clib) and adds yamltrip locked to 0.5.0.
pyproject.toml Swaps runtime dependency from ruamel-yaml to yamltrip>=0.5.0.
src/usethis/file/yaml/io.py Core YAML I/O manager rewritten to use yamltrip documents and patch operations.
src/usethis/_file/yaml/update.py Deleted ruamel-specific update/LCS logic.
src/usethis/file/yaml/typing.py Deleted ruamel scalar/type alias module.
src/_test.py Updates test YAML helper to load via yamltrip and map parse errors to YAMLDecodeError.
src/usethis/_integrations/pre_commit/yaml.py Updates pre-commit YAML manager to validate from yamltrip and commit via sync.
src/usethis/_integrations/pydantic/dump.py Adjusts dump logic (incl. list diffing) and tightens type exhaustiveness handling.
src/usethis/integrations/pydantic/typing.py Updates ModelRepresentation typing after removing ruamel-specific YAML literals.
src/usethis/_fallback.py Bumps fallback tool versions (uv/ruff).
tests/usethis/file/yaml/test_yaml_io.py Migrates YAML file manager tests from ruamel-specific behaviors to yamltrip semantics.
tests/usethis/_file/yaml/test_update.py Deletes tests for the removed ruamel update helpers.
tests/usethis/_integrations/pre_commit/test_yaml.py Updates tests to validate via yamltrip and new manager methods.
tests/usethis/_integrations/pre_commit/test_version.py Updates YAML fixtures/expectations after YAML formatting changes.
tests/usethis/_integrations/pre_commit/test_language.py Updates YAML fixtures/expectations after YAML formatting changes.
tests/usethis/_integrations/pre_commit/test_hooks.py Updates YAML parsing to use yamltrip and adjusts expected YAML output strings.
tests/usethis/_integrations/pydantic/test_dump.py Adjusts typing/pyright expectations related to model dumping changes.
tests/usethis/_tool/test_base.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_tool/impl/base/test_ty.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_tool/impl/base/test_tach.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_tool/impl/base/test_ruff.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_tool/impl/base/test_requirements_txt.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_tool/impl/base/test_import_linter.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_tool/impl/base/test_codespell.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/usethis/_core/test_core_tool.py Updates embedded pre-commit YAML fixtures for the new formatting/round-tripping.
tests/test__test.py Adds coverage for the updated _test.read_yaml() behavior and error mapping.
docs/about/faq.md Updates FAQ to reference yamltrip and describe formatting behavior.
docs/pipeweld.md Updates documentation references from ruamel-based updates to yamltrip-based mutations.
docs/module-tree.txt Removes deleted YAML modules from the documented module tree.
docs/functions.txt Removes deleted YAML helper functions from the documented function list.
docs/superpowers/specs/2026-05-15-yamltrip-migration-design.md Adds a design spec describing the migration approach and constraints.
docs/superpowers/plans/2026-05-15-yamltrip-migration.md Adds an implementation plan for the migration work.
AGENTS.md Updates agent/module/function inventories to reflect YAML module deletions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/usethis/_integrations/pydantic/dump.py Outdated
Comment thread src/usethis/_integrations/pydantic/typing_.py Outdated
Comment thread pyproject.toml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 31 changed files in this pull request and generated 2 comments.

Comment thread src/usethis/_file/yaml/io_.py
Comment thread src/usethis/_file/yaml/io_.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate to .root syntax in yamltrip ruamel.yaml might be abandoned (from PyPI)

2 participants