Skip to content

fix: extract converters from amends/extends base modules#52

Merged
jdx merged 7 commits intomainfrom
fix/converters-from-amends-base
Mar 25, 2026
Merged

fix: extract converters from amends/extends base modules#52
jdx merged 7 commits intomainfrom
fix/converters-from-amends-base

Conversation

@jdx
Copy link
Copy Markdown
Owner

@jdx jdx commented Mar 25, 2026

Summary

Fixes the missing field '_type' error in hk's CI (PR #769).

PR #49 added output.renderer.converters support, but only extracted converters from the current module's body entries. When a file amends or extends a base module that defines converters (like hk's hk.pkl amending Config.pkl), those converters were never extracted.

  • Extract converters from the base module's AST during amends processing
  • Extract converters from the base module's AST during extends processing
  • Added converter_inherited_from_amends_base test with two files on disk

Test plan

  • New test: converter_inherited_from_amends_base — creates Base.pkl with converters + child.pkl that amends it, verifies converters are applied
  • All 235 existing tests pass

This comment was generated by Claude Code.

When a pkl file amends or extends a base module that defines
output.renderer.converters, pklr was not extracting those converters
because it only looked at the current module's body entries. This
caused "missing field _type" deserialization errors in consumers
like hk where Config.pkl (the base) defines the converters and
hk.pkl amends it.

Now extract converters from base module AST during amends/extends
processing, before the current module's entries are evaluated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where Pkl converters defined in base modules were not being inherited by modules that amend or extend them, leading to runtime errors. The changes ensure that the evaluator correctly extracts and applies these converters from the base module's AST during both amends and extends operations, thereby resolving the missing field '_type' error and improving the robustness of Pkl module inheritance.

Highlights

  • Converter Inheritance Fix: Resolved the "missing field '_type'" error in hk's CI by ensuring Pkl converters defined in base modules are correctly inherited by amending/extending modules.
  • Amends Processing: Modified the evaluator to extract converters from the base module's Abstract Syntax Tree (AST) during amends processing.
  • Extends Processing: Modified the evaluator to extract converters from the base module's AST during extends processing.
  • New Test Case: Introduced converter_inherited_from_amends_base to verify the correct application of inherited converters.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces functionality to allow amending modules to inherit converters defined in their base modules' output blocks. The changes in src/eval.rs modify the Evaluator to extract converters from the 'output' property of base modules during evaluation, ensuring they are available for inherited objects. A new test case, converter_inherited_from_amends_base, has been added to tests/pkl_features.rs to validate this inheritance behavior. There is no feedback to provide on the review comments as none were supplied.

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Mar 25, 2026

Greptile Summary

This PR fixes a bug where output.renderer.converters defined in a base module were not being inherited when a child module used amends or extends, causing a missing field '_type' error at runtime. The fix re-parses the base module's AST after evaluating it and calls the existing extract_converters_from_ast helper for the output property entry.

Key changes and observations:

  • Converter extraction is correctly added to the file-based amends path and the file-based extends path.
  • The HTTP extends branch (lines ~589–596) is not updated — it still only extracts ClassDef entries, leaving converter inheritance broken for any module that extends a remote HTTP base module.
  • Because base-module converters are inserted into self.converters before the child module's own output block is processed, and apply_converters_recursive returns on the first match, a child converter that overrides a base converter for the same class will be silently ignored (base wins over child).
  • The new test only covers amends with files on disk; the modified extends code path has no corresponding test.
  • The test writes to a fixed temp directory name, which could race in parallel CI environments.

Confidence Score: 3/5

  • The core amends fix is correct and well-tested, but the HTTP extends gap and the base-before-child converter ordering are genuine correctness issues that could surface in real-world usage.
  • The amends code path that motivated the fix works correctly and is tested. However, the HTTP extends branch still lacks converter extraction (an identical omission to the bug being fixed), and the first-match ordering of merged base+child converters inverts expected override semantics. These are not just edge cases — HTTP-hosted PKL modules are a first-class use case.
  • src/eval.rs — specifically the HTTP extends loop (~line 589) and the converter-insertion ordering relative to child module processing

Important Files Changed

Filename Overview
src/eval.rs Adds converter extraction to amends and file-based extends base-module processing; the HTTP extends branch is left without the same fix, and base-before-child insertion order may cause base converters to shadow child overrides.
tests/pkl_features.rs New integration test validating amends-based converter inheritance; coverage gap for the extends path and minor parallel-safety concern with a fixed temp directory name.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Evaluator
    participant BaseModule
    participant ChildModule

    Caller->>Evaluator: eval_file(child.pkl, depth=0)
    Evaluator->>ChildModule: parse(child.pkl)
    note over Evaluator,ChildModule: child.pkl amends/extends Base.pkl

    Evaluator->>BaseModule: eval_file(Base.pkl, depth=1)
    BaseModule-->>Evaluator: Value::Object (base_obj)

    note over Evaluator: Re-parse Base.pkl AST (depth=0, converters.is_empty())
    Evaluator->>BaseModule: parse(Base.pkl)
    loop base_module.body entries
        alt Entry::ClassDef
            Evaluator->>Evaluator: eval_class_def → scope
        else Entry::Property("output") [NEW]
            Evaluator->>Evaluator: extract_converters_from_ast → self.converters (base converters first)
        end
    end

    note over Evaluator: Second pass – child module body
    loop child_module.body entries
        alt Entry::Property("output")
            Evaluator->>Evaluator: extract_converters_from_ast → self.converters (child appended after base)
        else Entry::Property(other)
            Evaluator->>Evaluator: eval_expr → out map
        end
    end

    Caller->>Evaluator: apply_converters(val)
    note over Evaluator: Iterates self.converters in order — first match wins (base beats child if same class)
    Evaluator-->>Caller: converted Value
Loading

Comments Outside Diff (1)

  1. src/eval.rs, line 589-596 (link)

    P1 HTTP extends missing converter extraction

    The file-based extends branch (lines ~548–576) now extracts converters from the base module's output block via the new Property("output") arm. However, the HTTP extends branch iterates ext_module.body with only a ClassDef pattern-match and never reaches converter extraction. Any module that extends an HTTP-hosted base module defining output.renderer.converters will silently inherit no converters, reproducing the same bug this PR is fixing.

    The loop should be aligned with the file-based case:

    Fix in Claude Code

Fix All in Claude Code

Reviews (1): Last reviewed commit: "[autofix.ci] apply automated fixes" | Re-trigger Greptile

Comment thread src/eval.rs
Comment on lines +510 to 517
if let Entry::Property(prop) = entry
&& prop.name == "output"
&& depth == 0
&& self.converters.is_empty()
{
self.extract_converters_from_ast(prop, &scope, depth).await;
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Base converters inserted before child converters, first-match wins

Converters are extracted from the base module here (amends processing, before the current module's body is evaluated). When the second pass later processes the child module's own output block (line ~658), any child-defined converters for the same class are appended after the base converters. Because apply_converters_recursive iterates the slice in order and returns on the first match (line ~2148), the base's converter takes precedence over the child's, which is the reverse of PKL's expected override semantics.

The fix applied to the extends branch has the same ordering issue (line ~571). Consider inserting base converters only when no child converter for that class already exists, or reversing the lookup order so child converters shadow base converters.

Fix in Claude Code

Comment thread tests/pkl_features.rs Outdated
Comment on lines +3195 to +3199
let dir = std::env::temp_dir().join("pklr_test_amends_converter");
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();

// Base module with class + converter
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Fixed temp-dir path risks parallel-test collisions

The test writes to a hard-coded directory pklr_test_amends_converter inside temp_dir(). If cargo test runs this test concurrently with itself (e.g. two CI jobs sharing the same machine's /tmp), or if another test using the same name is added, the remove_dir_all / create_dir_all sequence can race and cause flaky failures.

Prefer a unique per-invocation path using tempfile::TempDir (already a common pattern in Rust test suites) or at minimum append the process ID:

let dir = std::env::temp_dir()
    .join(format!("pklr_test_amends_converter_{}", std::process::id()));

Fix in Claude Code

Comment thread tests/pkl_features.rs
Comment on lines 3188 to +3249
assert_eq!(json["item"]["_type"], "foo");
assert_eq!(json["item"]["x"], 42);
}

#[test]
fn converter_inherited_from_amends_base() {
use std::io::Write;
let dir = std::env::temp_dir().join("pklr_test_amends_converter");
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();

// Base module with class + converter
let base_path = dir.join("Base.pkl");
let mut base_file = std::fs::File::create(&base_path).unwrap();
write!(
base_file,
r#"
class Step {{
check: String = ""
}}

output {{
renderer {{
converters {{
[Step] = (s) -> new Dynamic {{
_type = "step"
...s.toDynamic()
}}
}}
}}
}}
"#
)
.unwrap();

// Amending module
let child_path = dir.join("child.pkl");
let mut child_file = std::fs::File::create(&child_path).unwrap();
write!(
child_file,
r#"amends "Base.pkl"

myStep = new Step {{
check = "cargo test"
}}
"#
)
.unwrap();

let rt = tokio::runtime::Runtime::new().unwrap();
let json = rt.block_on(async {
let mut ev = Evaluator::new();
let val = ev
.eval_source(&std::fs::read_to_string(&child_path).unwrap(), &child_path)
.await
.unwrap();
let val = ev.apply_converters(val).await.unwrap();
val.to_json()
});
assert_eq!(json["myStep"]["_type"], "step");
assert_eq!(json["myStep"]["check"], "cargo test");
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No test for the extends code path

The PR modifies both the amends and extends branches of eval_module, but the new test only exercises the amends path. The file-based extends fix (lines ~566–573) and any future HTTP-extends fix go untested. Adding a second fixture using extends "Base.pkl" would give confidence that both branches behave correctly.

Fix in Claude Code

- Child converters now override base converters (clear before re-extract)
- Remove is_empty() guard so base converters always populate as fallback
- Use process ID in temp dir path to avoid parallel test collisions
- Add converter_inherited_from_extends_base test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jdx jdx force-pushed the fix/converters-from-amends-base branch from ffccf5e to 09c03c6 Compare March 25, 2026 15:16
… generics

- Pass class name to eval_class_def and set type_name on ObjectSource
- Parse generic type params for Mapping/Map (previously skipped)
- Use the value type param as a default template for Mapping entries
- Preserve base ObjectSource in merge_values for type propagation

This enables converters to match objects created via typed Mappings
(e.g., `new Mapping<String, Step> { ["x"] { check = "echo" } }`).

Note: objects created via property amendment (e.g., `steps { ["x"] { ... } }`)
in amending modules do not yet carry type info, as pklr does not
implement type annotation propagation from class property declarations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jdx jdx force-pushed the fix/converters-from-amends-base branch from bfa77dd to 80e6a53 Compare March 25, 2026 22:31
Three interconnected fixes that enable converters to work with the
common pattern of bare objects in typed Mappings (e.g., hk.pkl's
`steps { ["echo"] { check = "echo ok" } }` amending a
`Mapping<String, Step>`):

1. Mapping generic type params: Parse generic params for Mapping/Map
   (`new Mapping<String, Step> {}`) and inject a synthetic `default`
   entry referencing the value type class. This makes the Mapping
   carry type information for its values.

2. Body amendment scope: When a property has a body amendment
   (`foo { ... }`) and the property exists in scope with ObjectSource,
   use eval_amended_object instead of creating a fresh object. This
   preserves the base value's type context through amendments.

3. Type-aware DynProperty evaluation: When a Mapping's default template
   has ObjectSource, evaluate DynProperty values via eval_amended_object
   rather than merge_values, and propagate the template's type_name to
   the result. This enables converters to match nested typed objects.

Also seeds scope with base properties before processing child module
entries, so body amendments can find base values.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jdx jdx force-pushed the fix/converters-from-amends-base branch from e0d95d5 to 42df72e Compare March 25, 2026 22:55
@jdx jdx merged commit fd03293 into main Mar 25, 2026
6 checks passed
@jdx jdx deleted the fix/converters-from-amends-base branch March 25, 2026 23:09
@jdx jdx mentioned this pull request Mar 25, 2026
jdx added a commit that referenced this pull request Mar 25, 2026
## 🤖 New release

* `pklr`: 0.3.0 -> 0.4.0 (⚠ API breaking changes)

### ⚠ `pklr` breaking changes

```text
--- failure enum_tuple_variant_field_added: pub enum tuple variant field added ---

Description:
An enum's exhaustive tuple variant has a new field, which has to be included when constructing or matching on this variant.
        ref: https://doc.rust-lang.org/reference/attributes/type_system.html#the-non_exhaustive-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.46.0/src/lints/enum_tuple_variant_field_added.ron

Failed in:
  field 2 of variant Expr::New in /tmp/.tmproz423/pklr/src/parser.rs:101
```

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

## [0.4.0](v0.3.0...v0.4.0) -
2026-03-25

### Fixed

- extract converters from amends/extends base modules
([#52](#52))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant