feat!: redesign transfer public API with N:M routing and per-mapping resolvers#2069
Conversation
…resolvers Redesign the transfer package public API to support flexible N:M component routing where different source components can be transferred to different target repositories, each with its own resolver. Key changes: - BuildGraphDefinition now takes only (ctx, ...Option) — fully option-based - WithTransfer(Component(...), ToRepositorySpec(...), FromResolver(...)) pattern for composing transfer mappings - Per-mapping resolvers: each WithTransfer carries its own source resolver - FromRepository(repo, spec) convenience for simple single-repo sources - FromLister(lister) for dynamic component enumeration - ComponentLister/ComponentListerFunc interfaces for batch component discovery - Multi-target support: same component can be transferred to multiple targets - Value-based repository deduplication via RepositoryEqual/AppendUniqueRepositories - Debug logging throughout discovery and graph construction - Comprehensive unit tests for public API, options, discovery, and graph builder - Integration tests for CTF-to-OCI, CTF-to-CTF, multi-component, multi-resource, recursive, FromRepository, and CopyModeAllResources scenarios BREAKING CHANGE: BuildGraphDefinition signature changed from positional parameters to option-based. Callers must use WithTransfer with Component, ToRepositorySpec, and FromResolver/FromRepository instead. Signed-off-by: Jakob Möller <contact@jakob-moeller.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an N:M transfer mapping DSL and wiring: new mapping types/options, per-component resolvers, multi-root transfer roots, multi-resolver discovery with recursive propagation of targets/resolvers/digests, and graph construction that emits per-component-per-target upload transformations. Tests and docs updated. Changes
Sequence DiagramsequenceDiagram
participant Caller as "Caller"
participant API as "Transfer API"
participant Mapper as "Mapping Resolver"
participant Lister as "Component Lister"
participant MultiRes as "multiResolver"
participant Discoverer as "Discoverer"
participant Graph as "Graph Builder"
Caller->>API: BuildGraphDefinition(ctx, opts...)
API->>Mapper: collectTransferRoots(Mappings)
loop per Mapping
alt explicit components
Mapper->>Mapper: resolve explicit ComponentID list
else lister
Mapper->>Lister: ListComponentVersions(ctx)
Lister-->>Mapper: []ComponentID
end
Mapper->>MultiRes: register resolver for component key (resolverMap)
Mapper-->>Mapper: merge targets (AppendUniqueRepositories)
end
Mapper-->>API: []TransferRoot
API->>Graph: BuildGraphDefinition(ctx, roots, recursive,...)
loop per root
Graph->>Discoverer: Discover(root)
Discoverer->>MultiRes: GetRepositorySpecificationForComponent / GetComponentVersionRepositoryForComponent
MultiRes-->>Discoverer: descriptor + repoSpec (or repo)
Discoverer->>Discoverer: verify pinned digest (if expected)
alt recursive && has references
Discoverer->>Discoverer: propagate child targets/resolvers into maps
end
end
Discoverer-->>Graph: discovered vertices
Graph->>Graph: generate per-component-per-target upload transformations
Graph-->>API: TransformationGraphDefinition
API-->>Caller: graph
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested Reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
bindings/go/transfer/transfer_test.go (1)
54-65: Non-deterministic map iteration in mock resolver.The
GetComponentVersionRepositoryForSpecificationiterates overm.reposmap whensharedRepois nil, but Go map iteration order is non-deterministic. While this works in current tests (which always setsharedRepo), it could cause flaky tests if used differently in the future.Consider making the fallback deterministic
func (m *mockCVRepoResolver) GetComponentVersionRepositoryForSpecification(_ context.Context, _ runtime.Typed) (repository.ComponentVersionRepository, error) { if m.err != nil { return nil, m.err } if m.sharedRepo != nil { return m.sharedRepo, nil } - for _, r := range m.repos { - return r, nil - } - return nil, fmt.Errorf("no repos configured") + return nil, fmt.Errorf("no repos configured and sharedRepo is nil") }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bindings/go/transfer/transfer_test.go` around lines 54 - 65, The fallback path in mockCVRepoResolver.GetComponentVersionRepositoryForSpecification uses a map iteration over m.repos which is non-deterministic; make it deterministic by selecting a stable key (e.g., collect the keys from m.repos, sort them, and return the repo for the first sorted key) or by explicitly looking up a known default key, then return that repo instead of iterating the map arbitrarily.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@bindings/go/transfer/integration/integration_test.go`:
- Around line 746-754: The test currently only adds a local blob via
addComponentWithResources which doesn't exercise CopyModeAllResources; update
the fixture to include a non-local resource (e.g., an OCI or Helm artifact)
instead of (or in addition to) the local blob so the CopyModeAllResources path
is hit—modify the call that creates the source component
(addComponentWithResources) to register an OCI/Helm style resource descriptor
(unique type and reference) or use an existing helper to add an OCI/Helm
resource to the ctfRepo, then invoke the transfer with CopyModeAllResources and
assert the non-local resource was copied (or that behavior differs from
CopyModeLocalBlobResources) to validate the branch (referencing
addComponentWithResources, createCTFRepository, and the CopyModeAllResources
setting).
In `@bindings/go/transfer/internal/discovery.go`:
- Around line 56-59: The code races on concurrent access to resolverMap:
Discover mutates the shared map while Resolve and other readers access it; add a
sync.RWMutex field (e.g., resolverMapMu) to the same struct that holds
resolverMap and use Lock/Unlock around writes (where Discover inserts child
resolvers and at the other write sites around the code referenced at lines
~195-200) and RLock/RUnlock around reads (e.g., in Resolve when doing
repoResolver, ok := r.resolverMap[key]) to avoid concurrent map read/write
panics; ensure the mutex is initialized as part of the struct (zero value is
fine) and apply the same locking discipline for any other accesses to
resolverMap.
In `@bindings/go/transfer/transfer.go`:
- Around line 47-49: The error message for the empty transfer mappings check
(when len(o.Mappings) == 0 in transfer.go) incorrectly repeats "WithTransfer";
update the fmt.Errorf call to a correct, non-duplicated instruction such as
fmt.Errorf("no transfer mappings specified: use WithTransfer") (or list the
actual intended option names if there are multiple), so the message accurately
guides the user.
- Around line 77-86: The loop currently sets rd.resolver only on first encounter
and silently ignores later m.Resolver values; change the logic so that when
byKey[key] already exists you compare existing rd.resolver with m.Resolver and
handle conflicts instead of silently dropping them: if rd.resolver is nil, set
it to m.Resolver; if rd.resolver is non-nil and not equal to m.Resolver, return
an error (or log a warning and return an error per reviewer guidance) indicating
a conflicting resolver for that component key; continue to always append targets
via internal.AppendUniqueRepositories(rd.targets, []runtime.Typed{m.Target}) and
preserve keyOrder behavior. Ensure you reference the same symbols (byKey,
rd.resolver, m.Resolver, rd.targets, keyOrder) when implementing the check and
error path.
---
Nitpick comments:
In `@bindings/go/transfer/transfer_test.go`:
- Around line 54-65: The fallback path in
mockCVRepoResolver.GetComponentVersionRepositoryForSpecification uses a map
iteration over m.repos which is non-deterministic; make it deterministic by
selecting a stable key (e.g., collect the keys from m.repos, sort them, and
return the repo for the first sorted key) or by explicitly looking up a known
default key, then return that repo instead of iterating the map arbitrarily.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9d8da3a0-c0d6-461a-8bf9-23c123828c5a
📒 Files selected for processing (13)
bindings/go/transfer/component.gobindings/go/transfer/doc.gobindings/go/transfer/integration/integration_test.gobindings/go/transfer/internal/discovery.gobindings/go/transfer/internal/discovery_test.gobindings/go/transfer/internal/graph.gobindings/go/transfer/internal/graph_test.gobindings/go/transfer/internal/helpers.gobindings/go/transfer/internal/helpers_test.gobindings/go/transfer/options.gobindings/go/transfer/options_test.gobindings/go/transfer/transfer.gobindings/go/transfer/transfer_test.go
frewilhelm
left a comment
There was a problem hiding this comment.
preliminary review: Besides the findings of coderabbitai it looks okay
Signed-off-by: Jakob Möller <contact@jakob-moeller.com>
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
🧹 Nitpick comments (1)
bindings/go/transfer/transfer.go (1)
84-86: Document the resolver reuse pattern for components.The pointer equality check
rd.resolver != m.Resolverat line 84-86 is intentional. Each call toFromRepository()creates a newrepoResolverstruct, so the same component used across multiple mappings requires either:
- Reusing the same resolver instance via
FromResolver(), or- Using
FromRepository()only once per componentThe error message states "each component must use the same resolver across all mappings," but the recommended pattern—reusing resolver instances or using
FromRepository()once per component—could be more explicitly documented.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bindings/go/transfer/transfer.go` around lines 84 - 86, Update the error handling and docs to explicitly document the resolver reuse pattern: when the pointer check rd.resolver != m.Resolver fails, enhance the error message (and/or add an inline comment near the rd.resolver comparison and the repoResolver definition) to state that resolver identity is checked by pointer equality and that callers must either reuse the same resolver instance via FromResolver() or call FromRepository() only once per component; reference the rd.resolver vs m.Resolver check and the repoResolver/FromRepository()/FromResolver() symbols so readers know where the requirement originates.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@bindings/go/transfer/transfer.go`:
- Around line 84-86: Update the error handling and docs to explicitly document
the resolver reuse pattern: when the pointer check rd.resolver != m.Resolver
fails, enhance the error message (and/or add an inline comment near the
rd.resolver comparison and the repoResolver definition) to state that resolver
identity is checked by pointer equality and that callers must either reuse the
same resolver instance via FromResolver() or call FromRepository() only once per
component; reference the rd.resolver vs m.Resolver check and the
repoResolver/FromRepository()/FromResolver() symbols so readers know where the
requirement originates.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a1ef1424-5223-41dc-8936-56e6c6322cab
📒 Files selected for processing (7)
bindings/go/transfer/integration/integration_test.gobindings/go/transfer/internal/discovery.gobindings/go/transfer/internal/discovery_test.gobindings/go/transfer/internal/graph.gobindings/go/transfer/options.gobindings/go/transfer/transfer.gobindings/go/transfer/transfer_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- bindings/go/transfer/options.go
fabianburth
left a comment
There was a problem hiding this comment.
I'm not through yet, but here are some of my thoughts so far.
Generally, I like the API!
…est utilities - Add conditional `digest` and `label` handling in `oci.go` for enhanced nil safety. - Introduce `pushTestOCIImage` utility for integration tests and support `CopyModeAllResources`. Signed-off-by: Jakob Möller <contact@jakob-moeller.com>
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| - | - | Username Password | 0bce61e | bindings/go/transfer/integration/integration_test.go | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
- Switch `TransferRoot` collection from slice to map for improved key uniqueness and lookup efficiency. - Update tests and internal structures to align with the new map-based implementation. Signed-off-by: Jakob Möller <contact@jakob-moeller.com>
e3503f1 to
acf40ac
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
bindings/go/transfer/internal/graph.go (1)
71-95:⚠️ Potential issue | 🟠 MajorFail fast when different recursive roots claim the same child with different source state.
resolverMapanddiscoveredDigestsare shared globally bycomponent:version, but recursive discovery populates them with winner-takes-the-slot semantics. If two roots reach the same descendant through different source resolvers or different pinned digests, the source chosen for that child becomes schedule-dependent, so the transfer can end up resolving it from the wrong origin and only validating against one parent's digest. Please detect conflicting inherited resolver/digest assignments and return an error instead of keeping a single mutable slot per child key.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bindings/go/transfer/internal/graph.go` around lines 71 - 95, The current discoverer/multiResolver uses shared maps (resolverMap and discoveredDigests) with winner-takes-slot semantics, which can hide conflicts when different recursive roots assign different resolvers or pinned digests for the same child; update the discovery logic in discoverer (where discoveredDigests is populated) and in the multiResolver.expectedDigest resolution to detect conflicting assignments: when attempting to set an entry in discoveredDigests or resolverMap for an existing key, compare the existing value to the new one and return a descriptive error if they differ (do not overwrite), and propagate that error up to abort the transfer; use the unique symbols discoverer, discoveredDigests, resolverMap, and multiResolver.expectedDigest to locate the places to add these checks and error returns.
🧹 Nitpick comments (1)
bindings/go/transfer/transfer_test.go (1)
484-578: Add a same-component/different-resolver case to thecollectTransferRootstests.This block exercises merge and dedupe behavior, but not the branch where
collectTransferRootsmust reject two mappings for the same component key with different resolver instances. That check is one of the main safety rails of the new per-mapping resolver API, so it should have an explicit regression test.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bindings/go/transfer/transfer_test.go` around lines 484 - 578, Add a regression test that verifies collectTransferRoots rejects two mappings for the same component key when the Resolver instances differ: create two distinct mockCVRepoResolver instances (not the same pointer), build an Options with two Mapping entries that share the same ComponentID (e.g., "ocm.software/test:1.0.0") but assign different Resolver objects and the same or different Targets, call collectTransferRoots(t.Context(), o) and assert it returns a non-nil error (and does not silently merge); reference collectTransferRoots, Options, Mapping, ComponentID and mockCVRepoResolver to locate where to add the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@bindings/go/transfer/internal/graph.go`:
- Around line 71-95: The current discoverer/multiResolver uses shared maps
(resolverMap and discoveredDigests) with winner-takes-slot semantics, which can
hide conflicts when different recursive roots assign different resolvers or
pinned digests for the same child; update the discovery logic in discoverer
(where discoveredDigests is populated) and in the multiResolver.expectedDigest
resolution to detect conflicting assignments: when attempting to set an entry in
discoveredDigests or resolverMap for an existing key, compare the existing value
to the new one and return a descriptive error if they differ (do not overwrite),
and propagate that error up to abort the transfer; use the unique symbols
discoverer, discoveredDigests, resolverMap, and multiResolver.expectedDigest to
locate the places to add these checks and error returns.
---
Nitpick comments:
In `@bindings/go/transfer/transfer_test.go`:
- Around line 484-578: Add a regression test that verifies collectTransferRoots
rejects two mappings for the same component key when the Resolver instances
differ: create two distinct mockCVRepoResolver instances (not the same pointer),
build an Options with two Mapping entries that share the same ComponentID (e.g.,
"ocm.software/test:1.0.0") but assign different Resolver objects and the same or
different Targets, call collectTransferRoots(t.Context(), o) and assert it
returns a non-nil error (and does not silently merge); reference
collectTransferRoots, Options, Mapping, ComponentID and mockCVRepoResolver to
locate where to add the test.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 35f2e80d-de0b-459d-9fef-df97530d80d1
📒 Files selected for processing (4)
bindings/go/transfer/internal/graph.gobindings/go/transfer/internal/graph_test.gobindings/go/transfer/transfer.gobindings/go/transfer/transfer_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- bindings/go/transfer/transfer.go
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
bindings/go/transfer/internal/graph.go (1)
65-66: Sort roots and discovered component keys before emitting the graph.Both loops iterate Go maps directly, so discovery order and
tgd.Transformationsorder vary between runs. The graph may still execute, but serialized definitions, logs, and order-sensitive tests will flap. Sorting the keys before these loops would make the API output reproducible.♻️ Proposed stabilization
+ rootKeys := make([]string, 0, len(roots)) + for key := range roots { + rootKeys = append(rootKeys, key) + } + sort.Strings(rootKeys) - for key, root := range roots { + for _, key := range rootKeys { + root := roots[key] dagRoots = append(dagRoots, key) targetMap[key] = AppendUniqueRepositories(targetMap[key], root.Targets) resolverMap[key] = root.SourceResolver }+ componentKeys := make([]string, 0, len(d.Vertices)) + for key := range d.Vertices { + componentKeys = append(componentKeys, key) + } + sort.Strings(componentKeys) - for key, v := range d.Vertices { + for _, key := range componentKeys { + v := d.Vertices[key] val := v.Attributes[dagsync.AttributeValue].(*discoveryValue) // ... }You'd also need to add
import "sort".Also applies to: 154-199
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bindings/go/transfer/internal/graph.go` around lines 65 - 66, The code iterates Go maps (e.g., the loop building dagRoots from roots and the loop over tgd.Transformations) which yields non-deterministic order; sort the keys before iterating to produce stable output: collect map keys into a slice, call sort.Strings (or appropriate comparator) and then append/iterate in that sorted order (apply this to the roots->dagRoots construction and the loop over tgd.Transformations/discovered component keys), and add import "sort".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@bindings/go/transfer/internal/graph.go`:
- Around line 22-29: The TransferRoot struct currently groups many Targets with
a single SourceResolver which causes the last resolver to overwrite earlier
mappings; change the shape so each target is paired with its resolver (e.g.,
replace Targets []runtime.Typed with Targets []TargetMapping where TargetMapping
has Target runtime.Typed and Resolver
resolvers.ComponentVersionRepositoryResolver) and update all
construction/consumption sites (notably the code paths in transfer.go around
WithTransfer and the other usages referenced) to emit one TransferRoot per
target-mapping or iterate TargetMapping so each target uses its own resolver;
also adjust the struct fields at the other occurrence noted (lines ~67-68) to
match the new pair shape.
---
Nitpick comments:
In `@bindings/go/transfer/internal/graph.go`:
- Around line 65-66: The code iterates Go maps (e.g., the loop building dagRoots
from roots and the loop over tgd.Transformations) which yields non-deterministic
order; sort the keys before iterating to produce stable output: collect map keys
into a slice, call sort.Strings (or appropriate comparator) and then
append/iterate in that sorted order (apply this to the roots->dagRoots
construction and the loop over tgd.Transformations/discovered component keys),
and add import "sort".
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 122ed5dd-5669-4882-b175-7ff0a37aab61
📒 Files selected for processing (5)
bindings/go/transfer/integration/integration_test.gobindings/go/transfer/internal/graph.gobindings/go/transfer/internal/graph_test.gobindings/go/transfer/transfer.gobindings/go/transfer/transfer_test.go
✅ Files skipped from review due to trivial changes (1)
- bindings/go/transfer/transfer.go
🚧 Files skipped from review as they are similar to previous changes (2)
- bindings/go/transfer/internal/graph_test.go
- bindings/go/transfer/integration/integration_test.go
- Enforce error handling for ambiguous resolvers during recursive discovery. - Enhance `TransferRoot` design with clear documentation on resolver behavior and target handling. Signed-off-by: Jakob Möller <contact@jakob-moeller.com>
|
great addition - just a few questions |
What this PR does / why we need it
Redesign the transfer package public API to support flexible N:M component routing where different source components can be transferred to different target repositories, each with its own resolver.
Key changes:
BREAKING CHANGE: BuildGraphDefinition signature changed from positional parameters to option-based. Callers must use WithTransfer with Component, ToRepositorySpec, and FromResolver/FromRepository instead.
Which issue(s) this PR fixes
fixes many issues I had with exposure of the public api of transfer because it was hard to use. now it should be much simpler and more flexible
Testing
checkout the integration tests
Verification
ocmI have run these with a custom build of the CLI that was adjusted to use the new lib. 41fe25epart of open-component-model/ocm-project#933
Note: Most of the added code is actually tests.
snippet from CLI