Add support for additional read-only artifact stores#9702
Add support for additional read-only artifact stores#9702openshift-merge-bot[bot] merged 2 commits into
Conversation
|
Hi @pauloappbr. Thanks for your PR. I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds support for configurable read-only auxiliary OCI artifact stores. NewStore accepts additionalPaths; Store tracks and queries additionalStores for Status/List/Pull (RO-first, dedup by digest). Removal remains limited to the main writable store. Config gains ReadOnlyArtifactStores and call sites updated. Changes
Sequence Diagram(s)sequenceDiagram
participant Server as CRI-O Server
participant RO as Read-Only Store(s)
participant Main as Writable Store
participant Registry as Remote Registry
Note over Server,RO: Request to ensure/pull artifact by reference
Server->>RO: Query Status/List for artifact (in configured order)
alt found in RO
RO-->>Server: Return artifact (digest/path)
else not found in any RO
Server->>Main: Query Status/List
alt found in Main
Main-->>Server: Return artifact (digest/path)
else not found
Server->>Registry: Pull artifact (copy.Image)
Registry-->>Main: Write artifact into main writable store
Main-->>Server: Return artifact (digest/path)
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (6)
pkg/config/config.go (1)
425-427: Consider adding path validation for AdditionalArtifactStores.The new
AdditionalArtifactStoresfield lacks validation in theRuntimeConfig.Validate()method. Similar configuration fields likeHooksDir(lines 1364-1385) include checks to ensure paths are valid directories. Consider adding validation to:
- Check if paths are absolute
- Verify paths exist or can be created
- Log warnings for invalid paths rather than failing startup (to match the behavior in
internal/ociartifact/store.gowhich ignores invalid stores)[scratchpad_end] -->
♻️ Example validation pattern
You could add validation in the
RuntimeConfig.Validate()method similar to the HooksDir pattern:// Validate additional artifact stores for _, storePath := range c.AdditionalArtifactStores { if !filepath.IsAbs(storePath) { logrus.Warnf("Additional artifact store path is not absolute: %s", storePath) continue } if err := os.MkdirAll(storePath, 0o755); err != nil { logrus.Warnf("Cannot create additional artifact store directory %s: %v", storePath, err) } }Note: Since
internal/ociartifact/store.goalready silently ignores invalid stores, this validation would be informational and help users catch configuration errors early.internal/storage/image.go (1)
904-904: Pass AdditionalArtifactStores to the artifact fallback path for consistency.The OCI artifact store created in the fallback path (line 904) uses
nilfor additional stores, while the main image pull path inserver/server.go:461explicitly passesconfig.AdditionalArtifactStores. To ensure consistent artifact resolution across all code paths, consider threading the additional artifact stores throughImageCopyOptionstopullImageImplementation. This would allow the fallback artifact retrieval to check additional read-only stores, matching behavior with the primary pull path.internal/ociartifact/store.go (4)
57-69: Consider logging errors when ignoring invalid additional stores.While silently ignoring invalid stores prevents CRI-O startup failures, users may not realize their configured additional stores are being skipped. Adding a warning log would improve observability.
📋 Proposed improvement
for _, path := range additionalPaths { // Assume the pattern is to have an "artifacts" directory within the configured path, // similar to how additionalimagestores works in containers/storage. addPath := filepath.Join(path, "artifacts") addStore, err := libartStore.NewArtifactStore(addPath, systemContext) if err != nil { - // Just ignore invalid stores to prevent breaking CRI-O startup + // Just ignore invalid stores to prevent breaking CRI-O startup + log.Warnf(context.Background(), "Ignoring invalid additional artifact store %q: %v", path, err) continue } additionalStores = append(additionalStores, RealLibartifactStore{addStore}) validAdditionalPaths = append(validAdditionalPaths, addPath) }As per coding guidelines, use logrus with structured fields for logging.
110-110: Use structured logging fields instead of string formatting.The logging statement uses string formatting, but coding guidelines recommend using logrus with structured fields.
📋 Proposed improvement
- log.Infof(ctx, "Checking/Pulling OCI artifact from ref: %s", ref) + log.WithFields(log.Fields{ + "ref": ref, + }).Infof(ctx, "Checking/Pulling OCI artifact")As per coding guidelines, use logrus with structured fields for logging.
166-172: Consider logging errors when listing additional stores.Errors from additional stores are silently ignored, which could hide issues like permission problems or corrupted stores. Adding warning logs would improve observability without breaking functionality.
📋 Proposed improvement
for _, addStore := range s.additionalStores { addArts, err := addStore.List(ctx) if err != nil { + log.Warnf(ctx, "Failed to list artifacts from additional store: %v", err) continue } arts = append(arts, addArts...) }As per coding guidelines, use logrus with structured fields for logging.
473-488: Consider logging errors during path resolution.The
resolveRootPathfunction silently ignores errors fromList()calls (lines 475, 481). While this might be intentional for robustness with read-only stores, logging these errors would help diagnose configuration or permission issues.📋 Proposed improvement
func (s *Store) resolveRootPath(ctx context.Context, nameOrDigest string) (string, error) { - arts, _ := s.LibartifactStore.List(ctx) + arts, err := s.LibartifactStore.List(ctx) + if err != nil { + log.Debugf(ctx, "Failed to list artifacts from main store during path resolution: %v", err) + } if s.containsArtifact(arts, nameOrDigest) { return s.rootPath, nil } for i, store := range s.additionalStores { - arts, _ := store.List(ctx) + arts, err := store.List(ctx) + if err != nil { + log.Debugf(ctx, "Failed to list artifacts from additional store during path resolution: %v", err) + continue + } if s.containsArtifact(arts, nameOrDigest) { return s.additionalPaths[i], nil } } return "", fmt.Errorf("artifact %s not found in any store", nameOrDigest) }As per coding guidelines, use logrus with structured fields for logging.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
internal/config/seccomp/seccompociartifact/seccompociartifact.gointernal/oci/container.gointernal/ociartifact/store.gointernal/ociartifact/store_test.gointernal/storage/image.gopkg/config/config.goserver/server.go
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go
📄 CodeRabbit inference engine (AGENTS.md)
**/*.go: Use interface-based design and dependency injection patterns in Go code
Propagate context.Context through function calls in Go code
Usefmt.Errorfwith%wfor error wrapping in Go code
Use logrus with structured fields for logging in Go code
Add comments explaining 'why' not 'what' in Go code
Use platform-specific file naming:*_{linux,freebsd}.gofor platform-dependent code
Files:
internal/storage/image.gointernal/oci/container.goserver/server.gointernal/config/seccomp/seccompociartifact/seccompociartifact.gopkg/config/config.gointernal/ociartifact/store.gointernal/ociartifact/store_test.go
**/*_test.go
📄 CodeRabbit inference engine (AGENTS.md)
Use
*_test.gonaming convention for unit test files
Files:
internal/ociartifact/store_test.go
🧠 Learnings (3)
📓 Common learnings
Learnt from: CR
Repo: cri-o/cri-o PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-03T18:27:19.593Z
Learning: When adding/changing features, update related repositories: cri-o.io website and packaging repositories
📚 Learning: 2025-12-17T13:38:34.646Z
Learnt from: bitoku
Repo: cri-o/cri-o PR: 9667
File: server/container_create.go:1233-1236
Timestamp: 2025-12-17T13:38:34.646Z
Learning: In the cri-o/cri-o repository, protobuf-generated Get* methods for k8s.io/cri-api types are nil-safe: if the receiver is nil, GetX() returns the zero value instead of panicking. Do not add explicit nil checks before chaining calls on such getters. Apply this guidance to all Go code that uses these generated getters across the codebase.
Applied to files:
internal/storage/image.gointernal/oci/container.goserver/server.gointernal/config/seccomp/seccompociartifact/seccompociartifact.gopkg/config/config.gointernal/ociartifact/store.gointernal/ociartifact/store_test.go
📚 Learning: 2025-12-18T13:28:24.244Z
Learnt from: bitoku
Repo: cri-o/cri-o PR: 9676
File: internal/lib/stats/cgroup_stats_unsupported.go:1-7
Timestamp: 2025-12-18T13:28:24.244Z
Learning: In the cri-o/cri-o repository, for platform-specific types guarded by Go build tags (for example //go:build !linux), implement empty structs for unsupported platforms to permit compilation and clearly indicate the feature is not available rather than mirroring the Linux struct with unpopulated fields. Apply this pattern to all relevant platform-specific files across the codebase (i.e., any file under build-taged sections that should compile on all targets but lacks full implementation for some platforms).
Applied to files:
internal/storage/image.gointernal/oci/container.goserver/server.gointernal/config/seccomp/seccompociartifact/seccompociartifact.gopkg/config/config.gointernal/ociartifact/store.gointernal/ociartifact/store_test.go
🧬 Code graph analysis (6)
internal/storage/image.go (1)
internal/ociartifact/store.go (1)
NewStore(46-78)
server/server.go (1)
internal/ociartifact/store.go (2)
NewStore(46-78)Store(36-43)
internal/config/seccomp/seccompociartifact/seccompociartifact.go (2)
internal/ociartifact/store.go (1)
NewStore(46-78)internal/config/seccomp/seccompociartifact/impl.go (1)
Impl(10-12)
pkg/config/config.go (2)
internal/config/cgmgr/cgmgr_linux.go (2)
SetCgroupManager(112-133)CgroupManager(41-98)internal/config/cgmgr/cgmgr_unsupported.go (2)
SetCgroupManager(40-42)CgroupManager(9-27)
internal/ociartifact/store.go (5)
internal/ociartifact/libartifact_store.go (2)
LibartifactStore(13-25)RealLibartifactStore(27-29)internal/ociartifact/impl.go (1)
Impl(16-28)vendor/go.podman.io/common/pkg/libartifact/store/store.go (1)
NewArtifactStore(50-84)internal/ociartifact/artifact.go (2)
Artifact(13-18)ArtifactData(21-23)vendor/go.podman.io/common/pkg/libartifact/artifact.go (1)
Artifact(13-18)
internal/ociartifact/store_test.go (4)
test/mocks/ociartifact/ociartifact.go (2)
MockLibartifactStore(213-217)NewMockLibartifactStore(225-229)internal/ociartifact/store.go (3)
Store(36-43)NewStore(46-78)FakeLibartifactStore(518-520)internal/ociartifact/store_test_inject.go (1)
FakeLibartifactStore(21-23)internal/ociartifact/libartifact_store.go (1)
LibartifactStore(13-25)
🔇 Additional comments (10)
pkg/config/config.go (1)
2233-2249: LGTM - Testing helpers follow Go conventions.These setter methods provide appropriate dependency injection points for testing. The "Used for testing only" comments clearly indicate their purpose. This pattern is common in Go codebases for enabling test isolation.
server/server.go (1)
461-461: LGTM - Correctly wires additional artifact stores configuration.The artifact store initialization properly passes
config.AdditionalArtifactStoresto enable support for multiple read-only artifact stores. This change aligns with the updatedNewStoresignature and implements the feature as intended.internal/oci/container.go (1)
1003-1016: LGTM - Testing helpers are well-structured.These test-only setter methods provide appropriate mechanisms for test setup:
SetStateallows direct state injectionSetStateAndSpoofPidadditionally sets a deterministic PID (12345) for testing scenariosThe hardcoded PID value is clearly for testing and aligns with the existing
Spoofed()pattern in the container struct.internal/config/seccomp/seccompociartifact/seccompociartifact.go (2)
24-24: LGTM - NewStore call updated correctly.The artifact store initialization has been updated to match the new three-argument signature, passing
nilfor additional stores. This is appropriate for the seccomp artifact context.
94-98: LGTM - Test injection method follows best practices.The
SetImplmethod provides a clean dependency injection point for testing, allowing tests to mock the artifact store implementation. This follows common Go testing patterns.internal/ociartifact/store.go (2)
262-265: Verify fallback behavior is intentional.When
resolveRootPathfails, this function falls back tos.rootPath, whereasBlobMountPaths(lines 420-423) returns an error. This inconsistency suggests different design intentions, but it should be verified that the fallback behavior here is correct.If the artifact doesn't exist in any store, using
s.rootPathwill likely cause a subsequent error when trying to access the artifact data. Consider whether it's better to return the error immediately for clearer error messages.
504-526: LGTM: Clean testing infrastructure.The testing helpers follow good dependency injection patterns and are clearly documented. The approach of exposing
SetImplandSetFakeStoremethods allows for flexible test setup while keeping the production code clean.internal/ociartifact/store_test.go (3)
50-77: LGTM: Test properly adapted to new Store structure.The test correctly:
- Creates Store with
nilforadditionalPaths(testing the base case)- Injects mocks using
SetImplandSetFakeStore- Sets up appropriate expectations with
AnyTimes()for methods called multiple times- Verifies error messages contain expected substrings
The test maintains good coverage while adapting to the new multi-store architecture.
79-109: LGTM: Consistent test pattern.This test follows the same well-structured pattern as the first test, maintaining consistency in the test suite. The error path coverage for
DockerNewReferenceis valuable.
112-188: LGTM: Comprehensive test coverage for Status function.The Status tests provide good coverage of different scenarios:
- Unqualified names with empty store (should return ErrNotFound)
- Unqualified names with artifacts present (should return validation error)
- Fully-qualified names (should succeed)
The test setup is consistent with other tests in the file and properly injects dependencies.
bitoku
left a comment
There was a problem hiding this comment.
Thank you for the PR!
We're in the middle of migration from our implementation to libartifact in c/container-libs.
I think we should wait for a while until it's done to avoid rework.
|
Ref: #9691 |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9702 +/- ##
==========================================
+ Coverage 64.68% 67.59% +2.90%
==========================================
Files 212 212
Lines 29235 29327 +92
==========================================
+ Hits 18911 19823 +912
+ Misses 8635 7812 -823
- Partials 1689 1692 +3 🚀 New features to boost your workflow:
|
|
A friendly reminder that this PR had no activity for 30 days. |
b07cdc4 to
4f3cb91
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
internal/factory/container/device_test.go (1)
299-303: Consider using octal notation for file mode clarity.
os.FileMode(432)is correct but opaque. Usingos.FileMode(0o660)makes the intent (owner+group read-write) immediately obvious to reviewers.🔧 Suggested change
- FileMode: func() *os.FileMode { m := os.FileMode(432); return &m }(), + FileMode: func() *os.FileMode { m := os.FileMode(0o660); return &m }(),Apply similarly at lines 310, 390, and 397.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@internal/factory/container/device_test.go` around lines 299 - 303, Replace opaque decimal filemode literals with octal notation for clarity: change instances like FileMode: func() *os.FileMode { m := os.FileMode(432); return &m }() to FileMode: func() *os.FileMode { m := os.FileMode(0o660); return &m }(), and apply the same replacement for the other FileMode occurrences in the test (the FileMode struct fields shown in the device test cases) so each uses the equivalent 0o... octal literal instead of a decimal number.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@internal/ociartifact/store.go`:
- Around line 94-106: Store.List currently concatenates results from
s.libartifactStore and s.additionalStores causing duplicate Artifact entries;
modify Store.List to deduplicate artifacts by a uniqueness key (e.g.,
Artifact.Digest()) when merging. Implement a seen map keyed by the digest (or
the Artifact's unique identifier) and only append artifacts whose key is not yet
in the map; keep using s.libartifactStore and s.additionalStores and ensure
callers like getByNameOrDigest continue to work unchanged. Ensure nil-safe
handling if Digest() can return empty and preserve original ordering by adding
main store items first followed by first-seen items from additional stores.
- Around line 40-54: The loop over additionalPaths in
internal/ociartifact/store.go currently swallows errors from
libart.NewArtifactStore; update the error branch inside the for loop (the block
handling addStore, err := libart.NewArtifactStore(addPath, systemContext)) to
emit a warning using logrus with structured fields (include the attempted path
and the error) — e.g., use logrus.WithFields(map[string]interface{}{"path":
addPath, "err": err}).Warn("skipping additional artifact store") — and then
continue to preserve existing behavior; do not change the control flow or how
additionalStores and validAdditionalPaths are appended. Ensure you import logrus
if not already present. If you prefer propagating context, consider evolving
NewArtifactStore to accept context.Context in a follow-up, but for this change
use logrus structured warning in the error branch.
- Around line 153-171: In resolveRootPath, wrap the error returned by
libart.NewArtifactStorageReference using fmt.Errorf with %w to preserve the
original error; specifically change the error return when
NewArtifactStorageReference(nameOrDigest) fails to return fmt.Errorf("invalid
nameOrDigest: %w", err) so it matches the wrapping style used in Status and
Remove.
---
Nitpick comments:
In `@internal/factory/container/device_test.go`:
- Around line 299-303: Replace opaque decimal filemode literals with octal
notation for clarity: change instances like FileMode: func() *os.FileMode { m :=
os.FileMode(432); return &m }() to FileMode: func() *os.FileMode { m :=
os.FileMode(0o660); return &m }(), and apply the same replacement for the other
FileMode occurrences in the test (the FileMode struct fields shown in the device
test cases) so each uses the equivalent 0o... octal literal instead of a decimal
number.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
internal/factory/container/device_test.gointernal/ociartifact/datastore/store.gointernal/ociartifact/store.gointernal/storage/image.gopkg/config/config.goserver/server.go
🚧 Files skipped from review as they are similar to previous changes (1)
- internal/storage/image.go
|
/ok-to-test |
6cc110b to
e7d9dce
Compare
|
I pushed a commit on top of this branch to address the review comments. PTAL again @cri-o/cri-o-maintainers /cherry-pick release-1.35 |
|
@saschagrunert: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pauloappbr, saschagrunert The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| // Skip pulling if the artifact already exists in any store (including | ||
| // read-only additional stores). This avoids duplicating artifacts that | ||
| // are already available and prevents unnecessary network traffic. | ||
| // To force a re-pull (e.g., after a tag re-point), the artifact must | ||
| // be removed first. | ||
| if art, err := s.Status(ctx, strRef); err == nil { | ||
| log.Infof(ctx, "Artifact %s already exists locally in %s, skipping pull", strRef, art.RootPath()) | ||
| dgst := art.Digest() | ||
|
|
||
| return &dgst, nil | ||
| } | ||
|
|
There was a problem hiding this comment.
This will change the current behavior even without additional store.
registry.example.com/models/llm:v2 which resolves to digest sha256:aaa won't be updated once it's in the main store even if someone pushes a new artifact to registry.example.com/models/llm:v2, same tag with different digest sha256:bbb.
There was a problem hiding this comment.
Fixed. The skip now only applies to artifacts found in read-only additional stores. Artifacts in the main store are always re-pulled, so tag re-pointing works as expected.
| Size: uint64(a.TotalSizeBytes()), | ||
| RepoTags: repoTags, | ||
| RepoDigests: []string{a.CanonicalName()}, | ||
| Pinned: true, |
There was a problem hiding this comment.
It's expected that artifacts are garbage collected, isn't it? There should be a check in GC to check whether the artifact/image is used.
e7d9dce to
1de2bd2
Compare
|
Integration tests fail all over the place because critest requires go 1.26 now. We may wanna push #9774 for that. |
This commit implements the logic allowing users to configure additional
read-only paths for OCI artifacts. This addresses use cases where artifacts
(such as large AI models) are pre-populated on shared filesystems or
read-only media, preventing redundant downloads to the graph root.
Changes included:
1. Core Logic (internal/ociartifact):
- Updated `Store` struct and `NewStore` to accept `additionalPaths`.
- Implemented `resolveRootPath` helper to look up artifacts in additional
stores before falling back to the main writeable store.
- Updated `PullData`, `List`, and `BlobMountPaths` to respect these paths.
2. Configuration (pkg/config):
- Added `AdditionalArtifactStores` slice to `RuntimeConfig`.
3. Wiring (server):
- Injected the configuration value into the artifact store initialization.
4. Build & Test Fixes:
- Updated `NewStore` signatures in `internal/storage/image.go` and
`seccompociartifact.go`.
- Added missing test helpers (`SetState`, `SetImpl`, `SetCgroupManager`)
in `container.go`, `config.go`, and `seccompociartifact.go` to ensure
full project compilation and test passing.
Fixes: cri-o#9570
Related: cri-o#9583
Signed-off-by: Paulo Henrique <paulo.hco47@gmail.com>
- Only skip pulling for artifacts found in additional (read-only) stores, not the main store, so tag re-pointing still works as expected - Deduplicate List results by reference only (additional stores win) - Move implementation detail from doc comment into function body - Mark additional_artifact_stores as experimental, subject to change Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
1de2bd2 to
a21c685
Compare
|
Rebased |
|
/lgtm |
|
@saschagrunert: new pull request created: #9829 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@pauloappbr: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds support for additional read-only OCI artifact stores via the
additional_artifact_storesconfiguration option and--additional-artifact-storesCLI flag. This allows pre-populated artifacts (e.g. large AI/ML models on NFS) to be used without re-downloading them into the main store.Artifacts in additional stores take priority over the main store. Pulls are skipped when an artifact already exists in any store. Tag re-pointing is not supported for read-only stores.
Which issue(s) this PR fixes:
Fixes #9570
Related to #9583
Does this PR introduce a user-facing change?