feat: version-aware JIT source provisioning with TTL-based cleanup#2010
feat: version-aware JIT source provisioning with TTL-based cleanup#2010
Conversation
|
Warning This PR exceeds the recommended limit of 1,000 lines.Large PRs are difficult to review and may be rejected due to their size. Please verify that this PR does not address multiple issues. |
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
📝 WalkthroughWalkthroughThis PR adds version-aware JIT source provisioning with automatic component refresh, TTL-based workdir cleanup, incremental per-file synchronization, enhanced metadata tracking with platform-specific locking, and JIT provisioning support for Changes
Sequence DiagramsequenceDiagram
participant User
participant CLI as CLI/Terraform
participant Provisioner as Source Provisioner
participant Metadata as Metadata Manager
participant FileSystem as FileSystem
participant Workdir as Workdir Service
User->>CLI: terraform plan (with source configured)
CLI->>Provisioner: AutoProvisionSource()
Provisioner->>Metadata: ReadMetadata(workdirPath)
Metadata-->>Provisioner: WorkdirMetadata or nil
alt Metadata exists
Provisioner->>Provisioner: checkMetadataChanges()
alt Version/URI changed
Provisioner->>Workdir: SyncDir(source, workdir)
Workdir->>FileSystem: SyncDir (incremental sync)
FileSystem-->>Workdir: changed=true
Provisioner->>Metadata: WriteMetadata() with new version
else No changes
Provisioner-->>CLI: ready (skip re-provisioning)
end
else No metadata (new workdir)
Provisioner->>Workdir: SyncDir(source, workdir)
Workdir->>FileSystem: SyncDir (copy all files)
Provisioner->>Metadata: WriteMetadata() with SourceVersion
end
CLI->>CLI: Use workdir for component execution
CLI-->>User: Terraform result
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly Related PRs
Suggested Reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📝 WalkthroughWalkthroughThis PR introduces version-aware JIT source provisioning with TTL-based workdir cleanup. It adds duration parsing utilities, workdir metadata management with platform-specific locking, incremental file synchronization, and a new Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/provisioner/workdir/workdir_test.go (1)
414-458: Make the metadata write failure deterministic across OSes.Relying on chmod’d read-only dirs isn’t reliable on Windows, so this test can pass unexpectedly. Consider forcing a cross-platform error by creating a file at the
.atmospath soMkdirAllfails consistently.🔧 Suggested change
- // Create .atmos directory as read-only to prevent writing metadata. - atmosDir := filepath.Join(workdirPath, AtmosDir) - err = os.MkdirAll(atmosDir, 0o555) - require.NoError(t, err) - t.Cleanup(func() { - // Restore permissions so cleanup can delete the directory. - _ = os.Chmod(atmosDir, 0o755) - }) + // Create a file at .atmos to force WriteMetadata to fail cross-platform. + atmosDir := filepath.Join(workdirPath, AtmosDir) + err = os.WriteFile(atmosDir, []byte("not-a-dir"), 0o444) + require.NoError(t, err)As per coding guidelines, keep tests cross-platform.
🤖 Fix all issues with AI agents
In `@pkg/duration/duration.go`:
- Around line 139-146: ParseDuration currently converts seconds -> time.Duration
by multiplying seconds * time.Second without checking for overflow; before
multiplying, validate the parsed seconds (from Parse) against time.Duration
bounds (int64 nanoseconds) so the result won't overflow. In ParseDuration,
compute the safe max/min seconds as int64(math.MaxInt64)/int64(time.Second) and
int64(math.MinInt64)/int64(time.Second) (or use float64 equivalents if Parse
returns float64), check that seconds lies within those bounds, and return a
clear error if not; then perform the multiplication and return the duration.
Reference functions: ParseDuration and Parse.
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 276-283: In checkMetadataChanges, remove the `sourceSpec.Version
!= ""` guard so the code directly compares sourceSpec.Version to
metadata.SourceVersion; this ensures removing a pinned version (emptying
Version) is detected as a change and triggers re-provisioning—update the
comparison in the function (checkMetadataChanges, the Version change branch) to
always compare the two values and return the change message when they differ.
- Around line 79-98: needsProvisioning currently treats missing metadata as a
signal to re-provision, which forces non-workdir component-path targets to
re-vendor on every run; change needsProvisioning to accept the isWorkdir flag
(e.g., needsProvisioning(targetDir, sourceSpec, isWorkdir)) and update its logic
so that missing metadata only triggers re-provision when isWorkdir is true,
keeping existing checks for version/URI changes otherwise; update the call in
provision_hook.go (the current call around determineSourceTargetDirectory and
the other occurrence at the region noted 239-258) to pass isWorkdir, and ensure
any tests or callers of needsProvisioning are adjusted accordingly.
In `@website/blog/2025-01-22-version-aware-jit-provisioning.mdx`:
- Around line 93-95: The blog filename and embedded sample timestamps are off by
one year; rename the file "2025-01-22-version-aware-jit-provisioning.mdx" to use
2026-01-22 and update the Created, Updated, and Last Accessed timestamp values
in the front-matter/block (the lines containing "Created", "Updated", and "Last
Accessed") to 2026-01-22 (and adjust times if needed) so the filename and the
sample dates are consistent with the 2026-01-22 release.
🧹 Nitpick comments (7)
pkg/duration/duration_test.go (1)
55-61: Assert the sentinel error for invalid inputs.These tests can be tighter by asserting
ErrInvalidDurationviaErrorIs, not just any error. As per coding guidelines, useerrors.Is()semantics in tests.✅ Suggested test tightening
import ( "testing" "time" + errUtils "github.com/cloudposse/atmos/errors" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ t.Run(tt.name, func(t *testing.T) { result, err := Parse(tt.input) if tt.wantErr { assert.Error(t, err, "expected error for input %q", tt.input) + assert.ErrorIs(t, err, errUtils.ErrInvalidDuration) } else { require.NoError(t, err, "unexpected error for input %q", tt.input) assert.Equal(t, tt.expected, result, "unexpected result for input %q", tt.input) } }) @@ t.Run(tt.name, func(t *testing.T) { result, err := ParseDuration(tt.input) if tt.wantErr { assert.Error(t, err, "expected error for input %q", tt.input) + assert.ErrorIs(t, err, errUtils.ErrInvalidDuration) } else { require.NoError(t, err, "unexpected error for input %q", tt.input) assert.Equal(t, tt.expected, result, "unexpected result for input %q", tt.input) } })Also applies to: 86-91
pkg/provisioner/workdir/fs.go (1)
138-167: Verify empty directory cleanup.After deleting files, empty directories in
dst(except AtmosDir) may remain. This could accumulate over time if source directories are removed.♻️ Optional: Clean up empty directories after file deletion
func deleteRemovedFiles(dst string, srcFiles map[string]bool) (bool, error) { anyDeleted := false + dirsToCheck := []string{} err := filepath.WalkDir(dst, func(path string, d fs.DirEntry, err error) error { // ... existing logic ... if d.IsDir() || srcFiles[relPath] { + if d.IsDir() && relPath != "." && relPath != AtmosDir { + dirsToCheck = append(dirsToCheck, path) + } return nil } anyDeleted = true return os.Remove(path) }) + + // Remove empty directories (in reverse order for deepest-first) + for i := len(dirsToCheck) - 1; i >= 0; i-- { + _ = os.Remove(dirsToCheck[i]) // Silently fails if not empty + } return anyDeleted, err }tests/cli_jit_source_workdir_test.go (1)
26-92: Test design is solid and effectively validates the source precedence fix.The marker file approach clearly distinguishes local vs remote content, and the assertions properly verify provisioning succeeded. The fixture exists and the test setup is clean.
The error from
cmd.Execute()is safe to ignore—if provisioning fails, therequire.NoError(t, err, ...)onos.Stat(workdirPath)catches it. That said, logging the error would help with debugging when things go wrong:♻️ Capture error from Execute for better debugging
// Execute - this may error due to terraform not being configured, // but the provisioning should have been attempted. - _ = cmd.Execute() + err := cmd.Execute() + if err != nil { + t.Logf("Execute returned error (expected for incomplete terraform setup): %v", err) + }pkg/provisioner/workdir/metadata_lock_windows.go (1)
31-47: Inconsistent error handling compared to Unix implementation.The Unix version wraps errors with
errUtils.Build(errUtils.ErrWorkdirMetadata)for both read and unmarshal failures, but this Windows version returns raw errors. Consider aligning for consistent error types across platforms.♻️ Suggested alignment with Unix error handling
+import ( + "encoding/json" + "os" + "time" + + errUtils "github.com/cloudposse/atmos/errors" +) func loadMetadataWithReadLockWindows(metadataFile string) (*WorkdirMetadata, error) { // On Windows, skip read locks entirely to avoid timeout issues. data, err := os.ReadFile(metadataFile) if err != nil { if os.IsNotExist(err) { return nil, nil } - return nil, err + return nil, errUtils.Build(errUtils.ErrWorkdirMetadata). + WithCause(err). + WithExplanation("Failed to read metadata file"). + WithContext("path", metadataFile). + Err() } var metadata WorkdirMetadata if err := json.Unmarshal(data, &metadata); err != nil { - return nil, err + return nil, errUtils.Build(errUtils.ErrWorkdirMetadata). + WithCause(err). + WithExplanation("Failed to unmarshal metadata"). + WithContext("path", metadataFile). + Err() } return &metadata, nil }pkg/provisioner/workdir/metadata.go (1)
24-41: Minor TOCTOU race in ReadMetadata.The
os.Statcheck at line 30 and subsequent read vialoadMetadataWithReadLockhas a small race window where the file could be deleted between checks. SinceloadMetadataWithReadLockalready handlesos.IsNotExist, consider simplifying:♻️ Remove redundant stat checks
func ReadMetadata(workdirPath string) (*WorkdirMetadata, error) { defer perf.Track(nil, "workdir.ReadMetadata")() metadataPath := MetadataPath(workdirPath) - // Check if metadata file exists. - if _, err := os.Stat(metadataPath); os.IsNotExist(err) { - // Try legacy location. - legacyPath := filepath.Join(workdirPath, WorkdirMetadataFile) - if _, err := os.Stat(legacyPath); os.IsNotExist(err) { - return nil, nil - } - metadataPath = legacyPath - } - - // Use read lock for concurrent safety. - return loadMetadataWithReadLock(metadataPath) + // Try new location first. + metadata, err := loadMetadataWithReadLock(metadataPath) + if err != nil { + return nil, err + } + if metadata != nil { + return metadata, nil + } + + // Try legacy location. + legacyPath := filepath.Join(workdirPath, WorkdirMetadataFile) + return loadMetadataWithReadLock(legacyPath) }pkg/provisioner/workdir/metadata_lock_unix.go (1)
57-61: Variable shadowing in defer.The
errin the defer shadows the outererrvariable. While it works here since the outererrisn't used after, it's a subtle gotcha.♻️ Rename to avoid shadowing
defer func() { - if err := lock.Unlock(); err != nil { - log.Trace("Failed to unlock metadata file", "error", err, "path", lockPath) + if unlockErr := lock.Unlock(); unlockErr != nil { + log.Trace("Failed to unlock metadata file", "error", unlockErr, "path", lockPath) } }()cmd/terraform/workdir/workdir_helpers.go (1)
151-160: Consider distinguishing metadata error vs not found.When
ReadMetadatareturns(nil, nil), the current code wraps withWithCause(nil). This works but loses the distinction between "no metadata file" and "error reading metadata". A minor clarity improvement:♻️ Handle nil metadata explicitly
metadata, err := provWorkdir.ReadMetadata(workdirPath) - if err != nil || metadata == nil { + if err != nil { + return nil, errUtils.Build(errUtils.ErrWorkdirMetadata). + WithCause(err). + WithExplanation(fmt.Sprintf("Failed to read workdir metadata for component '%s' in stack '%s'", component, stack)). + WithContext("component", component). + WithContext("stack", stack). + Err() + } + if metadata == nil { return nil, errUtils.Build(errUtils.ErrWorkdirMetadata). - WithCause(err). WithExplanation(fmt.Sprintf("Workdir not found for component '%s' in stack '%s'", component, stack)). WithHint("Run 'atmos terraform init' to create the workdir"). WithContext("component", component). WithContext("stack", stack). Err() }
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Fix all issues with AI agents
In `@cmd/terraform/workdir/workdir_helpers.go`:
- Around line 118-123: The loop silently drops all errors from
provWorkdir.ReadMetadata(workdirPath); change it to return a wrapped error when
err != nil (including context like the workdirPath) while keeping the current
behavior of skipping when metadata == nil; locate the call to
provWorkdir.ReadMetadata and replace the silent continue-on-error with returning
fmt.Errorf("read metadata for %s: %w", workdirPath, err) (or errors.Wrap) so
failures (permission/corruption) are surfaced but still continue to skip on
metadata == nil.
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 239-246: The block comment above the needsProvisioning function
has bullet lines without trailing periods which fails godot linting; update the
comment text for needsProvisioning so each bullet ends with a period (e.g.,
change lines like "- Directory doesn't exist or is empty" to "- Directory
doesn't exist or is empty.") and ensure the final summary line "Returns (false,
"") if the existing workdir is up-to-date." also ends with a period to satisfy
comment lint rules.
- Around line 276-283: checkMetadataChanges currently ignores a removed version
because it only flags changes when sourceSpec.Version != "" and differs; update
the logic in checkMetadataChanges to detect any difference between
metadata.SourceVersion and sourceSpec.Version (including when sourceSpec.Version
is empty) and return true with a clear message (e.g., "Source version removed (X
→ default/latest)" or "Source version changed (X → Y)") so that removing the
version triggers a refresh; modify the branch that references sourceSpec.Version
and metadata.SourceVersion accordingly and keep the existing URI comparison for
sourceSpec.Uri vs metadata.SourceURI.
- Around line 86-98: The current logic treats any missing metadata as "needs
provisioning" causing non-workdir components to be re-vendored every run; change
the branch after needsProvisioning(targetDir, sourceSpec) so that if needs==true
and isWorkdir==false you first check for existing installed artifacts/metadata
(e.g., presence of the targetDir or a provisioning marker) and skip provisioning
if they already exist, returning nil; keep the existing
workdir.UpdateLastAccessed behavior for isWorkdir paths. Update the code around
needsProvisioning, isWorkdir, targetDir and componentConfig (and use a helper
like hasProvisionMetadata or a simple os.Stat(targetDir)/marker-file check) to
preserve the original "skip if exists" contract for non-workdir components.
- Around line 263-272: Update isNonEmptyDir to treat a directory containing only
the metadata folder ".atmos" as empty: after os.ReadDir in isNonEmptyDir,
iterate the entries and ignore any entry with Name() == ".atmos" (and continue
for non-nil errors); return true only if there exists at least one entry whose
name is not ".atmos" and false otherwise. Keep the initial os.Stat checks and
error handling unchanged and ensure the function still returns false on ReadDir
error.
In `@pkg/provisioner/workdir/fs.go`:
- Around line 96-117: syncSourceToDest's WalkDir currently processes the .atmos
directory and can overwrite destination metadata; update the WalkDir callback
(the function that computes relPath and uses srcFiles, fileNeedsCopy, copyFile)
to skip any entry whose relPath is ".atmos" or starts with ".atmos/" (or the OS
path separator variant) by returning fs.SkipDir for directories and nil for
files so the entire .atmos tree is ignored during the source walk.
In `@website/blog/2025-01-22-version-aware-jit-provisioning.mdx`:
- Around line 93-95: Update the example timestamps in the frontmatter block:
change the "Created", "Updated", and "Last Accessed" entries from 2025 to 2026
(e.g., set them to 2026-01-20 and 2026-01-22 or align them with the intended
publish date 2026-01-22) so the sample dates in the "Created/Updated/Last
Accessed" lines reflect the 2026 release; edit the literal lines containing
"Created", "Updated", and "Last Accessed" accordingly.
🧹 Nitpick comments (5)
pkg/provisioner/workdir/metadata_lock_windows.go (2)
22-25: Fixed delay adds latency regardless of need.The 50ms sleep runs after every operation. For bulk workdir operations, this accumulates. Consider making this configurable or only applying the delay when the function actually performed writes.
♻️ Potential optimization
-func withMetadataFileLockWindows(metadataFile string, fn func() error) error { - // No file locking on Windows to avoid timeout issues. - // The metadata is non-critical functionality, so we can operate - // without strict locking on Windows. - - // Add a small delay after operations to let Windows release file handles. - defer func() { - time.Sleep(50 * time.Millisecond) - }() - - // Just execute the function without any locking. - return fn() +func withMetadataFileLockWindows(_ string, fn func() error) error { + // No file locking on Windows to avoid timeout issues. + // The metadata is non-critical functionality, so we can operate + // without strict locking on Windows. + return fn() }
41-44: Consider wrapping unmarshal errors with context.Per coding guidelines, errors should be wrapped with context using
fmt.Errorfor the error builder. The raw unmarshal error lacks information about which file failed.♻️ Add error context
var metadata WorkdirMetadata if err := json.Unmarshal(data, &metadata); err != nil { - return nil, err + return nil, fmt.Errorf("failed to unmarshal metadata from %s: %w", metadataFile, err) }pkg/duration/duration.go (1)
27-33: Consider adding 'w' for weeks suffix.The
keywordDurationsmap includes "weekly", butunitMultipliersdoesn't support'w'. Users might expect"1w"to work alongside"7d".♻️ Add week suffix support
var unitMultipliers = map[byte]int64{ 's': 1, 'm': secondsPerMinute, 'h': secondsPerHour, 'd': secondsPerDay, + 'w': secondsPerWeek, }tests/cli_jit_source_workdir_test.go (2)
61-63: Consider capturing execution errors for debugging.The return value is intentionally ignored since terraform may error without full configuration. However, logging or storing the error could help debug test failures.
♻️ Optional improvement
// Execute - this may error due to terraform not being configured, // but the provisioning should have been attempted. - _ = cmd.Execute() + err := cmd.Execute() + t.Logf("cmd.Execute() returned: %v", err) // Helps debug test failures
113-125: Test coverage could be expanded.This test verifies
terraform source describesucceeds but doesn't assert on the output. Consider adding assertions on the describe output to verify source configuration is correctly recognized, or documenting why this is sufficient as a smoke test.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2010 +/- ##
==========================================
+ Coverage 75.73% 76.02% +0.28%
==========================================
Files 793 797 +4
Lines 74060 74558 +498
==========================================
+ Hits 56089 56679 +590
+ Misses 14476 14334 -142
- Partials 3495 3545 +50
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@pkg/duration/duration.go`:
- Around line 108-126: The multiplication valInt * multiplier can overflow int64
and produce invalid negative durations; update the code in the parsing path
(where valInt and multiplier are used—e.g., the block that currently returns
valInt * multiplier) to perform a bounds check before multiplying: check whether
abs(valInt) > math.MaxInt64/abs(multiplier) (or use safe compare with
math.MaxInt64 and math.MinInt64) and if it would overflow return
errUtils.Build(errUtils.ErrInvalidDuration).WithExplanation("duration
overflow").WithContext("value", valInt).WithContext("multiplier",
multiplier).Err(); otherwise perform the multiplication and return the result.
Ensure you reference the existing unitMultipliers, valInt, multiplier, and
ErrInvalidDuration symbols when implementing the check.
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 110-115: The current block always calls writeRemoteMetadata when
isWorkdir, which forces SourceTypeRemote and clears ContentHash; update the
logic to check sourceSpec.Type and call the appropriate metadata writer (e.g.,
call writeLocalMetadata for local sources or keep writeRemoteMetadata for remote
sources) or alter writeRemoteMetadata to accept the SourceType and preserve
local fields when sourceSpec.Type indicates a local source; locate the isWorkdir
branch in provisionHook (provision_hook.go) and branch on sourceSpec.Type (or
pass sourceSpec.Type into writeRemoteMetadata/writeLocalMetadata) so local
workdir targets retain their local-specific metadata and content hash.
In `@pkg/provisioner/workdir/fs.go`:
- Around line 130-143: fileNeedsCopy currently only compares content hashes and
returns false when hashes match, which misses mode-only changes; update
fileNeedsCopy to stat both srcPath and dstPath (using os.Stat) and compare their
permission bits (e.g., FileMode.Perm()) in addition to the hasher.HashFile
result, treating any stat error on src or dst as needing a copy, and return true
if either hashes differ or permission bits differ so permission-only changes
(which copyFile preserves via os.Chmod) are synced.
🧹 Nitpick comments (1)
pkg/provisioner/source/provision_hook_test.go (1)
191-331: Add coverage for the non-workdir branch inneedsProvisioning.
TestNeedsProvisioningalways passesisWorkdir=true, so the newisWorkdir=falsepath isn’t exercised. Consider adding a small case to lock the “existing non-empty dir → no provisioning” behavior. As per coding guidelines, new features should include comprehensive unit tests.♻️ Suggested test addition
@@ func TestNeedsProvisioning(t *testing.T) { @@ for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { path := tt.setup(t) // Test with isWorkdir=true since these tests verify metadata behavior. result, reason := needsProvisioning(path, tt.sourceSpec, true) assert.Equal(t, tt.expected, result) assert.Equal(t, tt.expectedReason, reason) }) } + + t.Run("non-workdir with existing files", func(t *testing.T) { + tempDir := t.TempDir() + dirPath := filepath.Join(tempDir, "component") + err := os.MkdirAll(dirPath, 0o755) + require.NoError(t, err) + err = os.WriteFile(filepath.Join(dirPath, "main.tf"), []byte("# test"), 0o644) + require.NoError(t, err) + + result, reason := needsProvisioning(dirPath, sourceSpec, false) + assert.False(t, result) + assert.Empty(t, reason) + }) }
|
💥 This pull request now has conflicts. Could you fix it @osterman? 🙏 |
a994750 to
7b06069
Compare
|
Warning Release Documentation RequiredThis PR is labeled
|
|
Warning Release Documentation RequiredThis PR is labeled
|
|
Warning Release Documentation RequiredThis PR is labeled
|
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@cmd/terraform/workdir/workdir_list.go`:
- Around line 101-123: The code currently formats lastAccessed (computed from
workdirs[i].LastAccessed or CreatedAt) and will render "0001-01-01..." when both
times are zero; change the logic in the block that builds rows (around the
lastAccessed variable and the rows = append(...) that uses lastAccessed.Format)
to detect IsZero() after falling back and instead set a placeholder string
(e.g., "-") for the timestamp; introduce a lastAccessedStr used in the rows
slice instead of calling lastAccessed.Format directly, and apply the same change
to the other occurrence that formats timestamps (the second formatting location
referenced in the comment).
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 314-345: The isLocalSource function misclassifies Windows absolute
paths; update it to use filepath.IsAbs(uri) (from the path/filepath package) for
OS-aware absolute-path detection instead of only checking for "/" and "."
prefixes, while keeping the existing file:// check and remoteIndicators logic;
ensure you add the filepath import and run uri through filepath.IsAbs early in
isLocalSource (before remoteIndicators) so Windows paths like C:\... or \... are
correctly treated as local.
🧹 Nitpick comments (1)
pkg/provisioner/workdir/clean_test.go (1)
286-413: Consider table-driven consolidation for expired-clean scenarios.These cases share setup patterns and assertions; a table-driven structure would reduce duplication and ease expansion. As per coding guidelines, prefer table-driven tests when exercising multiple scenarios.
|
💥 This pull request now has conflicts. Could you fix it @osterman? 🙏 |
|
Warning Release Documentation RequiredThis PR is labeled
|
- Add tests for workdir clean command edge cases - Add tests for workdir show command scenarios - Add duration parsing tests for TTL validation - Add filesystem tests for workdir operations - Add metadata lock tests for Unix file locking Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
9d115b8 to
55b3e31
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@cmd/terraform/workdir/workdir_clean_cmd_test.go`:
- Around line 465-535: The tests currently call resetViperForTest to mutate
viper state; instead initialize a fresh command test context by calling
cmd.NewTestKit(t) at the start of each test (e.g., in
TestCleanCmd_RunE_ExpiredWithoutTTL, TestCleanCmd_RunE_ExpiredWithAll,
TestCleanCmd_RunE_ExpiredWithComponent,
TestCleanCmd_RunE_ActualCall_AllWithComponent,
TestCleanCmd_RunE_ActualCall_NoFlagsNoArgs,
TestCleanCmd_RunE_ActualCall_ComponentWithoutStack) to isolate RootCmd/viper
state; remove or stop using resetViperForTest and ensure each test starts with
cmd.NewTestKit(t) before setting any viper flags and invoking cleanCmd.RunE so
the command state is reset between tests.
In `@pkg/provisioner/workdir/metadata_lock_windows.go`:
- Around line 31-47: The function loadMetadataWithReadLockWindows should wrap
returned errors with errUtils.ErrWorkdirMetadata: when os.ReadFile(metadataFile)
returns a non-NotExist error, return nil and wrap the error (e.g.,
fmt.Errorf("%w: %v", errUtils.ErrWorkdirMetadata, err)); likewise wrap the
json.Unmarshal error before returning; add imports for fmt and errUtils. Ensure
the special-case os.IsNotExist still returns (nil, nil).
In `@pkg/provisioner/workdir/workdir.go`:
- Around line 219-233: The error hint is hardcoded to "components/terraform"
which can be incorrect; update the error construction around
extractComponentPath(...) and the subsequent existence check to use the resolved
componentPath variable (or derive the actual components base from
atmosConfig/Components.Terraform.BasePath if preferred) instead of the literal
string: change the WithHint call to include componentPath (or a message
referencing it) so the hint accurately points to the real path when
s.fs.Exists(componentPath) fails; keep the existing WithContext("path",
componentPath) and errUtils.Build usage.
♻️ Duplicate comments (2)
internal/exec/packer_output.go (1)
64-66: Wrap auto-provision error with sentinel (duplicate).This was flagged in a previous review. Line 65 should join with
errUtils.ErrProvisionSourceor similar sentinel before adding context to supporterrors.Ischecks.cmd/terraform/workdir/workdir_helpers.go (1)
118-123: Don’t silently drop metadata read errors in ListWorkdirs.
Right nowerr != nilis treated the same as “no metadata,” which can mask corruption or permission problems. Consider returning a wrapped error onerr != nilbut still skipping whenmetadata == nil.🛠️ Suggested tweak
- metadata, err := provWorkdir.ReadMetadata(workdirPath) - if err != nil || metadata == nil { - // Skip directories with invalid or missing metadata. - // This allows the list operation to succeed even if some workdirs are corrupt. - continue - } + metadata, err := provWorkdir.ReadMetadata(workdirPath) + if err != nil { + return nil, errUtils.Build(errUtils.ErrWorkdirMetadata). + WithCause(err). + WithExplanation("Failed to read workdir metadata"). + WithContext("path", workdirPath). + Err() + } + if metadata == nil { + // Skip directories without metadata. + continue + }As per coding guidelines, errors should be wrapped with context.
🧹 Nitpick comments (9)
examples/source-provisioning/README.md (1)
27-31: Clarify the version‑pinning claim.
The bullet says remote URIs are version‑pinned, but the example doesn’t show a version. Consider softening the wording or adding a version in the example.Proposed tweak
-2. **Remote URIs** - GitHub URLs with version pinning +2. **Remote URIs** - GitHub URLs (optionally version‑pinned)pkg/provisioner/workdir/fs_test.go (1)
14-71: Consider table-driven cases forfileNeedsCopy.
These scenarios share setup and assertions; a table-driven test would trim repetition and scale better. As per coding guidelines, prefer table-driven tests for multi-scenario coverage.pkg/provisioner/workdir/clean_test.go (1)
286-413: Consider table-driven tests for the TTL cleanup scenarios.
These cases are parallel and would be easier to extend as a single table. As per coding guidelines, prefer table-driven tests for multi-scenario coverage.cmd/terraform/workdir/workdir_show_test.go (1)
470-484: Consider asserting SourceType defaults correctly.This test verifies zero LastAccessed isn't displayed, but doesn't set SourceType. If printShowHuman has conditional logic based on SourceType, you may want to explicitly test local vs unset SourceType behavior separately. Not blocking, just a thought for completeness.
pkg/provisioner/workdir/metadata_lock_unix_test.go (1)
48-75: Potential test flakiness with exclusive locks.The assertion at line 74 requires at least one goroutine to succeed. If the locking implementation uses non-blocking
LOCK_NBand the timing is unfortunate, all 5 goroutines could theoretically fail. Consider either:
- Removing the assertion entirely (the test still verifies no crashes/deadlocks)
- Running goroutines sequentially with a smaller subset to guarantee success
Not blocking since the probability is low, but worth noting.
cmd/terraform/workdir/workdir_show.go (1)
75-86: Consider usingSourceTypeRemoteconstant instead of magic string.Line 90 compares
sourceType == "remote"directly. Theworkdirpackage definesSourceTypeRemoteconstant which would be more maintainable.♻️ Suggested improvement
+import provWorkdir "github.com/cloudposse/atmos/pkg/provisioner/workdir" + // In printShowHuman: -if sourceType == "remote" { +if sourceType == string(provWorkdir.SourceTypeRemote) {cmd/terraform/workdir/workdir_clean_cmd_test.go (1)
475-535: Consider a table‑driven structure for RunE validation cases.
The validation tests are very similar; a small table of flag/args permutations would cut repetition and make coverage easier to extend.As per coding guidelines, prefer table-driven tests for multiple scenarios.
tests/cli_jit_source_workdir_test.go (2)
92-123: Placeholder test provides limited coverage.Lines 121-122 explicitly call this a "placeholder." The test only confirms
source describedoesn't error - it doesn't verify source precedence behavior.Consider either:
- Adding meaningful assertions (e.g., workdir state, output verification)
- Removing this test since the first test already covers the behavior
294-334: Inconsistent assertions compared to other tests.
TestJITSource_PackerOutputandTestJITSource_HelmfileGenerateVarfilelack the workdir existence and content verification thatTestJITSource_GenerateVarfileandTestJITSource_GenerateBackendhave.For consistency, add workdir assertions:
♻️ Suggested additions
For
TestJITSource_PackerOutput(after line 315):+ // Verify workdir was provisioned by JIT source. + workdirPath := filepath.Join(".workdir", "packer", "dev-ami-workdir") + info, statErr := os.Stat(workdirPath) + require.NoError(t, statErr, "Workdir should exist at %s (JIT provisioning should have run)", workdirPath) + require.True(t, info.IsDir(), "Workdir path should be a directory")For
TestJITSource_HelmfileGenerateVarfile(after line 333):+ // Verify workdir was provisioned. + workdirPath := filepath.Join(".workdir", "helmfile", "dev-nginx-workdir") + info, err := os.Stat(workdirPath) + require.NoError(t, err, "Workdir should exist at %s", workdirPath) + require.True(t, info.IsDir(), "Workdir path should be a directory")
- Skip permission-based tests on Windows (Unix permissions not supported) - TestFileNeedsCopy_DifferentPermissions - TestCopyFile_PreservesPermissions - TestServiceProvision_WriteMetadataFails (read-only dirs work differently) - Use actual componentPath in error hint instead of hardcoded path Addresses CodeRabbit review feedback. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Wrap auto-provision error with ErrSourceProvision sentinel (packer_output.go) - Add error wrapping with ErrWorkdirMetadata in Windows metadata loader - Document circular import limitation preventing cmd.NewTestKit usage Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@pkg/provisioner/workdir/workdir_test.go`:
- Around line 416-418: The test currently checks os.Getenv("GOOS") which is
incorrect because GOOS is a compile-time value; update the condition to use
runtime.GOOS (e.g., if runtime.GOOS == "windows" || filepath.Separator == '\\')
and add the runtime import to the test file; ensure you remove the os.Getenv
usage and import "runtime" so the read-only directory skip logic in
workdir_test.go uses the correct platform value.
♻️ Duplicate comments (1)
cmd/terraform/workdir/workdir_clean_cmd_test.go (1)
465-476: Usecmd.NewTestKit(t)to isolate RootCmd/Viper state.The helper bypasses the required testkit reset and can leak global state across cmd tests. Consider restructuring tests (e.g., external test package or shared helper that avoids the circular import) so
cmd.NewTestKit(t)runs before touching Viper. As per coding guidelines, ...
🧹 Nitpick comments (4)
pkg/provisioner/workdir/fs_test.go (2)
3-10: Useruntime.GOOSfor OS checks.
os.Getenv("GOOS")is non-idiomatic and may be unset;runtime.GOOSis the canonical, reliable check. Consider updating both Windows skips accordingly.🔧 Suggested change
import ( "os" "path/filepath" + "runtime" "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ - if os.Getenv("GOOS") == "windows" || filepath.Separator == '\\' { + if runtime.GOOS == "windows" { t.Skip("Skipping permission test on Windows - Unix file permissions not supported") } @@ - if os.Getenv("GOOS") == "windows" || filepath.Separator == '\\' { + if runtime.GOOS == "windows" { t.Skip("Skipping permission test on Windows - Unix file permissions not supported") }Also applies to: 49-52, 199-202
14-74: Consider table-driven tests to reduce repetition.These scenarios are great, but the repeated setup makes the file longer than needed. A table-driven approach for
fileNeedsCopyandcopyFilewould align with test guidelines and improve maintainability. As per coding guidelines, ...Also applies to: 155-220
pkg/provisioner/workdir/workdir.go (1)
184-184: Silent discard of ReadMetadata error may hide real failures.Ignoring the error is fine when metadata doesn't exist yet. However, this also silences permission errors or corrupt JSON. A warning log would help debugging without breaking the flow.
- existingMetadata, _ := ReadMetadata(workdirPath) + existingMetadata, err := ReadMetadata(workdirPath) + if err != nil { + ui.Warning(fmt.Sprintf("Failed to read existing metadata (will recreate): %s", err)) + }cmd/terraform/workdir/workdir_clean_cmd_test.go (1)
478-538: Consolidate RunE validation tests into a table-driven test.These cases are repetitive and easier to maintain as a single table-driven test. As per coding guidelines, ...
♻️ Suggested refactor
-func TestCleanCmd_RunE_ExpiredWithoutTTL(t *testing.T) { - resetViperForTest(t) - v := viper.GetViper() - v.Set("expired", true) - v.Set("ttl", "") // No TTL provided. - - err := cleanCmd.RunE(cleanCmd, []string{}) - assert.Error(t, err) - assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) -} - -func TestCleanCmd_RunE_ExpiredWithAll(t *testing.T) { - resetViperForTest(t) - v := viper.GetViper() - v.Set("expired", true) - v.Set("ttl", "7d") - v.Set("all", true) // Cannot use --all with --expired. - - err := cleanCmd.RunE(cleanCmd, []string{}) - assert.Error(t, err) - assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) -} - -func TestCleanCmd_RunE_ExpiredWithComponent(t *testing.T) { - resetViperForTest(t) - v := viper.GetViper() - v.Set("expired", true) - v.Set("ttl", "7d") - - err := cleanCmd.RunE(cleanCmd, []string{"vpc"}) // Component arg with --expired. - assert.Error(t, err) - assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) -} - -func TestCleanCmd_RunE_ActualCall_AllWithComponent(t *testing.T) { - resetViperForTest(t) - v := viper.GetViper() - v.Set("all", true) - - err := cleanCmd.RunE(cleanCmd, []string{"vpc"}) // --all with component arg. - assert.Error(t, err) - assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) -} - -func TestCleanCmd_RunE_ActualCall_NoFlagsNoArgs(t *testing.T) { - resetViperForTest(t) - - err := cleanCmd.RunE(cleanCmd, []string{}) // No flags, no args. - assert.Error(t, err) - assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) -} - -func TestCleanCmd_RunE_ActualCall_ComponentWithoutStack(t *testing.T) { - resetViperForTest(t) - v := viper.GetViper() - v.Set("stack", "") // No stack provided. - - err := cleanCmd.RunE(cleanCmd, []string{"vpc"}) // Component without --stack. - assert.Error(t, err) - assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) -} +func TestCleanCmd_RunE_Validation(t *testing.T) { + cases := []struct { + name string + setup func(v *viper.Viper) + args []string + }{ + { + name: "expired without ttl", + setup: func(v *viper.Viper) { + v.Set("expired", true) + v.Set("ttl", "") + }, + args: []string{}, + }, + { + name: "expired with all", + setup: func(v *viper.Viper) { + v.Set("expired", true) + v.Set("ttl", "7d") + v.Set("all", true) + }, + args: []string{}, + }, + { + name: "expired with component", + setup: func(v *viper.Viper) { + v.Set("expired", true) + v.Set("ttl", "7d") + }, + args: []string{"vpc"}, + }, + { + name: "all with component", + setup: func(v *viper.Viper) { + v.Set("all", true) + }, + args: []string{"vpc"}, + }, + { + name: "no flags no args", + setup: func(v *viper.Viper) {}, + args: []string{}, + }, + { + name: "component without stack", + setup: func(v *viper.Viper) { + v.Set("stack", "") + }, + args: []string{"vpc"}, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + resetViperForTest(t) + v := viper.GetViper() + tc.setup(v) + + err := cleanCmd.RunE(cleanCmd, tc.args) + assert.Error(t, err) + assert.ErrorIs(t, err, errUtils.ErrWorkdirClean) + }) + } +}
GOOS is a compile-time constant, not a runtime environment variable.
os.Getenv("GOOS") returns empty unless explicitly set.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
pkg/provisioner/workdir/workdir_test.go (2)
502-536: Usefilepath.FromSlashfor test paths to maintain cross-platform compatibility.The test should align with other tests in this file, which consistently use
filepath.FromSlash()for path values in componentConfig and mock expectations. Extract the hardcoded path as a variable to avoid repetition.♻️ Proposed update
mockFS := NewMockFileSystem(ctrl) mockHasher := NewMockHasher(ctrl) + componentPath := filepath.FromSlash("/custom/path/to/component") + mockFS.EXPECT().MkdirAll(workdirPath, gomock.Any()).Return(nil) - mockFS.EXPECT().Exists("/custom/path/to/component").Return(true) - mockFS.EXPECT().SyncDir("/custom/path/to/component", workdirPath, mockHasher).Return(true, nil) + mockFS.EXPECT().Exists(componentPath).Return(true) + mockFS.EXPECT().SyncDir(componentPath, workdirPath, mockHasher).Return(true, nil) mockHasher.EXPECT().HashDir(workdirPath).Return("abc123", nil) // Note: WriteMetadata uses real filesystem with atomic write, not mocked FileSystem. componentConfig := map[string]any{ "component": "vpc", "atmos_stack": "dev", - "component_path": "/custom/path/to/component", + "component_path": componentPath, "provision": map[string]any{ "workdir": map[string]any{ "enabled": true,
416-463: Guard read-only metadata test against root bypass.Tests running as root may bypass write-permission restrictions, causing this test to fail unexpectedly. Skip the test when EUID is 0, matching the pattern already established in
pkg/config/cache_test.goandcmd/working_directory_test.go.🔧 Suggested hardening
func TestServiceProvision_WriteMetadataFails(t *testing.T) { if runtime.GOOS == "windows" { t.Skip("Skipping read-only directory test on Windows - directory permissions work differently") } + if os.Geteuid() == 0 { + t.Skip("Skipping read-only directory test when running as root - permissions may be bypassed") + } // Create a temp dir with a read-only .atmos subdirectory to force WriteMetadata to fail.
🧹 Nitpick comments (1)
pkg/provisioner/workdir/workdir_test.go (1)
1194-1278: Consider table-driven BuildLocalMetadata tests.These three cases are structurally similar and could be consolidated to reduce duplication and make future additions easier.
As per coding guidelines, ...
The kubernetes.io domain frequently has connection failures/timeouts in CI, causing spurious link check failures. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
668c09b to
2a629e7
Compare
Instead of silently passing when main.tf doesn't exist, the tests now: - Explicitly fail if main.tf exists (unexpected) - Read and check for LOCAL_VERSION_MARKER to provide better diagnostics - Use t.Fatalf to fail fast with clear error messages Addresses CodeRabbit feedback about test assertion clarity. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Resolve conflicts in terraform_generate_varfile.go and terraform_generate_backend.go by adopting main's cleaner ensureTerraformComponentExists helper function approach. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive tests for: - pkg/provisioner/workdir/metadata.go: - MetadataPath function - WriteMetadata with all fields populated - ReadMetadata new location priority over legacy - UpdateLastAccessed preserves all fields - pkg/provisioner/workdir/clean.go: - checkWorkdirExpiry for expired/non-expired workdirs - getModTimeFromEntry - findExpiredWorkdirs with mixed workdirs - CleanExpiredWorkdirs with empty base path - Clean with Expired option precedence - formatDuration edge cases - pkg/provisioner/workdir/fs.go: - DefaultPathFilter.Match with patterns - SyncDir with nested directories - SyncDir updating changed files - pkg/provisioner/source/provision_hook.go: - checkMetadataChanges with version scenarios - isNonEmptyDir edge cases - needsProvisioning for non-workdir targets - writeWorkdirMetadata source type detection - writeWorkdirMetadata preserving ContentHash Coverage improvements: - workdir package: ~79% → 92.5% - source package: ~76% → 83.6% Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
These changes were released in v1.206.0-rc.1. |
what
--expired,--ttl, and--dry-runflags.workdir-metadata.jsonto.atmos/metadata.jsonfor better organizationsource_uri,source_version, andlast_accessedtimestamps in metadataAdditional Fixes
generate varfileandgenerate backenddon't work with JIT vendoring #2019 - Fix JIT provisioning gaps in generate commandsterraform generate varfile/terraform generate backend- now support JIT-sourced componentshelmfile generate varfile- now supports JIT-sourced componentspacker output- now supports JIT-sourced componentswhy
generate varfileandgenerate backenddon't work with JIT vendoring #2019)references
pkg/durationpackage extracted for reusable duration parsingworkdir listandworkdir showcommands display version and access informationwebsite/blog/2025-01-22-version-aware-jit-provisioning.mdxSummary by CodeRabbit
--ttlflag (e.g.,--ttl=7d).