feat: Add source cache TTL for JIT-vendored components#2138
Conversation
Implement a `ttl` (time-to-live) field on component source configuration to control how long cached JIT-vendored sources are reused before re-pulling from the remote.
## Changes
- Add `TTL` field to `VendorComponentSource` schema
- Add `TerraformSourceSettings` struct with global TTL default to `Terraform` config
- Parse `ttl` from source maps in extract.go
- Implement TTL expiration check in `needsProvisioning()` function
- Support per-component override and global default from atmos.yaml
- Handle zero TTL explicitly (always expires for active development)
- Add comprehensive unit tests for TTL behavior
## Configuration
```yaml
# Per-component (stack manifest)
components:
terraform:
my-module:
source:
uri: git::https://github.com/org/repo.git
version: main
ttl: "0s" # Always re-pull
# Global default (atmos.yaml)
components:
terraform:
source:
ttl: "1h" # Re-pull if older than 1 hour
```
## Documentation
- Updated terraform, helmfile, and packer source command docs
- Added design pattern section on cache TTL for floating refs
- Created PRD explaining problem, solution, and architecture
- Created blog post with user-facing guidance
- Updated roadmap with shipped milestone
Fixes #2135
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
MDX requires multi-line dd content to use block-level tags (dd on its own line) rather than inline tags to handle blank lines correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughAdds a source cache TTL mechanism for JIT-vendored components: schema fields for per-component and global TTL, parsing of Changes
Sequence Diagram(s)sequenceDiagram
participant Config as Config/Schema\n(atmos.yaml)
participant Extract as Extract Layer
participant Cache as Metadata Cache
participant Provision as Provisioning Logic
participant Source as Remote Source
Config->>Extract: provide source config (uri, version, ttl)
Extract->>Cache: persist source spec + TTL
Provision->>Cache: read metadata (UpdatedAt, TTL)
Provision->>Provision: evaluate isSourceCacheExpired(TTL, UpdatedAt)
alt TTL expired or TTL == "0s"
Provision->>Source: re-pull remote source
Source-->>Provision: fresh content
Provision->>Cache: update metadata.UpdatedAt
else cache still valid
Provision->>Provision: reuse cached workdir
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pkg/provisioner/source/extract.go`:
- Around line 92-95: The ttl field must be validated during extraction: in the
block that reads sourceMap["ttl"] and assigns spec.TTL, attempt to
parse/validate the ttl string (e.g., with time.ParseDuration or the project’s
TTL parser) and on parse/validation failure return ErrSourceInvalidSpec with
context (include the invalid value and a brief hint about expected format)
instead of silently assigning; update the extraction path that sets spec.TTL
(where sourceMap, spec.TTL are referenced) to perform this validation and return
the error when invalid.
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 341-345: The TTL parsing error is currently swallowed after
calling duration.ParseDuration(ttl) (variable ttlDuration), which hides
misconfiguration; instead surface the failure by logging or returning an
explicit error message: when duration.ParseDuration(ttl) returns err, call the
existing logger (or return a non-empty error string) with context like "invalid
TTL format" and the err details and avoid proceeding as if no TTL was provided;
update the clause that now returns (false, "") to return (false, "<clear error
message including err>") or propagate the error so callers of this function can
fail-fast and notify users (also update any tests that assumed silent skip).
- Around line 81-84: The TTL defaulting currently always uses
atmosConfig.Components.Terraform.Source.TTL; update AutoProvisionSource to
select the global TTL based on the componentType parameter instead of hardcoding
Terraform. Locate the TTL assignment that checks sourceSpec.TTL and replace the
hardcoded reference with a component-type-aware lookup on atmosConfig.Components
(e.g., switch or map keyed by componentType to retrieve the appropriate
Components.<Type>.Source.TTL), ensuring you still only set sourceSpec.TTL when
it is empty; reference symbols: AutoProvisionSource, sourceSpec.TTL,
componentType, and atmosConfig.Components.Source.TTL.
In `@pkg/schema/schema.go`:
- Around line 436-442: The PR adds TerraformSourceSettings (type
TerraformSourceSettings with TTL) but lacks equivalent typed global source
configs and merging logic for Helmfile and Packer, so add corresponding types
(e.g., HelmfileSourceSettings and PackerSourceSettings with the same TTL field
and tags), expose them in the top-level schema struct where
TerraformSourceSettings is declared, and update the provisioning/merge code that
applies global defaults (the routine that reads TerraformSourceSettings) to use
the appropriate global default per component type (helmfile ->
HelmfileSourceSettings, packer -> PackerSourceSettings) when resolving
component.source.ttl so components.helmfile.source.ttl and
components.packer.source.ttl are supported consistently with
Terraform.Source.TTL.
In `@website/src/data/roadmap.js`:
- Line 333: The shipped roadmap entry for the milestone with label "Source cache
TTL for JIT-vendored components" is missing the required pr field; update the
object (the milestone literal that currently has keys label, status, quarter,
changelog, prd, docs, description, benefits) to include a pr field set to the PR
number (e.g., pr: '###' or the actual PR string) so the milestone includes
status, changelog and pr as required by guidelines.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
Run ID: 18f715bb-d5c0-468a-9b4a-26657913dd44
📒 Files selected for processing (13)
docs/prd/source-cache-ttl.mdpkg/provisioner/source/extract.gopkg/provisioner/source/extract_test.gopkg/provisioner/source/provision_hook.gopkg/provisioner/source/provision_hook_test.gopkg/schema/schema.gopkg/schema/vendor_component.gowebsite/blog/2026-03-03-source-cache-ttl.mdxwebsite/docs/cli/commands/helmfile/source/source.mdxwebsite/docs/cli/commands/packer/source/source.mdxwebsite/docs/cli/commands/terraform/source/source.mdxwebsite/docs/design-patterns/version-management/source-based-versioning.mdxwebsite/src/data/roadmap.js
- Validate TTL at extraction time; invalid values now return ErrSourceInvalidSpec with context and hint - Rename TerraformSourceSettings to SourceSettings and add Source field to Helmfile and Packer schema structs - Use component-type-aware global TTL defaults instead of hardcoding Terraform - Force re-provision on invalid TTL format instead of silently skipping - Add pr: 2138 to roadmap milestone entry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/provisioner/source/provision_hook.go (1)
348-351: Include parse-error context in the invalid TTL message.Line 350 surfaces the invalid TTL, but adding the parse error (and a short format hint) would make troubleshooting faster.
As per coding guidelines, "Provide clear error messages to users, include troubleshooting hints when appropriate, and log detailed errors for debugging."Proposed tweak
- return true, fmt.Sprintf("Invalid source TTL %q; forcing re-provision to avoid stale cache", ttl) + return true, fmt.Sprintf( + "Invalid source TTL %q (%v); forcing re-provision to avoid stale cache. Use formats like \"1h\", \"30m\", or \"0s\"", + ttl, + err, + )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/provisioner/source/provision_hook.go` around lines 348 - 351, The invalid TTL branch should include the parse error and a short format hint in the returned message: when duration.ParseDuration(ttl) fails (see ttlDuration and duration.ParseDuration(ttl)), return a message that embeds err.Error() and a brief example/hint such as "expected formats like '30s' or '5m'" so callers can quickly diagnose and fix the TTL value.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 348-351: The invalid TTL branch should include the parse error and
a short format hint in the returned message: when duration.ParseDuration(ttl)
fails (see ttlDuration and duration.ParseDuration(ttl)), return a message that
embeds err.Error() and a brief example/hint such as "expected formats like '30s'
or '5m'" so callers can quickly diagnose and fix the TTL value.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 29ec299c-3ec2-4224-846c-2d5f2b23f14f
📒 Files selected for processing (6)
docs/prd/source-cache-ttl.mdpkg/provisioner/source/extract.gopkg/provisioner/source/extract_test.gopkg/provisioner/source/provision_hook.gopkg/schema/schema.gowebsite/src/data/roadmap.js
🚧 Files skipped from review as they are similar to previous changes (3)
- pkg/provisioner/source/extract_test.go
- pkg/provisioner/source/extract.go
- website/src/data/roadmap.js
… handling Add Go doc comments to 5 new TTL test functions to improve docstring coverage. Change Source fields from value to pointer types on Terraform, Helmfile, and Packer structs so unset source config is properly nil rather than a zero-value struct, with corresponding nil checks in the provisioning logic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (3)
pkg/provisioner/source/provision_hook.go (1)
346-369: Normalize TTL input before zero-value checks and parsing.Right now zero TTL detection is exact-match only. Trimming/lowercasing first avoids accidental “invalid TTL” behavior for harmless formatting differences.
🛠️ Suggested adjustment
func isSourceCacheExpired(ttl string, updatedAt time.Time) (bool, string) { + normalizedTTL := strings.ToLower(strings.TrimSpace(ttl)) + // Handle zero TTL explicitly (always expired). - if isZeroTTL(ttl) { - return true, fmt.Sprintf("Source cache expired (TTL: %s, always re-pull)", ttl) + if isZeroTTL(normalizedTTL) { + return true, fmt.Sprintf("Source cache expired (TTL: %s, always re-pull)", normalizedTTL) } - ttlDuration, err := duration.ParseDuration(ttl) + ttlDuration, err := duration.ParseDuration(normalizedTTL) if err != nil { - return true, fmt.Sprintf("Invalid source TTL %q; forcing re-provision to avoid stale cache", ttl) + return true, fmt.Sprintf("Invalid source TTL %q; forcing re-provision to avoid stale cache", ttl) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/provisioner/source/provision_hook.go` around lines 346 - 369, The TTL string should be normalized (trimmed and lowercased) before being checked or parsed to avoid accidental "invalid TTL" or missed zero checks; update isSourceCacheExpired to call a normalizedTTL := strings.ToLower(strings.TrimSpace(ttl)) and use normalizedTTL for the isZeroTTL check and for duration.ParseDuration, and either adjust isZeroTTL to expect a normalized value or perform the same normalization inside isZeroTTL so values like " 0S ", "0M", or " 0 " are correctly treated as zero TTL; keep all existing return messages but continue to display the original ttl where desirable or switch to normalizedTTL consistently.pkg/provisioner/source/extract_test.go (1)
673-738: Consolidate TTL extraction cases into one table-driven test.These four tests follow the same setup/assertion pattern; one table-driven test will reduce duplication and make future TTL cases easier to add.
♻️ Refactor sketch
-func TestExtractSource_WithTTL(t *testing.T) { ... } -func TestExtractSource_WithoutTTL(t *testing.T) { ... } -func TestExtractSource_WithInvalidTTL(t *testing.T) { ... } -func TestExtractSource_WithZeroTTL(t *testing.T) { ... } +func TestExtractSource_TTL(t *testing.T) { + tests := []struct { + name string + source map[string]any + expectErr error + expectedTTL string + }{ + // valid / missing / invalid / zero TTL cases + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + componentConfig := map[string]any{"source": tt.source} + result, err := ExtractSource(componentConfig) + // assertions... + }) + } +}As per coding guidelines, "Use table-driven tests for testing multiple scenarios in Go."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/provisioner/source/extract_test.go` around lines 673 - 738, The four TTL tests for ExtractSource are duplicated and should be consolidated into a single table-driven test: create a slice of test cases (with fields like name, ttlValue (nil/""/string), expectedTTL, expectError bool, expectedErr) and iterate with t.Run; for each case build the componentConfig with "source":{"uri":"github.com/example/repo//module","version":..., "ttl": ttlValue} (omit ttl when nil), call ExtractSource, and assert Uri and Version always, then check TTL equals expectedTTL when expectError is false and assert errors (require.Error / assert.ErrorIs with errUtils.ErrSourceInvalidSpec) and nil result when expectError is true; replace the four individual TestExtractSource_* functions with this single table-driven test to cover valid, missing, invalid, and zero TTL scenarios.pkg/provisioner/source/provision_hook_test.go (1)
335-402: Add an invalid-TTL case to lock in the defensive safety-net behavior.This suite is strong; one more case (
ttl: "invalid") asserting reprovision + reason containingInvalid source TTLwould prevent regressions in the fallback path.As per coding guidelines, "Every new feature must include comprehensive unit tests targeting >80% code coverage for all packages."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/provisioner/source/provision_hook_test.go` around lines 335 - 402, Add a test case to TestNeedsProvisioning_TTL that uses ttl: "invalid" to verify the defensive fallback in needsProvisioning; update the tests slice in TestNeedsProvisioning_TTL to include an entry with name "invalid TTL forces reprovision", ttl: "invalid", any updatedAt (e.g., time.Now()), expected: true, and expectExpired: true, then after calling needsProvisioning(dirPath, sourceSpec, true) assert the result is true and assert.Contains(t, reason, "Invalid source TTL") to lock in the error-path behavior for schema.VendorComponentSource.TTL handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/provisioner/source/extract_test.go`:
- Around line 673-738: The four TTL tests for ExtractSource are duplicated and
should be consolidated into a single table-driven test: create a slice of test
cases (with fields like name, ttlValue (nil/""/string), expectedTTL, expectError
bool, expectedErr) and iterate with t.Run; for each case build the
componentConfig with
"source":{"uri":"github.com/example/repo//module","version":..., "ttl":
ttlValue} (omit ttl when nil), call ExtractSource, and assert Uri and Version
always, then check TTL equals expectedTTL when expectError is false and assert
errors (require.Error / assert.ErrorIs with errUtils.ErrSourceInvalidSpec) and
nil result when expectError is true; replace the four individual
TestExtractSource_* functions with this single table-driven test to cover valid,
missing, invalid, and zero TTL scenarios.
In `@pkg/provisioner/source/provision_hook_test.go`:
- Around line 335-402: Add a test case to TestNeedsProvisioning_TTL that uses
ttl: "invalid" to verify the defensive fallback in needsProvisioning; update the
tests slice in TestNeedsProvisioning_TTL to include an entry with name "invalid
TTL forces reprovision", ttl: "invalid", any updatedAt (e.g., time.Now()),
expected: true, and expectExpired: true, then after calling
needsProvisioning(dirPath, sourceSpec, true) assert the result is true and
assert.Contains(t, reason, "Invalid source TTL") to lock in the error-path
behavior for schema.VendorComponentSource.TTL handling.
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 346-369: The TTL string should be normalized (trimmed and
lowercased) before being checked or parsed to avoid accidental "invalid TTL" or
missed zero checks; update isSourceCacheExpired to call a normalizedTTL :=
strings.ToLower(strings.TrimSpace(ttl)) and use normalizedTTL for the isZeroTTL
check and for duration.ParseDuration, and either adjust isZeroTTL to expect a
normalized value or perform the same normalization inside isZeroTTL so values
like " 0S ", "0M", or " 0 " are correctly treated as zero TTL; keep all existing
return messages but continue to display the original ttl where desirable or
switch to normalizedTTL consistently.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: fe76f4c8-c4e0-43c4-9ea7-fe42cecd52eb
📒 Files selected for processing (4)
pkg/provisioner/source/extract_test.gopkg/provisioner/source/provision_hook.gopkg/provisioner/source/provision_hook_test.gopkg/schema/schema.go
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2138 +/- ##
==========================================
+ Coverage 76.59% 76.62% +0.02%
==========================================
Files 833 833
Lines 79592 79635 +43
==========================================
+ Hits 60963 61017 +54
+ Misses 14825 14819 -6
+ Partials 3804 3799 -5
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
|
💥 This pull request now has conflicts. Could you fix it @osterman? 🙏 |
Resolve roadmap.js conflict by keeping both milestones (source-cache-ttl and per-target version overrides). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract applyGlobalTTLDefault into a testable function and add comprehensive tests for: global TTL defaults (terraform/helmfile/packer), per-component override precedence, isSourceCacheExpired edge cases (invalid TTL, all zero forms, 7d format), isZeroTTL variants, and TTL format validation in ExtractSource. Coverage: 83.4% → 85.6% for source package. All TTL functions at 100%. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (2)
pkg/provisioner/source/provision_hook.go (1)
355-358: Include the parse error details in the invalid TTL reason.Current message is good, but dropping
errhides the exact parsing failure and slows troubleshooting.Suggested tweak
ttlDuration, err := duration.ParseDuration(ttl) if err != nil { - return true, fmt.Sprintf("Invalid source TTL %q; forcing re-provision to avoid stale cache", ttl) + return true, fmt.Sprintf( + "Invalid source TTL %q (%v); forcing re-provision to avoid stale cache", + ttl, + err, + ) }As per coding guidelines, provide clear error messages to users, include troubleshooting hints when appropriate, and log detailed errors for debugging.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/provisioner/source/provision_hook.go` around lines 355 - 358, The invalid TTL error message hides the parsing error; update the block that calls duration.ParseDuration (assigning to ttlDuration, err) so the returned reason includes the parse error details (err) alongside the TTL value — e.g., include err.Error() or %v in the fmt.Sprintf used in the return statement where ttlDuration and err are in scope — so the returned string shows both the invalid TTL and the underlying parse error for debugging.pkg/provisioner/source/provision_hook_test.go (1)
450-454: Tighten non-expired-path assertions with empty reason checks.Right now non-expired cases only check
expired/result == false; assertingreasonis empty would catch noisy or stale reason regressions.Optional hardening
result, reason := needsProvisioning(dirPath, sourceSpec, true) assert.Equal(t, tt.expected, result) if tt.expectExpired { assert.Contains(t, reason, "Source cache expired") + } else { + assert.Empty(t, reason) }expired, reason := isSourceCacheExpired(tt.ttl, tt.updatedAt) assert.Equal(t, tt.expected, expired) if tt.expectedReason != "" { assert.Contains(t, reason, tt.expectedReason) + } else { + assert.Empty(t, reason) }Also applies to: 1139-1142
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/provisioner/source/provision_hook_test.go` around lines 450 - 454, The tests call needsProvisioning(dirPath, sourceSpec, true) and currently only assert result/expired is false for non-expired cases; update the test assertions to also assert that the returned reason string is empty when tt.expectExpired is false to catch stray or stale reason messages—i.e., where the table-driven test checks tt.expectExpired, add an assert.Equal(t, "", reason) (or assert.Empty) for the non-expired branch; apply the same tightening to the other occurrence around the 1139-1142 test block so both places validate an empty reason on non-expired outcomes.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/provisioner/source/provision_hook_test.go`:
- Around line 450-454: The tests call needsProvisioning(dirPath, sourceSpec,
true) and currently only assert result/expired is false for non-expired cases;
update the test assertions to also assert that the returned reason string is
empty when tt.expectExpired is false to catch stray or stale reason
messages—i.e., where the table-driven test checks tt.expectExpired, add an
assert.Equal(t, "", reason) (or assert.Empty) for the non-expired branch; apply
the same tightening to the other occurrence around the 1139-1142 test block so
both places validate an empty reason on non-expired outcomes.
In `@pkg/provisioner/source/provision_hook.go`:
- Around line 355-358: The invalid TTL error message hides the parsing error;
update the block that calls duration.ParseDuration (assigning to ttlDuration,
err) so the returned reason includes the parse error details (err) alongside the
TTL value — e.g., include err.Error() or %v in the fmt.Sprintf used in the
return statement where ttlDuration and err are in scope — so the returned string
shows both the invalid TTL and the underlying parse error for debugging.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: e2d173ea-f2e8-403e-99dd-e9c82d622f8c
📒 Files selected for processing (4)
pkg/provisioner/source/provision_hook.gopkg/provisioner/source/provision_hook_test.gopkg/schema/schema.gowebsite/src/data/roadmap.js
|
These changes were released in v1.208.1-rc.1. |
Summary
Implement a
ttl(time-to-live) field on component source configuration to control how long cached JIT-vendored sources are reused before re-pulling from the remote. This solves the problem of stale caches when using floating refs (branches) without requiring manual--forceflags.Key Feature: Declarative cache expiration policy. Set
ttl: 0sfor active development (always fresh),ttl: 1hfor team collaboration (hourly refresh), or omit TTL for infinite cache (backward compatible).Problem Solved
When JIT-vendored components use floating refs like
version: "main", Atmos skips re-pulling because the version string in metadata hasn't changed—it's still"main"even though upstream content has. Developers must manually delete.workdir/or runsource pull --force.Solution
Add optional
ttlfield to source configuration. When set, the source provisioner compares the workdir's update timestamp against the TTL. If expired, the source is re-pulled automatically.Changes
TTLfield toVendorComponentSourceschemaTerraformSourceSettingsstruct with global TTL defaultttlfrom source maps in extract.goneedsProvisioning()Documentation
ttlfielddocs/prd/source-cache-ttl.md)website/blog/2026-03-03-source-cache-ttl.mdx)Test Plan
go build ./...go test ./pkg/provisioner/source/...make lintcd website && npm run buildRelated
Fixes #2135
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests