Update to `stacks`. Add `import` to all configs. Remove hardcoded settings. Add Terraform S3 backend generation by aknysh · Pull Request #6 · cloudposse/atmos

aknysh · 2020-11-19T01:25:27Z

what

Update to stacks (instead of environment/stage)
Update to components (instead of projects)
Add import to all configs
Remove hardcoded settings
Add Terraform S3 backend generation
Simplify

why

Use stacks instead of environment/stage to define configurations - this will allow arbitrary naming the stacks and not tying them to env/stage, as well as separation of stacks for the same environment/stage into different files (e.g. terraform config in one file, helmfiles config in another), or separation of stacks by AWS resources
Add import to YAML stack configs - allow automatically importing global-globals and stack-globals into each configuration
Remove hardcoded settings - make the CLI generic (no hardcoded namespaces, regions, IAM roles, etc.)
Add (and simplify) the CLI commands to generate Terraform S3 backend configs for each Terraform component

osterman · 2020-11-19T04:51:13Z

example/stacks/globals.yaml

+namespace: eg
+
+terraform_backend_environment: ue2
+
+terraform_backend_stage: root
+
+terraform_backend_role_environment: gbl
+
+terraform_backend_role_stage: root
+
+kubeconfig_profile_environment: gbl


Suggested change

namespace: eg

terraform_backend_environment: ue2

terraform_backend_stage: root

terraform_backend_role_environment: gbl

terraform_backend_role_stage: root

kubeconfig_profile_environment: gbl

components:

globals:

namespace: eg

terraform_backend_environment: ue2

terraform_backend_stage: root

terraform_backend_role_environment: gbl

terraform_backend_role_stage: root

kubeconfig_profile_environment: gbl

- Fix Comment #6: Update ErrPermissionDenied message to be concise - Fix Comment #7: Update ErrNoComponentsWithTags to mention tags - Fix Comment #8: Wire NoColor from global persistent flags - Fix Comment #9/#14: Replace direct os.Stdout/Stderr with ui/data abstractions - Fix Comment #10: Remove direct viper.BindEnv, use os.LookupEnv for TERM - Fix Comment #11: Use data.Write in writeOutput function - Fix Comment #13: Add Intro component to diff.mdx (replace :::note) - Fix Comment #15: Use ui.Warningf in executeComponentVendorDiff stub - Fix Comment #21/#22: Fix broken documentation links in diff.mdx - Add test I/O initialization for data.Write() and ui.Infof() calls 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- cmd/auth/shell.go: Use envpkg.MergeGlobalEnv() for consistency with exec.go (addresses CodeRabbit comment #3 about env merging inconsistency) - cmd/auth/whoami.go: Use %w for error wrapping to preserve error chain (addresses CodeRabbit comment #4 about error wrapping) - tests/cli_describe_component_test.go: Use cross-platform TTY detection with term.IsTTYSupportForStdout() and close file handle properly (addresses CodeRabbit comments #5, #6) - tests/describe_test.go: Add skipIfNoTTY helper with cross-platform TTY detection and proper file handle cleanup (addresses CodeRabbit comments #7, #8) Note: Comments #1 and #2 (codeql clear-text logging) are false positives - the atmos auth env command intentionally outputs credentials for shell sourcing, similar to `aws configure export-credentials`. Suppression comments are already in place. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add duration overflow guard in ParseDuration (Comment #6) - Fix non-workdir re-provisioning: skip metadata check for non-workdir targets (Comments #7, #11) - Detect version removal: trigger re-provisioning when version is removed (Comments #8, #14) - Fix blog date 2025 → 2026 (Comments #9, #16) - Surface metadata read failures as warnings in ListWorkdirs (Comment #10) - Add periods to comment block in needsProvisioning (Comment #12) - Treat .atmos-only directories as empty in isNonEmptyDir (Comment #13) - Skip .atmos during source walk in syncSourceToDest (Comment #15) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Use filepath.Join for OS-safe test paths (Comments #1, #6) - Route FileCache operations through injected FileSystem interface (Comment #2) - Add ErrCacheFetch sentinel and wrap fetch() errors (Comment #3) - Fix misleading "log" comment in GetOrFetch (Comment #4) - Add missing BrowserSessionWarningShown assertion (Comment #5) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…2010) * fix: JIT source provisioning now takes precedence over local components When both source.uri and provision.workdir.enabled are configured on a component, the JIT source provisioner now always runs, even if a local component already exists. This ensures that source + workdir provisioning always vendors from the remote source to the workdir path, respecting the version specified in stack config rather than using a potentially stale local component. Added regression test to verify source provisioning takes precedence when both local component and source config are present. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * feat: version-aware JIT source provisioning with TTL-based cleanup - Implement intelligent re-provisioning for remote sources based on version/URI changes - Add incremental local sync with per-file checksum comparison using SyncDir - Support TTL-based cleanup for stale workdirs via duration parsing - Move workdir metadata from flat file to .atmos/metadata.json for better organization - Track source_uri, source_version, and last_accessed timestamps - Add new CLI flags: --expired, --ttl, --dry-run for workdir clean command - Update workdir list and show commands with version and access information - Extract duration parsing to new pkg/duration package for reusability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: Reduce cyclomatic and cognitive complexity in workdir/source packages - Extract helper functions to reduce function complexity: - duration.go: Use maps for unit multipliers and keywords, extract parseInteger/parseWithSuffix/parseKeyword - provision_hook.go: Extract isNonEmptyDir and checkMetadataChanges - clean.go: Extract checkWorkdirExpiry, getLastAccessedTime, getModTimeFromEntry - fs.go: Extract syncSourceToDest, fileNeedsCopy, deleteRemovedFiles - workdir.go: Extract validateComponentPath, computeContentHash, create localMetadataParams struct - Pass localMetadataParams by pointer to avoid hugeParam warning Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: Update source-provisioning example to use demo-library Replace terraform-null-label (which is a module, not a component) with demo-library components that can actually be run with terraform apply. The example now demonstrates both source types: - weather: LOCAL source (../demo-library/weather) - ipinfo: REMOTE source (github.com/cloudposse/atmos//examples/demo-library/ipinfo) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Address PR review comments for JIT source provisioning - Add duration overflow guard in ParseDuration (Comment #6) - Fix non-workdir re-provisioning: skip metadata check for non-workdir targets (Comments #7, #11) - Detect version removal: trigger re-provisioning when version is removed (Comments #8, #14) - Fix blog date 2025 → 2026 (Comments #9, #16) - Surface metadata read failures as warnings in ListWorkdirs (Comment #10) - Add periods to comment block in needsProvisioning (Comment #12) - Treat .atmos-only directories as empty in isNonEmptyDir (Comment #13) - Skip .atmos during source walk in syncSourceToDest (Comment #15) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Update golden snapshot for atmos describe config Add the new provision.workdir section to the expected output, matching the new JIT source provisioning configuration schema. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Address additional PR review comments - Guard against int64 overflow in parseWithSuffix (Comment #2) - Branch metadata writing by source type - local vs remote (Comment #3) - Add permission checks to fileNeedsCopy for mode changes (Comment #4) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Add tests to improve coverage for workdir and source provisioning Adds comprehensive tests for: - CleanExpiredWorkdirs with mock filesystem - formatDuration for human-readable output - getLastAccessedTime with atime fallback to mtime - checkWorkdirExpiry with valid/corrupt/missing metadata - isLocalSource for local vs remote URI detection Also fixes linter issues: - godot: Fix comment in duration.go - revive: Refactor formatWithOptions to map-based dispatch Addresses CodeRabbit comment #1 requesting improved patch coverage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Return wrapped error from ReadMetadata instead of warning Changes error handling in ListWorkdirs to return a wrapped error when ReadMetadata fails, surfacing permission/corruption issues to callers. Directories without metadata (metadata == nil) still skip silently. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Improve test coverage and address CodeRabbit review comments - Add metadata_test.go with tests for UpdateLastAccessed and readMetadataUnlocked - Add buildLocalMetadata tests covering all timestamp preservation branches - Add cleanExpiredWorkdirs and CleanExpiredWorkdirs tests - Fix ListWorkdirs to skip invalid metadata instead of failing entire operation - Fix zero timestamp display to show "-" instead of "0001-01-01" - Fix isLocalSource to use filepath.IsAbs for Windows path support - Fix godot lint issues in log_utils.go Coverage improvements: - pkg/provisioner/workdir: 82.1% -> 88.1% - cmd/terraform/workdir: 58.7% -> 92.2% (function coverage) - UpdateLastAccessed: 0% -> 84.2% - readMetadataUnlocked: 0% -> 100% - buildLocalMetadata: 57% -> 100% - cleanExpiredWorkdirs: 0% -> 100% Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Add JIT source provisioning tests for destroy and init commands Add test coverage to confirm that JIT source provisioning correctly takes precedence over local components for all terraform subcommands, not just plan. These tests verify that when: - source.uri is configured - provision.workdir.enabled: true - A local component exists at components/terraform/<component>/ The workdir is populated from the remote source, NOT copied from the local component. This confirms the fix in ExecuteTerraform() works universally for destroy and init commands. Uses table-driven test pattern to avoid code duplication. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Expand JIT source tests to cover all terraform subcommands Expand table-driven test to verify JIT source provisioning works for all 22 terraform subcommands that operate on a component with a stack: Core execution: apply, deploy, destroy, init, workspace State/resource: console, force-unlock, get, graph, import, output, refresh, show, state, taint, untaint Validation/info: metadata, modules, providers, test, validate All commands correctly trigger JIT source provisioning when: - source.uri is configured - provision.workdir.enabled: true - A local component exists The workdir is populated from remote source, not local component. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Improve test coverage and address CodeRabbit review comments - Make tests fail-fast instead of silently skipping when files don't exist - Verify context.tf exists (proving remote source was used) - Assert main.tf does NOT exist (proving local component wasn't copied) - Remove unused strings import - Update roadmap with JIT source provisioning precedence milestone - Update vendoring initiative progress from 86% to 89% Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Add JIT source provisioning to generate commands (#2019) - Add JIT source provisioning to terraform generate varfile - Add JIT source provisioning to terraform generate backend - Add JIT source provisioning to helmfile generate varfile - Add JIT source provisioning to packer output - Update golden snapshot for secrets-masking_describe_config test The generate commands were missing JIT source provisioning that exists in ExecuteTerraform(), causing them to fail with JIT-vendored components. This fix adds the same pattern to all affected commands. Closes #2019 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: Add automatic component refresh milestone to roadmap Add new milestone to Vendoring & Resilience initiative: - "Automatic component refresh on version changes" - Links to PR #2010 and version-aware-jit-provisioning blog post - Update progress from 89% to 95% Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Improve test coverage and address CodeRabbit review comments - Add tests for workdir clean command edge cases - Add tests for workdir show command scenarios - Add duration parsing tests for TTL validation - Add filesystem tests for workdir operations - Add metadata lock tests for Unix file locking Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Windows test compatibility and improve error hint accuracy - Skip permission-based tests on Windows (Unix permissions not supported) - TestFileNeedsCopy_DifferentPermissions - TestCopyFile_PreservesPermissions - TestServiceProvision_WriteMetadataFails (read-only dirs work differently) - Use actual componentPath in error hint instead of hardcoded path Addresses CodeRabbit review feedback. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Address CodeRabbit review comments - Wrap auto-provision error with ErrSourceProvision sentinel (packer_output.go) - Add error wrapping with ErrWorkdirMetadata in Windows metadata loader - Document circular import limitation preventing cmd.NewTestKit usage Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Use runtime.GOOS instead of os.Getenv for Windows detection GOOS is a compile-time constant, not a runtime environment variable. os.Getenv("GOOS") returns empty unless explicitly set. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: Ignore flaky kubernetes.io URLs in link checker The kubernetes.io domain frequently has connection failures/timeouts in CI, causing spurious link check failures. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: Improve JIT source test assertions with explicit failure Instead of silently passing when main.tf doesn't exist, the tests now: - Explicitly fail if main.tf exists (unexpected) - Read and check for LOCAL_VERSION_MARKER to provide better diagnostics - Use t.Fatalf to fail fast with clear error messages Addresses CodeRabbit feedback about test assertion clarity. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: improve coverage for JIT source provisioning Add comprehensive tests for: - pkg/provisioner/workdir/metadata.go: - MetadataPath function - WriteMetadata with all fields populated - ReadMetadata new location priority over legacy - UpdateLastAccessed preserves all fields - pkg/provisioner/workdir/clean.go: - checkWorkdirExpiry for expired/non-expired workdirs - getModTimeFromEntry - findExpiredWorkdirs with mixed workdirs - CleanExpiredWorkdirs with empty base path - Clean with Expired option precedence - formatDuration edge cases - pkg/provisioner/workdir/fs.go: - DefaultPathFilter.Match with patterns - SyncDir with nested directories - SyncDir updating changed files - pkg/provisioner/source/provision_hook.go: - checkMetadataChanges with version scenarios - isNonEmptyDir edge cases - needsProvisioning for non-workdir targets - writeWorkdirMetadata source type detection - writeWorkdirMetadata preserving ContentHash Coverage improvements: - workdir package: ~79% → 92.5% - source package: ~76% → 83.6% Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com> Co-authored-by: aknysh <andriy.knysh@gmail.com>

aknysh added 30 commits November 16, 2020 22:19

Update CLI

fd02534

Update CLI

858cfd8

Update CLI

0d5c975

Update CLI

9909086

Update CLI

0930ff5

Update CLI

3ad00cc

Update CLI

efb3d34

Update CLI

0e80109

Update CLI

6dbd2ca

Update CLI

f43b406

Update CLI

4761407

Update CLI

312165c

Update CLI

4c6f645

Update CLI

dbb8118

Update CLI

a49f358

Update CLI

03719a8

Update CLI

e330461

Update CLI

80985c1

Update CLI

e9028af

Update CLI

e77aac1

Update CLI

977b40e

Update CLI

8df582c

Update CLI

a12847a

Update CLI

e768bd1

Update CLI

4d454a6

Update CLI

a054d30

Update CLI

2a0a3d8

Update CLI

d17f898

Update CLI

af4fae5

Update CLI

9323aa2

aknysh added 9 commits November 18, 2020 16:44

Update CLI

b74cbbe

Update CLI

bde61a3

Update CLI

59e2306

Update CLI

a5d703c

Update CLI

815dfcd

Update CLI

95a5746

Update CLI

c3e7186

Update CLI

027ec9b

Update CLI

9b1f0cd

aknysh requested a review from osterman November 19, 2020 01:25

aknysh requested a review from a team as a code owner November 19, 2020 01:25

aknysh self-assigned this Nov 19, 2020

aknysh changed the title ~~Update to stacks. Add import to all configs. Remove hardcoded settings. Add Terraform S3 backend generation~~ Update to stacks. Add import to all configs. Remove hardcoded settings. Add Terraform S3 backend generation Nov 19, 2020

osterman approved these changes Nov 19, 2020

View reviewed changes

osterman reviewed Nov 19, 2020

View reviewed changes

aknysh merged commit 835c568 into master Nov 19, 2020

aknysh deleted the update-to-stacks branch November 19, 2020 05:00

coderabbitai bot mentioned this pull request Dec 10, 2025

Add working_directory support to commands and workflows #1852

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update to `stacks`. Add `import` to all configs. Remove hardcoded settings. Add Terraform S3 backend generation#6

Update to `stacks`. Add `import` to all configs. Remove hardcoded settings. Add Terraform S3 backend generation#6
aknysh merged 52 commits intomasterfrom
update-to-stacks

aknysh commented Nov 19, 2020 •

edited

Loading

Uh oh!

osterman Nov 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aknysh commented Nov 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

what

why

Uh oh!

osterman Nov 19, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aknysh commented Nov 19, 2020 •

edited

Loading