test: fix test isolation issues and improve sandbox testing by osterman · Pull Request #1512 · cloudposse/atmos

osterman · 2025-09-25T04:20:05Z

what

Add proper cleanup of Terraform artifacts before and after tests in terraform_test.go to prevent conflicts between test runs
Implement comprehensive sandbox testing framework with component isolation for better test reliability
Move sandbox functionality to testhelpers package to avoid import cycles
Add sandbox option to CLI test configuration for isolated test environments
Clean up terraform state files, lock files, and backend configs between tests
Add test coverage for sandbox creation, cleanup, environment variables, and artifact exclusion
Fix all linting issues and reduce cyclomatic complexity
Remove obsolete demo_enhanced_errors.md file
Refactor component path computation to be centralized and consistent - Previously, component paths were computed in multiple places with slightly different logic, leading to potential inconsistencies and test failures when running in sandbox mode

why

Tests were failing intermittently due to leftover Terraform artifacts from previous runs
Tests would sometimes conflict when running in parallel or repeatedly
Import cycles were preventing proper code organization
Needed a way to isolate test environments to prevent cross-test interference
The demo_enhanced_errors.md file was obsolete and no longer needed
Component path computation was duplicated across multiple packages (internal/exec, pkg/component, tests) with subtle differences that could lead to bugs and made the code harder to maintain. Centralizing this logic ensures consistency and makes the codebase more maintainable.

references

Fixes test isolation issues discovered during local development
Improves test reliability and developer experience when running tests locally
Centralizes component path resolution logic for better maintainability

🤖 Generated with Claude Code

- Add proper cleanup of Terraform artifacts before and after tests to prevent conflicts - Implement comprehensive sandbox testing framework with component isolation - Move sandbox functionality to testhelpers package to avoid import cycles - Clean up terraform state files, lock files, and backend configs between tests - Ensure tests can run repeatedly without interference from previous runs - Add test coverage for sandbox creation, cleanup, environment variables, and artifact exclusion - Fix all linting issues and reduce cyclomatic complexity - Remove obsolete demo_enhanced_errors.md file 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

mergify · 2025-09-25T04:20:45Z

Warning

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues.
Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

…orm tests - Fix all sandbox test fixture paths to use correct relative path (../fixtures/) - Add sandbox flag to tests that run terraform commands for better isolation: - atmos-functions.yaml: terraform output tests - exec-command.yaml: terraform plan/apply tests - metadata.yaml: terraform plan test - All sandbox tests now passing - Tests with terraform operations now use isolated component copies 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com)

- Add TestSandboxComponentIsolation to prove sandbox truly isolates components - Creates sandbox and modifies a component file in it - Verifies original component remains unchanged - Proves paths are different and contain sandbox identifier - Add TestSandboxActuallyUsedByAtmos to verify sandbox is properly used - Verifies environment variables point to sandbox temp directory - Creates marker files in sandbox to prove isolation - Lists actual components copied to sandbox (6 items found) - Verifies markers don't appear in original locations These tests conclusively prove the sandbox mechanism works correctly and provides true component isolation for parallel test execution. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Add GetComponentPath() function to handle all path resolution logic - Respect environment variable overrides (ATMOS_COMPONENTS_*_BASE_PATH) - Support custom component paths (e.g., opentofu/, terragrunt/) - Fix sandbox test isolation issues on all platforms - Ensure paths are always absolute and properly cleaned - Add comprehensive tests for path resolution logic This refactor fixes test failures by ensuring that sandbox environment variables correctly override component paths, regardless of the user's configuration or directory structure. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

codecov · 2025-09-25T05:58:50Z

Codecov Report

❌ Patch coverage is 65.01650% with 106 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.03%. Comparing base (e15c114) to head (d082db4).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
tests/testhelpers/sandbox.go	59.34%	63 Missing and 11 partials ⚠️
internal/exec/helmfile.go	0.00%	7 Missing ⚠️
internal/exec/packer.go	14.28%	5 Missing and 1 partial ⚠️
internal/exec/packer_output.go	14.28%	5 Missing and 1 partial ⚠️
internal/exec/terraform.go	14.28%	5 Missing and 1 partial ⚠️
pkg/utils/component_path_utils.go	94.20%	3 Missing and 1 partial ⚠️
internal/exec/path_utils.go	87.50%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1512      +/-   ##
==========================================
+ Coverage   56.98%   57.03%   +0.04%     
==========================================
  Files         284      286       +2     
  Lines       30292    30587     +295     
==========================================
+ Hits        17261    17444     +183     
- Misses      11207    11302      +95     
- Partials     1824     1841      +17

Flag	Coverage Δ
unittests	`57.03% <65.01%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Modified path construction functions to avoid converting simple test paths to absolute paths - Only use GetComponentPath with absolute conversion when resolved absolute paths are present - Added Windows-specific handling for file permission tests that don't apply on Windows - Fixes test failures on Windows CI while maintaining backward compatibility 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

github-actions · 2025-09-26T20:05:46Z

These changes were released in v1.192.0.

github-actions · 2025-09-26T23:52:34Z

These changes were released in v1.193.0-test.1.

osterman requested a review from a team as a code owner September 25, 2025 04:20

github-actions bot added the size/xl Extra large size PR label Sep 25, 2025

osterman and others added 2 commits September 24, 2025 23:37

osterman added the no-release Do not create a new release (wait for additional code changes) label Sep 25, 2025

osterman and others added 2 commits September 25, 2025 08:34

Merge branch 'main' into fix-local-tests

d082db4

aknysh approved these changes Sep 25, 2025

View reviewed changes

aknysh merged commit 83e8a1f into main Sep 25, 2025
52 checks passed

aknysh deleted the fix-local-tests branch September 25, 2025 16:59

osterman mentioned this pull request Sep 27, 2025

fix: prevent path duplication when using absolute paths in component configuration #1535

Merged

osterman mentioned this pull request Oct 19, 2025

Refactor exit code handling and error reporting #1669

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: fix test isolation issues and improve sandbox testing#1512

test: fix test isolation issues and improve sandbox testing#1512
aknysh merged 6 commits intomainfrom
fix-local-tests

osterman commented Sep 25, 2025 •

edited

Loading

Uh oh!

mergify bot commented Sep 25, 2025

This PR exceeds the recommended limit of 1,000 lines.

Uh oh!

codecov bot commented Sep 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

osterman commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

what

why

references

Uh oh!

mergify bot commented Sep 25, 2025

This PR exceeds the recommended limit of 1,000 lines.

Uh oh!

codecov bot commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

osterman commented Sep 25, 2025 •

edited

Loading

codecov bot commented Sep 25, 2025 •

edited

Loading