Skip to content

🐛 use LocalStack legacy CloudFormation engine#85

Merged
sodre merged 3 commits into
mainfrom
fix/issue-81-cfn-deletion
Jan 12, 2026
Merged

🐛 use LocalStack legacy CloudFormation engine#85
sodre merged 3 commits into
mainfrom
fix/issue-81-cfn-deletion

Conversation

@sodre

@sodre sodre commented Jan 12, 2026

Copy link
Copy Markdown
Member

Summary

  • Use PROVIDER_OVERRIDE_CLOUDFORMATION=engine-legacy to work around LocalStack v2 engine stack deletion bug
  • Add minimal reproduction template for filing upstream bug report
  • Update all documentation and CI configuration with legacy engine setting

Problem

LocalStack's CloudFormation v2 engine has a bug where stack deletion fails with:

Template format error: Unresolved resource dependencies [AggregatorDLQ] in the Resources block of the template

This occurs when:

  1. A resource has a Condition: attribute (conditionally created)
  2. Another resource references it via !GetAtt inside an !If
  3. The condition evaluated to false during creation (resource was NOT created)

The v2 engine incorrectly tries to resolve !GetAtt references during deletion even for resources that were never created.

Solution

Use the legacy CloudFormation engine via PROVIDER_OVERRIDE_CLOUDFORMATION=engine-legacy. This is a workaround until LocalStack fixes the v2 engine bug upstream.

Note: The legacy engine has its own bug where CloudWatch Alarm Threshold parameters are passed as strings instead of numbers. Tests use aggregator_stack_options (no alarms) to avoid this issue.

Changes

  • .github/workflows/ci.yml - Added legacy engine to integration and benchmark jobs
  • CLAUDE.md - Updated LocalStack Docker commands with legacy engine
  • docs/infra/localstack.md - Updated examples and added warning callout
  • examples/fastapi-demo/docker-compose.yml - Added legacy engine env var
  • tests/test_integration_localstack.py - Updated docstring, fixed cleanup
  • tests/test_stack_manager.py - Updated test to expect success (not xfail)
  • tests/fixtures/localstack-v2-bug-repro.yaml - Minimal reproduction template

Test plan

  • Run unit tests: pytest tests/ -v --ignore=tests/test_performance.py (375 passed)
  • Run integration tests with legacy engine: pytest -m integration -v (23 passed)
  • Verified v2 engine bug reproduction locally
  • Verified legacy engine workaround works

Closes #81

sodre and others added 3 commits January 12, 2026 12:33
- Fix AggregatorDLQAlarmName output to use DeployAggregatorAlarms condition
  instead of DeployAggregator (fixes template error when aggregator enabled
  but alarms disabled)
- Update CLAUDE.md with LocalStack Docker socket requirement
- Add pytest markers for aws and e2e tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…#81)

Add test_stack_create_and_delete_minimal to document and verify the
LocalStack CloudFormation v2 engine bug where stack deletion fails with:
  "Template format error: Unresolved resource dependencies [AggregatorDLQ]"

Root cause: LocalStack's new CloudFormation v2 engine incorrectly tries
to resolve !GetAtt references in conditional resources during deletion,
even when those resources were never created (condition evaluated to false).

Workarounds:
1. Use legacy engine: PROVIDER_OVERRIDE_CLOUDFORMATION=engine-legacy
2. Wrap delete_stack() in try/except and delete resources directly

The test:
- Creates a minimal stack (no aggregator, no alarms)
- Attempts deletion (expected to fail on LocalStack v2 engine)
- Uses xfail to document the known issue
- Falls back to direct DynamoDB table deletion for cleanup

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
LocalStack's CloudFormation v2 engine has a bug where stack deletion
fails with "Unresolved resource dependencies [AggregatorDLQ]" due to
incorrect resolution of !GetAtt references in conditional resources.

Workaround: Use PROVIDER_OVERRIDE_CLOUDFORMATION=engine-legacy

Note: Legacy engine has its own bug where CloudWatch Alarm Threshold
parameters are passed as strings. Tests use aggregator_stack_options
(no alarms) to avoid this issue.

Updated:
- CLAUDE.md documentation
- docs/infra/localstack.md
- .github/workflows/ci.yml (integration and benchmark jobs)
- tests/test_integration_localstack.py docstring and cleanup
- tests/test_stack_manager.py (expect success, not xfail)
- examples/fastapi-demo/docker-compose.yml
- tests/fixtures/localstack-v2-bug-repro.yaml (minimal reproduction)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.30.

Benchmark suite Current: 05d83ea Previous: 7e8b69a Ratio
tests/test_performance.py::TestConcurrentThroughputBenchmarks::test_sequential_acquisitions 4.970156682205006 iter/sec (stddev: 0.11962265277986306) 6.652134556790673 iter/sec (stddev: 0.08528911949683002) 1.34

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Jan 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.37%. Comparing base (7e8b69a) to head (05d83ea).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
+ Coverage   87.17%   87.37%   +0.20%     
==========================================
  Files          18       18              
  Lines        1933     1933              
==========================================
+ Hits         1685     1689       +4     
+ Misses        248      244       -4     
Flag Coverage Δ
integration 56.54% <ø> (+0.41%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@claude

claude Bot commented Jan 12, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.


@sodre sodre merged commit 1bbc649 into main Jan 12, 2026
12 of 13 checks passed
@sodre sodre deleted the fix/issue-81-cfn-deletion branch January 12, 2026 20:37
sodre added a commit that referenced this pull request Jan 12, 2026
This reverts commit 1bbc649.

Prefer using LocalStack's v2 CloudFormation engine (the default) rather
than the legacy engine. While v2 has a stack deletion bug with conditional
resources, the legacy engine has its own bug with CloudWatch Alarm
Threshold parameters.

The v2 engine is the future of LocalStack CloudFormation support, and
we should track the upstream bug rather than permanently work around it.

Upstream issue: localstack/localstack#13609

Re-opens #81 for tracking until LocalStack fixes the v2 engine.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
sodre added a commit that referenced this pull request Jan 12, 2026
This reverts commit 1bbc649.

Prefer using LocalStack's v2 CloudFormation engine (the default) rather
than the legacy engine. While v2 has a stack deletion bug with conditional
resources, the legacy engine has its own bug with CloudWatch Alarm
Threshold parameters.

The v2 engine is the future of LocalStack CloudFormation support, and
we should track the upstream bug rather than permanently work around it.

Upstream issue: localstack/localstack#13609

Re-opens #81 for tracking until LocalStack fixes the v2 engine.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
sodre added a commit that referenced this pull request Jan 12, 2026
…#86)

* ⏪ revert: undo LocalStack legacy CloudFormation engine workaround (#85)

This reverts commit 1bbc649.

Prefer using LocalStack's v2 CloudFormation engine (the default) rather
than the legacy engine. While v2 has a stack deletion bug with conditional
resources, the legacy engine has its own bug with CloudWatch Alarm
Threshold parameters.

The v2 engine is the future of LocalStack CloudFormation support, and
we should track the upstream bug rather than permanently work around it.

Upstream issue: localstack/localstack#13609

Re-opens #81 for tracking until LocalStack fixes the v2 engine.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* 🐛 fix(infra): address code review feedback for PR #86

1. Fix CloudFormation condition mismatch: AggregatorDLQAlarmName output
   now uses DeployAggregatorAlarms condition (not DeployAggregator) to
   match the resource it references.

2. Add test_cloudformation_stack_deployment_no_alarms: Tests the edge
   case where EnableAggregator=true but EnableAlarms=false.

3. Add test_stack_create_and_delete_minimal: Integration test for full
   stack lifecycle with graceful handling of LocalStack v2 deletion bug.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* 📝 docs: remove references to non-existent e2e pytest marker

The e2e marker was removed by the revert. Update CLAUDE.md to reflect
that integration tests now include full stack lifecycle tests.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
@sodre sodre added this to the v0.2.0 milestone Jan 14, 2026
@sodre sodre mentioned this pull request Jan 14, 2026
14 tasks
@sodre sodre changed the title fix(infra): use LocalStack legacy CloudFormation engine (#81) 🐛 use LocalStack legacy CloudFormation engine Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 LocalStack: CloudFormation stack deletion fails with template validation error

1 participant