Terraform/OpenTofu Multi-Component Commands (Filtered/Bulk Operations) by aknysh · Pull Request #1327 · cloudposse/atmos

aknysh · 2025-06-23T00:42:58Z

what

Add Terraform/OpenTofu Multi-Component Commands (Filtered/Bulk Operations)
- atmos terraform plan/apply/deploy --all
- atmos terraform plan/apply/deploy --all --stack <stack>
- atmos terraform plan/apply/deploy --affected
- atmos terraform plan/apply/deploy --affected --stack <stack>
- atmos terraform plan/apply/deploy --affected --include-dependents
- atmos terraform plan/apply/deploy --affected --stack <stack> --include-dependents
- atmos terraform plan/apply/deploy --components <component1>,<component2>
- atmos terraform plan/apply/deploy --components <component1>,<component2> --stack <stack>
- atmos terraform plan/apply/deploy --query <yq-expression>
- atmos terraform plan/apply/deploy --query <yq-expression> --stack <stack>
Add unit tests
Update docs
- https://pr-1327.atmos-docs.ue2.dev.plat.cloudposse.org/cli/commands/terraform/usage/

why

Allow Terraform/Opentofu operations (plan, apply, deploy) on a set of components in all stacks or a specific stack depending on stack, set of components, query, or filters

description

Atmos Terraform/OpenTofu commands fall into two categories:

Single-Component: Run Terraform for one component at a time
Multi-Component (Filtered/Bulk): Run Terraform across multiple components using stack names, selectors, or change detection

Single-Component Commands Usage

# Execute `terraform <command>` on a `component` in a `stack`
atmos terraform <command> <component> -s <stack> [options]
atmos terraform <command> <component> --stack <stack> [options]

Multi-Component Commands (Bulk Operations) Usage

# Execute `terraform <command>` on all components in the stack `prod`
atmos terraform <command> --stack prod

# Execute `terraform <command>` on components `component-1` and `component-2` in all stacks
atmos terraform <command> --components component-1,component-2

# Execute `terraform <command>` on components `component-1` and `component-2` in the stack `prod`
atmos terraform <command> --stack prod --components component-1,component-2

# Execute `terraform <command>` on all components in all stacks
atmos terraform <command> --all

# Execute `terraform <command>` on all components in the stack `prod`
atmos terraform <command> --all --stack prod

# Execute `terraform <command>` on all the directly affected components in all stacks in dependency order
# (if component dependencies are configured)
atmos terraform <command> --affected

# Execute `terraform <command>` on all the directly affected components in the `prod` stack in dependency order
# (if component dependencies are configured)
atmos terraform <command> --affected --stack prod

# Execute `terraform <command>` on all the directly affected components in all stacks in dependency order.
# For each directly affected component, detect the dependent components and process them in dependency order, recursively.
# Dependents are components that are indirectly affected, meaning that nothing in the current branch modifies their code
# or configs, but they are configured as dependencies of the components that are modified
atmos terraform <command> --affected --include-dependents

# Execute `terraform <command>` on all the directly affected components in the `prod` stack in dependency order.
# For each directly affected component, detect the dependent components and process them in dependency order, recursively.
atmos terraform <command> --affected --include-dependents --stack prod

# Execute `terraform <command>` on all components that have `vars.tags.team == "data"`, in all stacks
atmos terraform <command> --query '.vars.tags.team == "data"'

# Execute `terraform <command>` on all components that have `vars.tags.team == "eks"`, in the stack `prod`
atmos terraform <command> --query '.vars.tags.team == "eks"' --stack prod

# Execute `terraform <command>` on all components that have `settings.context.account_id == 12345`, in all stacks
atmos terraform <command> --query '.settings.context.account_id == 12345'

Multi-Component Commands (Bulk Operations) Examples

Let's assume that we have the following Atmos stack manifests in the prod and nonprod stacks,
with dependencies between the components:

components:
  terraform:
    vpc:
      vars:
        tags:
          # Team `network` manages the `vpc` component
          team: network
    eks/cluster:
      vars:
        tags:
          # Team `eks` manages the `eks/cluster` component
          team: eks
      settings:
        depends_on:
          # `eks/cluster` depends on the `vpc` component
          1:
            component: vpc
    eks/external-dns:
      vars:
        tags:
          # Team `eks` manages the `eks/external-dns` component
          team: eks
      settings:
        depends_on:
          # `eks/external-dns` depends on the `eks/cluster` component
          1:
            component: eks/cluster
    eks/karpenter:
      vars:
        tags:
          # Team `eks` manages the `eks/karpenter` component
          team: eks
      settings:
        depends_on:
          # `eks/karpenter` depends on the `eks/cluster` component
          1:
            component: eks/cluster
    eks/karpenter-node-pool:
      vars:
        tags:
          # Team `eks` manages the `eks/karpenter-node-pool` component
          team: eks
      settings:
        # `eks/karpenter-node-pool` depends on the `eks/cluster` and `eks/karpenter` components
        depends_on:
          1:
            component: eks/cluster
          2:
            component: eks/karpenter
    eks/istio/base:
      vars:
        tags:
          # Team `istio` manages the `eks/istio/base` component
          team: istio
      settings:
        # `eks/istio/base` depends on the `eks/cluster` component
        depends_on:
          1:
            component: eks/cluster
    eks/istio/istiod:
      vars:
        tags:
          # Team `istio` manages the `eks/istio/istiod` component
          team: istio
      settings:
        # `eks/istio/istiod` depends on the `eks/cluster` and `eks/istio/base` components
        depends_on:
          1:
            component: eks/cluster
          2:
            component: eks/istio/base
    eks/istio/test-app:
      vars:
        tags:
          # Team `istio` manages the `eks/istio/test-app` component
          team: istio
      settings:
        # `eks/istio/test-app` depends on the `eks/cluster`, `eks/istio/istiod` and `eks/istio/base` components
        depends_on:
          1:
            component: eks/cluster
          2:
            component: eks/istio/istiod
          3:
            component: eks/istio/base

Let's run the following Multi-Component commands in dry-run mode and review the output to understand what each command executes:

# Execute the `terraform apply` command on all components in all stacks

> atmos terraform apply --all --dry-run

Executing command="atmos terraform apply vpc -s nonprod"
Executing command="atmos terraform apply eks/cluster -s nonprod"
Executing command="atmos terraform apply eks/external-dns -s nonprod"
Executing command="atmos terraform apply eks/istio/base -s nonprod"
Executing command="atmos terraform apply eks/istio/istiod -s nonprod"
Executing command="atmos terraform apply eks/istio/test-app -s nonprod"
Executing command="atmos terraform apply eks/karpenter -s nonprod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s nonprod"

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"
Executing command="atmos terraform apply eks/external-dns -s prod"
Executing command="atmos terraform apply eks/istio/base -s prod"
Executing command="atmos terraform apply eks/istio/istiod -s prod"
Executing command="atmos terraform apply eks/istio/test-app -s prod"
Executing command="atmos terraform apply eks/karpenter -s prod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod"

# Execute the `terraform apply` command on all components in the `prod` stack

> atmos terraform apply --all --stack prod --dry-run

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"
Executing command="atmos terraform apply eks/external-dns -s prod"
Executing command="atmos terraform apply eks/istio/base -s prod"
Executing command="atmos terraform apply eks/istio/istiod -s prod"
Executing command="atmos terraform apply eks/istio/test-app -s prod"
Executing command="atmos terraform apply eks/karpenter -s prod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod"

# Execute the `terraform apply` command on all components in the `prod` stack

> atmos terraform apply --stack prod --dry-run

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"
Executing command="atmos terraform apply eks/external-dns -s prod"
Executing command="atmos terraform apply eks/istio/base -s prod"
Executing command="atmos terraform apply eks/istio/istiod -s prod"
Executing command="atmos terraform apply eks/istio/test-app -s prod"
Executing command="atmos terraform apply eks/karpenter -s prod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod"

# Execute the `terraform apply` command on the `vpc` and `eks/cluster` components
# in all stacks.

> atmos terraform apply --components vpc,eks/cluster --dry-run

Executing command="atmos terraform apply vpc -s nonprod"
Executing command="atmos terraform apply eks/cluster -s nonprod"

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"

# Execute the `terraform apply` command on the `vpc` and `eks/cluster` components
# in the `prod` stack.

> atmos terraform apply --stack prod --components vpc,eks/cluster --dry-run

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"

# Execute the `terraform apply` command on the components filtered by the query expression,
# in all stacks.

> atmos terraform apply --query '.vars.tags.team == "eks"' --dry-run

Skipping the component because the query criteria not satisfied command="atmos terraform apply vpc -s nonprod" query=".vars.tags.team == \"eks\""
Executing command="atmos terraform apply eks/cluster -s nonprod"
Executing command="atmos terraform apply eks/external-dns -s nonprod"
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/base -s nonprod" query=".vars.tags.team == \"eks\""
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/istiod -s nonprod" query=".vars.tags.team == \"eks\""
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/test-app -s nonprod" query=".vars.tags.team == \"eks\""
Executing command="atmos terraform apply eks/karpenter -s nonprod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s nonprod"

Skipping the component because the query criteria not satisfied command="atmos terraform apply vpc -s prod" query=".vars.tags.team == \"eks\""
Executing command="atmos terraform apply eks/cluster -s prod"
Executing command="atmos terraform apply eks/external-dns -s prod"
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/base -s prod" query=".vars.tags.team == \"eks\""
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/istiod -s prod" query=".vars.tags.team == \"eks\""
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/test-app -s prod" query=".vars.tags.team == \"eks\""
Executing command="atmos terraform apply eks/karpenter -s prod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod"

# Execute the `terraform apply` command on the components filtered by the query expression,
# in the `prod` stack.

> atmos terraform apply --query '.vars.tags.team == "eks"' --stack prod --dry-run

Skipping the component because the query criteria not satisfied command="atmos terraform apply vpc -s prod" query=".vars.tags.team == \"eks\""
Executing command="atmos terraform apply eks/cluster -s prod"
Executing command="atmos terraform apply eks/external-dns -s prod"
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/base -s prod" query=".vars.tags.team == \"eks\""
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/istiod -s prod" query=".vars.tags.team == \"eks\""
Skipping the component because the query criteria not satisfied command="atmos terraform apply eks/istio/test-app -s prod" query=".vars.tags.team == \"eks\""
Executing command="atmos terraform apply eks/karpenter -s prod"
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod"

# Execute the `terraform apply` command on all components affected by the changes
# in the current branch, in all stacks, in dependency order.
# Assume that the components `vpc` and `eks/cluster` in all stacks are affected (e.g. just added).

> atmos terraform apply --affected --dry-run

Executing command="atmos terraform apply vpc -s nonprod"
Executing command="atmos terraform apply eks/cluster -s nonprod"

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"

# Execute the `terraform apply` command on all components affected by the changes
# in the current branch, in the `prod` stack, in dependency order.
# Assume that the components `vpc` and `eks/cluster` in the `prod` stack are affected (e.g. just added).

> atmos terraform apply --affected --stack prod --dry-run

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod"

# Execute the `terraform apply` command on all the components affected by the changes
# in the current branch, in all stacks.
# For each directly affected component, detect the dependent components and process
# them in dependency order, recursively.
# Dependents are components that are indirectly affected, meaning that nothing in the
# current branch modifies their code or configs, but they are configured as
# dependencies of the components that are modified.

> atmos terraform apply --affected --include-dependents --dry-run

Executing command="atmos terraform apply vpc -s nonprod"
Executing command="atmos terraform apply eks/cluster -s nonprod" dependency of component=vpc in stack=nonprod
Executing command="atmos terraform apply eks/karpenter -s nonprod" dependency of component=eks/cluster in stack=nonprod
Executing command="atmos terraform apply eks/karpenter-node-pool -s nonprod" dependency of component=eks/karpenter in stack=nonprod
Executing command="atmos terraform apply eks/external-dns -s nonprod" dependency of component=eks/cluster in stack=nonprod
Executing command="atmos terraform apply eks/istio/base -s nonprod" dependency of component=eks/cluster in stack=nonprod
Executing command="atmos terraform apply eks/istio/istiod -s nonprod" dependency of component=eks/istio/base in stack=nonprod
Executing command="atmos terraform apply eks/istio/test-app -s nonprod" dependency of component=eks/istio/istiod in stack=nonprod

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod" dependency of component=vpc in stack=prod
Executing command="atmos terraform apply eks/external-dns -s prod" dependency of component=eks/cluster in stack=prod
Executing command="atmos terraform apply eks/istio/base -s prod" dependency of component=eks/cluster in stack=prod
Executing command="atmos terraform apply eks/istio/istiod -s prod" dependency of component=eks/istio/base in stack=prod
Executing command="atmos terraform apply eks/istio/test-app -s prod" dependency of component=eks/istio/istiod in stack=prod
Executing command="atmos terraform apply eks/karpenter -s prod" dependency of component=eks/cluster in stack=prod
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod" dependency of component=eks/karpenter in stack=prod

# Execute the `terraform apply` command on all the components affected by the changes
# in the current branch, in the `prod` stack.
# For each directly affected component, detect the dependent components and process
# them in dependency order, recursively.
# Dependents are components that are indirectly affected, meaning that nothing in the
# current branch modifies their code or configs, but they are configured as
# dependencies of the components that are modified.

> atmos terraform apply --affected --stack prod --include-dependents --dry-run

Executing command="atmos terraform apply vpc -s prod"
Executing command="atmos terraform apply eks/cluster -s prod" dependency of component=vpc in stack=prod
Executing command="atmos terraform apply eks/external-dns -s prod" dependency of component=eks/cluster in stack=prod
Executing command="atmos terraform apply eks/istio/base -s prod" dependency of component=eks/cluster in stack=prod
Executing command="atmos terraform apply eks/istio/istiod -s prod" dependency of component=eks/istio/base in stack=prod
Executing command="atmos terraform apply eks/istio/test-app -s prod" dependency of component=eks/istio/istiod in stack=prod
Executing command="atmos terraform apply eks/karpenter -s prod" dependency of component=eks/cluster in stack=prod
Executing command="atmos terraform apply eks/karpenter-node-pool -s prod" dependency of component=eks/karpenter in stack=prod

Summary by CodeRabbit

New Features
- Added --affected and --all flags to Terraform commands for selective or full-stack operations.
- Introduced filtering flags: --components, --query, and --include-dependents for precise targeting of components and their dependencies.
- Added Git-related flags (--repo-path, --ref, --sha, --clone-target-ref, --ssh-key, --ssh-key-password) to identify affected components based on code changes.
- Implemented --dry-run mode to simulate Terraform commands without applying changes.
- Enhanced Terraform execution to process affected components in dependency order, including recursive handling of dependents.
- Added detailed CLI argument parsing and validation for the describe affected command with deprecation handling for --verbose.
- Extended CLI commands with new persistent flags supporting advanced Git workflows and component filtering.
- Introduced a new shell command for native Terraform usage within component/stack context.
- Added new generate subcommands for backend config and varfile generation.
- Enhanced clean, workspace, import, plan, deploy, and apply commands with new flags and behaviors.
- Added utility functions to determine affected components by comparing Git commits, supporting cloning, checkout, and local repo path modes.
- Added recursive dependency processing to include dependents of affected components automatically.
- Introduced query-based filtering for multi-component Terraform commands.
- Added support for --verbose flag deprecation with warning and log level adjustment.
Bug Fixes
- Centralized and improved error handling for CLI argument parsing and flag validation.
- Fixed error handling during flag processing to ensure immediate termination on errors.
- Resolved concurrency issues on Windows by explicit temporary directory removal after Git operations.
Documentation
- Expanded CLI help and usage documentation with comprehensive examples for single and multi-component Terraform operations.
- Updated examples to demonstrate bulk operations, dependency-aware execution, and filtering by queries or affected components.
- Clarified usage of Terraform/OpenTofu commands and flags within Atmos.
- Updated CLI help texts to include new flags and detailed descriptions.
- Corrected documentation text for clarity and consistency.
Chores
- Upgraded multiple dependencies and updated default Atmos version in Dockerfile and CI workflows.
Tests
- Added new tests covering affected components execution with dependents and query-based component filtering.
- Introduced fixtures modeling complex component dependency graphs for multi-component Terraform workflows.
- Improved test readability and adjusted mocks to align with updated method signatures.

…affected-all

…affected-all # Conflicts: # internal/exec/describe_affected.go

…affected-all

aknysh added 30 commits May 28, 2025 10:56

updates

b77b60c

updates

8bd7bb2

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

e1155c8

…affected-all

updates

4a41a63

updates

1714bd8

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

99f8f89

…affected-all

updates

f5ffd76

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

e0ffc27

…affected-all # Conflicts: # internal/exec/describe_affected.go

updates

2b84c9e

updates

48761b6

updates

c7e7a10

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

6add310

…affected-all

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

fb6af91

…affected-all

updates

cc09cc1

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

1daf432

…affected-all

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

c983af7

…affected-all

updates

7849c4b

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

5c8c3a7

…affected-all

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

ac224b0

…affected-all

updates

eb30490

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

20714dc

…affected-all

Merge remote-tracking branch 'origin/main' into terraform-plan-apply-…

ab7a177

…affected-all

updates

826225f

updates

81dcfdf

updates

e3a40de

updates

abd3f13

updates

dedb973

updates

b17695d

updates

77c3857

updates

618f36c

github-actions bot temporarily deployed to preview June 25, 2025 03:10 Destroyed

coderabbitai bot approved these changes Jun 25, 2025

View reviewed changes

osterman approved these changes Jun 25, 2025

View reviewed changes

aknysh merged commit 2ae8f49 into main Jun 25, 2025
50 checks passed

aknysh deleted the terraform-plan-apply-affected-all branch June 25, 2025 15:03

mergify bot removed the needs-cloudposse Needs Cloud Posse assistance label Jun 25, 2025

coderabbitai bot mentioned this pull request Jun 28, 2025

Improve logging and error handling #1358

Merged

This was referenced Jul 16, 2025

Implement !terraform.state Atmos YAML function #1363

Merged

[Feature improvement] Added default values handling for custom CLI flags #1379

Merged

This was referenced Jul 28, 2025

Fix plan-diff command #1382

Merged

Update !terraform.state YAML function #1383

Merged

Update atmos terraform plan command, add --skip-planfile flag #1385

Merged

coderabbitai bot mentioned this pull request Aug 23, 2025

Update atmos describe affected and atmos describe dependents commands #1414

Merged

This was referenced Sep 5, 2025

Add Native Support for Ansible Components #1448

Closed

Atmos Auth Implementation #1452

Closed

Atmos Auth Implementation #1475

Merged

feat: Atmos List Instances, Pro Upload #1254

Merged

This was referenced Oct 10, 2025

Improve performance heatmap #1611

Merged

Atmos Performance Optimization #1639

Merged

This was referenced Oct 20, 2025

Update mockgen to go.uber.org/mock #1681

Merged

Fix atmos describe affected --include-dependents --stack <stack> command to correctly process the dependents only from the provided stack #1703

Merged

This was referenced Oct 28, 2025

feat: Implement comprehensive error handling with cockroachdb/errors #1599

Closed

feat: Implement unified flag parsing system with strongly-typed Options pattern #1746

Closed

coderabbitai bot mentioned this pull request Nov 10, 2025

feat: Add configuration profiles support #1752

Merged

coderabbitai bot mentioned this pull request Dec 23, 2025

feat: Native CI Integration with Summary Templates #1891

Open

coderabbitai bot mentioned this pull request Feb 9, 2026

Detect deleted components in affected stacks #2063

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Terraform/OpenTofu Multi-Component Commands (Filtered/Bulk Operations)#1327

Terraform/OpenTofu Multi-Component Commands (Filtered/Bulk Operations)#1327
aknysh merged 107 commits intomainfrom
terraform-plan-apply-affected-all

aknysh commented Jun 23, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aknysh commented Jun 23, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

what

why

description

Single-Component Commands Usage

Multi-Component Commands (Bulk Operations) Usage

Multi-Component Commands (Bulk Operations) Examples

Summary by CodeRabbit

Summary by CodeRabbit

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aknysh commented Jun 23, 2025 •

edited by coderabbitai bot

Loading