Skip to content

Add independent Go modules for terragrunt and dependencies#1608

Merged
james00012 merged 9 commits intomainfrom
workspace-setup
Nov 7, 2025
Merged

Add independent Go modules for terragrunt and dependencies#1608
james00012 merged 9 commits intomainfrom
workspace-setup

Conversation

@james00012
Copy link
Copy Markdown
Contributor

@james00012 james00012 commented Oct 26, 2025

Summary

Modularizes Terratest by creating independent Go modules for the terragrunt dependency chain. Users can now import only what they need, reducing dependency bloat and CVE vulnerabilities.

Problem

Currently, importing terratest pulls in ALL dependencies (AWS, Azure, GCP, K8s, etc.), causing:

Solution

Created independent modules with their own go.mod files:

  • files, formatting (new shared module), logger, retry, shell, testing, terragrunt

Key achievement: Decoupled terragrunt from terraform module by creating a shared formatting module for HCL functions.

Impact

Users importing modules/terragrunt no longer get:

  • ❌ hashicorp/hcl/v2
  • ❌ terraform-json
  • ❌ go-getter
  • ✅ Only: files, formatting, logger, retry, shell, testing

Test Plan

  • All formatting module tests pass
  • All terragrunt tests pass
  • Terraform tests pass
  • Full build successful
  • Dependencies properly decoupled

Fixes #1392

Set up Go workspace with go.work and individual module files for better dependency management.
@james00012 james00012 changed the title Add Go workspace configuration for multi-module development Add independent Go modules for terragrunt and dependencies Oct 26, 2025
Create independent formatting functions in terragrunt module to remove
dependency on terraform module. This significantly reduces dependencies
for users who only need terragrunt testing.

Changes:
- Add modules/terragrunt/format.go with formatting utilities
- Update init.go to use local formatting functions
- Remove terraform from terragrunt/go.mod dependencies
- Update workspace dependencies

Benefits:
- Users importing terragrunt no longer pull in HCL parsing libraries
- Reduced dependency bloat (no hashicorp/hcl, terraform-json, go-getter)
- Easier to update terragrunt independently from terraform
Create a shared formatting module to eliminate code duplication and
decouple terragrunt from terraform module dependencies.

Changes:
- Add modules/formatting module with HCL formatting utilities
- Update terraform module to use shared formatting functions
- Update terragrunt module to use shared formatting instead of terraform
- Remove duplicated format.go from terragrunt module
- Add comprehensive tests for formatting module
- Update go.work to include formatting module
- Update go.mod files with proper dependencies

Benefits:
- Terragrunt no longer depends on terraform module
- Eliminates code duplication (161 lines removed from terragrunt)
- Removes heavyweight dependencies from terragrunt:
  - hashicorp/hcl/v2
  - hashicorp/terraform-json
  - hashicorp/go-getter
  - tmccombs/hcl2json
  - zclconf/go-cty
- Maintains backward compatibility (all tests pass)
- Easier maintenance with single source of truth for formatting
Improve code clarity and maintainability:

Changes:
- Update comments in formatting module to reflect it's shared by both
  Terraform and Terragrunt (not Terraform-specific)
- Clarify comment in terragrunt/init.go about arguments being passed
  through to underlying terraform command
- Remove unnecessary replace directives from files/go.mod (logger and
  testing are not imported by this module)

All tests passing:
- formatting module tests: ✓
- terragrunt init tests: ✓
- builds successful: ✓
Fix CI build failure by reverting dependency upgrades that were
accidentally included in the modularization PR. This keeps the PR
focused solely on module structure changes.

Changes:
- Revert gruntwork-io/go-commons from v0.17.2 to v0.8.0
- Remove urfave/cli/v2 (keep v1.22.16)
- Downgrade dependencies in all module go.mod files
- Clean up indirect dependencies

The dependency upgrades introduced breaking API changes in
cmd/terratest_log_parser that should be addressed in a separate PR.

Fixes CI build errors in PR #1608
yhakbar
yhakbar previously approved these changes Nov 3, 2025
Copy link
Copy Markdown
Contributor

@yhakbar yhakbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I believe this will be a breaking change, right? We might want to call that out in release notes.

Also, we might want to be a bit more mindful with internal/exported packages in these modules. Should all of this be exposed to library consumers? Do we want to make any signature change in any of them a breaking change in the library?

@denis256
Copy link
Copy Markdown
Member

denis256 commented Nov 3, 2025

Looks like go.mod wasn't updated

$ go mod tidy
$ git status
On branch workspace-setup
Your branch is up to date with 'origin/workspace-setup'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   go.sum
$ git diff
diff --git a/go.sum b/go.sum
index e4372e14..082f8551 100644
--- a/go.sum
+++ b/go.sum
@@ -166,6 +166,7 @@ github.com/aws/smithy-go v1.22.1 h1:/HPHZQ0g7f4eUeK6HKglFz8uwVfZKgoI25rb/J+dnro=
 github.com/aws/smithy-go v1.22.1/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg=
 github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d h1:xDfNPAt8lFiC1UJrqV3uuy861HCTo708pDMbjHHdCas=
 github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d/go.mod h1:6QX/PXZ00z/TKoufEY6K/a0k6AhaJrQKdFe6OfVXsa4=
+github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=

@james00012
Copy link
Copy Markdown
Contributor Author

LGTM.

I believe this will be a breaking change, right? We might want to call that out in release notes.

Also, we might want to be a bit more mindful with internal/exported packages in these modules. Should all of this be exposed to library consumers? Do we want to make any signature change in any of them a breaking change in the library?

This is actually not a breaking change, the public API of modules/terraform and modules/terragrunt remains unchanged (same function signatures and behavior), and consumers still use the same import paths. Only the internal implementation was refactored by moving the private formatTerraformArgs function to a shared module. This change is backward compatible while providing additional flexibility, consumers can now import just the terragrunt module without pulling in the terraform module as a dependency.

Regarding internal vs. exported packages - that's a great point about modules/formatting now being public. I think it makes sense to address whether it should be internal/formatting in a follow-up PR to keep this one focused on the core modularization.

Updates go.sum with missing checksums for various dependencies' .mod files.
Copy link
Copy Markdown
Member

@denis256 denis256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be released as a breaking change. We may also need to update documentation to reflect new imports.

@james00012 james00012 merged commit 375ff5e into main Nov 7, 2025
2 of 3 checks passed
@james00012 james00012 deleted the workspace-setup branch November 7, 2025 04:16
@dkulchinsky
Copy link
Copy Markdown

Hey @james00012,

Following this PR, we are having issues with our test suites.

go mod tidy errors with:

❯ go mod tidy
go: finding module for package github.com/gruntwork-io/terratest/modules/k8s
go: finding module for package github.com/onsi/ginkgo/v2
go: finding module for package github.com/gruntwork-io/terratest/modules/logger
go: finding module for package github.com/gruntwork-io/terratest/modules/terraform
go: finding module for package github.com/gruntwork-io/terratest/modules/retry
go: finding module for package github.com/onsi/gomega
go: finding module for package k8s.io/api/core/v1
go: finding module for package github.com/gruntwork-io/terratest/modules/helm
go: found github.com/gruntwork-io/terratest/modules/helm in github.com/gruntwork-io/terratest v0.52.0
go: found github.com/gruntwork-io/terratest/modules/k8s in github.com/gruntwork-io/terratest v0.52.0
go: found github.com/gruntwork-io/terratest/modules/logger in github.com/gruntwork-io/terratest/modules/logger v0.0.0-20251107042628-de08859c6b2d
go: found github.com/gruntwork-io/terratest/modules/retry in github.com/gruntwork-io/terratest/modules/retry v0.0.0-20251107042628-de08859c6b2d
go: found github.com/gruntwork-io/terratest/modules/terraform in github.com/gruntwork-io/terratest/modules/terraform v0.0.0-20251107042628-de08859c6b2d
go: found github.com/onsi/ginkgo/v2 in github.com/onsi/ginkgo/v2 v2.27.2
go: found github.com/onsi/gomega in github.com/onsi/gomega v1.38.2
go: found k8s.io/api/core/v1 in k8s.io/api v0.34.1
go: github.com/gruntwork-io/terratest/modules/logger@v0.0.0: reading github.com/gruntwork-io/terratest/modules/logger/go.mod at revision modules/logger/v0.0.0: unknown revision modules/logger/v0.0.0

according to your comment above, this should be a noop but appears to be breaking things.

we import:

        "github.com/gruntwork-io/terratest/modules/helm"
        "github.com/gruntwork-io/terratest/modules/k8s"
        "github.com/gruntwork-io/terratest/modules/logger"
        "github.com/gruntwork-io/terratest/modules/retry"
        "github.com/gruntwork-io/terratest/modules/terraform"

@donjuanmon
Copy link
Copy Markdown

Breaking things on our end as well.. Bug #1613 describes issue.

james00012 added a commit that referenced this pull request Nov 10, 2025
This reverts commit 375ff5e, reversing
changes made to e011dc7.
james00012 added a commit that referenced this pull request Nov 10, 2025
Remove modules/terraform/go.mod and go.sum which were introduced in PR #1608. These files should not exist in the pre-modularization state.
james00012 added a commit that referenced this pull request Nov 11, 2025
This reverts PR #1608, which introduced independent Go modules for the
terragrunt dependency chain. Reverting back to the original monolithic
module structure.
james00012 added a commit that referenced this pull request Nov 11, 2025
This reverts PR #1608, which introduced independent Go modules for the
terragrunt dependency chain. Reverting back to the original monolithic
module structure.
james00012 added a commit that referenced this pull request Nov 11, 2025
Revert PR #1608: Revert modularization changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update several, sometimes year old go dependencies with well-known CVEs

5 participants