Skip to content

feat: add !unset YAML function to delete keys from configuration#1521

Open
osterman wants to merge 17 commits intomainfrom
feat/unset-yaml-function
Open

feat: add !unset YAML function to delete keys from configuration#1521
osterman wants to merge 17 commits intomainfrom
feat/unset-yaml-function

Conversation

@osterman
Copy link
Member

@osterman osterman commented Sep 25, 2025

what

  • Add new !unset YAML function that completely removes keys from configuration during inheritance and merging
  • Implement processing in both stack merging (yaml_func_utils.go) and config loading (process_yaml.go)
  • Add comprehensive unit tests for all functionality
  • Create documentation with examples and use cases
  • Update YAML functions index documentation

why

  • Users need a way to explicitly remove inherited configuration values, not just override them with null
  • Current workarounds require physically removing or commenting out keys in parent configurations
  • This addresses GitHub issue A YAML way of undefining a value without removing the key #227: "A YAML way of undefining a value without removing the key"
  • Provides fine-grained control over configuration inheritance in complex stack hierarchies

Key Features

  • Complete removal: Unlike setting to null, !unset completely removes the key from configuration
  • Inheritance control: Child configurations can remove values inherited from parents
  • Works everywhere: Functions in all Atmos configuration sections (vars, settings, env, metadata, etc.)
  • Type-safe: Operates after YAML parsing, ensuring no syntax breakage
  • Respects skip list: Can be disabled via skip list if needed

Examples

Basic Usage

# parent.yaml
components:
  terraform:
    vpc:
      vars:
        enable_nat_gateway: true
        enable_vpn_gateway: true

# child.yaml
import:
  - parent

components:
  terraform:
    vpc:
      vars:
        enable_vpn_gateway: !unset  # Completely removes this key

Removing Nested Values

config:
  database:
    host: "prod.db.example.com"
    backup_enabled: true

# Override:
config:
  database:
    backup_enabled: !unset  # Remove backup config
    host: "dev.db.example.com"

Testing

All tests pass:

  • ✅ Unit tests for config processing
  • ✅ Unit tests for stack processing
  • ✅ Integration tests with other YAML functions
  • ✅ Skip list functionality tests
  • ✅ Inheritance scenario tests

references

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a YAML !unset function to remove keys or list items during config processing and inheritance. Works at any depth, supports multiple unsets, and coexists with other YAML functions.
  • Tests
    • Introduced comprehensive tests covering flat and nested structures, arrays, multiple/nested unsets, inheritance scenarios, and edge cases.
  • Documentation
    • Added dedicated docs and examples for !unset, including usage in stack manifests, nested removals, list handling, and guidance on expected behavior.

- Add !unset YAML function constant and registration
- Implement unset processing in yaml_func_utils.go for stack merging
- Implement unset handling in config/process_yaml.go for initial parsing
- Handle DocumentNode, MappingNode, and ScalarNode processing correctly
- Respect skip list functionality for !unset
- Add comprehensive unit tests for both config and exec packages
- Create documentation for !unset function
- Update YAML functions index documentation

The !unset function allows users to explicitly remove keys from inherited
configurations, providing fine-grained control over stack inheritance.
This addresses issue #227 requesting a way to undefine values in YAML.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@osterman osterman requested a review from a team as a code owner September 25, 2025 17:01
@github-actions github-actions bot added the size/l Large size PR label Sep 25, 2025
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 25, 2025

Warning

Rate limit exceeded

@osterman has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 23 minutes and 28 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 3728a05 and 4626beb.

📒 Files selected for processing (8)
  • internal/exec/yaml_func_unset_test.go (1 hunks)
  • pkg/config/load_config_test.go (1 hunks)
  • pkg/config/process_yaml.go (3 hunks)
  • pkg/config/process_yaml_unset_test.go (1 hunks)
  • pkg/utils/array_cleanup.go (1 hunks)
  • pkg/utils/array_cleanup_test.go (1 hunks)
  • pkg/utils/yaml_utils.go (4 hunks)
  • website/docs/functions/yaml/unset.mdx (1 hunks)
📝 Walkthrough

Walkthrough

Adds a new !unset YAML function and supporting logic to mark and remove values during YAML processing. Implements UnsetMarker propagation across maps/sequences, updates processing flow for scalar/mapping handling, adds tests validating unset behavior in various scenarios, and documents the feature.

Changes

Cohort / File(s) Summary
YAML unset core implementation
internal/exec/yaml_func_utils.go, pkg/utils/yaml_utils.go, pkg/config/process_yaml.go
Introduces UnsetMarker type and !unset tag constant; extends node processing (mapping/scalar/sequence) to detect and propagate unset markers; updates scalar dispatch to remove keys on !unset; integrates tag into recognized YAML functions.
Unit tests for unset
internal/exec/yaml_func_unset_test.go, pkg/config/process_yaml_unset_test.go
Adds comprehensive tests covering flat/nested maps, arrays, multiple/nested unsets, skip-lists, interactions with other tags, inheritance-like scenarios, empty/invalid cases, and overall preprocessing behavior.
Documentation
website/docs/functions/yaml/index.mdx, website/docs/functions/yaml/unset.mdx
Documents the new !unset YAML function with overview, usage examples, detailed behavior, use cases, and references; updates function index.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant U as User YAML
  participant L as YAML Loader
  participant P as processYaml
  participant M as processMappingNode / processChildren
  participant S as processScalarNode
  participant C as processCustomTags
  participant U$ as UnsetMarker
  participant V as Viper Store

  U->>L: Provide YAML with !unset
  L->>P: Parse AST Node
  alt Mapping/Sequence
    P->>M: Traverse keys/items
    loop For each key/item
      M->>S: Process value node
      S->>C: Check custom tag
      alt tag == !unset
        C-->>S: Return UnsetMarker
        S-->>M: Propagate UnsetMarker
        M-->>V: Skip set/remove key/item
      else other/regular tag
        C-->>S: Resolved value
        S-->>M: Value
        M-->>V: Set value
      end
    end
  else Scalar (top-level)
    P->>S: Process scalar
    S->>C: Check tag
    alt tag == !unset
      C-->>S: UnsetMarker
      S-->>V: Remove key at path
    else regular
      C-->>S: Resolved value
      S-->>V: Set value
    end
  end

  note over U$,V: UnsetMarker is a signal to delete/omit<br/>without storing a concrete value.
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested labels

minor

Suggested reviewers

  • aknysh

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 4.76% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title clearly and concisely describes the introduction of the !unset YAML function for deleting keys, matching the primary change in the pull request.
Linked Issues Check ✅ Passed The pull request implements the sentinel !unset tag, propagates an UnsetMarker through processing, removes keys during merging and config loading, and includes tests verifying inheritance and omission behaviors as specified in issue #227.
Out of Scope Changes Check ✅ Passed All modifications relate directly to supporting the new !unset YAML function through implementation, testing, or documentation, with no unrelated changes detected.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 83e8a1f and 3728a05.

📒 Files selected for processing (7)
  • internal/exec/yaml_func_unset_test.go (1 hunks)
  • internal/exec/yaml_func_utils.go (4 hunks)
  • pkg/config/process_yaml.go (3 hunks)
  • pkg/config/process_yaml_unset_test.go (1 hunks)
  • pkg/utils/yaml_utils.go (2 hunks)
  • website/docs/functions/yaml/index.mdx (2 hunks)
  • website/docs/functions/yaml/unset.mdx (1 hunks)
🧰 Additional context used
📓 Path-based instructions (7)
website/**

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

website/**: Update website documentation in website/ when adding features
Ensure consistency between CLI help text and website documentation
Follow the website's documentation structure and style
Keep website code in website/ and follow its architecture/style; test changes locally
Keep CLI and website documentation in sync; document new features with examples and use cases

Files:

  • website/docs/functions/yaml/index.mdx
  • website/docs/functions/yaml/unset.mdx
pkg/**/*.go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

Place business logic in pkg rather than in cmd

Files:

  • pkg/config/process_yaml_unset_test.go
  • pkg/utils/yaml_utils.go
  • pkg/config/process_yaml.go
**/*_test.go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

**/*_test.go: Every new feature must include comprehensive unit tests
Test both happy paths and error conditions
Use table-driven tests for multiple scenarios

**/*_test.go: Use table-driven tests for unit tests and focus on pure functions where possible.
Always use t.Skipf with a clear reason when skipping tests; never use t.Skip.
For CLI tests requiring rebuilt binaries, set a package-level skipReason in TestMain and call os.Exit(m.Run()); individual tests must check and t.Skipf if set; never use log.Fatal for missing/stale binaries.
Mirror test file names to their implementation files (e.g., aws_ssm_store_test.go for aws_ssm_store.go) and co-locate tests with implementation.

Files:

  • pkg/config/process_yaml_unset_test.go
  • internal/exec/yaml_func_unset_test.go
**/*.go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

**/*.go: All code must pass golangci-lint checks
Follow Go error handling idioms and use meaningful error messages
Wrap errors with context using fmt.Errorf("context: %w", err)
Consider custom error types for domain-specific errors
Follow standard Go coding style; run gofmt and goimports
Use snake_case for environment variables
Document complex logic with inline comments

**/*.go: All Go comments must end with periods (enforced by golangci-lint's godot).
Wrap all errors with static errors defined in errors/errors.go; never return dynamic errors directly; use fmt.Errorf with %w and add details after the static error.
Always bind environment variables using viper.BindEnv and provide an ATMOS_ alternative for each external env var.
Use structured logging for system/debug events; logging must not affect execution and should use appropriate levels per docs/logging.md.
Prefer SDKs over shelling out to binaries for cross-platform compatibility; use filepath/os/runtime for portable paths and OS-specific logic.

Files:

  • pkg/config/process_yaml_unset_test.go
  • internal/exec/yaml_func_unset_test.go
  • pkg/utils/yaml_utils.go
  • internal/exec/yaml_func_utils.go
  • pkg/config/process_yaml.go
pkg/config/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Use Viper for configuration management with config name "atmos", add config path ".", and AutomaticEnv with ATMOS prefix.

Files:

  • pkg/config/process_yaml_unset_test.go
  • pkg/config/process_yaml.go
pkg/**/*_test.go

📄 CodeRabbit inference engine (CLAUDE.md)

Place unit tests under pkg/** matching the package they test.

Files:

  • pkg/config/process_yaml_unset_test.go
**/!(*_test).go

📄 CodeRabbit inference engine (.cursor/rules/atmos-rules.mdc)

Document all exported functions, types, and methods with Go doc comments

Files:

  • pkg/utils/yaml_utils.go
  • internal/exec/yaml_func_utils.go
  • pkg/config/process_yaml.go
🧠 Learnings (9)
📚 Learning: 2024-12-03T04:01:16.446Z
Learnt from: aknysh
PR: cloudposse/atmos#810
File: website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx:0-0
Timestamp: 2024-12-03T04:01:16.446Z
Learning: In the `terraform.output.mdx` documentation file (`website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx`), the cache invalidation and cache scope behavior for the `!terraform.output` function are already described.

Applied to files:

  • website/docs/functions/yaml/index.mdx
📚 Learning: 2025-01-19T22:30:27.600Z
Learnt from: aknysh
PR: cloudposse/atmos#0
File: :0-0
Timestamp: 2025-01-19T22:30:27.600Z
Learning: The Atmos YAML function `!env` is used to retrieve environment variables and assign them to sections in stack manifests. It supports both simple types (string, number, boolean) and complex types (JSON-encoded lists, maps, objects).

Applied to files:

  • website/docs/functions/yaml/index.mdx
  • website/docs/functions/yaml/unset.mdx
📚 Learning: 2025-01-19T22:30:27.600Z
Learnt from: aknysh
PR: cloudposse/atmos#0
File: :0-0
Timestamp: 2025-01-19T22:30:27.600Z
Learning: The Atmos YAML function `!include` allows downloading local or remote files from different sources and assigning their contents to sections in stack manifests. It supports various protocols (file, http, git, s3, etc.) and can filter content using YQ expressions.

Applied to files:

  • website/docs/functions/yaml/index.mdx
  • pkg/utils/yaml_utils.go
📚 Learning: 2024-12-01T00:33:20.298Z
Learnt from: aknysh
PR: cloudposse/atmos#810
File: examples/tests/stacks/catalog/terraform/template-functions-test2/defaults.yaml:28-32
Timestamp: 2024-12-01T00:33:20.298Z
Learning: In `examples/tests/stacks/catalog/terraform/template-functions-test2/defaults.yaml`, `!exec atmos terraform output` is used in examples to demonstrate its usage, even though `!terraform.output` is the recommended approach according to the documentation.

Applied to files:

  • website/docs/functions/yaml/index.mdx
  • website/docs/functions/yaml/unset.mdx
📚 Learning: 2025-09-23T02:30:42.362Z
Learnt from: CR
PR: cloudposse/atmos#0
File: .cursor/rules/atmos-rules.mdc:0-0
Timestamp: 2025-09-23T02:30:42.362Z
Learning: Applies to **/*_test.go : Every new feature must include comprehensive unit tests

Applied to files:

  • pkg/config/process_yaml_unset_test.go
  • internal/exec/yaml_func_unset_test.go
📚 Learning: 2025-09-25T03:30:16.196Z
Learnt from: CR
PR: cloudposse/atmos#0
File: CLAUDE.md:0-0
Timestamp: 2025-09-25T03:30:16.196Z
Learning: Applies to **/*_test.go : Use table-driven tests for unit tests and focus on pure functions where possible.

Applied to files:

  • internal/exec/yaml_func_unset_test.go
📚 Learning: 2025-04-24T01:40:13.576Z
Learnt from: haitham911
PR: cloudposse/atmos#1202
File: pkg/config/process_yaml.go:0-0
Timestamp: 2025-04-24T01:40:13.576Z
Learning: When processing YAML nodes with custom directives in Go using gopkg.in/yaml.v3, setting node.Tag = "" is sufficient to prevent re-processing of the node. It's not necessary to also clear node.Value after updating the configuration store (e.g., Viper), as the original value doesn't affect subsequent operations once the tag is removed.

Applied to files:

  • internal/exec/yaml_func_unset_test.go
  • internal/exec/yaml_func_utils.go
  • pkg/config/process_yaml.go
📚 Learning: 2025-04-24T01:06:15.259Z
Learnt from: haitham911
PR: cloudposse/atmos#1202
File: pkg/config/process_yaml.go:0-0
Timestamp: 2025-04-24T01:06:15.259Z
Learning: When processing YAML structures with gopkg.in/yaml.v3, the Value field of a Node is only meant for scalar values. Complex data types like maps and arrays are represented through child nodes in the YAML tree structure.

Applied to files:

  • pkg/config/process_yaml.go
📚 Learning: 2025-09-25T03:30:16.196Z
Learnt from: CR
PR: cloudposse/atmos#0
File: CLAUDE.md:0-0
Timestamp: 2025-09-25T03:30:16.196Z
Learning: Applies to pkg/stack/**/*.go : When changing stack processing, update core logic in pkg/stack and validate multiple inheritance scenarios and template rendering.

Applied to files:

  • pkg/config/process_yaml.go
🧬 Code graph analysis (3)
internal/exec/yaml_func_unset_test.go (3)
pkg/schema/schema.go (2)
  • AtmosConfiguration (26-61)
  • AtmosSectionMapType (13-13)
internal/exec/yaml_func_utils.go (2)
  • UnsetMarker (13-15)
  • ProcessCustomYamlTags (17-24)
pkg/utils/yaml_utils.go (1)
  • AtmosYamlFuncUnset (27-27)
internal/exec/yaml_func_utils.go (1)
pkg/utils/yaml_utils.go (1)
  • AtmosYamlFuncUnset (27-27)
pkg/config/process_yaml.go (1)
pkg/utils/yaml_utils.go (5)
  • AtmosYamlFuncUnset (27-27)
  • AtmosYamlFuncEnv (23-23)
  • AtmosYamlFuncExec (17-17)
  • AtmosYamlFuncInclude (24-24)
  • AtmosYamlFuncGitRoot (26-26)
🪛 GitHub Actions: Website Preview Build
website/docs/functions/yaml/index.mdx

[error] 1-1: Build failed: Error while processing doc metadata for version 'current'. Command failed: 'npm run build:site'.

website/docs/functions/yaml/unset.mdx

[error] 3-3: Markdown front matter parsing failed due to unknown YAML tag !unset in doc metadata (Docusaurus).


[error] 1-1: Build failed: Error while processing doc metadata for version 'current'. Command failed: 'npm run build:site'.

🪛 Gitleaks (8.28.0)
pkg/config/process_yaml_unset_test.go

[high] 31-31: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Build (macos-latest, macos)
  • GitHub Check: Lint (golangci)
  • GitHub Check: Analyze (go)
  • GitHub Check: Run pre-commit hooks
  • GitHub Check: Summary
🔇 Additional comments (4)
pkg/utils/yaml_utils.go (1)

27-42: Tag registration looks correct.
Thanks for threading !unset into the known tag list so the parser can spot it.

website/docs/functions/yaml/index.mdx (1)

78-162: Docs update reads well.
The new blurb and examples slot in cleanly with the existing function catalogue.

pkg/config/process_yaml_unset_test.go (1)

10-178: Nice breadth on the unset tests.
The table plus focused scenarios give solid confidence in the merge behaviour.

internal/exec/yaml_func_utils.go (1)

12-149: Skip-aware removal logic looks good.
Appreciate how the marker bubbles through maps and slices without breaking skip-list semantics.

Comment on lines +90 to +97
// Check if the value node has the !unset tag.
if valueNode.Tag == u.AtmosYamlFuncUnset {
// Delete the key from viper by setting it to nil.
// This effectively removes it from the configuration.
v.Set(newPath, nil)
log.Debug("Unsetting configuration key", "path", newPath)
continue
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Setting nil on the Viper store keeps the key alive.

Right now v.Set(newPath, nil) leaves the entry in Viper’s backing map (e.g. IsSet still returns true and AllSettings() reports the key, now as null). Downstream merges therefore continue to see the key, so !unset doesn’t actually remove it from the Atmos config and the feature goal is missed. Let’s delete the entry from the parent map instead of writing nil—for example, walk the dotted path, delete the final segment, and write the mutated map back. Please apply the same treatment in processScalarNode where we also call v.Set(..., nil).

🤖 Prompt for AI Agents
In pkg/config/process_yaml.go around lines 90 to 97, calling v.Set(newPath, nil)
leaves the key present in Viper (reported as null) so !unset doesn't actually
remove the config; instead, resolve the parent map for newPath by walking the
dotted path stored in Viper (get the root map, traverse segments except the
last), delete the final segment from that parent map, and write the mutated root
map back to Viper (e.g., v.Set("", mutatedRoot) or v.MergeConfigMap depending on
existing patterns) so the key is truly removed; apply the same parent-map
deletion logic in processScalarNode where v.Set(..., nil) is currently used.

@osterman osterman added the minor New features that do not break anything label Sep 25, 2025
osterman and others added 5 commits September 25, 2025 12:13
- Add periods to all test comment lines for consistency
- Add detailed explanatory comments about Viper nil behavior for key deletion
- Quote YAML front matter values containing exclamation marks to fix Docusaurus parsing

Co-Authored-By: CodeRabbit <noreply@coderabbit.ai>
The test was using indented YAML which is actually valid. Changed to use unclosed quotes which causes a true YAML parsing error.
Changed broken link from /core-concepts/stacks/configuration to /core-concepts/stacks/stacks
Use the correct link format /core-concepts/stacks without the extra /stacks suffix
- Fixed mergeConfigFile to process YAML functions in temp viper before merging
- This ensures that configs from directories properly override configs from files
- Fixed test assertion order in TestLoadConfigFromCLIArgsMultipleMerge
- All config processing now follows the same pattern: load, process functions, then merge
- Modified preprocessAtmosYamlFunc to only set regular scalar values if they haven't been set yet
- This fixes an issue where configs merged later were being overwritten by earlier configs
- Ensures proper precedence: later configs override earlier ones
- Special YAML functions (!env, !exec, !unset, etc.) are still always processed
… merging

When Viper merges YAML configurations (especially from remote imports), it creates both array entries and indexed map keys (e.g., both 'steps' array and 'steps[0]', 'steps[1]' map entries). This causes duplicate entries in YAML output.

This fix adds a CleanupArrayIndexKeys utility that:
- Recursively removes indexed map keys when corresponding arrays exist
- Handles both maps and structs (converts structs to maps via JSON marshaling)
- Applies cleanup in ConvertToYAML before encoding to YAML

This ensures clean YAML output without duplicate array representations.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions github-actions bot added size/xl Extra large size PR and removed size/l Large size PR labels Sep 26, 2025
@mergify
Copy link

mergify bot commented Sep 26, 2025

Warning

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues.
Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

@mergify
Copy link

mergify bot commented Oct 2, 2025

💥 This pull request now has conflicts. Could you fix it @osterman? 🙏

@mergify mergify bot added the conflict This PR has conflicts label Oct 2, 2025
…features

- Remove JSON marshaling/unmarshaling from CleanupArrayIndexKeys
- Only process map[string]interface{} types directly
- Preserve YAML anchors, tags, and field ordering
- Return structs and other types unchanged
Resolved conflicts:
- internal/exec/yaml_func_utils.go: Integrated !unset tag handling into
  the new refactored processSimpleTags function
- pkg/utils/yaml_utils.go: Combined buffer pooling optimization with
  CleanupArrayIndexKeys preprocessing
- internal/exec/yaml_func_unset_test.go: Updated test function calls
  to include new stackInfo parameter
- pkg/utils/array_cleanup.go: Refactored RebuildArrayFromIndexedKeys
  to reduce cognitive complexity

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Dec 1, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-01-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

@mergify mergify bot removed the conflict This PR has conflicts label Dec 1, 2025
@github-actions
Copy link

github-actions bot commented Dec 1, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-06-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

@mergify
Copy link

mergify bot commented Dec 6, 2025

💥 This pull request now has conflicts. Could you fix it @osterman? 🙏

@mergify mergify bot added the conflict This PR has conflicts label Dec 6, 2025
Resolved conflicts:
- pkg/config/process_yaml.go: Integrated !unset handler with main's
  refactored processNode/processScalarNode structure and !random support
- pkg/utils/yaml_utils.go: Added both !unset and !random constants
  and registered both in AtmosYamlTags slice
- website/docs/functions/yaml/index.mdx: Combined documentation for
  both !unset and !random YAML functions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@mergify mergify bot removed the conflict This PR has conflicts label Dec 6, 2025
@github-actions
Copy link

github-actions bot commented Dec 6, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-09-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

@mergify
Copy link

mergify bot commented Dec 9, 2025

💥 This pull request now has conflicts. Could you fix it @osterman? 🙏

@mergify mergify bot added the conflict This PR has conflicts label Dec 9, 2025
Resolved conflict in pkg/utils/yaml_utils.go:
- Added !unset constant alongside new AWS YAML functions
  (!aws.account_id, !aws.caller_identity_arn, etc.)
- Maintained consistent alignment formatting

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@mergify mergify bot removed the conflict This PR has conflicts label Dec 9, 2025
@github-actions
Copy link

github-actions bot commented Dec 9, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-09-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

@mergify
Copy link

mergify bot commented Dec 9, 2025

💥 This pull request now has conflicts. Could you fix it @osterman? 🙏

@mergify mergify bot added the conflict This PR has conflicts label Dec 9, 2025
Resolved conflicts:
- internal/exec/yaml_func_utils.go: Updated !unset handler to return
  (any, bool, error) to match main's refactored processSimpleTags signature
- internal/exec/yaml_func_unset_test.go: Updated test function calls
  to handle new return signatures (result, err) for processNodes and
  processCustomTags

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Dec 9, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-09-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

@mergify mergify bot removed the conflict This PR has conflicts label Dec 9, 2025
Addresses PR review comment: v.Set(nil) doesn't truly remove keys from
Viper - IsSet still returns true and AllSettings reports the key as null.

Changes:
- Add deleteViperKey function that properly removes keys by:
  1. Getting all settings as a map
  2. Deleting the key from the nested map structure
  3. Re-reading the modified map as YAML config
- Add deleteNestedKey helper for map traversal and deletion
- Update tests to load YAML via ReadConfig (matching real Atmos usage)
- Add comprehensive test coverage for key deletion scenarios

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Dec 9, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-10-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

@codecov
Copy link

codecov bot commented Dec 9, 2025

Codecov Report

❌ Patch coverage is 74.48276% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.13%. Comparing base (448d405) to head (164f6e2).

Files with missing lines Patch % Lines
pkg/config/process_yaml.go 52.00% 15 Missing and 9 partials ⚠️
internal/exec/yaml_func_utils.go 72.72% 3 Missing and 3 partials ⚠️
pkg/utils/array_cleanup.go 91.54% 4 Missing and 2 partials ⚠️
pkg/utils/yaml_utils.go 50.00% 0 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (74.48%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1521      +/-   ##
==========================================
+ Coverage   73.11%   73.13%   +0.01%     
==========================================
  Files         550      551       +1     
  Lines       53168    53307     +139     
==========================================
+ Hits        38873    38984     +111     
- Misses      11439    11454      +15     
- Partials     2856     2869      +13     
Flag Coverage Δ
unittests 73.13% <74.48%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/utils/yaml_utils.go 81.15% <50.00%> (+0.05%) ⬆️
internal/exec/yaml_func_utils.go 87.71% <72.72%> (-1.98%) ⬇️
pkg/utils/array_cleanup.go 91.54% <91.54%> (ø)
pkg/config/process_yaml.go 67.15% <52.00%> (-2.05%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link

github-actions bot commented Dec 10, 2025

Warning

Changelog Entry Required

This PR is labeled minor or major but doesn't include a changelog entry.

Action needed: Add a new blog post in website/blog/ to announce this change.

Example filename: website/blog/2025-12-18-feature-name.mdx

Alternatively: If this change doesn't require a changelog entry, remove the minor or major label.

Resolved conflict in pkg/utils/yaml_utils.go:
- Added both !cwd (from main) and !unset (from feature branch)
- Added entries to const block, AtmosYamlTags slice, and atmosYamlTagsMap

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

osterman added a commit that referenced this pull request Dec 23, 2025
Added PR links to milestones across all initiatives:

Auth Initiative:
- EKS Kubeconfig → PR #1884
- ECR Authentication → PR #1859
- GitHub Apps → PR #1683

Developer Experience:
- Streaming Terraform UI → PR #1908
- Native CI integration → PR #1891

Workflows:
- Unified task execution → PR #1901
- New workflow step types → PR #1899

Extensibility:
- Added !unset YAML function → PR #1521
- Added !append YAML function → PR #1513

Vendoring:
- Vendor registry pattern → PR #1889
- Just-in-time vendoring → PR #1877
- Component workdir provisioning → PR #1876

Terragrunt Parity:
- Multi-stack formats → PR #1842

CI/CD:
- CI Summary Templates → PR #1891
- Terraform command registry → PR #1891
aknysh added a commit that referenced this pull request Dec 26, 2025
* feat: Add comprehensive roadmap page with initiatives and milestones

- Add Roadmap component with interactive initiative cards
- Create MilestoneList, MilestoneDrawer, and ProgressBar components
- Add Tooltip component for PR/issue titles with theme-aware styling
- Implement responsive design with flexbox alignment fixes
- Add extensive roadmap data with documented milestones across 10 initiatives
- Add changelog and docs links to all relevant milestones
- Update Docusaurus navbar with Roadmap link

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix: Address CodeRabbit review comments for roadmap components

- Fix type mismatch: use undefined instead of null for selectedMilestone
- Make changelogSlugs optional in Initiative interface
- Replace fragile status className construction with explicit mapping
- Reduce tooltip z-index from 9999 to 1002 (just above drawer)
- Add scroll/resize listeners for tooltip position updates
- Improve keyboard accessibility with tabIndex, aria-describedby, role="tooltip"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: Update Customizable list output description

Focus on custom column views rather than output formats.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: Move Component-aware tab completion to DX initiative

Tab completion is more about developer experience than discoverability.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(roadmap): rename 'Customizable list output' to 'Customizable list columns'

Clarifies that this is an improvement to existing list commands (added Q1 2025),
not a new feature. The Q4 2025 milestone added customizable column support.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(roadmap): update initiative names and add new milestones

- Rename 'Migration & Code Generation' to 'Feature Parity with Terragrunt'
- Rename 'Quality & Community' to 'Code Quality and Community'
- Change 'Learning section' to 'New learning section'
- Add Native Terraform to migration guides list
- Add Roadmap milestone to Documentation initiative

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): correct design patterns docs link

Changed /design-patterns/design-patterns to /design-patterns

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add 'Request a Feature' link to initiatives section

Adds a prominent link to GitHub feature request issue template
for users who want to suggest new features for the roadmap.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add 'View Features' link to GitHub issues

Adds a link to browse existing feature requests and issues
alongside the 'Request a Feature' link.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): rename 'View Features' to 'View Issues'

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(roadmap): reorganize Feature Parity with Terragrunt initiative

Changes to Feature Parity with Terragrunt:
- Keep: File-scoped locals
- Rename: Stack name field → Imperative stack names
- Rename: Generate section inheritance → File generation (generate blocks)
- Rename: Backend provisioning → Automatic backend provisioning
- Add: AWS context YAML functions (!aws.account-id, !aws.region, etc.)
- Remove: Metadata inheritance (merged into other features)

Moved to Extensibility & Custom Components:
- !literal YAML function
- metadata.name for component workspace keys

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add planned Terragrunt parity features

- Automatic source provisioning (no vendoring required)
- Concurrent component provisioning
- plan --all and apply --all commands
- Automatic provider caching

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): update DX milestones

- Update parent directory search description to emphasize running from any directory
- Add filesystem paths for components feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): use neutral blue gradient for progress bars

Replace orange/red colors with lighter blues to avoid
'behind schedule' connotation - progress just means 'not
as far along yet', not delayed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(roadmap): reorganize quality and docs initiatives

- Rename 'Code Quality and Community' to 'Code Quality and Test Coverage'
- Rename 'Documentation Overhaul' to 'Documentation Overhaul and Community Announcements'
- Move 'Changelog introduction' from quality to docs initiative
- Rename migration guides milestone to 'Added migration guides'

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add nightly releases and PR feature releases to quality initiative

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add 'Push for stability' to quality initiative

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): clarify backend provisioning vs backend generation

Backend provisioning is about provisioning the backend itself (S3 bucket,
DynamoDB table) for cold-start scenarios, not just generating backend.tf files.

* fix(roadmap): remove DynamoDB from backend provisioning description

DynamoDB locking is no longer needed for S3 state backends.

* fix(roadmap): mention native S3 state locking for backend provisioning

* feat(roadmap): mark --all flags as shipped, add PR links for planned features

- Mark 'plan --all' and 'apply --all' as shipped (Q4 2025)
- Add PR #1877 link for automatic source provisioning
- Add PR #1882 link for automatic provider caching

* feat(roadmap): add comprehensive PR links to milestones

Added PR links to milestones across all initiatives:

Auth Initiative:
- EKS Kubeconfig → PR #1884
- ECR Authentication → PR #1859
- GitHub Apps → PR #1683

Developer Experience:
- Streaming Terraform UI → PR #1908
- Native CI integration → PR #1891

Workflows:
- Unified task execution → PR #1901
- New workflow step types → PR #1899

Extensibility:
- Added !unset YAML function → PR #1521
- Added !append YAML function → PR #1513

Vendoring:
- Vendor registry pattern → PR #1889
- Just-in-time vendoring → PR #1877
- Component workdir provisioning → PR #1876

Terragrunt Parity:
- Multi-stack formats → PR #1842

CI/CD:
- CI Summary Templates → PR #1891
- Terraform command registry → PR #1891

* feat(roadmap): improve milestone labels with descriptive context

Update milestone labels to describe what each feature does rather than
just listing the command or feature name:

- Discoverability: Change "Added `atmos list stacks`" to
  "`atmos list stacks` to browse all configured stacks" etc.
- Auth: Change "EKS Kubeconfig integration" to "Automatic EKS kubeconfig
  tied to identities" and similar for ECR

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add Featured Improvements section with 6 key highlights

Add a prominent Featured Improvements section near the top of the roadmap
page showcasing major capabilities:

1. Dev Container Support - Consistent development environments
2. Toolchain Management - Automatic tool installation and versioning
3. Cloud Authentication - Unified identity layer across clouds
4. Beautiful Workflows - Multi-step automation with typed inputs
5. Source Provisioning - Just-in-time component fetching (in-progress)
6. Native CI/CD Support - Local = CI experience (in-progress)

Also fixes:
- Provider auto-generation marked as shipped (Q3 2025)
- Devcontainer docs link corrected

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): enhance featured section and milestone grouping

- Add PRD links to featured items (devcontainer, keyring-backends, provisioner-system, terraform-registry-migration)
- Fix statuses: Toolchain Management and Beautiful Workflows changed from shipped to in-progress
- Update quarters: Native CI/CD, Toolchain, Workflows, and Source Provisioning set to Q1 2026
- Add sorting by status (shipped first) and quarter (earlier first)
- Implement milestone grouping with categories (Featured Improvements, Everything Else) and priorities (high, nice-to-have)
- Add PR links for Toolchain Management (#1686) and Beautiful Workflows (#1899)
- Update Cloud Authentication description with "Zero local config required"
- Rename "Initiatives" section to "Major Initiatives"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(roadmap): add benefits field, clickable featured cards, and overall progress stats

- Add benefits field to all 89 milestones explaining "Why It Matters"
- Add benefits to 6 featured items with benefit-focused explanations
- Create FeaturedDrawer component for featured card detail view
- Make featured cards clickable with drawer showing description/benefits
- Create RoadmapStats component showing overall progress (shipped/in-progress/planned)
- Add animated progress bar with color-coded segments
- Expand changelog-check.yml to also require roadmap.js updates for minor/major PRs
- Update CLAUDE.md with roadmap update requirements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): simplify drawer links to reduce visual noise

Remove heavy background colors from drawer link buttons and convert
them to simple inline text links that wrap horizontally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): use plain colors for drawer links, one per line

- Change links to use neutral emphasis colors instead of brand colors
- Display each link on its own line for cleaner vertical layout

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style(roadmap): use brightening instead of underline for drawer link hover

Change drawer link hover behavior to brighten text color instead of
adding underline for cleaner visual feedback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(roadmap): add CI/local parity to cloud auth benefits

Highlight that cloud authentication works identically in CI and
locally, a key differentiator for the unified auth system.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style(roadmap): increase gap between progress bar and Major Initiatives

Add more breathing room between the Overall Progress section and
the Major Initiatives heading below it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style(roadmap): double gap between progress bar and Major Initiatives

Increase bottom margin from 3rem to 6rem for more visual separation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(roadmap): mark Terragrunt parity milestones as featured

Mark all major Terragrunt feature parity milestones as featured:
- File-scoped locals
- Imperative stack names
- File generation (generate blocks)
- Automatic backend provisioning
- AWS context YAML functions
- Multi-stack formats (YAML, JSON and HCL)
- Automatic provider caching
- Automatic source provisioning
- Provider auto-generation
- Concurrent component provisioning

Also link PR #1876 (workdir provisioning) to concurrent component
provisioning milestone.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): address CodeRabbit review feedback

- Add target="_blank" and rel="noopener noreferrer" to external GitHub
  links in FeaturedSection.tsx (PRD and PR links)
- Add target="_blank" and rel="noopener noreferrer" to external GitHub
  links in InitiativeCard.tsx (issues and PRs)
- Add target="_blank" and rel="noopener noreferrer" to external GitHub
  links in FeaturedDrawer.tsx (PRD and PR links)
- Guard against division by zero in RoadmapStats.tsx progress bar
  width calculations

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(blog): introduce the Atmos product roadmap

Add a blog post announcing the new public product roadmap page, highlighting:
- Featured improvements like Dev Containers, Cloud Auth, and Workflows
- Major initiatives including Terragrunt parity and documentation overhaul
- Overall progress tracking across all milestones
- How to get involved and provide feedback

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(roadmap): reconcile changelog announcements with roadmap

Add missing changelog links to existing milestones:
- workflow-file-auto-discovery
- automatic-backend-provisioning
- path-based-component-resolution
- aws-yaml-functions
- product-roadmap

Add new milestones for shipped features:
- Auth: auth shell, SSO auto-provisioning, identity flag for describe
- DX: terminal themes, helpful errors, provenance tracking, global env,
  metadata inheritance, interactive terraform prompts
- Extensibility: !random, !env stack manifest support, circular
  dependency detection, deferred evaluation, boolean flags

Update progress percentages:
- Auth: 80% → 85%
- DX: 75% → 90%
- Extensibility: 50% → 80%

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(roadmap): correct broken documentation links

Update doc links to use correct paths:
- /cli/configuration/configuration → /cli/configuration
- /cli/commands/commands → /cli/commands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* address comments and feedback

* address comments and feedback

---------

Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Co-authored-by: Andriy Knysh <aknysh@users.noreply.github.com>
Co-authored-by: aknysh <andriy.knysh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor New features that do not break anything size/xl Extra large size PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A YAML way of undefining a value without removing the key

2 participants