Skip to content

fix: Add retry and missing workflow step properties to all schema copies#2113

Merged
aknysh merged 5 commits intomainfrom
copilot/fix-schema-retry-field
Mar 3, 2026
Merged

fix: Add retry and missing workflow step properties to all schema copies#2113
aknysh merged 5 commits intomainfrom
copilot/fix-schema-retry-field

Conversation

Copy link
Contributor

Copilot AI commented Feb 26, 2026

Schema validation rejected retry field in workflow steps despite Go code supporting it since the feature was added. Users hitting validation errors when using retry with YAML anchors or inline configuration.

Changes

Added missing workflow step properties to all 6 atmos-manifest schema copies in the repository:

  • website/static/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json (public schema)
  • pkg/datafetcher/schema/atmos/manifest/1.0.json (embedded schema - compiled into binary)
  • tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json (test fixtures)
  • examples/demo-localstack/schemas/atmos-manifest.json
  • examples/demo-context/schemas/atmos-manifest.json
  • examples/demo-helmfile/schemas/atmos-manifest.json

Properties added to workflow step definitions:

  • retry - Full RetryConfig spec with max_attempts, initial_delay, max_delay, backoff_strategy, multiplier, random_jitter, max_elapsed_time
  • working_directory - Step-level working directory override
  • identity - Authentication identity for step execution
  • env - Step-level environment variables

All properties support !include directive and follow existing schema patterns (e.g., source_retry).

Example

workflows:
  deploy:
    steps:
      - name: apply
        command: terraform apply demo
        retry:
          max_attempts: 3
          backoff_strategy: exponential
          max_elapsed_time: 5m
        working_directory: /tmp/deploy
        env:
          TF_LOG: DEBUG

Previously failed with additionalProperties 'retry' not allowed. Now validates correctly across all schema copies including the embedded schema used at runtime.

Original prompt

This section details on the original issue you should resolve

<issue_title>Schema: error with schema not recognizing retry in workflows</issue_title>
<issue_description>### Describe the Bug

Added retry to the workflows properties to out workflows and now atmos describe breaks

Expected Behavior

Retry should be added to schema so atmos doesn't blow up

Steps to Reproduce

# ============================================================
# ATMOS REPRO: `retry` field in workflow steps breaks `atmos describe stacks`
# ============================================================

# --- 0) Create isolated workspace ---
WORKDIR="$(mktemp -d -t atmos-repro-XXXXXX)"
echo "Working in: ${WORKDIR}"
cd "${WORKDIR}"

# --- 1) Write atmos.yaml ---
cat <<'EOF' > atmos.yaml
base_path: "."
components:
  terraform:
    base_path: "components/terraform"
    command: "tofu"
stacks:
  name_template: "{{ .vars.name }}"
  base_path: "stacks"
  included_paths:
    - "orgs/**/*"
workflows:
  base_path: "stacks/workflows"

# Validation schemas (for validating atmos stacks and components)
schemas:
  # https://json-schema.org
  jsonschema:
    # Can also be set using 'ATMOS_SCHEMAS_JSONSCHEMA_BASE_PATH' ENV var, or '--schemas-jsonschema-dir' command-line argument
    # Supports both absolute and relative paths
    base_path: "stacks/schemas/jsonschema"
  opa:
    # Can also be set using `ATMOS_SCHEMAS_OPA_BASE_PATH` ENV var, or `--schemas-opa-dir` command-line arguments
    # Supports both absolute and relative paths
    base_path: "stacks/schemas/opa"
  # JSON Schema to validate Atmos manifests
  atmos:
    # Can also be set using 'ATMOS_SCHEMAS_ATMOS_MANIFEST' ENV var, or '--schemas-atmos-manifest' command-line arguments
    # Supports both absolute and relative paths (relative to the `base_path` setting in `atmos.yaml`)
    manifest: "https://atmos.tools/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json"
EOF

# --- 2) Write a minimal stack so describe has something to parse ---
mkdir -p stacks/orgs/demo
cat <<'EOF' > stacks/orgs/demo/demo.yaml
vars:
  name: demo
terraform:
  backend_type: local
components:
  terraform:
    demo:
      vars: {}
EOF

# --- 3) Write the workflow WITH retry (the reproducer) ---
mkdir -p stacks/workflows

cat <<'EOF' > stacks/workflows/demo.yaml
name: "demo"

x-retry: &retry-config
  max_attempts: 3
  max_elapsed_time: 120m

workflows:
  01-plan:
    description: "Plan demo"
    steps:
      - name: demo
        command: terraform plan demo
        retry: *retry-config
      - name: end
        type: shell
        command: echo "done"
EOF

# --- 4) Sanity check layout ---
echo "== tree =="
find . -maxdepth 4 -type f | sed 's|^\./||' | sort

# --- 5) The actual repro ---
echo ""
echo "== describe stacks (should succeed without retry field, breaks with it) =="
atmos describe stacks 

echo ""
echo "Done. Workspace preserved at: ${WORKDIR}"
== describe stacks (should succeed without retry field, breaks with it) ==

   Error 

   Error: Atmos manifest JSON Schema validation error in the file 'workflows/demo.yaml':
   {
   "valid": false,
   "errors": [
   {
   "keywordLocation": "",
   "absoluteKeywordLocation": "https://json.schemastore.org/atmos-manifest.json#",
   "instanceLocation": "",
   "error": "doesn't validate with https://json.schemastore.org/atmos-manifest.json#"
   },
   {
   "keywordLocation": "/properties/workflows/$ref",
   "absoluteKeywordLocation": "https://json.schemastore.org/atmos-manifest.json#/properties/workflows/$ref",
   "instanceLocation": "/workflows",
   "error": "doesn't validate with '/definitions/workflows'"
   },
   {
   "keywordLocation": "/properties/workflows/$ref/oneOf",
   "absoluteKeywordLocation": "https://json.schemastore.org/atmos-manifest.json#/definitions/workflows/oneOf",
   "instanceLocation": "/workflows",
   "error": "oneOf failed"
   },
   {
   "keywordLocation": "/properties/workflows/$ref/oneOf/0/type",
   "absoluteKeywordLocation": "https://json.schemastore.org/atmos-manifest.json#/definitions/workflows/oneOf/0/type",
   "instanceLocation": "/workflows",
   "error": "expected string, but got object"
   },
   {
   "keywordLocation": "/properties/workflows/$ref/oneOf/1/patternProperties/%5E%5B1a-zA-Z0-9-_%7B%7D.%20%5D+$/$ref",
   "absoluteKeywordLocation": "https://json.schemastore.org/atmos-manifest.json#/definitions/workflows/oneOf/1/patternPropertie
s/%5E%5B1a-zA-Z0-9-%7B%7D.%20%5D+$/$ref",                                                                                         "instanceLocation": "/workflows/01-plan",
   "error": "doesn't validate with '/definitions/workflow_manifest'"
   },
   {
   "keywordLocation": "/properties/workflows/$ref/oneOf/1/patternProperties/%5E%5B~1a-zA-Z0-9-%7B%7D.%20%5D+$/$ref/oneOf",
   "absoluteKeywordLocation": "https://json.schemastore.org/atmos-manifest.json#/definitions/workflow_manifest/oneOf",
   "instanceLocation": "/workflows/01-plan",
   "error": "oneOf failed"
   },
   {...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes cloudposse/atmos#2112

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

@mergify mergify bot added triage Needs triage wip Work in Progress: Not ready for final review or merge labels Feb 26, 2026
@osterman osterman added the patch A minor, backward compatible change label Feb 26, 2026
…w step schema

Co-authored-by: osterman <52489+osterman@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix schema not recognizing retry in workflows fix: Add retry and missing workflow step properties to schema Feb 26, 2026
Copilot AI requested a review from osterman February 26, 2026 20:12
@mergify mergify bot removed the wip Work in Progress: Not ready for final review or merge label Feb 26, 2026
@osterman
Copy link
Member

@copilot I believe there are 3 schemas where this may need to be updated. Can you double check?

…operties

Extended schema updates to all atmos-manifest.json files:
- pkg/datafetcher/schema/atmos/manifest/1.0.json (embedded schema)
- tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json
- examples/demo-localstack/schemas/atmos-manifest.json
- examples/demo-context/schemas/atmos-manifest.json
- examples/demo-helmfile/schemas/atmos-manifest.json

All 6 schemas now include retry, working_directory, identity, and env properties for workflow steps.

Co-authored-by: osterman <52489+osterman@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 26, 2026

@copilot I believe there are 3 schemas where this may need to be updated. Can you double check?

You're absolutely right! I found and updated all 6 schema locations:

  1. website/static/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json (public schema)
  2. pkg/datafetcher/schema/atmos/manifest/1.0.json (embedded schema - critical for runtime)
  3. tests/fixtures/schemas/atmos/atmos-manifest/1.0/atmos-manifest.json (test fixtures)
  4. examples/demo-localstack/schemas/atmos-manifest.json
  5. examples/demo-context/schemas/atmos-manifest.json
  6. examples/demo-helmfile/schemas/atmos-manifest.json

All schemas now include the retry, working_directory, identity, and env properties for workflow steps. Changes committed in a6c10c6.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/repos/cloudposse/atmos/releases/latest
    • Triggering command: ./build/atmos ./build/atmos version -pack l/linux_amd64/compile -o init.go interface.go ux-amd64/pkg/tool/linux_amd64/compile (http block)
    • Triggering command: ./build/atmos ./build/atmos version mpile ux-amd64/pkg/too-buildtags -o sI3V/svRj2-rWVP8-errorsas mpile ux-amd64/pkg/tool/linux_amd64/vet l/consul.go github.com/aws/a-atomic t ux-amd64/pkg/too-buildtags (http block)
  • us.i.posthog.com
    • Triggering command: ./build/atmos ./build/atmos version -pack l/linux_amd64/compile -o init.go interface.go ux-amd64/pkg/tool/linux_amd64/compile (dns block)
    • Triggering command: ./build/atmos ./build/atmos version mpile ux-amd64/pkg/too-buildtags -o sI3V/svRj2-rWVP8-errorsas mpile ux-amd64/pkg/tool/linux_amd64/vet l/consul.go github.com/aws/a-atomic t ux-amd64/pkg/too-buildtags (dns block)
    • Triggering command: /home/REDACTED/work/atmos/atmos/build/atmos /home/REDACTED/work/atmos/atmos/build/atmos describe stacks ux-amd64/pkg/too-buildtags -o cYT6/OV8F-WePIw_-errorsas t ux-amd64/pkg/tool/linux_amd64/vet -p github.com/aws/a-atomic t ux-amd64/pkg/too-buildtags (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title fix: Add retry and missing workflow step properties to schema fix: Add retry and missing workflow step properties to all schema copies Feb 26, 2026
@osterman osterman marked this pull request as ready for review February 26, 2026 21:21
@osterman osterman requested a review from a team as a code owner February 26, 2026 21:21
@github-actions github-actions bot added the size/m Medium size PR label Feb 26, 2026
@github-actions
Copy link

github-actions bot commented Feb 26, 2026

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

@codecov
Copy link

codecov bot commented Feb 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.55%. Comparing base (53d7820) to head (4f863c8).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2113      +/-   ##
==========================================
+ Coverage   76.53%   76.55%   +0.02%     
==========================================
  Files         832      832              
  Lines       79458    79458              
==========================================
+ Hits        60810    60827      +17     
+ Misses      14854    14836      -18     
- Partials     3794     3795       +1     
Flag Coverage Δ
unittests 76.55% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mergify
Copy link

mergify bot commented Mar 1, 2026

💥 This pull request now has conflicts. Could you fix it @Copilot? 🙏

@mergify mergify bot added the conflict This PR has conflicts label Mar 1, 2026
Resolved conflicts in 2 schema files (website/static and tests/fixtures)
where main added workflow_retry $ref while PR added inline retry + 3 other
properties. Resolution: keep main's $ref for retry, add PR's
working_directory, identity, and env properties.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aknysh aknysh merged commit 4f0f243 into main Mar 3, 2026
61 checks passed
@aknysh aknysh deleted the copilot/fix-schema-retry-field branch March 3, 2026 21:20
@mergify mergify bot removed the triage Needs triage label Mar 3, 2026
@github-actions
Copy link

github-actions bot commented Mar 3, 2026

These changes were released in v1.208.0.

goruha added a commit that referenced this pull request Mar 9, 2026
…raform-plan

* osterman/native-ci-terraform: (28 commits)
  feat: Add source cache TTL for JIT-vendored components (#2138)
  feat: Per-target version overrides in vendor manifests (#2141)
  docs: Add PRD for browser-based auth in aws/user identity (#1887)
  docs: Add EKS kubeconfig authentication integration PRD (#1884)
  fix: correct marketplace.json schema and update docs with install/uninstall commands (#2142)
  fix: propagate auth to all YAML functions in multi-component execution (#2140)
  fix: Use atmos_component for source provisioner workdir paths (#2137)
  Fix identity prompts to respect --interactive flag (#2130)
  Increase PR size thresholds to accommodate AI-assisted development (#2136)
  docs: Add Azure authentication provider documentation (#2132)
  fix: propagate component-type level dependencies through stack processor (#2127)
  fix: Add retry and missing workflow step properties to all schema copies (#2113)
  Exclude unsupported windows/arm from goreleaser build matrix (#2133)
  Add AI Agent Skills for LLM-Powered Infrastructure Development (#2121)
  Fix: Convert toolchain paths to absolute in PATH to resolve exec.LookPath failures (#2095)
  Fix workdir collision for component instances sharing base component (#2093)
  fix(auth): propagate TTY state to subprocesses for SSO device flow in workflows (#2126)
  fix(security): prevent SSRF in GitHub OIDC token URL handling (CWE-918) (#2106)
  Fix #2112: add workflow_retry definition and retry property to workflow step schema (#2114)
  fix(auth): auto-detect GitHub Actions WIF with proper audience, host validation, and lazy GSM init (#2109)
  ...
@github-actions
Copy link

github-actions bot commented Mar 9, 2026

These changes were released in v1.208.1-test.9.

@github-actions
Copy link

github-actions bot commented Mar 9, 2026

These changes were released in v1.208.1-test.10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patch A minor, backward compatible change size/m Medium size PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants