fix: preserve dotenv env on retry by yohamta0 · Pull Request #2225 · dagucloud/dagu

yohamta0 · 2026-05-28T16:36:55Z

Summary

Preserve dotenv-derived environment variables when retrying a DAG from a failed step.

Changes

Keep dotenv values loaded from a resolved working_dir when retry code rebuilds the DAG from stored YAML
Preserve pre-resolved base/DAG env values alongside freshly parsed YAML env during retry restoration
Add regression coverage for API retry env resolution and CLI retry restoration

Related Issues

Closes #2223

Checklist

Code follows the project style guidelines
Self-review of the code has been performed
Tests have been added or updated as needed
Documentation has been updated as needed
Changes have been tested locally

Testing

go test ./internal/core/spec -run TestResolveEnvIncludesDotenvFromResolvedWorkingDir -count=1 -v
go test -overlay /private/tmp/dagu-ignore-startall-overlay.json ./internal/cmd -run TestRestoreDAGFromStatusIncludesDotenvFromResolvedWorkingDir -count=1 -v
go test ./internal/cmn/buildenv ./internal/core/spec -count=1
go test -overlay /private/tmp/dagu-ignore-startall-overlay.json ./internal/cmd -count=1
make test
GOCACHE=/private/tmp/dagu-gocache GOLANGCI_LINT_CACHE=/private/tmp/dagu-golangci-cache make check

Summary by cubic

Preserves dotenv-derived env vars when retrying a DAG, so .env files from a resolved working_dir remain in effect. Fixes inconsistent command expansion on retry.

Bug Fixes
- Keep dotenv values from resolved working_dir when retry rebuilds DAG from stored YAML.
- Merge envs with new helpers in internal/cmn/buildenv (FromMap, AppendMissing) and apply in internal/core/spec.ResolveEnv and internal/cmd.rebuildDAGFromYAML.
- Preserve pre-resolved base/DAG env alongside freshly parsed YAML env during retry.
- Add regression tests for API and CLI retry paths.

^{Written for commit 7daf1ce. Summary will update on new commits.}

Review in cubic

Summary by CodeRabbit

Bug Fixes
- Environment variables from .env files in DAG working directories are now properly included in resolved runtime environments.
- Environment variable handling improved to preserve and merge variables rather than replacing them.

Preserve dotenv-derived environment entries when retry paths rebuild a DAG from stored YAML. This keeps retry-from-step command expansion consistent with the original run when working_dir is resolved from base or DAG env. Co-Authored-By: Codex <codex@openai.com>

coderabbitai · 2026-05-28T16:37:11Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5a67aac6-841b-4b56-984c-3d40685d5ea2

📥 Commits

Reviewing files that changed from the base of the PR and between c3768fc and 7daf1ce.

📒 Files selected for processing (6)

internal/cmd/export_test.go
internal/cmd/helper.go
internal/cmd/helper_external_test.go
internal/cmn/buildenv/env.go
internal/core/spec/runtime_env.go
internal/core/spec/runtime_env_external_test.go

📝 Walkthrough

Walkthrough

The PR fixes a bug where environment variables from dotenv files in resolved working directories are lost when retrying a failed DAG step. It introduces deterministic env merging utilities and applies them to both DAG resolution and rebuild paths to preserve loaded entries.

Changes

Environment variable preservation during DAG restore and reload

Layer / File(s)	Summary
Environment merging utilities `internal/cmn/buildenv/env.go`	`FromMap` converts env maps to deterministic sorted slices, and `AppendMissing` appends entries from multiple sources only for keys not in base, with last-value-wins deduplication.
Environment resolution refactoring `internal/core/spec/runtime_env.go`	`ResolveEnv` snapshots pre-loaded env, builds env map, loads fresh env from YAML/location/default, then merges all sources via `AppendMissing` to preserve dotenv entries across branches.
DAG rebuild environment preservation `internal/cmd/helper.go`	`rebuildDAGFromYAML` snapshots pre-rebuild env, derives transport env from presolved build env, and merges all sources instead of overwriting, preserving dotenv-loaded variables during rebuild.
Test infrastructure and validation `internal/cmd/export_test.go`, `internal/cmd/helper_external_test.go`, `internal/core/spec/runtime_env_external_test.go`	Exports restore function for testing and adds integration tests validating that DAG restore and `ResolveEnv` both preserve dotenv entries from resolved working directories.

Sequence Diagram

sequenceDiagram
  participant Client as Retry Request
  participant Restore as DAG Restore
  participant Resolve as ResolveEnv
  participant DotEnv as .env<br/>in working_dir
  participant Exec as Step Execution
  Client->>Restore: restore from previous status
  Restore->>DotEnv: load from resolved QUANT_SIGNAL_DIR
  DotEnv-->>Restore: PYTHON_BIN, PROJECT_DIR
  Restore-->>Restore: merge with loadedEnv
  Restore->>Resolve: resolve env for execution
  Resolve->>DotEnv: load from resolved working_dir
  DotEnv-->>Resolve: PYTHON_BIN, PROJECT_DIR
  Resolve-->>Resolve: merge with pre-loaded env
  Resolve-->>Restore: complete env
  Restore->>Exec: execute step with full env
  Exec-->>Client: success (env vars available)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

dagucloud/dagu#2140: This PR complements #2140's fix to loadDotEnvFiles by ensuring env-scoped expanded WorkingDir entries are preserved when DAGs are restored or reloaded.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 55.56% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'fix: preserve dotenv env on retry' accurately summarizes the main change—preserving dotenv environment variables during DAG retry operations.
Description check	✅ Passed	The PR description includes all required sections: Summary, Changes, Related Issues, Checklist (completed), and Testing details; comprehensive and well-structured.
Linked Issues check	✅ Passed	The code changes successfully address issue `#2223` by preserving dotenv env vars from resolved working_dir during retry, with new helpers (FromMap, AppendMissing) and updated merge logic in ResolveEnv and rebuildDAGFromYAML.
Out of Scope Changes check	✅ Passed	All code changes are directly related to preserving and merging dotenv environment variables on retry: helper functions, env merging logic, and regression tests—no unrelated changes detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/retry-dotenv-env

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai · 2026-05-28T16:40:50Z

Actionable comments posted: 0

cubic-dev-ai

No issues found across 6 files

_{Re-trigger cubic}

Co-authored-by: Codex <codex@openai.com>

cubic-dev-ai Bot reviewed May 28, 2026

View reviewed changes

yohamta0 merged commit 60436fa into main May 29, 2026
11 checks passed

yohamta0 deleted the fix/retry-dotenv-env branch May 29, 2026 01:20

yohamta0 added a commit that referenced this pull request May 29, 2026

fix: preserve dotenv env on retry (#2225)

ec0b02f

Co-authored-by: Codex <codex@openai.com>

coderabbitai Bot mentioned this pull request May 29, 2026

fix(core): serialize Container.Env so container.env: vars appear in step output #2231

Merged

yohamta0 mentioned this pull request May 31, 2026

bug: dotenv variables vanish on error steps in subsequent retries #2234

Closed

coderabbitai Bot mentioned this pull request Jun 5, 2026

fix: preserve retry params for dotenv resolution #2267

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: preserve dotenv env on retry#2225

fix: preserve dotenv env on retry#2225
yohamta0 merged 1 commit into
mainfrom
fix/retry-dotenv-env

yohamta0 commented May 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 28, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot commented May 28, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

yohamta0 commented May 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Related Issues

Checklist

Testing

Summary by cubic

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot commented May 28, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

yohamta0 commented May 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 28, 2026 •

edited

Loading