Skip to content

fix: don't build the DAG when running unlock. Fixes #4000 and #198#4007

Merged
dkoppstein merged 1 commit intosnakemake:mainfrom
dkoppstein:minimal-unlock
Mar 11, 2026
Merged

fix: don't build the DAG when running unlock. Fixes #4000 and #198#4007
dkoppstein merged 1 commit intosnakemake:mainfrom
dkoppstein:minimal-unlock

Conversation

@dkoppstein
Copy link
Copy Markdown
Contributor

@dkoppstein dkoppstein commented Mar 9, 2026

This fixes #4000 and #198 by simply omitting the prepare DAG and build DAG steps in the unlock method in the workflow class.

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

  • Bug Fixes

    • Unlock command no longer performs extra workflow rebuilding, making unlock faster and more reliable.
  • Tests

    • Added an end-to-end test validating the CLI unlock behavior and cleanup of lock artifacts.
  • Chores

    • Added a minimal test workflow and expected output to support the new unlock test.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 9, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a1d40ffa-8877-483d-840e-8de85feaa4ef

📥 Commits

Reviewing files that changed from the base of the PR and between 58e219a and 23b6e59.

📒 Files selected for processing (4)
  • src/snakemake/workflow.py
  • tests/test_unlock/Snakefile
  • tests/test_unlock/expected-results/output.txt
  • tests/tests.py
💤 Files with no reviewable changes (1)
  • src/snakemake/workflow.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/test_unlock/Snakefile

📝 Walkthrough

Walkthrough

Workflow.unlock no longer calls _build_dag(); it now runs _prepare_dag(...) (if applicable), then directly performs persistence.cleanup_locks() and logging, skipping DAG reconstruction during unlock.

Changes

Cohort / File(s) Summary
Workflow core
src/snakemake/workflow.py
Removed the explicit _build_dag() call from Workflow.unlock() so unlock operations skip DAG reconstruction and proceed directly to lock cleanup and logging.
Unlock test inputs
tests/test_unlock/Snakefile, tests/test_unlock/expected-results/output.txt
Added a minimal Snakemake workflow and its expected output ("done") used by the new unlock test.
Unlock test
tests/tests.py
Added test_unlock_cli() end-to-end test that creates a stale lock, runs snakemake --unlock via subprocess, and asserts the locks directory is removed or emptied.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant CLI as CLI
participant Workflow as Workflow
participant Persistence as Persistence
participant DAG as DAGBuilder

CLI->>Workflow: invoke --unlock
Workflow->>DAG: _prepare_dag() (optional)
Workflow->>Persistence: cleanup_locks()
Persistence-->>Workflow: locks cleaned
Workflow-->>CLI: return success

mermaid
sequenceDiagram
participant CLI as CLI
participant Workflow as Workflow
participant Persistence as Persistence
participant DAG as DAGBuilder

CLI->>Workflow: invoke --unlock
Workflow->>DAG: _prepare_dag() (optional)
Workflow->>DAG: _build_dag()  /* OLD flow - removed */
DAG-->>Workflow: DAG built
Workflow->>Persistence: cleanup_locks()
Persistence-->>Workflow: locks cleaned
Workflow-->>CLI: return success

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: removing DAG building from the unlock operation and references the fixed issues.
Description check ✅ Passed The description follows the template structure, explains the fix, and confirms both QC checkboxes are completed.
Linked Issues check ✅ Passed The changes meet the requirement from #4000 by removing DAG building from the unlock method, avoiding unnecessary DAG construction.
Out of Scope Changes check ✅ Passed All changes are directly related to the unlock optimization objective: modifying the unlock method, adding test coverage, and including expected test results.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cademirch cademirch changed the title don't build the DAG when running unlock. Fixes #4000 and #198 fix: don't build the DAG when running unlock. Fixes #4000 and #198 Mar 9, 2026
@cademirch cademirch self-requested a review March 9, 2026 13:09
Copy link
Copy Markdown
Contributor

@cademirch cademirch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! thanks!

@dkoppstein
Copy link
Copy Markdown
Contributor Author

Added a test which is failing, need to think about how to restructure. WIP, do not merge!

@cademirch
Copy link
Copy Markdown
Contributor

Added a test which is failing, need to think about how to restructure. WIP, do not merge!

Sounds good! thanks for checking thoroughly

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/tests.py`:
- Around line 73-92: Update test_unlock_cli so it proves the unlock path does
not initialize the workflow/DAG: after the initial run that creates
.snakemake/locks, overwrite the Snakefile in the test workspace with a file that
will raise during DAG expansion (e.g., top-level "raise RuntimeError('DAG build
should not run')") so that any code path that initializes the workflow (see
src/snakemake/workflow.py:969-984) would fail; then run the CLI unlock command
and assert it succeeds and cleans the locks. Keep references to the test
function name test_unlock_cli and the problematic workflow initialization in
READMEed code path so reviewers can locate the change.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8b58196f-3915-4009-a822-ea459a6bb94e

📥 Commits

Reviewing files that changed from the base of the PR and between bf88390 and c46e23d.

📒 Files selected for processing (4)
  • src/snakemake/workflow.py
  • tests/test_unlock/Snakefile
  • tests/test_unlock/expected-results/output.txt
  • tests/tests.py
💤 Files with no reviewable changes (1)
  • src/snakemake/workflow.py

@dkoppstein
Copy link
Copy Markdown
Contributor Author

While changing this, I ran into the issue that self.persistence is set to None unless _prepare_dag() is run. I solved this by re-adding back _prepare_dag to the workflow unlock() method and making the implicit assumption that _build_dag() takes the majority of the time. That said, in principle this step could also be omitted, since unlocking simply is removing .snakemake/locks, but I somehow want to use the built in method for unlocking rather than e.g. calling shutil.rmtree separately.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tests/tests.py (1)

73-92: Use try/finally for cleanup to ensure tmpdir removal on test failure.

If the assertion on line 91 fails, the shutil.rmtree on line 92 won't execute, leaving the temporary directory behind. Other tests in this file (e.g., test_github_issue1384, test_github_issue1882) use try/finally for this pattern.

♻️ Suggested refactor
 def test_unlock_cli():
     # run the workflow once to create the .snakemake/locks folder
     tmpdir = run(dpath("test_unlock"), cleanup=False)
-    locks = os.path.join(tmpdir, ".snakemake", "locks")
-    # the lock directory may be cleaned up automatically by Snakemake,
-    # so ensure there's at least one file inside to simulate a stale lock.
-    os.makedirs(locks, exist_ok=True)
-    lockfile = os.path.join(locks, "dummy.lock")
-    open(lockfile, "w").close()
-    assert os.listdir(locks), "expected at least one lock file"
-
-    # run the CLI unlock command and verify it succeeds
-    sp.run(
-        [sys.executable, "-m", "snakemake", "--unlock"],
-        cwd=tmpdir,
-        check=True,
-    )
-    # after unlocking the locks directory should be removed or empty
-    assert not os.path.exists(locks) or not os.listdir(locks)
-    shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)
+    try:
+        locks = os.path.join(tmpdir, ".snakemake", "locks")
+        # the lock directory may be cleaned up automatically by Snakemake,
+        # so ensure there's at least one file inside to simulate a stale lock.
+        os.makedirs(locks, exist_ok=True)
+        Path(locks, "dummy.lock").touch()
+        assert os.listdir(locks), "expected at least one lock file"
+
+        # run the CLI unlock command and verify it succeeds
+        sp.run(
+            [sys.executable, "-m", "snakemake", "--unlock"],
+            cwd=tmpdir,
+            check=True,
+        )
+        # after unlocking the locks directory should be removed or empty
+        assert not os.path.exists(locks) or not os.listdir(locks)
+    finally:
+        shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS)

Note: The diff also replaces open(lockfile, "w").close() with Path(...).touch() since Path is already imported.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/tests.py` around lines 73 - 92, The test_unlock_cli function should
ensure tmpdir is always removed by wrapping the test body (after tmpdir =
run(...)) in a try/finally where the finally calls shutil.rmtree(tmpdir,
ignore_errors=ON_WINDOWS); also replace open(lockfile, "w").close() with
Path(lockfile).touch() since Path is already imported and used elsewhere. Locate
the test by the test_unlock_cli function and adjust the run/locks setup and the
sp.run(... "--unlock") invocation to live inside the try, leaving only the
shutil.rmtree(...) call in the finally block to guarantee cleanup on assertion
or other failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests/tests.py`:
- Around line 73-92: The test_unlock_cli function should ensure tmpdir is always
removed by wrapping the test body (after tmpdir = run(...)) in a try/finally
where the finally calls shutil.rmtree(tmpdir, ignore_errors=ON_WINDOWS); also
replace open(lockfile, "w").close() with Path(lockfile).touch() since Path is
already imported and used elsewhere. Locate the test by the test_unlock_cli
function and adjust the run/locks setup and the sp.run(... "--unlock")
invocation to live inside the try, leaving only the shutil.rmtree(...) call in
the finally block to guarantee cleanup on assertion or other failures.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b0a4baff-f6f6-49c3-8594-f4d1954d8a8f

📥 Commits

Reviewing files that changed from the base of the PR and between c46e23d and 58e219a.

📒 Files selected for processing (1)
  • tests/tests.py

@KaiHabermann
Copy link
Copy Markdown

KaiHabermann commented Mar 9, 2026

While changing this, I ran into the issue that self.persistence is set to None unless _prepare_dag() is run. I solved this by re-adding back _prepare_dag to the workflow unlock() method and making the implicit assumption that _build_dag() takes the majority of the time. That said, in principle this step could also be omitted, since unlocking simply is removing .snakemake/locks, but I somehow want to use the built in method for unlocking rather than e.g. calling shutil.rmtree separately.

Chiming in here, since I started this in a way :)
Your assumption is indeed what I observe, as the initial log of _build_dag()
logger.info("Building DAG of jobs...")
is written almost immediately followed by some wait time. Thanks for implementing this so quickly. There is no reason from my perspective to add more lines to maintain here

@dkoppstein
Copy link
Copy Markdown
Contributor Author

OK, it's ready for merge. @johanneskoester do you think cleanup_shadow also could get the same treatment? It's not clear to me why the DAG needs to be built here as well, and why cleanup shadow occurs in a persistence.lock() context.

@dkoppstein
Copy link
Copy Markdown
Contributor Author

OK, we decided that cleanup_shadow is a separate case, since the directory must be locked to avoid race conditions, and that the DAG must be built to lock the directory.

@dkoppstein dkoppstein merged commit acf79fd into snakemake:main Mar 11, 2026
59 checks passed
johanneskoester pushed a commit that referenced this pull request Mar 13, 2026
🤖 I have created a release *beep* *boop*
---


##
[9.17.0](v9.16.3...v9.17.0)
(2026-03-13)


### Features

* Allow storing snakemake metadata in files or databases
([#4012](#4012))
([dd75f31](dd75f31))
* Allow to specify comparison command per-unit test
([#3956](#3956))
([b88171c](b88171c))
* job table orderd topological when run is started
([#4018](#4018))
([75cf506](75cf506))
* lambda functions for priority in rules
([#3253](#3253))
([d2aa226](d2aa226))
* Make on... directive of modules accessible
([#4050](#4050))
([e9f2e1c](e9f2e1c))


### Bug Fixes

* adjust conda tests to not fail on apple silicon; fix
[#4040](#4040)
([#4049](#4049))
([f5b0142](f5b0142))
* allow "--containerize apptainer" to output apptainer format instead of
dockerfile ([#4030](#4030))
([f5cac30](f5cac30))
* apptainer command not recognized when singularity is absent
([#4010](#4010))
([b8162e2](b8162e2))
* capture stderr when tests fail
([#3995](#3995))
([97d74ba](97d74ba))
* **docs:** make Data-dependent conditional execution a complete example
([#4043](#4043))
([3a1d7f2](3a1d7f2))
* don't build the DAG when running unlock. Fixes
[#4000](#4000) and
[#198](#198)
([#4007](#4007))
([acf79fd](acf79fd))
* Ensure pixi tasks may be run as advertised
([#4046](#4046))
([88253c2](88253c2))
* fix checkpoint handling corner cases
([#3870](#3870) and
[#3559](#3559))
([#4015](#4015))
([63f4257](63f4257))
* issue 3642
([#4054](#4054))
([76e6fc2](76e6fc2))
* issue 3815
([#4026](#4026))
([b0eec96](b0eec96))
* logging None in shellcmd context causes error
([#4064](#4064))
([d0652cd](d0652cd))
* lookup function returns default value for empty DataFrame queries
([#4056](#4056))
([f71de97](f71de97))
* make `cache: omit-software` a rule specific property
([#4085](#4085))
([034a9e7](034a9e7))
* reduce number of tests leaving temporary files behind
([#4033](#4033))
([a3a1c97](a3a1c97))
* regression in dynamic resource handling
([#4038](#4038))
([f2c554a](f2c554a))
* somewhat shorter announce message
([#4080](#4080))
([57efc71](57efc71))


### Performance Improvements

* switch reretry with tenacity; decouple container classes (with Python
3.7 compat for old scripts) from rest of the codebase (enabling moving
to newer python versions)
([#4032](#4032))
([ffb19e7](ffb19e7))


### Documentation

* Add AI-assisted contributions policy to contributing guidelines
([#4051](#4051))
([dd70526](dd70526))
* **codebase:** Update & simplify plugin architecture section
([#4052](#4052))
([176cf63](176cf63))
* Correct workflow.source_path() description in documentation
([#4036](#4036))
([45883c5](45883c5))
* fixed wrong code example for collect() function
([#4037](#4037))
([5c85ed8](5c85ed8))
* Minor docs improvements
([#4089](#4089))
([29ea226](29ea226))
* switch to sphinx_design for tabs
([#3976](#3976))
([9674614](9674614))
* typo in the migration table breaking a pip install command
([#4024](#4024))
([66f9dda](66f9dda))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dont build entrie DAG when flags liek --unlock are set

4 participants