Skip to content

fix: symlink to directory "has older modification time" than target (#3782) #3784

Merged
johanneskoester merged 14 commits into
snakemake:mainfrom
thekswenson:fix/issue3687
Jun 1, 2026
Merged

fix: symlink to directory "has older modification time" than target (#3782) #3784
johanneskoester merged 14 commits into
snakemake:mainfrom
thekswenson:fix/issue3687

Conversation

@thekswenson

@thekswenson thekswenson commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

This is a test case for the problem described in #3687.

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

  • Tests

    • Added an integration test that verifies a workflow creating a file and a symlinked directory target, ensuring reruns are triggered when inputs change (skipped on Windows).
  • Bug Fixes

    • Improved handling of directory vs. symlink timestamps and touch behavior, and clarified related error messages for more reliable filesystem workflows.

@coderabbitai

coderabbitai Bot commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1f9fcbae-996e-4793-992d-75aa3358da91

📥 Commits

Reviewing files that changed from the base of the PR and between 88851e1 and 1a565a8.

📒 Files selected for processing (3)
  • src/snakemake/io/__init__.py
  • tests/test_github_issue3687/Snakefile
  • tests/tests.py
✅ Files skipped from review due to trivial changes (1)
  • src/snakemake/io/init.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/tests.py
  • tests/test_github_issue3687/Snakefile

📝 Walkthrough

Walkthrough

Adds a Snakemake workflow that creates dir1/D and a symlink dir2/Ddir1/D, an expected-results file for the symlink target, a new test to run the workflow (skipped on Windows), and refines mtime/touch logic to better handle symlinks and per-directory timestamps.

Changes

Cohort / File(s) Summary
Snakemake workflow
tests/test_github_issue3687/Snakefile
Adds three rules: make_file (creates dir1/D and dir1/D/file.txt), link_to_D (creates symlink dir2/Ddir1/D), and all (depends on dir2/D).
Expected results
tests/test_github_issue3687/expected-results/dir2/D
Adds expected content ../dir1/D representing the symlink target.
Tests
tests/tests.py
Adds test_github_issue3687() (skipped on Windows) which runs the workflow in a temp dir, manipulates inputs/mtime via copying/touching and asserts the run is triggered by updated input through the symlink.
I/O handling
src/snakemake/io/__init__.py
Adjusts mtime/touch logic: uses a local file = str(self.file), narrows storage managed_mtime access, improves error messages with resolved paths, distinguishes symlinks vs real directories, employs per-directory .snakemake_timestamp for real dirs, and avoids creating timestamp files for symlinks.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is incomplete. It states 'This is a test case for the problem' but the raw_summary shows changes in src/snakemake/io/init.py that represent the actual bug fix. The description also fails to disclose whether AI assistance was used and leaves the documentation checkbox unchecked without explanation. Update the description to clarify that this PR includes both the bug fix in src/snakemake/io/init.py and a test case, and complete the AI-assistance disclosure section by checking or explicitly stating whether AI tools were used.
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title correctly describes the main change: fixing a symlink-to-directory modification time issue and adding a test case (issue #3687).

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
tests/test_github_issue3687/Snakefile (1)

18-27: Consider ensuring parent directory exists.

The code creates a symlink at dir2/D, but the parent directory dir2 is not explicitly created. While Snakemake may handle parent directory creation for outputs automatically, consider adding explicit parent creation for robustness:

   run:
+    Path(str(output)).parent.mkdir(parents=True, exist_ok=True)
     Path(str(output)).symlink_to(f'../{input}')

Alternatively, verify that Snakemake's test framework handles this automatically.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2eb9079 and 805ce54.

📒 Files selected for processing (2)
  • tests/test_github_issue3687/Snakefile (1 hunks)
  • tests/test_github_issue3687/expected-results/dir2/D (1 hunks)
🔇 Additional comments (4)
tests/test_github_issue3687/expected-results/dir2/D (1)

1-1: LGTM! Expected symlink target is correct.

The expected symlink target ../dir1/D correctly matches what the Snakefile produces in the link_to_D rule (line 27).

tests/test_github_issue3687/Snakefile (3)

3-5: LGTM! Standard target rule.

The all rule correctly specifies the final workflow target.


1-27: Please paste the content of GitHub issue #3687 so I can confirm that this test setup accurately reproduces the reported behavior.


8-15: Ensure directory output is created before touch. Path.touch() doesn’t create parent directories—confirm that Snakemake’s directory('dir1/D') output is instantiated before the run block or explicitly call Path(output.file).parent.mkdir(parents=True, exist_ok=True) before touch().

@fgvieira

Copy link
Copy Markdown
Contributor

You need to add the test to tests/tests.py.

@fgvieira fgvieira changed the title test: symlink to directory "has older modification time" than target (#3782) test: symlink to directory "has older modification time" than target (#3782) Feb 27, 2026
@thekswenson

Copy link
Copy Markdown
Contributor Author

@fgvieira : I'm looking at the tests.py file and I don't see any explicit comparisons of expected-results against the actual results. Is this does automatically?

@thekswenson

Copy link
Copy Markdown
Contributor Author

Is it sufficent to just do the following?

def test_github_issue3687():
    run(dpath("test_github_issue3687"))

@fgvieira

fgvieira commented Feb 27, 2026

Copy link
Copy Markdown
Contributor

Yes, add that code to trigger the test and, if you have output, also include them in the test under an expected-results folder. This was, these files are automatically compared at the of the test to make that, not only the test run but that its output is the same.

@fgvieira

Copy link
Copy Markdown
Contributor

The test is being triggered now, but there seems to be something wrong. 😄

@thekswenson

Copy link
Copy Markdown
Contributor Author

The test is being triggered now, but there seems to be something wrong. 😄

Yeah... I don't think there is a fix out yet for the bug that this tests. I added this test case at the behest of @johanneskoester:

#3687 (comment)

@fgvieira fgvieira added the bug Something isn't working label Mar 10, 2026
@Hocnonsense Hocnonsense self-assigned this Mar 11, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/snakemake/io/__init__.py (1)

587-599: Preserve the original stat() failure via exception chaining.

These branches now add useful context, but they still discard the original FileNotFoundError / PermissionError, which makes symlink-vs-target failures harder to diagnose. Please capture the exception and raise WorkflowError(...) from e here.

♻️ Proposed change
-        except FileNotFoundError:
+        except FileNotFoundError as e:
             if self.is_storage:
                 return Mtime(storage=mtime_in_storage)
             raise WorkflowError(
                 f"Unable to obtain modification time of file {file} although it existed before. "
                 "It could be that a concurrent process has deleted it while Snakemake "
                 "was running."
-            )
-        except PermissionError:
+            ) from e
+        except PermissionError as e:
             raise WorkflowError(
                 f"Unable to obtain modification time of file {file} because of missing "
                 "read permissions."
-            )
+            ) from e

Based on learnings: "In the snakemake codebase, use exception chaining (raise NewError(...) from original) to preserve the original exception context when re-raising or wrapping exceptions in Python files."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/snakemake/io/__init__.py` around lines 587 - 599, The except handlers
that wrap FileNotFoundError and PermissionError should preserve the original
exception via exception chaining: capture the original exception (e) in both
except blocks and re-raise WorkflowError(...) using "raise WorkflowError(... )
from e" so the original stat() failure context is kept; update the
FileNotFoundError handler (the branch that returns
Mtime(storage=mtime_in_storage) when self.is_storage, otherwise raises
WorkflowError about modification time of file variable file) and the
PermissionError handler (which raises WorkflowError about missing read
permissions) to use "from e" when re-raising.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/snakemake/io/__init__.py`:
- Around line 587-599: The except handlers that wrap FileNotFoundError and
PermissionError should preserve the original exception via exception chaining:
capture the original exception (e) in both except blocks and re-raise
WorkflowError(...) using "raise WorkflowError(... ) from e" so the original
stat() failure context is kept; update the FileNotFoundError handler (the branch
that returns Mtime(storage=mtime_in_storage) when self.is_storage, otherwise
raises WorkflowError about modification time of file variable file) and the
PermissionError handler (which raises WorkflowError about missing read
permissions) to use "from e" when re-raising.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a90cdcd7-ddfb-4d61-adbd-7225fa885cc9

📥 Commits

Reviewing files that changed from the base of the PR and between 8448b49 and f70f6a1.

📒 Files selected for processing (2)
  • src/snakemake/io/__init__.py
  • tests/tests.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/tests.py

Comment thread src/snakemake/io/__init__.py
@Hocnonsense Hocnonsense changed the title test: symlink to directory "has older modification time" than target (#3782) fix: symlink to directory "has older modification time" than target (#3782) Mar 11, 2026
@Hocnonsense Hocnonsense moved this from Ready to In review in Snakemake Hackathon 2026 Mar 11, 2026
@johanneskoester

Copy link
Copy Markdown
Contributor

Waiting for a second test case here by @Hocnonsense to test whether the fix in snakemake/snakemake-interface-storage-plugins#87 sufficiently addressed the issue.

@Hocnonsense

Copy link
Copy Markdown
Contributor

The second test case is added and passed after snakemake_interface_storage_plugins updated. I think it is ready to be reviewed~ @johanneskoester

@Hocnonsense Hocnonsense linked an issue Apr 13, 2026 that may be closed by this pull request
@johanneskoester johanneskoester merged commit 41903d8 into snakemake:main Jun 1, 2026
60 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in Snakemake Hackathon 2026 Jun 1, 2026
johanneskoester pushed a commit that referenced this pull request Jun 1, 2026
🤖 I have created a release *beep* *boop*
---


##
[9.22.0](v9.21.1...v9.22.0)
(2026-06-01)


### Features

* add semantic helper functions choose_file/choose_folder/choose_tmp and
allow all semantic helper functions to be used when parsing resources
([#3820](#3820))
([d3c2386](d3c2386))
* add temp to default pathvars
([#4108](#4108))
([917fb11](917fb11))
* add workflow info to log
([#4079](#4079))
([e40e15b](e40e15b))
* support python 3.14
([#3739](#3739))
([7e3be0c](7e3be0c))


### Bug Fixes

* Adapt for dataclasses._MISSING_TYPE replaced with sentinel in Python
3.15 ([#4211](#4211))
([b7fb8df](b7fb8df))
* ensure that storage is not actually retrieved with --touch
([#4212](#4212))
([2726cae](2726cae))
* symlink to directory "has older modification time" than target
([#3782](#3782))
([#3784](#3784))
([41903d8](41903d8))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

symlink to directory "has older modification time" than target

4 participants