Skip to content

Fix listFiles() to follow symbolic links#6808

Merged
bentsherman merged 2 commits intomasterfrom
fix/listfiles-follow-symlinks
Feb 6, 2026
Merged

Fix listFiles() to follow symbolic links#6808
bentsherman merged 2 commits intomasterfrom
fix/listfiles-follow-symlinks

Conversation

@adamrtalbot
Copy link
Collaborator

Summary

  • Fix listFiles() to properly follow symbolic links by adding FileVisitOption.FOLLOW_LINKS to the walkFileTree call
  • Add test case for listing files in a symlinked directory

Problem

The listFiles() method was not following symbolic links because Files.walkFileTree() defaults to not following them. This caused listFiles() to fail to locate files when:

  1. The directory itself is a symlink
  2. Files within the directory are symlinks

A workaround using toRealPath() was suggested, but this doesn't work for cloud storage (S3 throws UnsupportedOperationException, Azure just returns toAbsolutePath()).

Solution

Added FileVisitOption.FOLLOW_LINKS to the walkFileTree call in listFiles0(). This is safe for cloud filesystems since:

  1. Cloud storage doesn't have symlinks, so FOLLOW_LINKS is effectively a no-op
  2. The S3 client already uses EnumSet.of(FileVisitOption.FOLLOW_LINKS) internally for downloads

Fixes #6807

Test plan

  • Added test case should list files in symlinked directory
  • All existing FilesExTest tests pass

🤖 Generated with Claude Code

The listFiles() method was not following symbolic links because
Files.walkFileTree() defaults to not following them. This caused
listFiles() to fail to locate files when called on a symlinked
directory.

Added FileVisitOption.FOLLOW_LINKS to the walkFileTree call to
ensure symlinks are properly resolved.

Fixes #6807

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@netlify
Copy link

netlify bot commented Feb 5, 2026

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit c6af95a
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/698606f63a18a40008d8b47b

@adamrtalbot
Copy link
Collaborator Author

adamrtalbot commented Feb 5, 2026

Testing:

#!/usr/bin/env nextflow

params.dir = "$projectDir/test_data"

workflow {
    channel.of(file(params.dir))
        .map { dir -> dir.listFiles() }
        .flatten()
        .view { "Found: ${it.name}" }
}
mkdir -p test_data_real                                                                           
echo "hello" > test_data_real/file.txt                                 
ln -s test_data_real test_data

echo "hello" > file_real.txt 
ln -s file_real.txt test_data/file_link.txt

Nextflow 25.10.3:

>  nextflow run test-symlinks.nf
N E X T F L O W  ~  version 25.10.3
Launching `test-symlinks.nf` [romantic_poitras] DSL2 - revision: dc961e3873
Found: test_data

This branch:

>  make compile && ./launch.sh run test-symlinks.nf
./gradlew compile exportClasspath
<redacted>
DONE Thu Feb  5 10:39:28 GMT 2026

N E X T F L O W  ~  version 25.12.0-edge
Launching `test-symlinks.nf` [pedantic_tuckerman] - revision: dc961e3873
Found: file.txt
Found: file_link.txt

@adamrtalbot
Copy link
Collaborator Author

adamrtalbot commented Feb 5, 2026

For completion, here's using the new listDirectory():

workflow {
    channel.of(file(params.dir))
        .map { dir -> dir.listDirectory() }
        .flatten()
        .view { "Found: ${it.name}" }
}

Current master:

N E X T F L O W  ~  version 25.12.0-edge
Launching `test-symlinks.nf` [romantic_boyd] - revision: a10ccfe3f8
Found: test_data

This branch:

N E X T F L O W  ~  version 25.12.0-edge
Launching `test-symlinks.nf` [mighty_mcclintock] - revision: a10ccfe3f8
Found: file.txt
Found: file_link.txt

@adamrtalbot
Copy link
Collaborator Author

@bentsherman bentsherman merged commit c8ff0ce into master Feb 6, 2026
23 checks passed
@bentsherman bentsherman deleted the fix/listfiles-follow-symlinks branch February 6, 2026 15:35
ewels pushed a commit to ewels/nextflow that referenced this pull request Feb 6, 2026
…w-io#6808)

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
pditommaso added a commit that referenced this pull request Feb 8, 2026
@pditommaso
Copy link
Member

Folks, i've reverted this PR. The change looks harmless at first glance but has a wider blast radius than expected.

Affected production callers

  • FilesEx.createLinkImpl — enumerates a directory to create hard links recursively.
  • LocalUpdateRepository — lists plugin directories.
  • Session.findBinEntries — discovers executable scripts in the pipeline bin/ directory.

Task hash impact

findBinEntries populates the bin entries that are factored into task hashing. If FOLLOW_LINKS changes which files are returned (or how symlinked entries are classified), this silently invalidates cached task results — users would see unexpected cache misses with no explanation.

This is a semantic difference that could introduce bugs both in internal code and in user pipelines calling .listFiles() as a public API. Tho I understand that's a desirable patch, at this point it would be better to provide an option to control it.

@adamrtalbot
Copy link
Collaborator Author

Thanks @pditommaso, well spotted.

@bentsherman
Copy link
Member

That's fine, we can just apply this change to listDirectory() since it is new in 26.04

@adamrtalbot please re-submit this PR, making listFiles0 parameterized so that listFiles calls it without follow_links and listDirectory calls it with follow_links.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

listFiles() does not follow symbolic links

3 participants