Overview

Relevant issue: #528

Current status:

• In python, when you exclude a dir, it still needs to list all files in that dir
• This is optimized with --experimental and here we try to achieve similar behaviour

Tests

Tested with

pytest tests/default/unit/targeting/test_target_manager.py tests/default/e2e/test_target_selection.py -v

which seem sufficiently comprehensive

Evaluation of performance

The env for testing is set up using the script atttached to #528, in particular:

$ ls -a
.             ..            .m2           HelloWorld.kt

$ find .m2 | wc -l
  280004

Comapring the result. All commands are opengrep scan -c $RULES ..., where $RULES point to a local copy of the semgrep rule repo.

Implementation details

Problem

When excluding large directories (e.g. node_modules, build), the Python frontend was slow and still reported tens/hundreds of thousands of skipped files in the scan summary. The root cause was that
Target.files_from_filesystem() uses path.glob("**/*"), which expands the entire directory tree before any filtering occurs. filter_excludes and FileIgnore.filter_paths then pattern-match against every
single collected file — O(N) in the number of files inside excluded directories.

By contrast, osemgrep (src/targeting/Find_targets.ml) checks each directory against exclusion patterns before descending, making excluded directories O(1) regardless of their size.

Solution

Add directory-level early pruning to the Python filesystem walker, matching osemgrep's behavior, with identical scan semantics.

Target.files_from_filesystem_with_dir_pruning (new method)

Replaces path.glob("**/*") with os.walk(topdown=True, followlinks=False). Before descending into each subdirectory, two checks are applied in-place on dirnames:

1. --exclude patterns via wcglob.globfilter — patterns are passed in already preprocessed (e.g. node_modules → **/node_modules, **/node_modules/**) so the preprocessing cost is paid once per scan.
   Paths are expressed relative to the scan root (not absolute) so that **/-prefixed wcmatch patterns match correctly.

2. .semgrepignore patterns via FileIgnore._survives(dir / ".__check__") — checking a virtual sentinel file inside the directory (rather than the directory path itself) correctly matches both dir
   patterns (via the generated dir/** fnmatch pattern) and dir/ folder patterns. _survives is pure pattern-matching with no filesystem access, so this is safe with non-existent paths.

TargetManager.get_all_files_with_dir_pruning (new method)

Preprocesses exclude patterns once, then iterates over targets mirroring the exact fallback chain of Target.files() — substituting files_from_filesystem_with_dir_pruning wherever files_from_filesystem
would have been called:

The critical fix for the default scan mode: previously when files_from_git_ls() failed (non-git directory or git unavailable), it silently fell back to the original files_from_filesystem() with no pruning.
Now it falls back to files_from_filesystem_with_dir_pruning.

Decorated with @lru_cache so the filesystem is walked at most once per unique (exclude_patterns, ignore_baseline_handler, file_ignore) combination.

TargetManager.get_files_for_language (modified)

Replaces self.get_all_files(ignore_baseline_handler) with a call to get_all_files_with_dir_pruning, passing:

• --exclude patterns for the current product + PATHS_ALWAYS_SKIPPED (.git)
• The product's FileIgnore object (if respect_semgrepignore is set)

The existing filter_excludes and filter_paths calls remain as safety nets for file-level patterns (e.g. --exclude "*.pyc") and for results from git-based traversals.