Skip to content

Fix @source with folders that are ignored#20214

Merged
RobinMalfait merged 7 commits into
mainfrom
fix/issue-19844
Jun 10, 2026
Merged

Fix @source with folders that are ignored#20214
RobinMalfait merged 7 commits into
mainfrom
fix/issue-19844

Conversation

@RobinMalfait

@RobinMalfait RobinMalfait commented Jun 10, 2026

Copy link
Copy Markdown
Member

This PR fixes an issue where a @source that's pointing to a folder that is git ignored, is also ignored by the @source even if it's explicitly added.

Internally, we convert @source directives from PublicSourceEntrys to SourceEntrys where we have dedicated enum branches for Auto, Pattern, Ignored and External.

The Auto one accepts a base path, and will be used for auto content detection. However, these paths will make use of all the default auto content detection rules, which includes git ignore rules.
We also have External where we link to something that's "external" to the current repo. We can probably improve this name, but it's external in the sense that it won't show up on GitHub for example, aka ignored. We have some content dirs that we ignore by default, such as the node_modules folder. When you do use @source with node_modules in the path, then we will mark it as an external resource which does not look at the gitignore related rules and allowing it to be included this way.

The idea with this is that, even though the folder is ignored by default, you can still include files from the folder by explicitly using the @source directive.

The issue as seen in #19844 is using vendor/ instead of node_modules/ which is not ignored by default. While we can add vendor/ to this same ignored dirs list, it will result in a breaking change because this folder is often used by the Laravel community to store some resources in.

This PR fixes this problem by not only looking at the content dirs we ignore by default, but also looking at the actual git ignore state of this folder. If it turns out that this is ignored, then we promote the Auto source to an External source.

Fixes: #19844
Closes: #20057

Test plan

  1. Added integration tests for this situation
  2. Ran the fix on the reproduction from @source silently ignores paths excluded by an allow-list .gitignore #19844. If we run the CLI with the DEBUG=* environment variable, the log file produces these results:
diff --git a/./tailwindcss-29207.log b/./tailwindcss-30381.log
index bc3017c..921af0e 100644
--- a/./tailwindcss-29207.log
+++ b/./tailwindcss-30381.log
@@ -6,8 +6,9 @@ INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin
 INFO tailwindcss_oxide::scanner: Optimized sources:
 INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme", pattern: "/**/*.phtml" }
 INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme", pattern: "/**/*.xml" }
-INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/vendor/acme/theme" }
+INFO tailwindcss_oxide::scanner: Source: External { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/vendor/acme/theme" }
 INFO tailwindcss_oxide::scanner: Source: Ignored { base: "/Users/robin/.fnm/node-versions/v26.1.0/installation/bin", pattern: "/node" }
 INFO discover_sources: tailwindcss_oxide::scanner: enter
-INFO discover_sources: tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme/index.phtml"
+INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme/index.phtml"
+INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/vendor/acme/theme/module/templates/component.phtml"
 INFO discover_sources: tailwindcss_oxide::scanner: exit

We're checking some .gitignore related files, so let's check on each OS [ci-all]

more tests

WIP

tests
No need to check whether we're dealing with an auto source, or whether
it's ignored when we know that ahead of time.
If the current `base` path for the `SourceEntry::Auto { base }` is git
ignored, then we promote it to a `SourceEntry::External { base }`
instead, bypassing the default git ignore rules.
@RobinMalfait RobinMalfait requested a review from a team as a code owner June 10, 2026 14:09
@greptile-apps

greptile-apps Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to promoting explicitly-referenced gitignored directories from Auto to External, affecting only @source entries that would previously have been silently excluded.

The core fix is well-reasoned: walking ancestors bottom-up, caching gitignore matchers, and using two distinct stop conditions (git-root boundary and cwd boundary for non-git projects) covers the expected scenarios without risking out-of-tree gitignore lookups. The pattern-normalisation refactor in mod.rs is a no-op because optimize() already guarantees a leading /. Tests cover the new path at both unit and integration level.

No files require special attention.

Reviews (2): Last reviewed commit: "ensure we stop at the current working di..." | Re-trigger Greptile


// `Gitignore::new` roots the matcher at the directory
// containing the file, so patterns match relative to it.
path.is_file().then(|| Gitignore::new(&path).0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Silent discard of .gitignore parse errors

Gitignore::new returns (Gitignore, Option<Error>) where the second element reports lines that could not be parsed. Discarding it with .0 means a malformed .gitignore will silently produce an incomplete matcher, potentially failing to promote an Auto source to External when it should be. Logging the error at WARN level (or at least DEBUG) would make diagnosing unexpected behaviour much easier without any cost to the happy path.

@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e4c4cd07-8f8d-4efd-a78b-38c19eb2b965

📥 Commits

Reviewing files that changed from the base of the PR and between 8397869 and 36dd4c4.

📒 Files selected for processing (1)
  • crates/oxide/src/scanner/sources.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/oxide/src/scanner/sources.rs

Hidden review stack artifact

Walkthrough

This PR detects when auto-detected source directories are ignored by default rules or applicable .gitignore files and promotes those entries from Auto to External by walking ancestor directories and caching Gitignore matchers. It also adjusts the walker ignore negation pattern and Pattern construction, adds unit and CLI tests for allow-list .gitignore behavior, updates test utilities (type narrowing, --allow-empty commit, stack-trace capture), and adds a changelog entry.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Fix @source with folders that are ignored' clearly and concisely describes the main change: fixing the handling of @source directives when pointing to git-ignored directories.
Description check ✅ Passed The description is detailed and directly related to the changeset, explaining the problem, solution approach, internal enum changes, and test coverage for fixing @source with ignored folders.
Linked Issues check ✅ Passed The PR fulfills objectives from both linked issues: it promotes Auto sources to External when git-ignored [#19844, #20057], detects allow-list .gitignore files [#20057], adds scanner tests [#20057], and demonstrates the fix works via debug logs on the reproduction case [#19844].
Out of Scope Changes check ✅ Passed All changes are directly related to the fix: Rust scanner logic for gitignore-aware promotion, test additions, and a minor utility update for better error stack traces in tests are all in scope.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

If we can't find a `.git` folder in any of the parents.
@RobinMalfait RobinMalfait merged commit 1bf4291 into main Jun 10, 2026
23 checks passed
@RobinMalfait RobinMalfait deleted the fix/issue-19844 branch June 10, 2026 16:44
RobinMalfait added a commit that referenced this pull request Jun 11, 2026
…s to avoid scanning too many files (#20217)

This PR fixes an issue where we were over-scanning because we lost
pattern information when a `@source` ended in `**/*` and contained other
dynamic parts in the glob pattern.

Noticed this while debugging and working on #20214. Let's say you have
the following:
```css
@source './blog/*/foo/bar/baz/**/*';
```

We make sure that folders or patterns ending in `**/*` are converted to
"auto sources", meaning that auto content detection should be used in
these folders.

However, when doing so, we would only take the `base` path of the glob.
And since we have "dynamic" parts (the `*`) in the pattern, that should
not be the case.

So the `@source` from above, would be turned into:
```rs
SourceEntry::Auto { base = "/Users/projects/project/blog" }
```

Notice that we lose all the information related to `/*/foo/bar/baz/`.
While this would technically still work, it also means that we are
scanning **too many files and folders** because we're only interested in
folders in the `blog` folder that also contain `foo/bar/baz`.

If the `*` wasn't there, then this would be correct, because then we
would've moved the `/blog/foo/bar/baz` part to the `base` ahead of time,
and the pattern would just be `/**/*`. That would result in:
```rs
SourceEntry::Auto { base = "/Users/projects/project/blog/foo/bar/baz" }
```

This PR fixes that, by _not_ converting it to an auto-source, and
instead convert it to a pattern:
```rs
SourceEntry::Pattern {
  base: "/Users/projects/project/blog",
  pattern: "/*/foo/bar/baz/**/*"
}
```

Notice that the static part `blog` is still moved to the base path. But
the rest stays in the `pattern` part as expected.

## Test plan

1. Added a failing test to make sure this doesn't happen anymore
2. Existing tests pass 
3. Tested this on the tailwindcss.com codebase using `@source
"../blog/tailwindcss*/**/*";`

```diff
 diff --git a/./tailwindcss-55526.log b/./tailwindcss-55527.log
index ce79c5da..03e97598 100644
--- a/./tailwindcss-55526.log
+++ b/./tailwindcss-55527.log
@@ -2,50 +2,9 @@ INFO tailwindcss_oxide::scanner: Provided sources:
 INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/app", pattern: "../blog/tailwindcss*/**/*", negated: false }
 INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/.bun/bin", pattern: "bun", negated: true }
 INFO tailwindcss_oxide::scanner: Optimized sources:
-INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog" }
+INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog", pattern: "/tailwindcss*/**/*" }
 INFO tailwindcss_oxide::scanner: Source: Ignored { base: "/Users/robin/.bun/bin", pattern: "/bun" }
 INFO discover_sources: tailwindcss_oxide::scanner: enter
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-05-23-headless-ui-v1-6-tailwind-ui-team-management/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-06-23-tailwind-templates-and-all-access/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-08-17-tailwind-framer-motion-template-and-tailwind-jobs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-09-09-new-personal-website-heroicons-2-headless-ui-v17/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-12-15-protocol-api-documentation-template/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-04-24-new-changelog-template-and-the-biggest-tailwind-ui-update-ever/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-07-18-tailwind-connect-2023-recap/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-08-07-meet-studio-our-new-agency-template/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-05-24-catalyst-application-layouts/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-05-30-prettier-plugin-collapse-whitespace/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-06-21-headless-ui-v2-1/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-09-12-radiant-a-beautiful-new-marketing-site-template/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2025-05-14-compass-course-starter-kit/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/automatic-class-sorting-with-prettier/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/building-react-and-vue-support-for-tailwind-ui/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/building-the-tailwind-blog/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/designing-tailwind-ui-ecommerce/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/from-900-to-1-how-we-hired-robin-malfait/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-unstyled-accessible-ui-components/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-4/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-5/demo.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-5/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/HeadlessUIV2Examples.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/StateAttributesExample.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/anchor-positioning.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/heroicons-micro/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/heroicons-v1/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/hiring-a-design-engineer-and-staff-engineer/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-catalyst/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-heroicons/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-linting-for-tailwindcss-intellisense/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-tailwind-play/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/just-in-time-the-next-generation-of-tailwind-css/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/multi-line-truncation-with-tailwindcss-line-clamp/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/simon-vrachliotis-joins-tailwind-labs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/standalone-cli/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-plus/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-ui-ecommerce/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-ui-now-with-react-and-vue-support/index.mdx"
 INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-5/index.mdx"
 INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-6/index.mdx"
 INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-7/index.mdx"
@@ -68,10 +27,4 @@ INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/t
 INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4-beta/index.mdx"
 INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4/color-palette.tsx"
 INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/utility-friendly-transitions-with-tailwindui-react/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/vanilla-js-support-for-tailwind-plus/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-brad-cornes-to-the-tailwind-team/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-david-luhr-to-tailwind-labs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-james-mcdonald-to-tailwind-labs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/whats-new-in-tailwindcss-on-youtube/index.mdx"
 INFO discover_sources: tailwindcss_oxide::scanner: exit
```

Notice now that a lot of files are skipped as expected since we-re not
accidentally over-scanning now.

[ci-all]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

@source silently ignores paths excluded by an allow-list .gitignore

1 participant