Fix @source with folders that are ignored#20214
Conversation
more tests WIP tests
No need to check whether we're dealing with an auto source, or whether it's ignored when we know that ahead of time.
If the current `base` path for the `SourceEntry::Auto { base }` is git
ignored, then we promote it to a `SourceEntry::External { base }`
instead, bypassing the default git ignore rules.
Confidence Score: 5/5Safe to merge — the change is narrowly scoped to promoting explicitly-referenced gitignored directories from Auto to External, affecting only @source entries that would previously have been silently excluded. The core fix is well-reasoned: walking ancestors bottom-up, caching gitignore matchers, and using two distinct stop conditions (git-root boundary and cwd boundary for non-git projects) covers the expected scenarios without risking out-of-tree gitignore lookups. The pattern-normalisation refactor in mod.rs is a no-op because optimize() already guarantees a leading /. Tests cover the new path at both unit and integration level. No files require special attention. Reviews (2): Last reviewed commit: "ensure we stop at the current working di..." | Re-trigger Greptile |
|
|
||
| // `Gitignore::new` roots the matcher at the directory | ||
| // containing the file, so patterns match relative to it. | ||
| path.is_file().then(|| Gitignore::new(&path).0) |
There was a problem hiding this comment.
Silent discard of
.gitignore parse errors
Gitignore::new returns (Gitignore, Option<Error>) where the second element reports lines that could not be parsed. Discarding it with .0 means a malformed .gitignore will silently produce an incomplete matcher, potentially failing to promote an Auto source to External when it should be. Logging the error at WARN level (or at least DEBUG) would make diagnosing unexpected behaviour much easier without any cost to the happy path.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Hidden review stack artifactWalkthroughThis PR detects when auto-detected source directories are ignored by default rules or applicable .gitignore files and promotes those entries from Auto to External by walking ancestor directories and caching Gitignore matchers. It also adjusts the walker ignore negation pattern and Pattern construction, adds unit and CLI tests for allow-list .gitignore behavior, updates test utilities (type narrowing, --allow-empty commit, stack-trace capture), and adds a changelog entry. 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
If we can't find a `.git` folder in any of the parents.
…s to avoid scanning too many files (#20217) This PR fixes an issue where we were over-scanning because we lost pattern information when a `@source` ended in `**/*` and contained other dynamic parts in the glob pattern. Noticed this while debugging and working on #20214. Let's say you have the following: ```css @source './blog/*/foo/bar/baz/**/*'; ``` We make sure that folders or patterns ending in `**/*` are converted to "auto sources", meaning that auto content detection should be used in these folders. However, when doing so, we would only take the `base` path of the glob. And since we have "dynamic" parts (the `*`) in the pattern, that should not be the case. So the `@source` from above, would be turned into: ```rs SourceEntry::Auto { base = "/Users/projects/project/blog" } ``` Notice that we lose all the information related to `/*/foo/bar/baz/`. While this would technically still work, it also means that we are scanning **too many files and folders** because we're only interested in folders in the `blog` folder that also contain `foo/bar/baz`. If the `*` wasn't there, then this would be correct, because then we would've moved the `/blog/foo/bar/baz` part to the `base` ahead of time, and the pattern would just be `/**/*`. That would result in: ```rs SourceEntry::Auto { base = "/Users/projects/project/blog/foo/bar/baz" } ``` This PR fixes that, by _not_ converting it to an auto-source, and instead convert it to a pattern: ```rs SourceEntry::Pattern { base: "/Users/projects/project/blog", pattern: "/*/foo/bar/baz/**/*" } ``` Notice that the static part `blog` is still moved to the base path. But the rest stays in the `pattern` part as expected. ## Test plan 1. Added a failing test to make sure this doesn't happen anymore 2. Existing tests pass 3. Tested this on the tailwindcss.com codebase using `@source "../blog/tailwindcss*/**/*";` ```diff diff --git a/./tailwindcss-55526.log b/./tailwindcss-55527.log index ce79c5da..03e97598 100644 --- a/./tailwindcss-55526.log +++ b/./tailwindcss-55527.log @@ -2,50 +2,9 @@ INFO tailwindcss_oxide::scanner: Provided sources: INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/app", pattern: "../blog/tailwindcss*/**/*", negated: false } INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/.bun/bin", pattern: "bun", negated: true } INFO tailwindcss_oxide::scanner: Optimized sources: -INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog" } +INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog", pattern: "/tailwindcss*/**/*" } INFO tailwindcss_oxide::scanner: Source: Ignored { base: "/Users/robin/.bun/bin", pattern: "/bun" } INFO discover_sources: tailwindcss_oxide::scanner: enter -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-05-23-headless-ui-v1-6-tailwind-ui-team-management/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-06-23-tailwind-templates-and-all-access/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-08-17-tailwind-framer-motion-template-and-tailwind-jobs/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-09-09-new-personal-website-heroicons-2-headless-ui-v17/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-12-15-protocol-api-documentation-template/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-04-24-new-changelog-template-and-the-biggest-tailwind-ui-update-ever/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-07-18-tailwind-connect-2023-recap/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-08-07-meet-studio-our-new-agency-template/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-05-24-catalyst-application-layouts/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-05-30-prettier-plugin-collapse-whitespace/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-06-21-headless-ui-v2-1/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-09-12-radiant-a-beautiful-new-marketing-site-template/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2025-05-14-compass-course-starter-kit/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/automatic-class-sorting-with-prettier/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/building-react-and-vue-support-for-tailwind-ui/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/building-the-tailwind-blog/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/designing-tailwind-ui-ecommerce/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/from-900-to-1-how-we-hired-robin-malfait/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-unstyled-accessible-ui-components/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-4/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-5/demo.tsx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-5/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/HeadlessUIV2Examples.tsx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/StateAttributesExample.tsx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/anchor-positioning.tsx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/heroicons-micro/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/heroicons-v1/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/hiring-a-design-engineer-and-staff-engineer/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-catalyst/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-heroicons/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-linting-for-tailwindcss-intellisense/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-tailwind-play/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/just-in-time-the-next-generation-of-tailwind-css/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/multi-line-truncation-with-tailwindcss-line-clamp/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/simon-vrachliotis-joins-tailwind-labs/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/standalone-cli/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-plus/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-ui-ecommerce/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-ui-now-with-react-and-vue-support/index.mdx" INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-5/index.mdx" INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-6/index.mdx" INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-7/index.mdx" @@ -68,10 +27,4 @@ INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/t INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4-beta/index.mdx" INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4/color-palette.tsx" INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/utility-friendly-transitions-with-tailwindui-react/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/vanilla-js-support-for-tailwind-plus/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-brad-cornes-to-the-tailwind-team/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-david-luhr-to-tailwind-labs/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-james-mcdonald-to-tailwind-labs/index.mdx" -INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/whats-new-in-tailwindcss-on-youtube/index.mdx" INFO discover_sources: tailwindcss_oxide::scanner: exit ``` Notice now that a lot of files are skipped as expected since we-re not accidentally over-scanning now. [ci-all]
This PR fixes an issue where a
@sourcethat's pointing to a folder that is git ignored, is also ignored by the@sourceeven if it's explicitly added.Internally, we convert
@sourcedirectives fromPublicSourceEntrys toSourceEntrys where we have dedicated enum branches forAuto,Pattern,IgnoredandExternal.The
Autoone accepts abasepath, and will be used for auto content detection. However, these paths will make use of all the default auto content detection rules, which includes git ignore rules.We also have
Externalwhere we link to something that's "external" to the current repo. We can probably improve this name, but it's external in the sense that it won't show up on GitHub for example, aka ignored. We have some content dirs that we ignore by default, such as thenode_modulesfolder. When you do use@sourcewithnode_modulesin the path, then we will mark it as anexternalresource which does not look at thegitignorerelated rules and allowing it to be included this way.The idea with this is that, even though the folder is ignored by default, you can still include files from the folder by explicitly using the
@sourcedirective.The issue as seen in #19844 is using
vendor/instead ofnode_modules/which is not ignored by default. While we can addvendor/to this same ignored dirs list, it will result in a breaking change because this folder is often used by the Laravel community to store some resources in.This PR fixes this problem by not only looking at the content dirs we ignore by default, but also looking at the actual git ignore state of this folder. If it turns out that this is ignored, then we promote the
Autosource to anExternalsource.Fixes: #19844
Closes: #20057
Test plan
@sourcesilently ignores paths excluded by an allow-list.gitignore#19844. If we run the CLI with theDEBUG=*environment variable, the log file produces these results:We're checking some
.gitignorerelated files, so let's check on each OS [ci-all]