feat: loki.source.file file_match block by kalleep · Pull Request #4743 · grafana/alloy

kalleep · 2025-10-31T09:57:05Z

PR Description

This PR adds a new file_match block to loki.source.file that enables built-in file discovery using glob patterns. This allows users to discover files directly within loki.source.file without needing a separate local.file_match component.

If file_match block is not enabled component behaves like it did before.

If this block enabled local.file_match is no longer needed and we avoid the overhead of waiting for alloy graph updates before we can schedule new files to tail.

Which issue(s) this PR fixes

Notes to the Reviewer

I added "resolver" interface that will change depending on configuration. The static one behaves exactly like before, requiring absolute paths and the glob implementation will perform globing with doublestar.

The watcher will do nothing when file_match is not configured.

I also changed to use nop logger in test, logging in tests adds additional noise that makes it hard to see test output. Maybe we should be able to control logging for test with an env variable instead?

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

github-actions · 2025-10-31T09:58:47Z

💻 Deploy preview available (Kalleep/loki source file match):

https://deploy-preview-alloy-4743-zb444pucvq-vp.a.run.app/docs/alloy/latest/

Copilot

Pull Request Overview

This PR introduces built-in file discovery capabilities to loki.source.file component through a new file_match block. The feature allows glob pattern-based file discovery directly within the component as an alternative to using the separate local.file_match component.

Key changes:

Added file_match configuration block with glob pattern support using the doublestar library
Implemented a resolver abstraction (static vs. glob) to handle different path resolution strategies
Added a watcher mechanism to periodically sync filesystem state when file matching is enabled

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`internal/component/loki/source/file/watcher.go`	New watcher component for periodic filesystem synchronization
`internal/component/loki/source/file/resolver.go`	New resolver abstraction with static and glob implementations
`internal/component/loki/source/file/resolver_test.go`	Unit tests for resolver implementations
`internal/component/loki/source/file/file.go`	Core component changes to integrate file matching and resolver system
`internal/component/loki/source/file/file_test.go`	Test infrastructure updates to validate both file match modes
`internal/component/loki/source/file/*_test.go`	Updated test utilities to use new logging API
`docs/sources/reference/components/loki/loki.source.file.md`	Documentation for new file_match block with examples
`docs/sources/reference/components/local/local.file_match.md`	Updated to recommend built-in file_match for simple cases
`CHANGELOG.md`	Added feature entry

internal/component/loki/source/file/file.go

internal/component/loki/source/file/watcher.go

internal/component/loki/source/file/resolver_test.go

internal/component/loki/source/file/watcher.go

internal/component/loki/source/file/file.go

github-actions · 2025-10-31T10:17:54Z

💻 Deploy preview available (faet: loki.source.file file_match block):

https://deploy-preview-alloy-4743-zb444pucvq-vp.a.run.app/docs/alloy/latest/

docs/sources/reference/components/local/local.file_match.md

Copilot

Pull Request Overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

internal/component/loki/source/file/file.go

docs/sources/reference/components/loki/loki.source.file.md

thampiotr · 2025-10-31T10:43:51Z

docs/sources/reference/components/local/local.file_match.md

 `local.file_match` discovers files on the local filesystem using glob patterns and the [doublestar][] library.

+{{< admonition type="note" >}}
+For simple use cases where you only need to discover files for a single `loki.source.file` component, consider using the built-in [`file_match`](../../loki/loki.source.file/#file_match) block in `loki.source.file` instead.


I think it's also better option for very large numbers of files that are frequently changing?

Perhaps we should recommend to always use the built-in block and only use local.file_match for things that are not possible with the built-in block. That way we ensure people get maximum correctness & performance.

Should we try create a list of use-cases that are not possible with built-in block?

Yes there is examples with file globing using either builtin or local.file_match. There I added this header
### File globbing with file_match (recommended)

Do you think we should make it clearer here?

Yes, I think we need to make it more clear when to use which option. And probably the logic should be to use the built-in file_match whenever you can and a separate component for cases that are not possible with built-in (we need to list a few of them)

thampiotr · 2025-10-31T10:44:54Z

docs/sources/reference/components/loki/loki.source.file.md

+- Better performance: File discovery is integrated directly into `loki.source.file`, eliminating the overhead of component-to-component communication
+- Same functionality: Supports the same glob patterns and features, including `__path_exclude__` for excluding files
+
+When `enabled` is set to `true`, you can use glob patterns (e.g., `/tmp/*.log` or `/var/log/**/*.log`) directly in the `targets` argument's `__path__` label.


What happens when enabled is false?

Then we require absolute file paths, like this component behaved before this pr

Can we write that in docs?

We already have it here https://github.com/grafana/alloy/pull/4743/files#diff-4b8c6b16f7e932bea249731a9f49d947024699eb129a37f0846aed9c1086e945R21-R23.

Do you think we need it in both places?

Changed this in 61c3e8c

I listed the use cases I could think of for local.file_match

docs/sources/reference/components/loki/loki.source.file.md

thampiotr · 2025-10-31T10:46:38Z

docs/sources/reference/components/loki/loki.source.file.md

+The `file_match` block enables built-in file discovery directly within `loki.source.file`.
+When enabled, the component discovers files on the local filesystem using glob patterns and the [doublestar][] library.


What happens when someone provides file_match and also sends targets in using the previous pattern?

I think that should ideally be an error, unless there's a simple way to make both work.

It will work but you get some overhead because now you are globing and syncing twice. Because all paths will match directly to one file and that is fine to pass to Glob

github-actions · 2025-10-31T12:59:13Z

💻 Deploy preview deleted (feat: loki.source.file file_match block).

Copilot

Pull Request Overview

Copilot reviewed 36 out of 36 changed files in this pull request and generated 3 comments.

internal/component/loki/source/file/file.go

internal/converter/internal/promtailconvert/testdata/azure.alloy

thampiotr · 2025-10-31T14:11:48Z

docs/sources/reference/components/loki/loki.source.file.md

+|---------------------|---------------------|-------------------------------------------------------------|---------|----------|
+| `enabled`           | `bool`              | Enable file discovery using glob patterns.                  | `false` | no       |
+| `ignore_older_than` | `duration`          | Ignores files with modification times before this duration. | `"0s"`  | no       |
+| `sync_period`       | `duration`          | How often to sync filesystem and targets.                   | `"10s"` | no       |


is that a good default? we have a chance to change it

I think the defaults are good, this is the same defaults that are using in promtail too.

docs/sources/reference/components/local/local.file_match.md

docs/sources/reference/components/loki/loki.source.file.md

thampiotr

Thanks, excited for this to get merged! :)
Just a few comments so far.

thampiotr · 2025-11-04T12:25:22Z

internal/component/loki/source/file/file_test.go

There is a //go:build !race on top of this file.

Are we sure this test runs in CI? it's not obvious to me right now

Could we get rid of this and fix the race conditions, since we done some improvements recently?

thampiotr · 2025-11-04T12:29:36Z

internal/component/loki/source/file/file.go

+		c.resolver = newStaticResolver()
+	}
+
+	if newArgs.FileMatch.SyncPeriod != c.args.FileMatch.SyncPeriod && newArgs.FileMatch.SyncPeriod < 0 {


Suggested change

if newArgs.FileMatch.SyncPeriod != c.args.FileMatch.SyncPeriod && newArgs.FileMatch.SyncPeriod < 0 {

if newArgs.FileMatch.SyncPeriod != c.args.FileMatch.SyncPeriod && newArgs.FileMatch.SyncPeriod > 0 {

Although I'm not sure if that can even happen? Because default is not 0, right?
maybe we should think what to do with c.watcher when we disable FileMatch separately.

internal/component/loki/source/file/file.go

thampiotr · 2025-11-04T12:33:37Z

internal/component/loki/source/file/file.go

+				if !c.args.FileMatch.Enabled {
+					return
+				}


So if we disable FileMatch and then enable FileMatch, I think the goroutine will not be restarted... can we add a test for this and fix the issue?

We have two choices: always keep the goroutines and watcher ticker on and check for FileMatch enabled - a bit more expensive but a simpler code. Or we schedule them only when FileMatch is enabled - a bit more complex, but fewer resources used. I don't have a strong opinion, I would like this code to be easier to follow though...

Good catch! My intention was to keep it running regardless, so we should not return here.

And only because it's way more simple to do that then having to manage it in between Updates

thampiotr · 2025-11-04T12:37:12Z

docs/sources/reference/components/local/local.file_match.md

 `local.file_match` discovers files on the local filesystem using glob patterns and the [doublestar][] library.

+{{< admonition type="note" >}}
+Use the built-in [`file_match`](../../loki/loki.source.file/#file_match) block in `loki.source.file` whenever possible.


We might want to reach out to k8s helm charts maintainers to change their defaults if they have them.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

kalleep requested review from a team and clayton-cornell as code owners October 31, 2025 09:57

kalleep changed the title ~~Kalleep/loki source file match~~ faet: loki.source.file file_match block Oct 31, 2025

kalleep requested review from a team, clayton-cornell and Copilot and removed request for a team and clayton-cornell October 31, 2025 10:03

Copilot AI reviewed Oct 31, 2025

View reviewed changes

kalleep requested a review from Copilot October 31, 2025 10:32

thampiotr reviewed Oct 31, 2025

View reviewed changes

docs/sources/reference/components/local/local.file_match.md Outdated Show resolved Hide resolved

Copilot AI reviewed Oct 31, 2025

View reviewed changes

internal/component/loki/source/file/file.go Outdated Show resolved Hide resolved

docs/sources/reference/components/loki/loki.source.file.md Outdated Show resolved Hide resolved

thampiotr reviewed Oct 31, 2025

View reviewed changes

docs/sources/reference/components/loki/loki.source.file.md Show resolved Hide resolved

thampiotr reviewed Oct 31, 2025

View reviewed changes

kalleep changed the title ~~faet: loki.source.file file_match block~~ feat: loki.source.file file_match block Oct 31, 2025

kalleep mentioned this pull request Oct 31, 2025

Converters: change to use file_match block in loki.source.file #4744

Closed

kalleep requested a review from Copilot October 31, 2025 13:03

Copilot AI reviewed Oct 31, 2025

View reviewed changes

internal/component/loki/source/file/file.go Show resolved Hide resolved

internal/component/loki/source/file/file.go Outdated Show resolved Hide resolved

internal/component/loki/source/file/file.go Show resolved Hide resolved

thampiotr reviewed Oct 31, 2025

View reviewed changes

internal/converter/internal/promtailconvert/testdata/azure.alloy Outdated Show resolved Hide resolved

thampiotr reviewed Oct 31, 2025

View reviewed changes

clayton-cornell reviewed Oct 31, 2025

View reviewed changes

kalleep force-pushed the kalleep/loki-source-file-match branch from 0e6b45a to ba28b05 Compare November 3, 2025 07:53

kalleep mentioned this pull request Nov 3, 2025

fix: add globbing support in loki.source.file #4702

Closed

4 tasks

kalleep requested a review from thampiotr November 3, 2025 13:38

thampiotr reviewed Nov 4, 2025

View reviewed changes

kalleep and others added 23 commits November 5, 2025 10:58

store pointer to ticker

12f2e1e

Skip updates if nothing have changed

bdb2b21

Update docs/sources/reference/components/loki/loki.source.file.md

c990a26

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @thampiotr

cd1c7f8

Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>

simplify watcher

6aea1c5

Update converter tests

44b2822

fmt

bb59083

fixes

4a0df20

ensure correct tick time

e7fea75

Set defaults

051b0ae

fix convert test

0e47f06

Fix test

dff83a4

Update docs

255602f

Set correct default

2d37ab7

Update docs/sources/reference/components/loki/loki.source.file.md

dfd2020

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Update docs/sources/reference/components/loki/loki.source.file.md

3cda448

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Update docs/sources/reference/components/local/local.file_match.md

ca8367f

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Update docs/sources/reference/components/local/local.file_match.md

91c54f4

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Update docs/sources/reference/components/loki/loki.source.file.md

5658564

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

fix watcher

9b0cbd5

Add validation

af72b3e

Fix tests

29f5cde

Remove build tag and fix race in test

28814a9

kalleep force-pushed the kalleep/loki-source-file-match branch from b56ee4b to 28814a9 Compare November 5, 2025 09:59

Fix imports

8df8f5f

kalleep merged commit 16e0e21 into main Nov 6, 2025
43 checks passed

kalleep deleted the kalleep/loki-source-file-match branch November 6, 2025 09:26

kalleep mentioned this pull request Nov 6, 2025

feat: update converter to use file_match block #4791

Merged

4 tasks

github-actions bot added the frozen-due-to-age label Dec 7, 2025

github-actions bot locked as resolved and limited conversation to collaborators Dec 7, 2025

		The `file_match` block enables built-in file discovery directly within `loki.source.file`.
		When enabled, the component discovers files on the local filesystem using glob patterns and the [doublestar][] library.

	if newArgs.FileMatch.SyncPeriod != c.args.FileMatch.SyncPeriod && newArgs.FileMatch.SyncPeriod < 0 {
	if newArgs.FileMatch.SyncPeriod != c.args.FileMatch.SyncPeriod && newArgs.FileMatch.SyncPeriod > 0 {

Conversation

kalleep commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

Uh oh!

github-actions bot commented Oct 31, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kalleep Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kalleep Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thampiotr left a comment

Choose a reason for hiding this comment

Uh oh!

kalleep commented Oct 31, 2025 •

edited

Loading

github-actions bot commented Oct 31, 2025 •

edited

Loading

kalleep Oct 31, 2025 •

edited

Loading

kalleep Oct 31, 2025 •

edited

Loading

github-actions bot commented Oct 31, 2025 •

edited

Loading