Introduce Shared YAML Dependency File To Manage Replace Directives by blewis12 · Pull Request #4862 · grafana/alloy

blewis12 · 2025-11-17T07:54:02Z

Part of #4719

This is an initial implementation taken from the PoC for Proposal 2 from this design doc

What is included

A basic generate-module-dependencies tool in the Alloy project that can be run with make generate-module-dependencies
The tool will read the dependency-replacements.yaml file located at the root of the project, transform the data, and apply it to the main go.mod file for Alloy. Note that for the time being this YAML file should not contain local replacements (e.g. we are not including the syntax module in that file in this PR)
The output of running the tool is included in this PR - it should not be changing any replace directives in Alloy's go.mod, but it is formatting them a bit due to the generation template being structured
There is also a GHA that will check the output of running make generate-module-dependencies and check that nothing is changed in the go.mod file in pipelines, this is to ensure that the state between the YAML replacements and go.mod file(s) are aligned
There are also some basic E2E tests

What is the intention

I'd like to merge this change in to understand the ergonomics of using this workflow in practice. Ideally it should not change much, and should make the addition of extra modules in the future more frictionless - but i'd like to gather some feedback

What will be done as follow ups

The current implementation only deals with go.mod file generation, not ocb yaml files - the latter will be introduced in a follow up PR. I've tried to keep things as "simple" as possible for now

.github/workflows/check-sync-module-dependencies.yml

github-actions · 2025-11-24T08:23:20Z

🔍 Dependency Review

No dependency version changes detected in go.mod.

No require entries were added, removed, or changed in version.
No replaces changed versions; they were reorganized and normalized into a generated block, but versions stayed the same.
An existing local replace for github.com/grafana/alloy/syntax => ./syntax was moved and annotated, but it already existed in the previous go.mod and does not change behavior.
The existing k8s client-go exclude directives were repositioned, not changed.

As a result, there are no dependency upgrades to analyze and no code changes required at this time.

Notes

This PR introduces a generator tool (tools/generate-module-dependencies) and CI workflow to centrally manage replace directives via dependency-replacements.yaml. The generated block is bounded by:
- BEGIN GENERATED REPLACES - DO NOT EDIT MANUALLY
- END GENERATED REPLACES
  Do not edit within these markers; update dependency-replacements.yaml instead and re-run the generator.
The tool’s own module (tools/generate-module-dependencies/go.mod) adds new dependencies (e.g., cobra, yaml.v3) within a separate module. Per instructions, net-new dependencies in a separate tool module were not assessed since they do not alter the main module’s dependency graph.

blewis12 · 2025-12-08T14:10:59Z

Thanks for the review so far @kgeckhart!

GHA will ensure things have been regenerated if someone changes dependency-replayments.yaml

Exactly, it won't do the generation itself but it will fail if it detects that the output of the generation command does not match the existing state

We do not enforce that all replaces have to be made via depedency-replacements.yaml

Yes, technically a user could put replace manually inside the go.mod file. This is currently valid for local replacements, since we don't want these to be duplicated, but we could also build on top of the existing tool to do further checks that we don't have errant replace directives

If a user adds or edits a replace directive manually within the template boundaries though, the CI will fail

thampiotr

Just a few minor comments, but once resolved I think this is ready.

tools/generate-module-dependencies/internal/helpers/files.go

tools/generate-module-dependencies/README.md

tools/generate-module-dependencies/internal/apply-replaces.go

tools/generate-module-dependencies/README.md

dependency-replacements.yaml

Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>

Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

thampiotr

LGTM!

…roperly in workflow and use git diff instead so we can ignore whitespace changes

…rafana#4862) * Introduce new dependency management tool * Move syntax replace to top of go.mod to avoid confusion * Apply replaces from tool * Introduce PR workflow to check modules are synced * Add E2E test * Address some PR comments Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Refactor to use a centralised cobra CLI, called by generate.go * Refactor to not generate .txt files and pass replace text in memory * Refactor E2E tests to use explicit test data * Update tools/generate-module-dependencies/replaces-mod.tpl Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com> * Return errors in FileHelper instead of using Fatalf * Address PR comments * go mod/mod changes * Reference actual calculated file to dependency yaml file in error logs * Group the generate-module-dependencies and Check for go.mod changes properly in workflow and use git diff instead so we can ignore whitespace changes * nit: a few improvements for clarity --------- Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

blewis12 force-pushed the yaml-dependency-file-overtop-alloy-poc branch 5 times, most recently from 4095548 to 4eed3b6 Compare November 17, 2025 13:34

blewis12 mentioned this pull request Nov 20, 2025

[otel-engine] Dependencies, Modules, Folders, Commands #4719

Closed

blewis12 force-pushed the yaml-dependency-file-overtop-alloy-poc branch 3 times, most recently from 92c493d to 903bfcc Compare November 23, 2025 17:43

github-advanced-security bot found potential problems Nov 23, 2025

View reviewed changes

.github/workflows/check-sync-module-dependencies.yml Fixed Show fixed Hide fixed

blewis12 force-pushed the yaml-dependency-file-overtop-alloy-poc branch 10 times, most recently from a28eeea to 2fd651c Compare November 23, 2025 21:57

blewis12 changed the title ~~PoC: Yaml Dependency File To Manage Replace Directives~~ Introduce Shared YAML Dependency File To Manage Replace Directives Nov 23, 2025

blewis12 force-pushed the yaml-dependency-file-overtop-alloy-poc branch 4 times, most recently from 4691655 to 843ea22 Compare November 24, 2025 08:13

blewis12 marked this pull request as ready for review November 24, 2025 08:21

blewis12 requested a review from a team as a code owner November 24, 2025 08:21

blewis12 requested a review from Copilot November 24, 2025 08:22

Copilot started reviewing on behalf of blewis12 November 24, 2025 08:22 View session

blewis12 force-pushed the yaml-dependency-file-overtop-alloy-poc branch 3 times, most recently from 3a41a88 to 2600ab5 Compare December 8, 2025 14:46

thampiotr reviewed Dec 8, 2025

View reviewed changes

blewis12 and others added 13 commits December 9, 2025 09:29

Introduce new dependency management tool

109f75e

Move syntax replace to top of go.mod to avoid confusion

5d0cbbe

Apply replaces from tool

b8c335b

Introduce PR workflow to check modules are synced

51f1920

Add E2E test

d854d12

Address some PR comments

8c1bd05

Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>

Refactor to use a centralised cobra CLI, called by generate.go

12e36ff

Refactor to not generate .txt files and pass replace text in memory

925e253

Refactor E2E tests to use explicit test data

63c0801

Update tools/generate-module-dependencies/replaces-mod.tpl

82603eb

Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

Return errors in FileHelper instead of using Fatalf

a9a809a

Address PR comments

ee84cc6

go mod/mod changes

1388c11

blewis12 force-pushed the yaml-dependency-file-overtop-alloy-poc branch from 08e3bb2 to 1388c11 Compare December 9, 2025 08:33

Reference actual calculated file to dependency yaml file in error logs

4ca68ba

blewis12 requested a review from thampiotr December 9, 2025 10:25

thampiotr approved these changes Dec 9, 2025

View reviewed changes

blewis12 added 2 commits December 9, 2025 13:32

Group the generate-module-dependencies and Check for go.mod changes p…

f71c79d

…roperly in workflow and use git diff instead so we can ignore whitespace changes

nit: a few improvements for clarity

e5d2644

blewis12 merged commit 23c0acf into main Dec 9, 2025
45 of 46 checks passed

blewis12 deleted the yaml-dependency-file-overtop-alloy-poc branch December 9, 2025 13:16

blewis12 mentioned this pull request Dec 10, 2025

Dependency Management In the OTel Engine #5053

Closed

blewis12 mentioned this pull request Dec 11, 2025

feat: Add Handling for OCB In YAML Dependency File #5065

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Shared YAML Dependency File To Manage Replace Directives#4862

Introduce Shared YAML Dependency File To Manage Replace Directives#4862
blewis12 merged 16 commits intomainfrom
yaml-dependency-file-overtop-alloy-poc

blewis12 commented Nov 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Nov 24, 2025 •

edited

Loading

Uh oh!

blewis12 commented Dec 8, 2025 •

edited

Loading

Uh oh!

thampiotr left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thampiotr left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

blewis12 commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is included

What is the intention

What will be done as follow ups

Uh oh!

Uh oh!

github-actions bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Dependency Review

Notes

Uh oh!

blewis12 commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thampiotr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thampiotr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

blewis12 commented Nov 17, 2025 •

edited

Loading

github-actions bot commented Nov 24, 2025 •

edited

Loading

blewis12 commented Dec 8, 2025 •

edited

Loading