Skip to content

feat: handle duplicate files from multiple packages deterministically#550

Merged
punkwalker merged 5 commits intocnoe-io:mainfrom
suwhang-cisco:main
Nov 12, 2025
Merged

feat: handle duplicate files from multiple packages deterministically#550
punkwalker merged 5 commits intocnoe-io:mainfrom
suwhang-cisco:main

Conversation

@suwhang-cisco
Copy link
Copy Markdown
Contributor

Related to the issue: #548

Currently, when we pass in multiple packages to idpbuilder create e.g.

idpbuilder create --package https://github.com/cnoe-io/stacks//ref-implementation   --package https://github.com/cnoe-io/stacks//caipe/base

where both packages have the same yaml files:

ref-implementation/
  backstage.yaml
  backstage/manifest/
caipe/base
  backstage.yaml
  backstage/manifest/

idpbuilder will be re-queuing both files in Gitea multiple times until sync is complete, causing race condition and indeterministic behaviour which file ends up in the cluster.

As ref-implementation is a package that contains all core elements for the idpbuilder cluster to be functional, it is useful for other packages to be built on top. However, there are cases where other packages would require to modify some of the existing applications in ref-implementation such as bringing their own customised backstage.

The fix I propose in this PR adds new label "cnoe.io/package-priority" to rank the order of packages and let the later packages higher priority, much like how helm values files work. It then uses this priority to determine if the package should be written in Gitea and reconciled or not.

This is Gitea experiencing multiple re-syncing between the duplicate files in current idpbuilder:
Screenshot 2025-10-24 at 12 47 30

New idpbuilder binary with the fix only updates the file once with the highest priority package:
Screenshot 2025-10-24 at 12 47 38

Signed-off-by: suwhang-cisco <suwhang@cisco.com>
Signed-off-by: suwhang-cisco <suwhang@cisco.com>
Signed-off-by: suwhang-cisco <suwhang@cisco.com>
Signed-off-by: suwhang-cisco <suwhang@cisco.com>
Copy link
Copy Markdown
Contributor

@punkwalker punkwalker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@punkwalker
Copy link
Copy Markdown
Contributor

/e2e

Signed-off-by: suwhang-cisco <suwhang@cisco.com>
@punkwalker punkwalker merged commit aba0fb0 into cnoe-io:main Nov 12, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants