Description

This PR introduces a few breaking changes to largely resolve some of the symptoms of the problem of introducing new files into the package layout and backwards incompatibility.

Given the scope of changes released for the bundle format signature - the compatibility and errors involved could have been improved by placing a feature flag in front of the bundle format process. This would allow the default workflows with new releases to still be backwards compatible. This allows for multiple releases to be introduced that contain the updated verification logic for bundles before we remove generation of the legacy signature entirely.

Future Proofing

The root of the original issue is that new files outside of the checksums integrity check cannot exist in the package and be backwards compatible. This is a very niche case because in most cases the file should always be included in the checksums list.

This PR introduces ProvenanceFiles to the build data for two purposes:

• It introduces optionality going forward to include a new file that all versions hereon forward would be able to handle files outside of the checksums (typically post-signing)
• It also provides an opportunity to create observability around these invisible files that are included in the package but not transparent in the definition

There is merit on concern on security of a MITM injection of an untracked file - but this is ever present and should be mitigated by a package being signed whereby the zarf.yaml cannot be modified.

Summary

• Move the bundle format behind a feature flag - not enabled by default for backwards compatibility
• Introduce provenance files for forward compatibility on the files tracked outside of the checksums file

Related Issue

Fixes #4591

Relates to #

Checklist before merging

• Test, docs, adr added or updated as needed
• Contributor Guide Steps followed