Skip to content

build system dependency management / lockfile support #833

@malt3

Description

@malt3

I think I identified a missing feature that many users of dnf would benefit from:

Let's say I have a build system that uses my source code to bundle an rpm.
I also want to build further artifacts based on my rpm. This could include:

  • Container images (built using docker build, podman, Buildah, ...)
  • OS images (built using mkosi, CoreOS builder, or a raw dnf5 --installroot)

For supply chain security and general repeatability of build steps, I think the rpm/dnf ecosystem should support the following:

A dependency file format

If you are familiar with go, this would be the equivalent of a go.mod file.
You specify your direct dependencies here.
As an example, if I have a program written C that depends on libcurl and xz, the dependency file might look like this:

rpm.deps

[fedora-updates]
curl-devel
xz-devel

A dependency lock file format

If you are familiar with go, this would be the equivalent of a go.sum file.
DNF could take this as an optional input. And, if provided, any rpm installed has to be included in the lockfile.
The lockfile would include: a specifier for an rpm file including the name of the rpm, the version and architecture (nevra) and one or more hashes of the expected rpm file itself.

Here is an example for a lockfile format. The actual format could be completely different.

rpm.lock

https://github.com/microsoft/rpmoci/blob/5b3d17345e9b012d36c05b3c2de06426d6db9922/tests/fixtures/update_from_lockfile/rpmoci.lock#L1-L67

A vendoring command

This would read a dependency file and a lock file and download all rpms requested by the dependency file (with their dependencies) into a local folder.
This could be a part of the dnf5 download command.

Benefits to the ecosystem

Many projects are implementing their own version of this today and could standardize instead

Examples include Fedora CoreOS, bazeldnf, rpmoci, repro-get.
Standardizing also allows dependency management systems (like renovate) to notify developers that dependencies are outdated (and could also warn about known CVEs by parsing the lockfile).

Incremental build systems

Incremental build systems (including Bazel, mkosi, Dockerfile / Containerfile) could read a dependency lockfile to decide if an image (or layer) needs to be rebuilt. This can lead to better caching and incremental builds.

Correct cache invalidation

This proposal eliminates the well known issue of having unknown state in a container image layer:

FROM fedora:38
RUN dnf install openssl

If I do a docker build once, docker will give me an old (and probably vulnerable) version of openssl.

Let's instead say I could do this:

FROM fedora:38
COPY rpm.deps rpm.lock /
RUN dnf install --dep-file rpm.deps --lock-file rpm.lock  openssl

Now if I update my lockfile and rerun docker build I could get a newer version of openssl (instead of the old vulnerable one that is cached).

Reproducible / repeatable builds

This is a basic requirement for reproducible builds. If I build an OS or container image based on RPMs, I basically need to vendor / mirror / pin every RPM myself. Otherwise, when rebuilding the image, dnf could install other versions of rpms from a repository.

Next steps

I would encourage feedback from different stakeholders and would like to get feedback if this rough idea would be welcome by the maintainers of dnf.
If this is the case, I would be happy to create a more detailed proposal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions