Description

relates to:

• containerd-integration: listing multi-arch images is (partially) broken  #44573
• Handle multi-platform images when listing them rumpl/moby#113
• API: add Plaform (OS and Architecture) to /containers/json #42464
• containerd-integration: cannot interact with images that don't have the default platform #44578
• containerd-integration: images pulled by digest don't show in image list #44579
• containerd-integration: confusuing behavior when pulling a new platform for an existing image #44581

With the containerd-integration in progress, we need to decide on UX for managing multi-arch images. Where previously an image would always be a single architecture, images may now be multi-arch, and we store multiple architectures / variants of an image. We discussed this topic some weeks agon in the containerd sync call, and the proposal was to create a ticket for discussion. Since that call, PR rumpl#113 made some changes (not yet upstreamed) to show individual architectures as individual images, which helps with visualising that the local image store has multiple variants stored for an image (and somewhat matches nerdctl), however, as we currently don't present the image's architecture in overview, this presentation is not ideal. It may also be a bit disjoint from the concept of multi-arch images, as image-variants now become "separate", somewhat defeating the concept of "multi".

This ticket describes some options; none of these are decided on (or final for that matter), but hopefully this can act as a starting point to come to a concensus on UX.

Assumptions

While writing these options, I made the following assumptions:

• where possible, we want the UX to stay close to the existing (non-multi-arch) UX
• for many (most) users, "multi-arch" remains to be "single-arch" (as before), but in some cases they may be using multiple architectures
  • the concept of multi-arch is for the image-index (multi-arch manifest) to be treated as a single entity
  • for most cases, the platform would be "current platform" (and we can keep the UX the same as non-multiarch)
  • for most cases, users wouldn't be micro-managing individual architectures; deleting / managing individual variants (when I run docker image rm alpine:3.16, I just want to remove alpine:3.16)
• in short: in most situations, the local image would be a "shallow" pull (one, or two architectures out of possible many)
• and; in most situations, there's no requirement to deal with all architectures. possible outliers here would be transferring an image between registries (pull from registry A, push to registry B)

Listing images

We need to decide where (and when) to expose architectures. We had some discussion about this during one of the maintainers calls, when discussing #42464. At the time, the question was raised "should this be visible by default?". There may not be a single answer to this (different scenarios require different information), and we could decide to include the platform information in API responses, but don't print the information by default.

If we decide to not include the platform, the output of docker images would remain the same as before multi-arch. Each image represents an image-manifest (either single- or multi-arch);

REPOSITORY       TAG          IMAGE ID       CREATED        SIZE
alpine           3.16         b95359c25051   25 hours ago   2.81MB
alpine           latest       8914eb54f968   5 days ago     3.26MB
busybox          latest       fcd85228d7a2   2 days ago     832kB

As we are showing manifest index (for multi-arch), we can add a PLATFORMS templating placeholder. Users can either use their own template, or we can add a flag to show that column. The column would show the variants that are present in the image cache (store) to keep the list short (and of we would truncate the output by default, unless --no-trunc is set). The SIZE column showing the size of all variants currently stored;

REPOSITORY       TAG          IMAGE ID       CREATED        PLATFORMS                                      SIZE
alpine           3.16         b95359c25051   25 hours ago   linux/amd64, linux/arm64/v8, linux/s390x, ...  6.19MB
alpine           latest       8914eb54f968   5 days ago     linux/arm64/v8                                 3.26MB
busybox          latest       fcd85228d7a2   2 days ago     linux/arm64/v8                                 832kB

Shallow / non-shallow pulls

There's some ambiguity, because alpine:3.16 is a multi-arch image, and provides many architectures. In the example above, alpine:3.16 is effectively a "shallow" pull; not all of the variants have been pulled (which is the most likely scenario). For reference, this is the list from Docker Hub; https://hub.docker.com/_/alpine/tags?page=1&name=3.16.3

DIGEST          OS/ARCH         COMPRESSED SIZE
b2774aff8c30    linux/386       2.68 MB
3d426b0bfc36    linux/amd64     2.68 MB
269d2ad7050b    linux/arm/v6    2.49 MB
92cd2f468f33    linux/arm/v7    2.31 MB
559254f7ee68    linux/arm64/v8  2.58 MB
a7ed77a6bc01    linux/ppc64le   2.67 MB
0c447070f97d    linux/s390x     2.47 MB

For cases where the user needs to interact with individual variants of the image, we can;

• add a --verbose or --show-platforms flag
• add a --platform flag on commands such as docker image rm / docker rmi, and docker image inspect to provide more granular control.

We can use something similar to docker service ps;

ID             NAME        IMAGE          NODE             DESIRED STATE   CURRENT STATE           ERROR                              PORTS
zef3c8wgn6x2   foo.1       nginx:alpine   docker-desktop   Running         Running 4 hours ago
vab7t41r0pg6    \_ foo.1   nginx:alpine   docker-desktop   Shutdown        Rejected 4 hours ago    "rpc error: code = Canceled de…"
c300yxopsccu    \_ foo.1   nginx:alpine   docker-desktop   Shutdown        Shutdown 4 hours ago
0qqppurnkc8x    \_ foo.1   nginx:alpine   docker-desktop   Shutdown        Shutdown 20 hours ago
scbfjmdcwm05    \_ foo.1   nginx:alpine   docker-desktop   Shutdown        Shutdown 27 hours ago
a1u6mkrs21gq   foo.2       nginx:alpine   docker-desktop   Running         Running 8 minutes ago

Which could look something like this:

REPOSITORY       TAG          IMAGE ID       CREATED        PLATFORMS                         SIZE
alpine           3.16         b95359c25051   25 hours ago   linux/amd64, linux/arm64/v8, ...  6.19MB
 \_ alpine       3.16         3d426b0bfc36   25 hours ago   linux/amd64                       2.68 MB
 \_ alpine       3.16         559254f7ee68   25 hours ago   linux/arm64/v8                    2.58 MB
 \_ alpine       3.16         0c447070f97d   25 hours ago   linux/s390x                       2.47 MB
alpine           latest       8914eb54f968   5 days ago     linux/arm64/v8                    3.26MB
 \_ alpine       latest       af06af3514c4   5 days ago     linux/arm64/v8                    3.26MB
busybox          latest       fcd85228d7a2   2 days ago     linux/arm64/v8                    832kB
 \_ busybox      latest       e68659cdc5b2   2 days ago     linux/arm64/v8                    832kB

It's worth noting that manifest (lists) are not necessarily unique; multiple tags can resolve to the same manifest(list). For example, if both alpine:latest and alpine:3.16 would have resolved to the same digest, the detailed list would be like below:

REPOSITORY       TAG          IMAGE ID       CREATED        PLATFORMS                         SIZE
alpine           3.16         b95359c25051   25 hours ago   linux/amd64, linux/arm64/v8, ...  6.19MB
 \_ alpine       3.16         3d426b0bfc36   25 hours ago   linux/amd64                       2.68 MB
 \_ alpine       3.16         559254f7ee68   25 hours ago   linux/arm64/v8                    2.58 MB
 \_ alpine       3.16         0c447070f97d   25 hours ago   linux/s390x                       2.47 MB
alpine           latest       b95359c25051   25 hours ago   linux/arm64/v8                    6.19MB
 \_ alpine       latest       3d426b0bfc36   25 hours ago   linux/amd64                       2.68 MB
 \_ alpine       latest       559254f7ee68   25 hours ago   linux/arm64/v8                    2.58 MB
 \_ alpine       latest       0c447070f97d   25 hours ago   linux/s390x                       2.47 MB

Note that in this case showing the REPOSITORY and TAG for each variant therefore is not "strictly correct", as those variants are not directly associated with a REPO or TAG. It's also not possible to "untag" such references (as this would mean "remove a reference from the manifest list", which would mean "create a new image manifest"). Because of this, we may want to consider to not present the TAG (and perhaps even REPOSITORY) for the variants;

REPOSITORY       TAG          IMAGE ID       CREATED        PLATFORMS                         SIZE
alpine           3.16         b95359c25051   25 hours ago   linux/amd64, linux/arm64/v8, ...  6.19 MB
 \_ alpine                    3d426b0bfc36   25 hours ago   linux/amd64                       2.68 MB
 \_ alpine                    559254f7ee68   25 hours ago   linux/arm64/v8                    2.58 MB
 \_ alpine                    0c447070f97d   25 hours ago   linux/s390x                       2.47 MB
alpine           latest       8914eb54f968   25 hours ago   linux/arm64/v8                    6.19 MB
 \_ alpine                    3d426b0bfc36   25 hours ago   linux/amd64                       2.68 MB
 \_ alpine                    559254f7ee68   25 hours ago   linux/arm64/v8                    2.58 MB
 \_ alpine                    0c447070f97d   25 hours ago   linux/s390x                       2.47 MB

Removing the REPOSITORY is a bit awkward, as it's the first column; we could consider changing the order of columns, putting the ID (digest) first;

IMAGE ID           REPOSITORY   TAG       PLATFORMS                         CREATED        SIZE
b95359c25051       alpine       3.16      linux/amd64, linux/arm64/v8, ...  25 hours ago   6.19 MB
 \_ 3d426b0bfc36                          linux/amd64                       25 hours ago   2.68 MB
 \_ 559254f7ee68                          linux/arm64/v8                    25 hours ago   2.58 MB
 \_ 0c447070f97d                          linux/s390x                       25 hours ago   2.47 MB
b95359c25051       alpine       latest    linux/arm64/v8                    25 hours ago   6.19 MB
 \_ 3d426b0bfc36                          linux/amd64                       25 hours ago   2.68 MB
 \_ 559254f7ee68                          linux/arm64/v8                    25 hours ago   2.58 MB
 \_ 0c447070f97d                          linux/s390x                       25 hours ago   2.47 MB

Given that in this presentation the columns don't align either way, we could consider omitting PLATFORMS for the top-level, re-purposing the space below REPOSITORY and TAG;

IMAGE ID           REPOSITORY   TAG       CREATED        SIZE
b95359c25051       alpine       3.16      25 hours ago   6.19 MB
 \_ 3d426b0bfc36    \_ linux/amd64        25 hours ago   2.68 MB
 \_ 559254f7ee68    \_ linux/arm64/v8     25 hours ago   2.58 MB
 \_ 0c447070f97d    \_ linux/s390x        25 hours ago   2.47 MB
b95359c25051       alpine       latest    25 hours ago   6.19 MB
 \_ 3d426b0bfc36    \_ linux/amd64        25 hours ago   2.68 MB
 \_ 559254f7ee68    \_ linux/arm64/v8     25 hours ago   2.58 MB
 \_ 0c447070f97d    \_ linux/s390x        25 hours ago   2.47 MB

Inspecting images

With the individual variants broken up, users can remove (or inspect) individual variants of an image, for example, to inspect the linux/s390x variant of the alpine image:

docker image inspect 0c447070f97d

For convenience, we should consider adding --platform to filter / show a specific variant:

docker image inspect --platform=linux/s390x alpine:3.16

• ⚠️ Currently, docker image inspect defaults to showing the image for the default platform.
• ❓ Do we want the command to default to show all architectures that are present?

Doing so would improve visibility for multi-arch images. The output of docker image inspect is already an array (which is used when inspecting multiple images);

docker image inspect busybox:latest hello-world:latest
[
  {
      "Id": "sha256:fcd85228d7a25feb59f101ac3a955d27c80df4ad824d65f5757a954831450185",
      "RepoTags": [
        "busybox:latest"
      ],
      "RepoDigests": null,
      "...": "...",
  },
  {
      "Id": "sha256:1ec996c686eb87d8f091080ec29dd1862b39b5822ddfd8f9a1e2c9288fad89fe",
      "RepoTags": [
        "hello-world:latest"
      ],
      "RepoDigests": [
        "hello-world@sha256:e18f0a777aefabe047a671ab3ec3eed05414477c951ab1a6f352a06974245fe7"
      ],
      "...": "...",
  }
]

Deleting images

Similarly to docker image inspect, the digest can be used to delete individual variants;

docker image rm 0c447070f97d

Note that this will remove the variant, so it should disappear from both alpine:3.16 and alpine:latest (or any image referencing it):

IMAGE ID           REPOSITORY   TAG       CREATED        SIZE
b95359c25051       alpine       3.16      25 hours ago   5.26 MB
 \_ 3d426b0bfc36    \_ linux/amd64        25 hours ago   2.68 MB
 \_ 559254f7ee68    \_ linux/arm64/v8     25 hours ago   2.58 MB
b95359c25051       alpine       latest    25 hours ago   5.26 MB
 \_ 3d426b0bfc36    \_ linux/amd64        25 hours ago   2.68 MB
 \_ 559254f7ee68    \_ linux/arm64/v8     25 hours ago   2.58 MB

As an alternative, we can add a --platform flag to docker image rm as well; the command below would achieve the same as above;

docker image rm --platform=linux/s390x alpine:3.16

❓ what do we want the behavior to be if (as in the example) the variant is referenced by multiple architectures? The docker image rm --platform command is ambiguous, as the user request the variant to be removed from the alpine:3.16 image. It may be surprising that it's also removed from the alpine:latest image (I guess the "most correct" presentation would be to show one line per digest, and multiple tags after it, but this is a HUGE change, and may not be very user-friendly).

• Print a warning? (Require some "force" option)?
• Or just "go ahead and remove"?

Shallow, shallower, shallowest

As images in the local store may be a "shallow" pull of a multi-arch image, the question is: do we want to provide insight into that? If so, how?

• Do we want an (optional) column to show that the image is "shallow"? (only some variants present)
• Do we want an indicator what the total size and number of variants would be if it's fully pulled?
• Do we want an option to pull (all) the "missing" variants?
• What do we want the behavior to be when doing a docker image pull without specifying a --platform?

For the "how much is missing comparing to (e.g.) docker service ls;

ID             NAME      MODE         REPLICAS   IMAGE          PORTS
i47e9yrzjef8   foo       replicated   2/2        nginx:alpine

We could add (optional) columns for "counts" to see "how" shallow the image is;

REPOSITORY       TAG          IMAGE ID       CREATED        PLATFORMS     SIZE
alpine           3.16         b95359c25051   25 hours ago   4/7           6.19MB / 17.88MB
alpine           latest       8914eb54f968   5 days ago     1/7           3.26MB / 17.88MB
busybox          latest       fcd85228d7a2   2 days ago     1/10          832kB / 10.76MB

For the "verbose" output, we could consider having an option to show what's missing; not sure (yet) how to best present that it's "missing", perhaps the SIZE column to show how much is there, or a MISSING somewhere?

IMAGE ID           REPOSITORY   TAG       CREATED        SIZE
b95359c25051       alpine       3.16      25 hours ago   6.19 / 17.88MB
 \_ b2774aff8c30    \_ linux/368          -                 0 / 2.68 MB
 \_ 3d426b0bfc36    \_ linux/amd64        25 hours ago   2.68 / 2.68 MB
 \_ 269d2ad7050b    \_ linux/arm/v6       -                 0 / 2.49 MB
 \_ 92cd2f468f33    \_ linux/arm/v7       -                 0 / 2.31 MB
 \_ 559254f7ee68    \_ linux/arm64/v8     25 hours ago   2.58 / 2.58 MB
 \_ a7ed77a6bc01    \_ linux/ppc64le      -                 0 / 2.67 MB
 \_ 0c447070f97d    \_ linux/s390x        25 hours ago   2.47 / 2.47 MB
b95359c25051       alpine       latest    25 hours ago   6.19 / 17.88MB
 \_ ...            ...                    ...             ... / ...

Filtering

Add a --platform option to docker image ls (and consider a --filter platform=xxx). Using the filter would show any image that currently has the given os/arch present;

docker image ls --platform=linux/s390x

REPOSITORY       TAG          IMAGE ID       CREATED        PLATFORMS     SIZE
alpine           3.16         b95359c25051   25 hours ago   4/7           6.19MB / 17.88MB

In "verbose" view, this would hide the other architectures;

IMAGE ID           REPOSITORY   TAG       CREATED        SIZE
b95359c25051       alpine       3.16      25 hours ago   6.19 / 17.88MB
 \_ 0c447070f97d    \_ linux/s390x        25 hours ago   2.47 / 2.47 MB
b95359c25051       alpine       latest    25 hours ago   6.19 / 17.88MB
 \_ ...            ...                    ...             ... / ...

Pruning

To be discussed; do we want pruning to default to

1. "remove unused architectures" from an image
2. or: only remove images as a whole (if any of the architectures are in use, don't remove anything)?
3. combination of 1. and 2.; if --all is set, do 1., otherwise do 2.
4. like 3. but with a dedicated flag?

Pulling images

If an image is not present in the local store, the current behavior is to pull with the default platform (platform of the host).

We need to define the expected behavior when pulling an image that is already present.

what should happen when

• doing a docker image pull without specifying a --platform ?
• doing a docker image pull with an explicit --platform=<DEFAULT PLATFORM>
• doing a docker image pull with --platform=<other platform>
• doing a docker run --platform and the given platform is not in the current image

Currently, the behavior is confusing (and there's some bugs / undefined behavior);

• containerd-integration: cannot interact with images that don't have the default platform #44578
• containerd-integration: images pulled by digest don't show in image list #44579
• containerd-integration: confusuing behavior when pulling a new platform for an existing image #44581

The part to define if image should be updated (re-resolving the digest) or not. There's advantages to either, but equally "confusing" behavior.

It depends on what we envision pull to mean, and whether a --platform was specified (explicitly?). The changes are subtle, but may be important. Some options;

1. When pulling an image (using name:tag), irregardless of --platform to be specified (explicitly), we resolve the digest, and pull the image.
2. (A) When pulling an image WITH a --platform specified
   • don't re-resolve the manifest-index
   • pull the new platform, using the digest that's found in manifest index that's currently present
   • This option allows "back-filling" missing plaforms for an already present image.
3. (B) When pulling an image WITH a --platform specified
   • resolve the manifest-index and pull it
   • pull the NEW platform with the digest found in the new manifest-index
   • store the NEW platform under the NEW manifest-index
   • Two images will show (currently); one for the old (with the existing platforms) and one for the new (with the newly pulled platform)
4. (C) Like combination of 2. and 3.: use 2. as default, but offer a --resolve, --update or --pull=<some option> option to force updating.
5. (A) When pulling an image WITHOUT a --platform specified, then
   • resolve the manifest-index and pull it
   • pull all existing platforms, using the digests found in the new manifest-index
   • pull the default platform (if not present) (?), using the digest found in the new manifest-index
6. (B) When pulling an image WITHOUT a --platform specified, then
   • resolve the manifest-index and pull it
   • pull only the default platform, using the digest found in the new manifest-index
   • store the default platform under the NEW manifest-index
   • Two images will show (currently); one for the old (with the existing platforms) and one for the new (with the newly pulled default platform)

My preference goes out to

• 2. or 4. when a --platform is set, as it doesn't implicitly update the image for other architectures.
• 5. for when no --platform is set; this is the closes match to the "pre-multi-arch" behavior: when pulling an image, it's updated to the latest version. But with multi-arch, this means "all arches that I already have".