docker save tar blob digest mtime not consistent when using overlay2 storage driver #51364
Description
Description
When using the overlay2 storage driver, docker save produces a tar file that sets the mtime of the blobs/{digest_algorithm} directory with a contemporary timestamp:
drwxr-xr-x 0/0 0 2025-06-20 14:51 blobs/
> drwxr-xr-x 0/0 0 2025-10-31 12:00 blobs/sha256/
-rw-r--r-- 0/0 1383 2025-06-20 14:51 blobs/sha256/03a64d61d91fc9b5ac1b2d39f5193e88bb315cddd0429d331aacb291de51fb2d
...
-rw-r--r-- 0/0 444 1970-01-01 00:00 index.json
-rw-r--r-- 0/0 5405 1970-01-01 00:00 manifest.json
-rw-r--r-- 0/0 31 1970-01-01 00:00 oci-layout
-rw-r--r-- 0/0 141 1970-01-01 00:00 repositories
I've reproduced this behaviour with 28.4.0 and 28.5.1. This behaviour is not observed by default with 29.0.0-rc1 as the containerd storage driver yields different tar files entirely. However I can also reproduce this with 29.0.0-rc1 by setting features.containerd-snapshotter=false.
Moby was previously updated to set stable timestamps for tar members generated by docker save, so I think this behaviour is unexpected.
I believe that update missed existing code where writing the OCI manifest file to the blob dir would modify the blob dir; updating its mtime after the mtime had been sanitised to 0.
Reproduce
# Install moby 29.0.0-rc1 and set features.containerd-snapshotter=false, or 28.4.0 or 28.5.1
$ docker pull ubuntu:24.04
$ docker save ubuntu:24.04 > save.tar
$ tar tvf save.tar
drwxr-xr-x 0/0 0 2025-10-01 13:01 blobs/
> drwxr-xr-x 0/0 0 2025-10-31 12:32 blobs/sha256/
-rw-r--r-- 0/0 80634368 2025-10-01 13:01 blobs/sha256/073ec47a8c22dcaa4d6e5758799ccefe2f9bde943685830b1bf6fd2395f5eabc
-rw-r--r-- 0/0 1318 2025-10-01 13:01 blobs/sha256/583aaddadccee9f8dbd3f47661b5a9a78daf4995f535396ff0c1ed0487e4db54
-rw-r--r-- 0/0 2297 2025-10-01 13:01 blobs/sha256/97bed23a34971024aa8d254abbe67b7168772340d1f494034773bc464e8dd5b6
-rw-r--r-- 0/0 402 1970-01-01 00:00 blobs/sha256/a80ae87142a0ae1f9e02e3561844d7004a28886aaca46db5392dd556df7326f2
-rw-r--r-- 0/0 360 1970-01-01 00:00 index.json
-rw-r--r-- 0/0 457 1970-01-01 00:00 manifest.json
-rw-r--r-- 0/0 31 1970-01-01 00:00 oci-layout
-rw-r--r-- 0/0 88 1970-01-01 00:00 repositories
This means that saved copies of the same tar are not consistent with eachother:
$ docker save ubuntu:24.04 > save.tar
$ docker save ubuntu:24.04 > save2.tar
$ sha256sum save.tar save2.tar
93e8664dac4ff602004f6461361da937296058b7369dd64e17770063327ec554 save.tar
b3825149268e4d849bf92f8a60d737b84c76f93179cf937a5ccec3b1c5223d80 save2.tar
Expected behavior
As a user, I would expect the output of docker save to be consistent for the same docker image.
docker version
Client: Docker Engine - Community
Version: 28.5.1
API version: 1.51
Go version: go1.24.8
Git commit: e180ab8
Built: Wed Oct 8 12:17:26 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.5.1
API version: 1.51 (minimum version 1.24)
Go version: go1.24.8
Git commit: f8215cc
Built: Wed Oct 8 12:17:26 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.27
GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da
runc:
Version: 1.2.5
GitCommit: v1.2.5-0-g59923ef
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client: Docker Engine - Community
Version: 28.5.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.39.4
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 28.5.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.11.0-29-generic
Operating System: Ubuntu 24.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: [redacted]
Total Memory: [redacted]
Name: [redacted]
ID: b6e0e13c-0b04-4fda-b475-10f1463a0186
Docker Root Dir: [redacted]
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
This does not occur when using the containerd image store, which seems to generate tar files with no mtime information at all:
$ tar tvf save_with_containerd.tar
drwxr-xr-x 0/0 0 1970-01-01 00:00 blobs/
drwxr-xr-x 0/0 0 1970-01-01 00:00 blobs/sha256/
-r--r--r-- 0/0 29723147 1970-01-01 00:00 blobs/sha256/4b3ffd8ccb5201a0fc03585952effb4ed2d1ea5e704d2e7330212fb8b16c86a3
-r--r--r-- 0/0 6688 1970-01-01 00:00 blobs/sha256/66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252
-r--r--r-- 0/0 2297 1970-01-01 00:00 blobs/sha256/97bed23a34971024aa8d254abbe67b7168772340d1f494034773bc464e8dd5b6
-r--r--r-- 0/0 424 1970-01-01 00:00 blobs/sha256/d22e4fb389065efa4a61bb36416768698ef6d955fe8a7e0cdb3cd6de80fa7eec
-rw-r--r-- 0/0 421 1970-01-01 00:00 index.json
-rw-r--r-- 0/0 211 1970-01-01 00:00 manifest.json
-r--r--r-- 0/0 30 1970-01-01 00:00 oci-layout
$ sha256sum save_with_containerd.tar save_with_containerd_again.tar
08e88baf5892afc1fc7a9aa0262c953ac193fffdc77eb36e6cf5ee4ec0a7ee68 save_with_containerd.tar
08e88baf5892afc1fc7a9aa0262c953ac193fffdc77eb36e6cf5ee4ec0a7ee68 save_with_containerd_again.tar