graph: exported images times matching creation by vbatts · Pull Request #16076 · moby/moby

vbatts · 2015-09-04T02:27:20Z

the image export, that is used in docker save previous has just had
the layers times (atimes, mtimes) be when the save was done.

vbatts@valse ~ (master) $ docker save busybox | tar tv
drwxr-xr-x 0/0               0 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/
-rw-r--r-- 0/0               3 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/VERSION
-rw-r--r-- 0/0            1405 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/json
-rw-r--r-- 0/0         2643968 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/layer.tar
drwxr-xr-x 0/0               0 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/
-rw-r--r-- 0/0               3 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/VERSION
-rw-r--r-- 0/0            1346 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/json
-rw-r--r-- 0/0            1024 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/layer.tar
drwxr-xr-x 0/0               0 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/
-rw-r--r-- 0/0               3 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/VERSION
-rw-r--r-- 0/0            1181 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/json
-rw-r--r-- 0/0            1024 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/layer.tar
-rw-r--r-- 0/0              90 2015-09-03 22:22 repositories

With this change, the layer's directory and artifact will have times
matching the image layer's created time. The "repositories" file is set
to epoch.

vbatts@valse ~ (master) $ docker save busybox | tar tv
drwxr-xr-x 0/0               0 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/
-rw-r--r-- 0/0               3 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/VERSION
-rw-r--r-- 0/0            1405 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/json
-rw-r--r-- 0/0         2643968 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/layer.tar
drwxr-xr-x 0/0               0 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/
-rw-r--r-- 0/0               3 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/VERSION
-rw-r--r-- 0/0            1346 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/json
-rw-r--r-- 0/0            1024 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/layer.tar
drwxr-xr-x 0/0               0 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/
-rw-r--r-- 0/0               3 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/VERSION
-rw-r--r-- 0/0            1181 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/json
-rw-r--r-- 0/0            1024 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/layer.tar
-rw-r--r-- 0/0              90 1969-12-31 19:00 repositories

Side effect of this is that the tar stream from docker save is now
more deterministic.

vbatts@valse ~ (master) $ docker save busybox | sha1sum
baf03e30ef79ca4d9c5e512d3a1b873880f404ca  -
vbatts@valse ~ (master) $ docker save busybox | sha1sum
baf03e30ef79ca4d9c5e512d3a1b873880f404ca  -
vbatts@valse ~ (master) $ docker save busybox | sha1sum
baf03e30ef79ca4d9c5e512d3a1b873880f404ca  -
vbatts@valse ~ (master) $ docker save busybox | sha1sum
baf03e30ef79ca4d9c5e512d3a1b873880f404ca  -

This idea came about from thinking about #8819 (comment)

Signed-off-by: Vincent Batts vbatts@redhat.com

the image export, that is used in `docker save` previous has just had the layers times (atimes, mtimes) be when the save was done. ```bash vbatts@valse ~ (master) $ docker save busybox | tar tv drwxr-xr-x 0/0 0 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/ -rw-r--r-- 0/0 3 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/VERSION -rw-r--r-- 0/0 1405 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/json -rw-r--r-- 0/0 2643968 2015-09-03 22:22 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/layer.tar drwxr-xr-x 0/0 0 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/ -rw-r--r-- 0/0 3 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/VERSION -rw-r--r-- 0/0 1346 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/json -rw-r--r-- 0/0 1024 2015-09-03 22:22 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/layer.tar drwxr-xr-x 0/0 0 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/ -rw-r--r-- 0/0 3 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/VERSION -rw-r--r-- 0/0 1181 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/json -rw-r--r-- 0/0 1024 2015-09-03 22:22 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/layer.tar -rw-r--r-- 0/0 90 2015-09-03 22:22 repositories ``` With this change, the layer's directory and artifact will have times matching the image layer's created time. The "repositories" file is set to epoch. ```bash vbatts@valse ~ (master) $ docker save busybox | tar tv drwxr-xr-x 0/0 0 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/ -rw-r--r-- 0/0 3 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/VERSION -rw-r--r-- 0/0 1405 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/json -rw-r--r-- 0/0 2643968 2015-04-17 18:01 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea/layer.tar drwxr-xr-x 0/0 0 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/ -rw-r--r-- 0/0 3 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/VERSION -rw-r--r-- 0/0 1346 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/json -rw-r--r-- 0/0 1024 2015-04-17 18:01 8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55/layer.tar drwxr-xr-x 0/0 0 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/ -rw-r--r-- 0/0 3 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/VERSION -rw-r--r-- 0/0 1181 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/json -rw-r--r-- 0/0 1024 2015-04-17 18:01 cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff/layer.tar -rw-r--r-- 0/0 90 1969-12-31 19:00 repositories ``` Side effect of this is that the tar stream from `docker save` is now more deterministic. ```bash vbatts@valse ~ (master) $ docker save busybox | sha1sum baf03e30ef79ca4d9c5e512d3a1b873880f404ca - vbatts@valse ~ (master) $ docker save busybox | sha1sum baf03e30ef79ca4d9c5e512d3a1b873880f404ca - vbatts@valse ~ (master) $ docker save busybox | sha1sum baf03e30ef79ca4d9c5e512d3a1b873880f404ca - vbatts@valse ~ (master) $ docker save busybox | sha1sum baf03e30ef79ca4d9c5e512d3a1b873880f404ca - ``` Signed-off-by: Vincent Batts <vbatts@redhat.com>

tianon · 2015-09-05T00:21:29Z

Wouldn't it make more sense to have repositories be set to the latest timestamp of the layers it represents?

vbatts · 2015-09-05T00:25:08Z

Maybe. There can be a number of images and tagged layers. Then it would be a sort and still variadic. Regardless of the arguments passed to docker save the time stamps will only be consistent in the way I've presently done this.

calavera · 2015-09-16T21:41:25Z

LGTM

tianon · 2015-09-16T22:04:27Z

@vbatts thanks for clarifying; makes sense 👍

vbatts · 2015-09-30T23:51:08Z

I literally can't even lxc. No idea what these failures are, but w/e.

cpuguy83 · 2015-10-01T15:05:34Z

How about an integration test?

vbatts · 2015-10-01T15:28:47Z

ooh

vbatts · 2015-10-01T16:08:43Z

@cpuguy83 done. PTAL

vdemeester · 2015-10-01T16:13:55Z

integration-cli/docker_cli_save_load_test.go

A nit, but :

err := json.Unmarshal([]byte(out), &data) c.Assert(err, checker.IsNil, "failed to marshal from %q: err %v", repoName, err)

cpuguy83 · 2015-10-01T16:41:49Z

integration-cli/docker_cli_save_load_test.go:110: wrong number of args for format in Fatalf call: 1 needed but 2 args

vbatts · 2015-10-01T17:29:37Z

@cpuguy83 huh?

+ go test -check.f TestSaveCheckTimes -test.timeout=60m -check.v github.com/docker/docker/integration-cli
INFO: Testing against a local daemon
PASS: docker_cli_save_load_test.go:99: DockerSuite.TestSaveCheckTimes   0.062s
OK: 1 passed
PASS

vbatts · 2015-10-01T18:00:57Z

@vdemeester updated to use assertion, though that loses the context message if it fails :-\

vdemeester · 2015-10-01T18:28:51Z

@vbatts why not using the check.Commentf thingy though ? ❓

vbatts · 2015-10-01T18:31:46Z

because

with this check package

The `docker save`ed output ought to have matching timestamp to the layer creation. Signed-off-by: Vincent Batts <vbatts@redhat.com>

vbatts · 2015-10-01T18:39:11Z

added commentf

On Thu, Oct 1, 2015 at 2:28 PM, Vincent Demeester notifications@github.com
wrote:

@vbatts https://github.com/vbatts why not using the check.Commentf
thingy though ? [image: ❓]

—
Reply to this email directly or view it on GitHub
#16076 (comment).

cpuguy83 · 2015-10-01T20:18:29Z

LGTM

vdemeester · 2015-10-01T20:21:08Z

LGTM, wating for janking 😝

vdemeester · 2015-10-01T20:49:21Z

Same old windows failure TestExecStartFails, not related..

graph: exported images times matching creation

paddie · 2017-02-09T15:51:39Z

I'm attempting to do some local caching of separate layers of the build process for a particularly grueling npm install pass, and I'm hitting on an issue that seems related to this.

Using this minimal image:

FROM alpine

CMD echo help

And running the following command on OSX repeatably, it produces different sha1 checksums.

I'm on OSX though, so I'm using the openssl sha1 to compute the sha1:

> docker build -t help .
> docker save $(docker history -q help) | openssl sha1
35da520b07a54d5f771d9bdae6e8d646d99109e8
> docker save $(docker history -q help) | openssl sha1
530b5ce927101400c2c4c7ceb908e87108a0736a

The following command still produces consistent sha1:

> docker save

Does anyone know what's the reason for the differences in the checksum that I'm seeing?

GordonTheTurtle added area/distribution Image Distribution status/0-triage labels Sep 4, 2015

vbatts mentioned this pull request Sep 4, 2015

docker save myimage has nondeterministic output #8819

Closed

jessfraz added status/2-code-review and removed status/0-triage labels Sep 8, 2015

vdemeester reviewed Oct 1, 2015
View reviewed changes

vbatts force-pushed the export_image_times branch from 0811924 to 3ac60dd Compare October 1, 2015 18:00

save: integration test for timestamp matching

e4478ca

The `docker save`ed output ought to have matching timestamp to the layer creation. Signed-off-by: Vincent Batts <vbatts@redhat.com>

vbatts force-pushed the export_image_times branch from 3ac60dd to e4478ca Compare October 1, 2015 18:39

vdemeester added status/4-merge and removed status/2-code-review labels Oct 1, 2015

vdemeester added a commit that referenced this pull request Oct 1, 2015

Merge pull request #16076 from vbatts/export_image_times

77da5d8

graph: exported images times matching creation

vdemeester merged commit 77da5d8 into moby:master Oct 1, 2015

vbatts deleted the export_image_times branch April 27, 2016 18:02

vvoland mentioned this pull request Nov 24, 2023

c8d/integration: Adjust TestSaveCheckTimes #46848

Merged

Conversation

vbatts commented Sep 4, 2015

Uh oh!

tianon commented Sep 5, 2015

Uh oh!

vbatts commented Sep 5, 2015

Uh oh!

calavera commented Sep 16, 2015

Uh oh!

tianon commented Sep 16, 2015

Uh oh!

vbatts commented Sep 30, 2015

Uh oh!

cpuguy83 commented Oct 1, 2015

Uh oh!

vbatts commented Oct 1, 2015

Uh oh!

vbatts commented Oct 1, 2015

Uh oh!

vdemeester Oct 1, 2015

Choose a reason for hiding this comment

Uh oh!

cpuguy83 commented Oct 1, 2015

Uh oh!

vbatts commented Oct 1, 2015

Uh oh!

vbatts commented Oct 1, 2015

Uh oh!

vdemeester commented Oct 1, 2015

Uh oh!

vbatts commented Oct 1, 2015

Uh oh!

vbatts commented Oct 1, 2015

Uh oh!

cpuguy83 commented Oct 1, 2015

Uh oh!

vdemeester commented Oct 1, 2015

Uh oh!

vdemeester commented Oct 1, 2015

Uh oh!

paddie commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

paddie commented Feb 9, 2017 •

edited

Loading