fix socket handling during copy by alexcb · Pull Request #1581 · moby/buildkit

alexcb · 2020-07-18T05:01:46Z

This reproduces an error with tar trying to archive a socket, which is similar to the one that was fixed in #1144

        client_test.go:1696: 
            	Error Trace:	client_test.go:1696
            	            				run.go:165
            	Error:      	Received unexpected error:
            	            	rpc error: code = Unknown desc = failed to compute cache key: failed to create hash for /root/socket.sock: archive/tar: sockets not supported
            	            	github.com/moby/buildkit/util/stack.Enable
            	            		/src/util/stack/stack.go:51
            	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
            	            		/src/util/grpcerrors/grpcerrors.go:162
            	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
            	            		/src/util/grpcerrors/intercept.go:22
            	            	google.golang.org/grpc.(*ClientConn).Invoke
            	            		/src/vendor/google.golang.org/grpc/call.go:35
            	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
            	            		/src/api/services/control/control.pb.go:1321
            	            	github.com/moby/buildkit/client.(*Client).solve.func2
            	            		/src/client/solve.go:201
            	            	golang.org/x/sync/errgroup.(*Group).Go.func1
            	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1357
            	            	failed to solve
            	            	github.com/moby/buildkit/client.(*Client).solve.func2
            	            		/src/client/solve.go:214
            	            	golang.org/x/sync/errgroup.(*Group).Go.func1
            	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1357
            	Test:       	TestIntegration/TestTarExporterWithSocketCopy/worker=oci-rootless
        oci.go:74: stdout: /usr/bin/sudo
        oci.go:74: stderr: /usr/bin/sudo
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="auto snapshotter: using overlayfs"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="running in rootless mode"
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="found worker \"0d7fqlw0dzm6coe47xuqsliqo\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:5e9fae6f5c60 org.mobyproject.buildkit.worker.sandbox:true org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
        oci.go:77: time="2020-07-18T04:54:13Z" level=warning msg="rootless mode is not supported for containerd workers. disabling containerd worker."
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="found 1 workers, default=\"0d7fqlw0dzm6coe47xuqsliqo\""
        oci.go:77: time="2020-07-18T04:54:13Z" level=warning msg="currently, only the default worker can be used."
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="running server on /tmp/bktest_buildkitd003312146/buildkitd.sock"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="session started"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolving host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" host="localhost:38005" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD url="http://localhost:38005/v2/library/alpine/manifests/latest"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" host="localhost:38005" response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65\"" response.status="200 OK" url="http://localhost:38005/v2/library/alpine/manifests/latest"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolved desc.digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolving host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" host="localhost:38005" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" host="localhost:38005" response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65\"" response.status="200 OK" url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolved desc.digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json size=528
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=GET size=528 url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65\"" response.status="200 OK" size=528 url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json size=1509
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=GET size=1509 url="http://localhost:38005/v2/library/alpine/blobs/sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1509 response.header.content-type=application/octet-stream response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e\"" response.status="200 OK" size=1509 url="http://localhost:38005/v2/library/alpine/blobs/sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json size=528
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2797541
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json size=1509
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=GET size=2797541 url="http://localhost:38005/v2/library/alpine/blobs/sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=2797541 response.header.content-type=application/octet-stream response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c\"" response.status="200 OK" size=2797541 url="http://localhost:38005/v2/library/alpine/blobs/sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="unpigz not found, falling back to go gzip" error="exec: \"unpigz\": executable file not found in $PATH"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="diff applied" d=255.743209ms dgst="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=0
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="> creating bmn7txh17f5go1zdf1dg57hk2 [sh -c nc -l -s local:/root/socket.sock & usleep 100000; kill %1]"
        oci.go:77: time="2020-07-18T04:54:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to create hash for /root/socket.sock: archive/tar: sockets not supported"
        oci.go:77: time="2020-07-18T04:54:14Z" level=debug msg="session finished: <nil>"
FAIL

tonistiigi · 2020-07-18T22:52:14Z

@alexcb Are you working on a fix or do you want us to take over from here?

alexcb · 2020-07-18T23:33:46Z

@tonistiigi I haven't started on any fix. I'm fairly new to buildkit, so it might take me a while to figure out what needs fixing. If you have time and it's any easy fix then I would gladly let you take over.

alexcb · 2020-07-23T16:17:35Z

I discovered one of the fstypes.Stat structs was being directly created in buildkit rather than in fsutil/stat.go.

tonistiigi · 2020-07-23T16:22:45Z

cache/contenthash/filehash.go

 	}

+	// Clear the socket bit since archive/tar.FileInfoHeader does not handle it
+	stat.Mode &^= uint32(os.ModeSocket)


would this make more sense inside NewFromStat ?

If you think that's a better place we can definitely move it there.

in NewFromStat(stat *fstypes.Stat) stat is passed as a pointer, would it be fine to modify the contents, or would it be better to make a copy of it so it's essentially passed in as a const?

I'm fine with the current as it is a trivial case, but if you want to be more correct you could check it the socket bit exists and only do copy then.

client/client_test.go

tonistiigi · 2020-07-23T18:17:56Z

LGTM but please squash the commits so it doesn't cause issues for git bisect in the future.

This fix is similar to the fix in moby#1144; but was hit in a different code path. Signed-off-by: Alex Couture-Beil <alex@earthly.dev>

alexcb · 2020-07-23T18:30:01Z

squashed! thanks for the help with the PR.

alexcb · 2020-07-23T18:43:48Z

that's odd the tests passed before my squash, but after the squash it panicked in a different part of the code:

panic: runtime error: slice bounds out of range [19792:19280]
goroutine 7169 [running]:
bytes.(*Buffer).Write(0xc000facab0, 0xc0009fa400, 0x52, 0x900, 0x3, 0x3, 0x1496f00)
	/usr/local/go/src/bytes/buffer.go:174 +0x102
fmt.Fprintf(0x15fc7a0, 0xc000facab0, 0x14b30b1, 0x14, 0xc000f62718, 0x3, 0x3, 0xc000f626f0, 0xd271a1, 0xc001c84740)
	/usr/local/go/src/fmt/print.go:205 +0xa5
github.com/moby/buildkit/util/testutil/integration.startCmd.func1(0xc000f62768, 0xc00008d980)
	/src/util/testutil/integration/util.go:39 +0x1b8
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000facb10, 0xc0005101e0)
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
FAIL	github.com/moby/buildkit/frontend/dockerfile	74.189s

tonistiigi · 2020-07-23T18:47:30Z

@alexcb That is unrelated. Restarted and will fix that in separate PR

An issue in buildkit caused a "tar/archive: sockets not supported" error while building the dot net example; this issue has been fixed upstream in buildkit ( see moby/buildkit#1581 ), and we can now remove this work-around.

An issue in buildkit caused a "tar/archive: sockets not supported" error while building the dot net example; this issue has been fixed upstream in buildkit ( see moby/buildkit#1581 ), and we can now remove this work-around. Co-authored-by: Alex Couture-Beil <alex@earthly.dev>

alexcb force-pushed the tar-exporter-with-socket-copy-reproduce-error branch from f680d09 to ed029cf Compare July 18, 2020 05:07

tonistiigi reviewed Jul 23, 2020

View reviewed changes

alexcb marked this pull request as ready for review July 23, 2020 17:08

alexcb force-pushed the tar-exporter-with-socket-copy-reproduce-error branch 2 times, most recently from 7c5a106 to 1a5fbb3 Compare July 23, 2020 17:18

alexcb changed the title ~~reproduce tar exporter with socket error~~ fix socket handling during copy Jul 23, 2020

Treat unix sockets as regular files

5382a20

This fix is similar to the fix in moby#1144; but was hit in a different code path. Signed-off-by: Alex Couture-Beil <alex@earthly.dev>

alexcb force-pushed the tar-exporter-with-socket-copy-reproduce-error branch from 1a5fbb3 to 5382a20 Compare July 23, 2020 18:26

tonistiigi approved these changes Jul 23, 2020

View reviewed changes

tonistiigi added the needs-cherry-pick/v0.7 label Jul 23, 2020

tonistiigi mentioned this pull request Jul 23, 2020

integration: lock buffer for concurrent writes #1590

Merged

tonistiigi approved these changes Jul 23, 2020

View reviewed changes

tonistiigi merged commit a67af5b into moby:master Jul 23, 2020

alexcb mentioned this pull request Jul 23, 2020

Remove socket file rm hack earthly/earthly#137

Merged

alexcb mentioned this pull request Jul 24, 2020

Socket files left over from 'dotnet restore' cause buildkit to fail earthly/earthly#115

Closed

tonistiigi added the needs-cherry-pick/v0.6 label Jul 25, 2020

This was referenced Jul 25, 2020

[v0.6] cherry picks #1596

Merged

[v0.7] cherry picks #1597

Merged

tonistiigi removed needs-cherry-pick/v0.6 needs-cherry-pick/v0.7 labels Jul 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix socket handling during copy#1581

fix socket handling during copy#1581
tonistiigi merged 1 commit intomoby:masterfrom
alexcb:tar-exporter-with-socket-copy-reproduce-error

alexcb commented Jul 18, 2020 •

edited

Loading

Uh oh!

tonistiigi commented Jul 18, 2020

Uh oh!

alexcb commented Jul 18, 2020

Uh oh!

alexcb commented Jul 23, 2020

Uh oh!

tonistiigi Jul 23, 2020

Uh oh!

alexcb Jul 23, 2020

Uh oh!

tonistiigi Jul 23, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonistiigi commented Jul 23, 2020

Uh oh!

alexcb commented Jul 23, 2020

Uh oh!

alexcb commented Jul 23, 2020 •

edited

Loading

Uh oh!

tonistiigi commented Jul 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alexcb commented Jul 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi commented Jul 18, 2020

Uh oh!

alexcb commented Jul 18, 2020

Uh oh!

alexcb commented Jul 23, 2020

Uh oh!

tonistiigi Jul 23, 2020

Choose a reason for hiding this comment

Uh oh!

alexcb Jul 23, 2020

Choose a reason for hiding this comment

Uh oh!

tonistiigi Jul 23, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonistiigi commented Jul 23, 2020

Uh oh!

alexcb commented Jul 23, 2020

Uh oh!

alexcb commented Jul 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi commented Jul 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alexcb commented Jul 18, 2020 •

edited

Loading

alexcb commented Jul 23, 2020 •

edited

Loading