integration: containerd snapshotter with dockerd#3176
integration: containerd snapshotter with dockerd#3176tonistiigi merged 7 commits intomoby:masterfrom
Conversation
416e557 to
f50e3a5
Compare
f50e3a5 to
c22715f
Compare
client/client_test.go
Outdated
| func testPushByDigest(t *testing.T, sb integration.Sandbox) { | ||
| integration.SkipIfDockerd(t, sb, "direct push") | ||
| integration.SkipIfDockerdMoby(t, sb, "direct push") | ||
| integration.SkipIfDockerdSnapshotter(t, sb, "can't push ref tagged by digest") |
There was a problem hiding this comment.
https://github.com/crazy-max/buildkit/actions/runs/3257225568/jobs/5348378063#step:8:1262
=== CONT TestIntegration/TestPushByDigest/worker=dockerd-containerd
client_test.go:788:
Error Trace: client_test.go:788
run.go:86
run.go:189
Error: Received unexpected error:
can't push tagged ref localhost:40335/foo/bar:latest by digest
github.com/moby/buildkit/util/stack.Enable
/src/util/stack/stack.go:77
github.com/moby/buildkit/util/grpcerrors.FromGRPC
/src/util/grpcerrors/grpcerrors.go:196
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
/src/util/grpcerrors/intercept.go:41
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
/src/api/services/control/control.pb.go:1535
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:231
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
failed to solve
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:244
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
Test: TestIntegration/TestPushByDigest/worker=dockerd-containerd
client/client_test.go
Outdated
| func testPullWithLayerLimit(t *testing.T, sb integration.Sandbox) { | ||
| integration.SkipIfDockerd(t, sb, "direct push") | ||
| integration.SkipIfDockerdMoby(t, sb, "direct push") | ||
| integration.SkipIfDockerdSnapshotter(t, sb, "unknown API capability source.image.layerlimit") |
There was a problem hiding this comment.
https://github.com/crazy-max/buildkit/actions/runs/3257160268/jobs/5348276727#step:8:1173
=== CONT TestIntegration/TestPullWithLayerLimit/worker=dockerd-containerd
client_test.go:6030:
Error Trace: client_test.go:6030
run.go:86
run.go:189
Error: Received unexpected error:
failed to load LLB: unknown API capability source.image.layerlimit
github.com/moby/buildkit/util/stack.Enable
/src/util/stack/stack.go:77
github.com/moby/buildkit/util/grpcerrors.FromGRPC
/src/util/grpcerrors/grpcerrors.go:196
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
/src/util/grpcerrors/intercept.go:41
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
/src/api/services/control/control.pb.go:1535
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:231
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
failed to solve
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:244
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
Test: TestIntegration/TestPullWithLayerLimit/worker=dockerd-containerd
client/client_test.go
Outdated
|
|
||
| func testAttestationDefaultSubject(t *testing.T, sb integration.Sandbox) { | ||
| integration.SkipIfDockerdMoby(t, sb, "direct push") | ||
| integration.SkipIfDockerdSnapshotter(t, sb, "number of platforms does not match references 1 2") |
There was a problem hiding this comment.
https://github.com/crazy-max/buildkit/actions/runs/3257051098/jobs/5348086746#step:8:1111
=== CONT TestIntegration/TestAttestationDefaultSubject/worker=dockerd-containerd
client_test.go:6745:
Error Trace: client_test.go:6745
run.go:86
run.go:189
Error: Received unexpected error:
number of platforms does not match references 1 2
github.com/moby/buildkit/util/stack.Enable
/src/util/stack/stack.go:77
github.com/moby/buildkit/util/grpcerrors.FromGRPC
/src/util/grpcerrors/grpcerrors.go:196
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
/src/util/grpcerrors/intercept.go:41
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
/src/api/services/control/control.pb.go:1535
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:231
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
failed to solve
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:244
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
Test: TestIntegration/TestAttestationDefaultSubject/worker=dockerd-containerd
There was a problem hiding this comment.
👀 uh oh, something seems actually wrong there...
That error message is usually a signal that something has gone quite wrong - the number of refs stop matching up with the number of things that use them.
I'm fairly sure this isn't correct anymore, the error message is also changed recently to be slightly more indicative of the actual problem: https://github.com/moby/buildkit/blob/master/exporter/containerimage/writer.go#L118.
There was a problem hiding this comment.
Yes the snapshot adapter does not seem to match next v0.11 but v0.10 atm: https://github.com/rumpl/moby/blob/c8d/builder/builder-next/adapters/snapshot/snapshot.go
| }, nil) | ||
| require.NoError(t, err) | ||
|
|
||
| integration.SkipIfDockerdSnapshotter(t, sb, "no match for platform in manifest") |
There was a problem hiding this comment.
https://github.com/crazy-max/buildkit/actions/runs/3257051098/jobs/5348086904#step:8:1817
=== CONT TestIntegration/TestNamedImageContextPlatform/worker=dockerd-containerd/frontend=builtin
dockerfile_test.go:5265:
Error Trace: dockerfile_test.go:5265
run.go:86
run.go:189
Error: Received unexpected error:
no match for platform in manifest sha256:301ee552783ae53eaeb667945fd876066675c539c259cdb3e70fc8dfb6e0ea48: not found
github.com/moby/buildkit/util/stack.Enable
/src/util/stack/stack.go:77
github.com/moby/buildkit/util/grpcerrors.FromGRPC
/src/util/grpcerrors/grpcerrors.go:196
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
/src/util/grpcerrors/intercept.go:41
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
/src/api/services/control/control.pb.go:1535
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:231
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
failed to solve
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:244
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
Test: TestIntegration/TestNamedImageContextPlatform/worker=dockerd-containerd/frontend=builtin
| func testNamedImageContextTimestamps(t *testing.T, sb integration.Sandbox) { | ||
| integration.SkipIfDockerd(t, sb, "direct push") | ||
| integration.SkipIfDockerdMoby(t, sb, "direct push", "named contexts timestamps") | ||
| integration.SkipIfDockerdSnapshotter(t, sb, "named contexts timestamps") |
There was a problem hiding this comment.
https://github.com/crazy-max/buildkit/actions/runs/3257051098/jobs/5348086904#step:8:1232
=== CONT TestIntegration/TestNamedImageContextTimestamps/worker=dockerd-containerd/frontend=builtin
dockerfile_test.go:5349:
Error Trace: dockerfile_test.go:5349
run.go:86
run.go:189
Error: Should not be: time.Date(2022, time.October, 15, 20, 48, 54, 585592478, time.UTC)
Test: TestIntegration/TestNamedImageContextTimestamps/worker=dockerd-containerd/frontend=builtin
There was a problem hiding this comment.
I don't understand the connection in here. Assuming DockerdSnapshotter uses the image exporter directly and not moby.
There was a problem hiding this comment.
This means for DockerdMoby we have to skip because direct push and named contexts timestamps are not working but with DockerdSnapshotter only named contexts timestamps is not working:
buildkit/frontend/dockerfile/dockerfile_test.go
Line 5349 in c717d6a
| func testCacheMultiPlatformImportExport(t *testing.T, sb integration.Sandbox) { | ||
| integration.SkipIfDockerd(t, sb, "direct push") | ||
| integration.SkipIfDockerdMoby(t, sb, "direct push") | ||
| integration.SkipIfDockerdSnapshotter(t, sb, "cache multi-platform hangs") |
There was a problem hiding this comment.
This one hangs with containerd-snapshotter which might be because docker daemon panics. Will repro and post the logs.
There was a problem hiding this comment.
Ok this one fails here:
buildkit/frontend/dockerfile/dockerfile_test.go
Lines 3996 to 3997 in c717d6a
Looking at the logs it seems it fails to pull the frontend gateway image:
=== CONT TestIntegration/TestCacheMultiPlatformImportExport/worker=dockerd-containerd/frontend=gateway
dockerfile_test.go:3954:
Error Trace: dockerfile_test.go:3954
run.go:86
run.go:189
Error: Received unexpected error:
pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
github.com/moby/buildkit/util/stack.Enable
/src/util/stack/stack.go:77
github.com/moby/buildkit/util/grpcerrors.FromGRPC
/src/util/grpcerrors/grpcerrors.go:196
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
/src/util/grpcerrors/intercept.go:41
google.golang.org/grpc.(*ClientConn).Invoke
/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
/src/api/services/control/control.pb.go:1535
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:231
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
failed to solve
github.com/moby/buildkit/client.(*Client).solve.func2
/src/client/solve.go:244
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
Test: TestIntegration/TestCacheMultiPlatformImportExport/worker=dockerd-containerd/frontend=gateway
client/mergediff_test.go
Outdated
| case "TestDiffSelf", "TestDiffScratch": | ||
| integration.SkipIfDockerdSnapshotter(t, sb, "https://github.com/moby/buildkit/pull/3176") |
There was a problem hiding this comment.
TestDiffSelf and TestDiffScratch hang too (dockerd panics):
=== CONT TestIntegration/TestDiffScratch/worker=dockerd-containerd
client_test.go:6071: checkAllReleasable: skipping check for exported tars in non-containerd test
mergediff_test.go:1256:
Error Trace: /src/client/client_test.go:5960
/src/client/mergediff_test.go:1256
/src/client/run.go:195
Error: Received unexpected error:
rpc error: code = Canceled desc = context canceled
failed to receive status
github.com/moby/buildkit/client.(*Client).solve.func4
/src/client/solve.go:306
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
Test: TestIntegration/TestDiffScratch/worker=dockerd-containerd
sandbox.go:249: panic: runtime error: invalid memory address or nil pointer dereference
sandbox.go:249: [signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x16db65e]
sandbox.go:249:
sandbox.go:249: goroutine 421 [running]:
sandbox.go:249: github.com/docker/docker/builder/builder-next/exporter/containerimage.(*imageExporterInstance).unpackImage(0xc000f52d80, {0x27ed380, 0xc0013845a0}, {{0xc000234150, 0x2f}, 0x0, {{0x1e715b8, 0x34}, {0xc0013a6140, 0x47}, ...}, ...}, ...)
sandbox.go:249: /go/src/github.com/docker/docker/builder/builder-next/exporter/containerimage/export.go:473 +0x37e
sandbox.go:249: github.com/docker/docker/builder/builder-next/exporter/containerimage.(*imageExporterInstance).Export(0xc000f52d80, {0x27ed380, 0xc000bdbdd0}, {{0x0?, 0x0?}, 0x0?, 0xc000bdbcb0?}, {0xc000660300, 0x19})
sandbox.go:249: /go/src/github.com/docker/docker/builder/builder-next/exporter/containerimage/export.go:355 +0x1bda
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func4({0x27ed380?, 0xc000bdbdd0}, {0xc000bee770?, 0x40f700?})
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:260 +0x6e
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1({0x27ed380, 0xc000bdbd10}, {0x27cf360, 0xc000e30720})
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:444 +0x1d9
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).InContext(0xc0010ea580, {0x27ed380, 0xc000bdb8f0}, 0xc000e30708)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:569 +0x130
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inBuilderContext({0x27ed380, 0xc000bdb8f0}, {0x27e5180, 0xc0010ea580}, {0x1df8879, 0x12}, {0x0, 0x0}, 0xc000bf2aa0)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:440 +0x1b3
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc000598240, {0x27ed380, 0xc000bdb8f0}, {0xc0006602e0, 0x19}, {0xc000660300, 0x19}, {0x0, 0xc000bdb920, {0x0, ...}, ...}, ...)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:259 +0x19fd
sandbox.go:249: github.com/docker/docker/builder/builder-next/control.(*Controller).Solve(0xc00056a160, {0x27ed380, 0xc000bdb8f0}, 0xc00016b320)
sandbox.go:249: /go/src/github.com/docker/docker/builder/builder-next/control/control.go:352 +0xc95
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1({0x27ed380, 0xc000bdb8f0}, {0x276a320?, 0xc00016b320})
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control/control.pb.go:1611 +0x78
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/util/grpcerrors.UnaryServerInterceptor({0x27ed380?, 0xc000bdb8f0?}, {0x276a320?, 0xc00016b320?}, 0xc0000cbad0?, 0x247a160?)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/grpcerrors/intercept.go:14 +0x3d
sandbox.go:249: github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control._Control_Solve_Handler({0x2697360?, 0xc00056a160}, {0x27ed380, 0xc000bdb8f0}, 0xc000bee700, 0x27c2808)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control/control.pb.go:1613 +0x138
sandbox.go:249: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002694a0, {0x27fc220, 0xc0010aae10}, 0xc00016aea0, 0xc000c62bd0, 0x38c8f78, 0x0)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1301 +0xb2b
sandbox.go:249: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0002694a0, {0x27fc220, 0xc0010aae10}, 0xc00016aea0, 0x0)
sandbox.go:249: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1642 +0xa2f
sandbox.go:249: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.2()
sandbox.go:249: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:938 +0x98
sandbox.go:249: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
sandbox.go:249: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:936 +0x28a
sandbox.go:249: > stopped 2022-11-22 17:01:01.50387662 +0000 UTC m=+10.060884675 exit status 2 2
|
@tonistiigi Found out image cache doesn't work with I guess that's because snapshots are put in another location. |
|
Enabled integration tests with oci exporter (see last commit) for containerd snapshotter using https://github.com/vvoland/moby/tree/c8d-store-rebased (rumpl/moby#99)
TestNamedOCILayoutContextExport.log https://github.com/crazy-max/buildkit/actions/runs/3286839062/jobs/5415408628#step:8:3727 https://github.com/crazy-max/buildkit/actions/runs/3286839062/jobs/5415408441#step:8:2206 https://github.com/crazy-max/buildkit/actions/runs/3286839062/jobs/5415408441#step:8:2680 TestOCILayoutPlatformSource.log https://github.com/crazy-max/buildkit/actions/runs/3286839062/jobs/5415408441#step:8:3106 https://github.com/crazy-max/buildkit/actions/runs/3286839062/jobs/5415408441#step:8:3654 then hangs for |
c22715f to
84ca71a
Compare
84ca71a to
0dd5c72
Compare
tonistiigi
left a comment
There was a problem hiding this comment.
Needs changes as discussed offline.
Add list of all supported test-features and the list of currently enabled ones. Skip can only happen for a defined feature. Based on dockerd configuration(if containerd image store enabled) the list on the enabled ones will change.
c23bd0e to
af771fa
Compare
|
TestClientGatewayIntegration.log |
6a24ba0 to
b195e49
Compare
|
What commit is that? Line numbers do not seem to match with |
|
Following test fails after rebasing: |
|
Other test that panics with containerd snapshotter |
56c3151 to
b34d876
Compare
b34d876 to
f3ece6f
Compare
|
@tonistiigi Added extra commit to fix #3355. |
f3ece6f to
84d9605
Compare
| var ErrRedirect = errors.New("unexpected redirect in response") | ||
|
|
||
| type Client struct { | ||
| scheme string |
There was a problem hiding this comment.
I think almost none of this is actually used. We don't need protos/tls or header/version negotiation. Only thing we need is to dial the UNIX socket and do the upgrade.
We can do this part later if you want. Eg. create a tracking issue for it or we can remove this commit from the current PR for now.
There was a problem hiding this comment.
Removed for now, will do this in another PR.
c6e1f22 to
12a80f8
Compare
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
12a80f8 to
b827bcb
Compare
Integration tests for
dockerdworker withcontainerd-snapshotterwas not enabled. This adds a new matrix item to the dockerd workflow to run tests using this snapshotter. Direct push is enabled with containerd snapshotter on most tests. Some of them have still issues (see comments review below).Tested here: https://github.com/crazy-max/buildkit/actions/runs/3257273194 with Docker version
https://github.com/rumpl/moby.git#c8d.Also add missing skips for some tests with dockerd worker:
TestTargetsDescribeDefinition: https://github.com/crazy-max/buildkit/actions/runs/3235292532/jobs/5299596379#step:9:1016TestTargetsList: https://github.com/crazy-max/buildkit/actions/runs/3235292532/jobs/5299596379#step:9:1361TestOutlineSecrets: https://github.com/crazy-max/buildkit/actions/runs/3235292532/jobs/5299596379#step:9:2078TestNamedOCILayoutContextExport: https://github.com/crazy-max/buildkit/actions/runs/3235292532/jobs/5299596379#step:9:2982TestNamedImageContextScratch: https://github.com/crazy-max/buildkit/actions/runs/3235292532/jobs/5299596379#step:9:3697TestNamedImageContextPlatform: https://github.com/crazy-max/buildkit/actions/runs/3235292532/jobs/5299596379#step:9:4067TestAttestationDefaultSubject: https://github.com/crazy-max/buildkit/actions/runs/3241141725/jobs/5312765705#step:9:1168Same for
testExportAnnotationsMediaTypesandtestExportAttestations.And another one related to #3171.