Remove in memory libcontainer containers map by crosbymichael · Pull Request #38931 · moby/moby

crosbymichael · 2019-03-25T20:18:23Z

This removes the in memory map and makes the libcontainerd code bind to containerd is a more conventional way.

derek · 2019-03-25T20:18:25Z

Thank you for your contribution. I've just checked and your Pull Request doesn't appear to have any description.
That's something we need before your Pull Request can be merged. Please see our contributing guide.

thaJeztah · 2019-03-26T19:34:13Z

Janky / Experimental

21:39:06 PASS: docker_cli_proxy_test.go:23: DockerDaemonSuite.TestCLIProxyProxyTCPSock	0.539s
21:44:06 --- FAIL: Test (2364.68s)
21:44:06 panic: DockerDaemonSuite.TestCleanupMountsAfterDaemonAndContainerKill test timed out after 5m0s [recovered]
21:44:06 	panic: DockerDaemonSuite.TestCleanupMountsAfterDaemonAndContainerKill test timed out after 5m0s

Power / Z (interestingly; those don't seem to be timing out the individual test after 5 minutes):

21:31:15 PASS: docker_cli_proxy_test.go:23: DockerDaemonSuite.TestCLIProxyProxyTCPSock	0.554s
01:21:28 Build timed out (after 300 minutes). Marking the build as failed.
01:21:28 Build timed out (after 300 minutes). Marking the build as aborted.

crosbymichael · 2019-03-26T19:43:07Z

@thaJeztah look at the daemon log for janky, its 60GB+

thaJeztah · 2019-03-26T20:03:48Z

This is the last bit of the s390x test-logs (in case it helps);

time="2019-03-25T21:31:14Z" level=debug msg="serving api on unix socket" socket="[inherited from parent]" 
time="2019-03-25T21:31:14.335086810Z" level=debug msg="garbage collected" d=1.567714ms 
time="2019-03-25T21:31:14.474735963Z" level=debug msg="event published" ns=moby topic="/tasks/create" type=containerd.events.TaskCreate 
time="2019-03-25T21:31:14.475104069Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
time="2019-03-25T21:31:14.475179759Z" level=error msg="failed to process event" container=9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e error="no such container: 9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e" event=create event-info="{9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 5958 0 0001-01-01 00:00:00 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
time="2019-03-25T21:31:14.495274373Z" level=debug msg="event published" ns=moby topic="/tasks/start" type=containerd.events.TaskStart 
time="2019-03-25T21:31:14.496286123Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/start
time="2019-03-25T21:31:14.496336100Z" level=error msg="failed to process event" container=9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e error="no such container: 9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e" event=start event-info="{9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 5958 0 0001-01-01 00:00:00 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
time="2019-03-25T21:31:14.588644109Z" level=debug msg="event published" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit 
time="2019-03-25T21:31:14.588867014Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit
time="2019-03-25T21:31:14.588923196Z" level=error msg="failed to process event" container=9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e error="no such container: 9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e" event=exit event-info="{9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 5958 0 2019-03-25 21:31:14.570435895 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
time="2019-03-25T21:31:14.620794836Z" level=info msg="shim reaped" id=9505d6e8d3a90a8df83902aa73fb5d534bca60aa62e56f0b6ea260542b4e7e1e 
time="2019-03-25T21:31:14.630561326Z" level=debug msg="event published" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete 
time="2019-03-25T21:31:14.630765629Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/delete
time="2019-03-25T21:31:14.630783073Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-03-25T21:31:14.634730087Z" level=debug msg="event published" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete 
time="2019-03-25T21:31:15.037830415Z" level=debug msg="Calling GET /v1.40/containers/json?all=1&filters=%7B%22status%22%3A%7B%22paused%22%3Atrue%7D%7D&limit=0"
time="2019-03-25T21:31:15.038310520Z" level=debug msg="Calling GET /v1.40/containers/json?all=1&limit=0"
time="2019-03-25T21:31:15.038706425Z" level=debug msg="Calling GET /v1.40/images/json"
time="2019-03-25T21:31:15.041332234Z" level=debug msg="Calling GET /v1.40/volumes"
time="2019-03-25T21:31:15.041358201Z" level=debug msg=VolumeStore.Find ByType=service.CustomFilter ByValue=0x110ad4300
time="2019-03-25T21:31:15.042132845Z" level=debug msg="Calling GET /v1.40/networks"
time="2019-03-25T21:31:15.042883206Z" level=debug msg="Calling GET /v1.40/plugins"
time="2019-03-25T21:31:15.616323267Z" level=debug msg="Calling GET /v1.40/containers/json?all=1&filters=%7B%22status%22%3A%7B%22paused%22%3Atrue%7D%7D&limit=0"
time="2019-03-25T21:31:15.616665464Z" level=debug msg="Calling GET /v1.40/containers/json?all=1&limit=0"
time="2019-03-25T21:31:15.616921549Z" level=debug msg="Calling GET /v1.40/images/json"
time="2019-03-25T21:31:15.619305025Z" level=debug msg="Calling GET /v1.40/volumes"
time="2019-03-25T21:31:15.619336132Z" level=debug msg=VolumeStore.Find ByType=service.CustomFilter ByValue=0x110ad4300
time="2019-03-25T21:31:15.619817565Z" level=debug msg="Calling GET /v1.40/networks"
time="2019-03-25T21:31:15.620349710Z" level=debug msg="Calling GET /v1.40/plugins"
time="2019-03-25T21:31:16.136823273Z" level=debug msg="Calling GET /v1.40/images/get?names=busybox%3Alatest"
time="2019-03-25T21:31:16.137245685Z" level=debug msg="CreateV1ID {\"architecture\":\"s390x\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"sh\"],\"ArgsEscaped\":true,\"Image\":\"sha256:1a7eb4a865a7c853e6c79072f53f4d568ca71f8f9c978858b6999400e4101cad\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"43993a255bacf17c7dcbe3bca72a4d01f34f3ab03e229153ca07f3dd2e75b44e\",\"container_config\":{\"Hostname\":\"43993a255bac\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"sh\\\"]\"],\"ArgsEscaped\":true,\"Image\":\"sha256:1a7eb4a865a7c853e6c79072f53f4d568ca71f8f9c978858b6999400e4101cad\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{}},\"created\":\"2017-11-02T08:18:28.648547045Z\",\"docker_version\":\"17.06.1-ce\",\"layer_id\":\"sha256:9e8a9a5f2215360420e8ec8da77e61b2512dd80f864957a0dacaa1c8fa58b8fe\",\"os\":\"linux\"}"
time="2019-03-25T21:31:16.137482498Z" level=debug msg="Assembling tar data for 3bde9804504f03ecf736a618aa81f611e295af4cf1aaaafe985ba770105a4b7b"
time="2019-03-25T21:31:16.534097265Z" level=debug msg="event published" ns=moby topic="/containers/create" type=containerd.events.ContainerCreate 
time="2019-03-25T21:31:16.542976310Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a/shim.sock" debug=true pid=6241 
time="2019-03-25T21:31:16Z" level=debug msg="registering ttrpc server" 
time="2019-03-25T21:31:16Z" level=debug msg="serving api on unix socket" socket="[inherited from parent]" 
time="2019-03-25T21:31:16.906495777Z" level=debug msg="event published" ns=moby topic="/tasks/create" type=containerd.events.TaskCreate 
time="2019-03-25T21:31:16.907231456Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
time="2019-03-25T21:31:16.907305504Z" level=error msg="failed to process event" container=649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a error="no such container: 649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a" event=create event-info="{649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 6259 0 0001-01-01 00:00:00 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
time="2019-03-25T21:31:16.923948950Z" level=debug msg="event published" ns=moby topic="/tasks/start" type=containerd.events.TaskStart 
time="2019-03-25T21:31:16.924274322Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/start
time="2019-03-25T21:31:16.924363763Z" level=error msg="failed to process event" container=649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a error="no such container: 649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a" event=start event-info="{649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 6259 0 0001-01-01 00:00:00 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
time="2019-03-25T21:31:17.375662852Z" level=debug msg="event published" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit 
time="2019-03-25T21:31:17.377665435Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit
time="2019-03-25T21:31:17.377764624Z" level=error msg="failed to process event" container=649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a error="no such container: 649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a" event=exit event-info="{649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 6259 143 2019-03-25 21:31:17.348966 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
time="2019-03-25T21:31:17.425173253Z" level=info msg="shim reaped" id=649feb44257a0b019cd7f45b73261ac82acd74c1a7db28bfe2484949b53fbd3a 
time="2019-03-25T21:31:17.434868712Z" level=debug msg="event published" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete 
time="2019-03-25T21:31:17.435019483Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/delete
time="2019-03-25T21:31:17.435046717Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-03-25T21:33:30.691156576Z" level=debug msg="clean 45 unused exec commands"

codecov · 2019-03-27T18:34:55Z

Codecov Report

Merging #38931 into master will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #38931      +/-   ##
==========================================
- Coverage   36.89%   36.89%   -0.01%     
==========================================
  Files         613      612       -1     
  Lines       45421    45346      -75     
==========================================
- Hits        16760    16730      -30     
+ Misses      26370    26326      -44     
+ Partials     2291     2290       -1

thaJeztah · 2019-03-28T10:58:39Z

Logs of the powerpc run; https://jenkins.dockerproject.org/job/Docker-PRs-powerpc/13837/console

time="2019-03-27T19:05:49.556836094Z" level=debug msg="Calling POST /v1.40/plugins/test/enable?timeout=30"
time="2019-03-27T19:05:49.558943639Z" level=debug msg="bundle dir created" bundle=/run/docker/plugins/6da598978f0f98980cc1df35a56658d3b6d0ad22de832f2a2f54b33ef469b9ca module=libcontainerd namespace=plugins.moby root=/go/src/github.com/docker/docker/bundles/test-integration/d288c678baf32/root/plugins/6da598978f0f98980cc1df35a56658d3b6d0ad22de832f2a2f54b33ef469b9ca/rootfs
time="2019-03-27T19:05:49.736262849Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: unknown\ngithub.com/docker/docker/vendor/github.com/containerd/containerd/errdefs.init.ializers\n\t/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/errdefs/errors.go:39\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:188\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_ppc64x.s:856" error_type="*errors.fundamental" module=api
time="2019-03-27T19:05:49.736340519Z" level=error msg="Handler for POST /v1.40/plugins/test/enable returned error: failed to start io pipe copy: containerd-shim: opening init-stdout failed: open init-stdout: no such file or directory: unknown"
time="2019-03-27T19:05:49.736521010Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: unknown\ngithub.com/docker/docker/vendor/github.com/containerd/containerd/errdefs.init.ializers\n\t/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/errdefs/errors.go:39\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:188\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_ppc64x.s:856" error_type="*errors.fundamental" module=api
time="2019-03-27T19:05:49.737268201Z" level=info msg="Processing signal 'interrupt'"
time="2019-03-27T19:05:49.737474316Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
time="2019-03-27T19:05:49.737515672Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
time="2019-03-27T19:05:49.738003917Z" level=debug msg="Unix socket /tmp/dxr/d288c678baf32/libnetwork/1f2b89b2381bbfd537699813aa7c109965e74d07f249f9dc3717babb83bf8a2b.sock doesn't exist. cannot accept client connections"
time="2019-03-27T19:05:49.738073622Z" level=debug msg="Cleaning up old mountid : start."
time="2019-03-27T19:05:49.738130392Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2019-03-27T19:05:49.738586326Z" level=debug msg="Cleaning up old mountid : done."
time="2019-03-27T19:05:49.738906591Z" level=debug msg="Clean shutdown succeeded"
time="2019-03-27T19:05:49.738973282Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

Cleaning up the error-message;

github.com/docker/docker/vendor/github.com/containerd/containerd/errdefs.init.ializers
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/errdefs/errors.go:39
runtime.main
	/usr/local/go/src/runtime/proc.go:188
runtime.goexit
	/usr/local/go/src/runtime/asm_ppc64x.s:856

This looks funny: errdefs.init.ializers, but not much info there /usr/local/go/src/runtime/proc.go:188, /usr/local/go/src/runtime/asm_ppc64x.s:856

I do notice that that socket path is 104 characters long (but I think that's exactly the maximum length?);

Unix socket /tmp/dxr/d288c678baf32/libnetwork/1f2b89b2381bbfd537699813aa7c109965e74d07f249f9dc3717babb83bf8a2b.sock doesn't exist. cannot accept client connections

cpuguy83 · 2019-04-04T18:33:33Z

Has experimental been timing out for each of these runs or just different errors?

crosbymichael · 2019-04-04T18:36:07Z

timeouts, it maybe legit, other prs recently are failing on it so idk

thaJeztah · 2019-04-04T22:35:13Z

Linting failure;

22:15:07 libcontainerd/types/types.go:65:6:warning: exported type Process should have comment or be unexported (golint)
22:15:07 daemon/util_test.go:29:2:warning: wrong number of return values (want 4, got 3) (gosimple)

thaJeztah · 2019-04-05T00:22:08Z

@crosbymichael needs a rebase now, likely because your other PR was merged

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

crosbymichael · 2019-04-08T19:17:02Z

Everything is good now

crosbymichael · 2019-04-08T19:17:21Z

rs5-process is ci failures and its not re-triggering

thaJeztah · 2019-04-08T19:36:58Z

😞 yeah, for some reason it won't trigger through labels (it works if you use rebuild/*); I triggered it manually in Jenkins

thaJeztah

LGTM

GordonTheTurtle added the status/0-triage label Mar 25, 2019

derek bot added the invalid label Mar 25, 2019

thaJeztah added status/failing-ci Indicates that the PR in its current state fails the test suite and removed invalid labels Mar 26, 2019

crosbymichael force-pushed the opts branch from d04f921 to 8160cd3 Compare March 27, 2019 18:34

thaJeztah added rebuild/* and removed status/failing-ci Indicates that the PR in its current state fails the test suite labels Mar 27, 2019

GordonTheTurtle removed the rebuild/* label Mar 27, 2019

crosbymichael added the rebuild/janky label Mar 28, 2019

GordonTheTurtle removed the rebuild/janky label Mar 28, 2019

thaJeztah added area/runtime Runtime status/2-code-review and removed status/0-triage labels Apr 1, 2019

crosbymichael force-pushed the opts branch 2 times, most recently from 3c77a1e to a2ddf63 Compare April 1, 2019 21:26

crosbymichael added rebuild/windowsRS1 labels Apr 2, 2019

GordonTheTurtle removed rebuild/experimental labels Apr 2, 2019

crosbymichael force-pushed the opts branch from a2ddf63 to d96ff00 Compare April 2, 2019 15:37

crosbymichael added rebuild/experimental labels Apr 2, 2019

GordonTheTurtle removed the rebuild/experimental label Apr 2, 2019

crosbymichael force-pushed the opts branch from d96ff00 to e0bc3c3 Compare April 4, 2019 18:46

crosbymichael added the rebuild/janky label Apr 4, 2019

GordonTheTurtle removed the rebuild/janky label Apr 4, 2019

crosbymichael force-pushed the opts branch from e0bc3c3 to 29389f2 Compare April 4, 2019 22:13

dmcgowan added the rebuild/janky label Apr 4, 2019

GordonTheTurtle removed the rebuild/janky label Apr 4, 2019

Export WithBundle code

adb15c2

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

crosbymichael force-pushed the opts branch 2 times, most recently from c10cc1b to 412e079 Compare April 5, 2019 12:56

Remove inmemory container map

b9b5dc3

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

crosbymichael force-pushed the opts branch from 412e079 to b9b5dc3 Compare April 5, 2019 19:48

crosbymichael added rebuild/windowsRS5-process labels Apr 8, 2019

GordonTheTurtle removed the rebuild/experimental label Apr 8, 2019

thaJeztah removed the rebuild/windowsRS5-process label Apr 8, 2019

thaJeztah approved these changes Apr 8, 2019

View reviewed changes

thaJeztah added status/4-merge and removed status/2-code-review labels Apr 8, 2019

crosbymichael merged commit 7e03619 into moby:master Apr 8, 2019

crosbymichael deleted the opts branch April 8, 2019 20:42

ddebroy mentioned this pull request Aug 8, 2019

Fix regression in handling of NotFound err during startup ENGCORE-929 #39703

Merged

thaJeztah mentioned this pull request Aug 9, 2019

[19.03 backport] Fix regression in handling of NotFound err during startup ENGCORE-929 docker-archive/engine#317

Merged

cpuguy83 mentioned this pull request Feb 5, 2020

Killing containerd breaks interaction with running containers #36367

Closed

Conversation

crosbymichael commented Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

derek bot commented Mar 25, 2019

Uh oh!

thaJeztah commented Mar 26, 2019

Uh oh!

crosbymichael commented Mar 26, 2019

Uh oh!

thaJeztah commented Mar 26, 2019

Uh oh!

codecov bot commented Mar 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

thaJeztah commented Mar 28, 2019

Uh oh!

cpuguy83 commented Apr 4, 2019

Uh oh!

crosbymichael commented Apr 4, 2019

Uh oh!

thaJeztah commented Apr 4, 2019

Uh oh!

thaJeztah commented Apr 5, 2019

Uh oh!

crosbymichael commented Apr 8, 2019

Uh oh!

crosbymichael commented Apr 8, 2019

Uh oh!

thaJeztah commented Apr 8, 2019

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

crosbymichael commented Mar 25, 2019 •

edited

Loading

codecov bot commented Mar 27, 2019 •

edited

Loading