Skip to content

[cri-integration] testcase TestIssue10467 #12404

@fuweid

Description

@fuweid

Description

https://github.com/containerd/containerd/actions/runs/18762002558/job/53528959734?pr=12399

2025-10-23T21:34:52.9498455Z     default: time="2025-10-23T21:34:52Z" level=info msg="[RuntimeService] RemovePodSandbox Response (podSandboxID=fb808a0c16d42a947e6c67f0d340801391bb605965a3461c862010b4e0343695)"
2025-10-23T21:34:52.9512828Z     default:     release_upgrade_linux_test.go:519: Checking container a3b3e08f9a218f98d296c0159889a7644348b749e5dd3d54704da7e3754a2732's shim client
2025-10-23T21:34:52.9525101Z     default:     release_upgrade_linux_test.go:521:
2025-10-23T21:34:52.9527594Z     default:         	Error Trace:	/root/go/src/github.com/containerd/containerd/integration/release_upgrade_linux_test.go:521
2025-10-23T21:34:52.9533100Z     default:         	            				/root/go/src/github.com/containerd/containerd/integration/issue10467_linux_test.go:120
2025-10-23T21:34:52.9534016Z     default:         	Error:      	An error is expected but got nil.
2025-10-23T21:34:52.9535524Z     default:         	Test:       	TestIssue10467
2025-10-23T21:34:52.9538144Z     default:         	Messages:   	should be closed after deleting pod
2025-10-23T21:34:52.9541306Z     default:     release_upgrade_linux_test.go:519: Checking container edc132f95a37adf7d7377da18960d3f350842064871b7feb63c003af1c679bcd's shim client
2025-10-23T21:34:52.9556615Z     default:     release_upgrade_linux_test.go:521:
2025-10-23T21:34:52.9559786Z     default:         	Error Trace:	/root/go/src/github.com/containerd/containerd/integration/release_upgrade_linux_test.go:521
2025-10-23T21:34:52.9561024Z     default:         	            				/root/go/src/github.com/containerd/containerd/integration/issue10467_linux_test.go:120
2025-10-23T21:34:52.9562309Z     default:         	Error:      	Error "rpc error: code = Unimplemented desc = service containerd.task.v3.Task" does not contain "ttrpc: closed"
2025-10-23T21:34:52.9563363Z     default:         	Test:       	TestIssue10467
2025-10-23T21:34:52.9563935Z     default:         	Messages:   	should be closed after deleting pod
2025-10-23T21:34:52.9573788Z     default:     release_upgrade_linux_test.go:519: Checking container 3b2d6f2d00681b11760590356e972103d7368efcf5bbd0502edd547aebdd61ee's shim client
2025-10-23T21:34:52.9576464Z     default:     release_upgrade_linux_test.go:521:
2025-10-23T21:34:52.9789799Z     default:         	Error Trace:	/root/go/src/github.com/containerd/containerd/integration/release_upgrade_linux_test.go:521
2025-10-23T21:34:52.9791622Z     default:         	            				/root/go/src/github.com/containerd/containerd/integration/issue10467_linux_test.go:120
2025-10-23T21:34:52.9793252Z     default:         	Error:      	Error "rpc error: code = Unimplemented desc = service containerd.task.v3.Task" does not contain "ttrpc: closed"
2025-10-23T21:34:52.9794280Z     default:         	Test:       	TestIssue10467
2025-10-23T21:34:52.9794981Z     default:         	Messages:   	should be closed after deleting pod
2025-10-23T21:34:52.9797916Z     default:     release_upgrade_linux_test.go:519: Checking container 5b66e3161d620c3fed23ea63e616a19ef7023397c2200477f8ea0339c6e2bdb2's shim client
2025-10-23T21:34:52.9801492Z     default:     issue10467_linux_test.go:97: Cleanup all the pods
2025-10-23T21:34:52.9804266Z     default: time="2025-10-23T21:34:52Z" level=info msg="Connecting to runtime service /tmp/TestIssue10467272859221/002/containerd.sock"
2025-10-23T21:34:52.9807203Z     default: time="2025-10-23T21:34:52Z" level=info msg="[RuntimeService] ListPodSandbox (filter=<nil>, timeout=1m0s)"

The shim is leaky after all containers are deleted.

2025-10-23T21:34:53.1359693Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.757208213Z" level=info msg="Container to stop \"edc132f95a37adf7d7377da18960d3f350842064871b7feb63c003af1c679bcd\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
2025-10-23T21:34:53.1362823Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.757222941Z" level=info msg="Container to stop \"a3b3e08f9a218f98d296c0159889a7644348b749e5dd3d54704da7e3754a2732\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
2025-10-23T21:34:53.1365625Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.765325691Z" level=debug msg="Received containerd event timestamp - 2025-10-23 21:34:51.764794635 +0000 UTC, namespace - \"k8s.io\", topic - \"/tasks/exit\""
2025-10-23T21:34:53.1369374Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.765514380Z" level=debug msg="TaskExit event in podsandbox handler container_id:\"7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902\" id:\"7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902\" pid:41647 exit_status:137 exited_at:{seconds:1761255291 nanos:764766443}"


2025-10-23T21:34:53.1372988Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.791063995Z" level=error msg="failed to delete task" error="rpc error: code = NotFound desc = cannot delete a deleted process: not found" id=7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902


2025-10-23T21:34:53.1375616Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.791300804Z" level=error msg="failed to delete task" error="ttrpc: closed" 
id=7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902


2025-10-23T21:34:53.1378053Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.791355035Z" level=info msg="shim disconnected" id=7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902 namespace=k8s.io
2025-10-23T21:34:53.1380370Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.791366135Z" level=info msg="cleaning up after shim disconnected" id=7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902 namespace=k8s.io
2025-10-23T21:34:53.1382819Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.791372126Z" level=info msg="cleaning up dead shim" id=7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902 namespace=k8s.io
2025-10-23T21:34:53.1385969Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.804276373Z" level=warning msg="warnings while cleaning up dead shim" id=7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902 namespace=k8s.io warnings="time=\"2025-10-23T21:34:51Z\" level=debug msg=\"starting signal loop\" namespace=k8s.io pid=42014 runtime=io.containerd.runc.v2\n"
2025-10-23T21:34:53.1389312Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.805400757Z" level=debug msg="Received containerd event timestamp - 2025-10-23 21:34:51.804961903 +0000 UTC, namespace - \"k8s.io\", topic - \"/tasks/exit\""
2025-10-23T21:34:53.1392734Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.805436041Z" level=info msg="Events for \"7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902\" is in backoff, enqueue event container_id:\"7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902\" id:\"7054fb27f09d57b9af28e8a869edb071db091e6c3399d49f2fda41b57dcdb902\" pid:41647 exit_status:137 exited_at:{seconds:1761255291 nanos:803326674}"
2025-10-23T21:34:53.1396779Z     default:     release_upgrade_linux_test.go:1052: time="2025-10-23T21:34:51.805699821Z" level=info msg="received exit event 

Steps to reproduce the issue

Race condition. hard to reproduce.
Same issue to #12344

Describe the results you received and expected

Should shutdown shim server

What version of containerd are you using?

main

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions