containerd 1.7.24 crash due to "fatal error: concurrent map iteration and map write" in CRI plugin #11302
Description
Description
Similar to #7797, I have a report of containerd crash for a k8s workload on a recent build of containerd v1.7.24. From the stack trace, the issue stems from this line where HTTP stream handler connection (h.conn) (type interface) is provided as value to klog.ErrorS which results in a fmt.Sprintf("%+v", value) on this line.
All other references of h.conn are using %p to avoid formatting of the connection.
Steps to reproduce the issue
Describe the results you received and expected
Stack trace
fatal error: concurrent map iteration and map write
goroutine 585192 [running]:
reflect.mapiternext(0x55a990967469?)
/usr/libexec/go-1.22/src/runtime/map.go:1397 +0x13
reflect.(*MapIter).Next(0xc0008eb428?)
/usr/libexec/go-1.22/src/reflect/value.go:2005 +0x74
internal/fmtsort.Sort({0x55a99201f620?, 0xc0011c8878?, 0xc0011c8870?})
/usr/libexec/go-1.22/src/internal/fmtsort/sort.go:62 +0x1d3
fmt.(*pp).printValue(0xc000bfe410, {0x55a99201f620?, 0xc0011c8878?, 0x2c?}, 0x76, 0x2)
/usr/libexec/go-1.22/src/fmt/print.go:816 +0x988
fmt.(*pp).printValue(0xc000bfe410, {0x55a9921b1a80?, 0xc0011c8870?, 0x55a9909bec90?}, 0x76, 0x1)
/usr/libexec/go-1.22/src/fmt/print.go:853 +0x11be
fmt.(*pp).printValue(0xc000bfe410, {0x55a99218a440?, 0xc0011c8870?, 0x0?}, 0x76, 0x0)
/usr/libexec/go-1.22/src/fmt/print.go:921 +0xaf4
fmt.(*pp).printArg(0xc000bfe410, {0x55a99218a440, 0xc0011c8870}, 0x76)
/usr/libexec/go-1.22/src/fmt/print.go:759 +0x4bc
fmt.(*pp).doPrintf(0xc000bfe410, {0x55a991b85625, 0x3}, {0xc0008ebc50, 0x1, 0x1})
/usr/libexec/go-1.22/src/fmt/print.go:1075 +0x37e
fmt.Sprintf({0x55a991b85625, 0x3}, {0xc0008ebc50, 0x1, 0x1})
/usr/libexec/go-1.22/src/fmt/print.go:239 +0x53
k8s.io/klog/v2/internal/serialize.Formatter.AnyToString({0xc0023c2b60?}, {0x55a99218a440?, 0xc0011c8870?})
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/internal/serialize/keyvalues.go:250 +0x5f
k8s.io/klog/v2/internal/serialize.Formatter.KVFormat({0x1?}, 0xc0023c2b60, {0x55a991f45a20, 0x55a9922b46a0}, {0x55a99218a440, 0xc0011c8870})
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/internal/serialize/keyvalues.go:237 +0x1e6
k8s.io/klog/v2/internal/serialize.Formatter.KVListFormat({0xc0023c2b60?}, 0xc0023c2b60, {0xc0022fd560, 0x6, 0xf?})
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/internal/serialize/keyvalues.go:166 +0x3b
k8s.io/klog/v2/internal/serialize.KVListFormat(...)
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/internal/serialize/keyvalues.go:171
k8s.io/klog/v2.(*loggingT).printS(0x55a993103100, {0x55a9922b8f48, 0xc000e89c80}, 0x2, 0x1, {0x55a991b9d3a2, 0xf}, {0xc0022fd560, 0x6, 0x6})
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:798 +0x17d
k8s.io/klog/v2.(*loggingT).errorS(0x55a991e310d8?, {0x55a9922b8f48, 0xc000e89c80}, 0x7fa3f869c400?, {0x0?, 0x0?}, 0x55a9918604e0?, {0x55a991b9d3a2?, 0xc001582ea0?}, {0xc0022fd560, ...})
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:771 +0x165
k8s.io/klog/v2.ErrorS(...)
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:1584
github.com/containerd/containerd/pkg/cri/streaming/portforward.(*httpStreamHandler).portForward(0xc0012e2cb0, 0xc001c783c0)
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/pkg/cri/streaming/portforward/httpstream.go:282 +0x6cd
created by github.com/containerd/containerd/pkg/cri/streaming/portforward.(*httpStreamHandler).run in goroutine 571738
/home/builder/rpmbuild/BUILD/containerd-1.7.24/GOPATH/src/github.com/containerd/containerd/pkg/cri/streaming/portforward/httpstream.go:237 +0x44c
What version of containerd are you using?
containerd v1.7.24
Any other relevant information
Although report is from containerd v1.7.24, containerd v1.6 is almost certainly affected. Currently unsure if containerd v2.0 is affected as the CRI plugin has been refactored to use k8s.io/kubelet/pkg/cri/streaming/portforward instead of instead.
Show configuration if it is related to CRI plugin.
No response