When upgrading to master, on 1/2 nodes Cilium failed out the first couple of times before successfully running. In the previous logs, there was this error at the end:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1b5bae1]
goroutine 314 [running]:
github.com/cilium/cilium/pkg/identity/cache.(*localIdentityCache).lookupByID(0x0, 0xc0000023d9, 0x0)
/go/src/github.com/cilium/cilium/pkg/identity/cache/local.go:168 +0x41
github.com/cilium/cilium/pkg/identity/cache.(*CachingIdentityAllocator).LookupIdentityByID(0xc00096a2a0, 0x298ec00, 0xc00017f640, 0x23d9, 0x25ea694)
/go/src/github.com/cilium/cilium/pkg/identity/cache/cache.go:245 +0x86
github.com/cilium/cilium/daemon/cmd.(*Daemon).notifyOnDNSMsg.func3(0xc000f001c0)
/go/src/github.com/cilium/cilium/daemon/cmd/fqdn.go:471 +0x5ef
github.com/cilium/cilium/pkg/proxy/logger.NewLogRecord(0x2947340, 0x3fddbc0, 0x29b9cc0, 0xc000978900, 0x25cdc35, 0x7, 0xffffff00, 0xc000f79ae0, 0x5, 0x5, ...)
/go/src/github.com/cilium/cilium/pkg/proxy/logger/logger.go:110 +0x28e
github.com/cilium/cilium/daemon/cmd.(*Daemon).notifyOnDNSMsg(0xc00000a1e0, 0xbfd8083315353350, 0xa66aed71, 0x3a817a0, 0xc000978900, 0xc000501690, 0xf, 0xc0005016f0, 0xc, 0xc0003ec750, ...)
/go/src/github.com/cilium/cilium/daemon/cmd/fqdn.go:439 +0xca5
github.com/cilium/cilium/pkg/fqdn/dnsproxy.(*DNSProxy).ServeDNS(0xc0003ceb40, 0x29ae500, 0xc000407200, 0xc0003ec750)
/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:570 +0x171a
github.com/miekg/dns.(*Server).serveDNS(0xc00014c5a0, 0xc000d1b800, 0x3a, 0x200, 0xc000407200)
/go/src/github.com/cilium/cilium/vendor/github.com/miekg/dns/server.go:597 +0x211
github.com/miekg/dns.(*Server).serveUDPPacket(0xc00014c5a0, 0xc0003f20b0, 0xc000d1b800, 0x3a, 0x200, 0xc000bc6120, 0xc000599a50)
/go/src/github.com/cilium/cilium/vendor/github.com/miekg/dns/server.go:552 +0xb2
github.com/miekg/dns.(*Server).serveUDP.func2(0xc00014c5a0, 0xc0003f20b0, 0xc000d1b800, 0x3a, 0x200, 0xc000bc6120, 0xc000599a50)
/go/src/github.com/cilium/cilium/vendor/github.com/miekg/dns/server.go:478 +0x67
created by github.com/miekg/dns.(*Server).serveUDP
/go/src/github.com/cilium/cilium/vendor/github.com/miekg/dns/server.go:477 +0x24f
After two restarts, the problem seemed to resolve itself.
I installed v1.7, then upgraded to v1.8, then upgraded to master (
docker.io/cilium/cilium:latest) as outlined in my comment here:#13259 (comment)
The connectivity check was running in the background, including the
pod-to-external-fqdn-allow-google-cnppod with DNS + FQDN policy applied.When upgrading to master, on 1/2 nodes Cilium failed out the first couple of times before successfully running. In the previous logs, there was this error at the end:
After two restarts, the problem seemed to resolve itself.