Skip to content

syncd crash observed during sonic-mgmt reboot tests #17346

@kenneth-arista

Description

@kenneth-arista
root@a0016a6416db:/tmp# gdb /usr/bin/syncd -c ./syncd.1699281446.29.core
...
Core was generated by `/usr/bin/syncd --diag -u -s -p /etc/sai.d/sai.profile -x /usr/share/sonic/hwsku'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f73ed7d8ce1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f73ecf149c0 (LWP 29))]
(gdb) bt
#0  0x00007f73ed7d8ce1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f73ed7c2537 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f73eda287ec in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f73eda33966 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f73eda339d1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f73eda33c65 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f73eda2b458 in std::__throw_system_error(int) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f73eda5d0fc in std::thread::join() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00005638720a6d6e in syncd::MdioIpcServer::stopMdioThread (this=0x563872cbf200) at MdioIpcServer.cpp:442
#9  0x00005638720716b9 in syncd::Syncd::run (this=this@entry=0x563872cad850) at /usr/include/c++/10/bits/shared_ptr_base.h:1324
#10 0x00005638720579af in syncd_main (argc=argc@entry=12, argv=argv@entry=0x7ffc0f9c62c8) at syncd_main.cpp:71
#11 0x0000563872055f9f in main (argc=12, argv=0x7ffc0f9c62c8) at main.cpp:9

It looks like the exception is hit in the join() while stopping the MdioThread:

430 void MdioIpcServer::stopMdioThread(void)
431 {
432     SWSS_LOG_ENTER();
433
434 #ifdef MDIO_ACCESS_USE_NPU
435     /* MDIO IPC server thread is only relevant in syncd but not in gbsyncd */
436     if (!MdioIpcServer::m_syncdContext)
437     {
438         return;
439     }
440
441     m_taskAlive = 0;
442     m_taskThread.join();
443     SWSS_LOG_NOTICE("IPC task thread is stopped\n");
444 #endif
445 }

Metadata

Metadata

Assignees

Labels

BRCMTriagedthis issue has been triaged

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions