Description

Fix sonic-net/sonic-buildimage#10530
Fix checkReplyType failed issue via recreating xcvr_table_helper on forking subprocess

In a rare scenario, xcvrd can fail due to the following exception:

xcvrd Traceback (most recent call last):
xcvrd   File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
xcvrd     self.run()
xcvrd   File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
xcvrd     self._target(*self._args, **self._kwargs)
xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 1140, in task_worker
xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 548, in del_port_sfp_dom_info_from_db
xcvrd     dom_tbl._del(port_name)
xcvrd   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 2474, in _del
xcvrd     return self.delete(*args, **kwargs)
xcvrd   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 2468, in delete
xcvrd     return _swsscommon.Table_delete(self, *args, **kwargs)
xcvrd RuntimeError: Expected to get redis type 3 got type 5, err: NON-STRING-REPLY: Input/output error

Signed-off-by: Stephen Sun stephens@nvidia.com

Motivation and Context

xcvr_table_helper is an integrated object containing all table and database connector objects for redis database accessing which are socket-based. It is created at the beginning of the xcvrd daemon. The sockets will be shared across parent and children processes after children forked, which is the chief culprit of the issue.

The application communicates with redis-db server in a synchronous way, which means the message flow is like this:

a process sends the request to redis-db server and then blocks
the redis-db server handles and replies the request
the process receives the reply and then continue to run

All the messages will transport via sockets.

For xcvrd, it works in a sparse mode which means the frequency at which it operates redis-db tables is relatively low and a child and the parent rarely send/receive messages at the same time, which means it works well even if two application processes, like a child and the parent, shares the socket via which they communicate with the redis-server in most cases.
However, it is still possible that both children and parent operates redis-db tables at the same time and messages interleave, which can cause a process receives a message which was sent to other processes sharing the socket with it. In this case, checkReplyType fails because the reply type is not expected.

To resolve the issue, the XcvrTableHelper should be a class member instead of a global variable. By doing so, the sockets created by it will not be duplicated among subprocess forking.

How Has This Been Tested?

Regression and manually test.

Additional Information (Optional)