Skip to content

[action] [PR:20024] Fix flaky capture service start (#20024)#20130

Merged
mssonicbld merged 1 commit intosonic-net:202311from
mssonicbld:cherry/202311/20024
Sep 4, 2024
Merged

[action] [PR:20024] Fix flaky capture service start (#20024)#20130
mssonicbld merged 1 commit intosonic-net:202311from
mssonicbld:cherry/202311/20024

Conversation

@mssonicbld
Copy link
Copy Markdown
Collaborator

Why I did it

Dropping control character (message sent when XSUB connects to XPUB as part of ZMQ Proxy setup to notify that subscription has been made) in do capture has been flaky since control character is not guaranteed to be the first message sent if there are events (like event-down-ctr) being published to XSUB.

Scenarios

  1. Control character is sent and is first message when starting capture service

eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1
eventd#eventd#eventd: :- do_capture: Received subscription message when XSUB connects to XPUB

  1. Events like event-down ctr is sent before control character

eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 17 sonic-events-host
eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 0 0 4 0 0 0 1 d 103 {"sonic-events-host:event-stopped-ctr":{"ctr_name":"EVENTD","timestamp":"2024-08-27T00:02:51.407518Z"}} 1 r 36 3357542f-bae1-458f-a804-660e620d21f5 1 s 1 9 1 t 19 1724716971407591080
heartbeat_ctrl: Set heartbeat_ctrl pause=1
do_capture: Received subscription message when XSUB connects to XPUB

  1. Control character is not sent at all

eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1

  1. Control character is delayed and not caught when starting capture service, but is then caught after causing deserialize error.

do_capture: Receiving event from source: 22 serialization::archive 18 17 sonic-events-host, will read second part of event
deserialize: deserialize Failed: input stream errorstr[0:64]:(#001) data type: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&
zmq_read_part: Failed to deserialize part rc=-2
zmq_read_part: last:errno=11
zmq_message_read: Failure to read part1 rc=-2
zmq_message_read: last:errno=11

We can cover these scenarios by just dropping the control character inside zmq_message_read as part of events_common in swsscommon (different PR). In this PR we will remove such handling logic and make sure that empty events that will be sent by control character are ignored.

Work item tracking
  • Microsoft ADO (number only):28728116

How I did it

Remove logic for handling control character

How to verify it

UT and sonic-mgmt test cases.

#### Why I did it

Dropping control character (message sent when XSUB connects to XPUB as part of ZMQ Proxy setup to notify that subscription has been made) in do capture has been flaky since control character is not guaranteed to be the first message sent if there are events (like event-down-ctr) being published to XSUB.

Scenarios

1) Control character is sent and is first message when starting capture service

`eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1`
`eventd#eventd#eventd: :- do_capture: Received subscription message when XSUB connects to XPUB`

2) Events like event-down ctr is sent before control character

`eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 17 sonic-events-host`
`eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 0 0 4 0 0 0 1 d 103 {"sonic-events-host:event-stopped-ctr":{"ctr_name":"EVENTD","timestamp":"2024-08-27T00:02:51.407518Z"}} 1 r 36 3357542f-bae1-458f-a804-660e620d21f5 1 s 1 9 1 t 19 1724716971407591080`
`heartbeat_ctrl: Set heartbeat_ctrl pause=1`
`do_capture: Received subscription message when XSUB connects to XPUB`

3) Control character is not sent at all

`eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1`

4) Control character is delayed and not caught when starting capture service, but is then caught after causing deserialize error.

`do_capture: Receiving event from source: 22 serialization::archive 18 17 sonic-events-host, will read second part of event`
`deserialize: deserialize Failed: input stream errorstr[0:64]:(sonic-net#1) data type: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&`
`zmq_read_part: Failed to deserialize part rc=-2`
`zmq_read_part: last:errno=11`
`zmq_message_read: Failure to read part1 rc=-2`
`zmq_message_read: last:errno=11`

We can cover these scenarios by just dropping the control character inside zmq_message_read as part of events_common in swsscommon (different PR). In this PR we will remove such handling logic and make sure that empty events that will be sent by control character are ignored.

##### Work item tracking
- Microsoft ADO **(number only)**:28728116

#### How I did it

Remove logic for handling control character

#### How to verify it

UT and sonic-mgmt test cases.
@mssonicbld
Copy link
Copy Markdown
Collaborator Author

Original PR: #20024

@mssonicbld mssonicbld mentioned this pull request Sep 4, 2024
11 tasks
@mssonicbld mssonicbld merged commit e9fec15 into sonic-net:202311 Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants