Skip to content

Workaround for the announce_routes stuck issue#3557

Merged
wangxin merged 1 commit intosonic-net:masterfrom
wangxin:ann-routes-robust-pr
Jun 1, 2021
Merged

Workaround for the announce_routes stuck issue#3557
wangxin merged 1 commit intosonic-net:masterfrom
wangxin:ann-routes-robust-pr

Conversation

@wangxin
Copy link
Copy Markdown
Collaborator

@wangxin wangxin commented May 31, 2021

Description of PR

Summary:
Fixes # (issue)

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Approach

What is the motivation for this PR?

We observed that the announcing routes step may got stuck ocassionally.
Possibly the stuck happened during HTTP posting to PTF docker to announce
routes.

How did you do it?

This change made some changes to avoid the stuck forever issue.

  • Add timeout to the requests.post method.
  • Add try...except for the function calls of announcing routes in the
    announce_routes.py module.
  • Force the exabgp module to refresh supervisord in case of start and restart.
  • Remove the line of referring non-exist logger object in the
    announce_routes.py module.

How did you verify/test it?

Tested testbed-cli.sh restart-ptf and testbed-cli.sh announce-routes.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

We observed that the announcing routes step may got stuck ocassionally.
Possibly the stuck happened during HTTP posting to PTF docker to announce
routes. This change made some changes to avoid the stuck forever issue.
* Add timeout to the requests.post method.
* Add try...except for the function calls of announcing routes in the
  announce_routes.py module.
* Force the exabgp module to refresh supervisord in case of start and restart.
* Remove the line of referring non-exist logger object in the
  announce_routes.py module.

Signed-off-by: Xin Wang <xiwang5@microsoft.com>
@wangxin wangxin requested a review from a team May 31, 2021 14:01
@bingwang-ms
Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@wangxin
Copy link
Copy Markdown
Collaborator Author

wangxin commented Jun 1, 2021

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@sonic-net sonic-net deleted a comment from azure-pipelines Bot Jun 1, 2021
@wangxin
Copy link
Copy Markdown
Collaborator Author

wangxin commented Jun 1, 2021

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@wangxin wangxin merged commit 111f07d into sonic-net:master Jun 1, 2021
@wangxin wangxin deleted the ann-routes-robust-pr branch September 9, 2021 08:08
vmittal-msft pushed a commit to vmittal-msft/sonic-mgmt that referenced this pull request Sep 28, 2021
We observed that the announcing routes step may got stuck ocassionally.
Possibly the stuck happened during HTTP posting to PTF docker to announce
routes. This change made some changes to avoid the stuck forever issue.
* Add timeout to the requests.post method.
* Add try...except for the function calls of announcing routes in the
  announce_routes.py module.
* Force the exabgp module to refresh supervisord in case of start and restart.
* Remove the line of referring non-exist logger object in the
  announce_routes.py module.

Signed-off-by: Xin Wang <xiwang5@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants