Skip to content

Enhanced route_check.py for multi_asic platforms#3077

Merged
deepak-singhal0408 merged 7 commits intosonic-net:masterfrom
deepak-singhal0408:deepsinghal/enhance_route_checker_for_multi_asic
Dec 19, 2023
Merged

Enhanced route_check.py for multi_asic platforms#3077
deepak-singhal0408 merged 7 commits intosonic-net:masterfrom
deepak-singhal0408:deepsinghal/enhance_route_checker_for_multi_asic

Conversation

@deepak-singhal0408
Copy link
Copy Markdown
Contributor

@deepak-singhal0408 deepak-singhal0408 commented Dec 12, 2023

What I did

Enhanced route_check.py script to cover multi-asic platforms.
Accordingly enhanced the test files as well.

MSFT ADO: 25416673

How I did it

Enhanced the route_check.py script to take additional optional parameter(--n/namespace). Without this parameter, the check will be run on all asics in multi-asic platforms.
Different connections to DBs are modified accordingly to connect to relevant ns dbs.
Result will be encapsulated under different namespace. For single asic, results will be displayed under Default-Namespace("").

testData and the testfiles are enhanced accordinly.

How to verify it

  1. Verified that all pytest UT cases are passing.
  2. Verified the route_check_test.sh script on single asic and multi-asic platforms.
  3. Verified Monit routecheck outputs by simulating a failure scenario on both single asic and multi-asic platforms.

output from Monit Check:
Single Asic:
xxx/usr/local/bin# monit status routecheck
Monit 5.20.0 uptime: 1d 20h 32m

Program 'routeCheck'
status Status failed
monitoring status Monitored
monitoring mode active
on reboot start
last exit value 255
last output Failure results: {{
"": {
"missed_ROUTE_TABLE_routes": [
"20c0:d9b8:99:80::/64"
]
}
}}
Failed. Look at reported mismatches above
add: {
"": []
}
del: {
"": []
}
data collected Tue, 12 Dec 2023 20:30:11

'''
Multi Asic:
'''
/bin# monit status routecheck
Monit 5.20.0 uptime: 1d 23h 51m

Program 'routeCheck'
status Status failed
monitoring status Monitored
monitoring mode active
on reboot start
last exit value 255
last output Failure results: {{
"asic0": {
"missed_ROUTE_TABLE_routes": [
"1.0.0.0/16"
]
},
"asic1": {
"missed_ROUTE_TABLE_routes": [
"1.0.0.0/16"
]
},
"asic2": {
"missed_ROUTE_TABLE_routes": [
"1.0.0.0/16"
]
}
}}
Failed. Look at reported mismatches above
add: {
"asic0": [],
"asic1": [],
"asic2": []
}
del: {
"asic0": [],
"asic1": [],
"asic2": []
}
data collected Tue, 12 Dec 2023 23:54:23
'''

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Copy link
Copy Markdown
Contributor

@arlakshm arlakshm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as comments

@rlhui rlhui added the p0 label Dec 13, 2023
@deepak-singhal0408
Copy link
Copy Markdown
Contributor Author

Thanks @arlakshm for the review comments..
JFYI, other than that, I have addressed the packet-chassis issue by skipping the eth1 routes in APPL_DB.. Also added a multi-asic testcase for frr routes mismatch..

Please have a look and let me know if anything.

abdosi
abdosi previously approved these changes Dec 16, 2023
Copy link
Copy Markdown
Contributor

@abdosi abdosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deepak-singhal0408 deepak-singhal0408 merged commit e7a8def into sonic-net:master Dec 19, 2023
@StormLiangMS
Copy link
Copy Markdown
Contributor

@deepak-singhal0408 could you run test with 202305 with this PR?

@deepak-singhal0408
Copy link
Copy Markdown
Contributor Author

Hi @StormLiangMS , ran the test on a T1 running 202305.12
Results are fine..

Results:

/usr/local/bin# route_check.py -m INFO
Checking routes for namespaces: ['']
All good!
usr/local/bin# route_check_test.sh
Platform: xxxxxxxx
num_asic: 1
expect errors!
Running Route Check...
Failure results: {{
"": {
"missed_ROUTE_TABLE_routes": [
"20c0:d9b8:99:80::/64"
],
"missed_INTF_TABLE_entries": [
"10.0.0.99/32"
],
"Unaccounted_ROUTE_ENTRY_TABLE_entries": [
"192.193.120.255/25"
]
}
}}
Failed. Look at reported mismatches above
add: {
"": []
}
del: {
"": []
}
return value: 255

expect success on dualtor, expect error on all other devices!
Running Route Check...
Failure results: {{
"": {
"Unaccounted_ROUTE_ENTRY_TABLE_entries": [
"fc02:1000::99/128"
]
}
}}
Failed. Look at reported mismatches above
add: {
"": []
}
del: {
"": []
}
return value: 255

expect success!
Running Route Check...
return value: 0

@StormLiangMS
Copy link
Copy Markdown
Contributor

hi @deepak-singhal0408 cherry pick conflict, could you file separate PR for 202305?

rlhui added a commit to Azure/sonic-utilities.msft that referenced this pull request Jan 5, 2024
…ute_check_masic

What I did
Enhanced route_check.py script to cover multi-asic platforms.
Accordingly enhanced the test files as well.
Master Branch PR: sonic-net/sonic-utilities#3077

MSFT ADO: 25416673

How I did it
Enhanced the route_check.py script to take additional optional parameter(--n/namespace). Without this parameter, the check will be run on all asics in multi-asic platforms.
Different connections to DBs are modified accordingly to connect to relevant ns dbs.
Result will be encapsulated under different namespace. For single asic, results will be displayed under Default-Namespace("").

testData and the testfiles are enhanced accordinly.
yxieca added a commit that referenced this pull request Jan 13, 2024
@yxieca
Copy link
Copy Markdown
Contributor

yxieca commented Jan 15, 2024

@anamehra @deepak-singhal0408 #3075 and #3077 have blocked sonic-utilities repo sub-module advance in sonic-buildimage repo.

deepak-singhal0408 added a commit that referenced this pull request Jan 20, 2024
* Enhanced route_check.py for multi_asic platforms
* skip eth1 routes for packet-chassis, pytest enhancements
StormLiangMS pushed a commit that referenced this pull request Jan 29, 2024
What I did
Cherry pick of PR #3077

MSFT ADO: 25416673

How I did it
Enhanced the route_check.py script to take additional optional parameter(--n/namespace). Without this parameter, the check will be run on all asics in multi-asic platforms.
Different connections to DBs are modified accordingly to connect to relevant ns dbs.
Result will be encapsulated under different namespace. For single asic, results will be displayed under Default-Namespace("").

testData and the testfiles are enhanced accordinly.

How to verify it
Verified that all pytest UT cases are passing.
Verified the route_check_test.sh script on single asic and multi-asic platforms.
Verified Monit routecheck outputs by simulating a failure scenario on both single asic and multi-asic platforms.
output from Monit Check:
Single Asic:
xxx/usr/local/bin# monit status routecheck
Monit 5.20.0 uptime: 1d 20h 32m

Program 'routeCheck'
status Status failed
monitoring status Monitored
monitoring mode active
on reboot start
last exit value 255
last output Failure results: {{
"": {
"missed_ROUTE_TABLE_routes": [
"20c0:d9b8:99:80::/64"
]
}
}}
Failed. Look at reported mismatches above
add: {
"": []
}
del: {
"": []
}
data collected Tue, 12 Dec 2023 20:30:11

'''
Multi Asic:
'''
/bin# monit status routecheck
Monit 5.20.0 uptime: 1d 23h 51m

Program 'routeCheck'
status Status failed
monitoring status Monitored
monitoring mode active
on reboot start
last exit value 255
last output Failure results: {{
"asic0": {
"missed_ROUTE_TABLE_routes": [
"1.0.0.0/16"
]
},
"asic1": {
"missed_ROUTE_TABLE_routes": [
"1.0.0.0/16"
]
},
"asic2": {
"missed_ROUTE_TABLE_routes": [
"1.0.0.0/16"
]
}
}}
Failed. Look at reported mismatches above
add: {
"asic0": [],
"asic1": [],
"asic2": []
}
del: {
"asic0": [],
"asic1": [],
"asic2": []
}
data collected Tue, 12 Dec 2023 23:54:23
nmoray pushed a commit to nmoray/sonic-utilities that referenced this pull request Jun 25, 2025
* Enhanced route_check.py for multi_asic platforms
* skip eth1 routes for packet-chassis, pytest enhancements
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

9 participants