[Monit] Monitoring the running status of containers.#6251

Merged

lguohan merged 18 commits intosonic-net:masterfrom

yozhao101:monitoring_containers

Jan 8, 2021

Contributor

yozhao101 commented Dec 18, 2020

- Why I did it
This PR aims to monitor the running status of each container. Currently the auto-restart feature was enabled. If a critical process exited unexpected, the container will be restarted. If the container was restarted 3 times during 20 minutes, then it will not run anymore unless we cleared the flag using the command sudo systemctl reset-failed <container_name> manually.

- How I did it
We will employ Monit to monitor a script. This script will generate the expected running container list and compare it with the current running containers. If there are containers which were expected to run but were not running, then an alerting message will be written into syslog.

- How to verify it
I tested this feature on a lab device str-a7050-acs-3 which has single ASIC and str2-n3164-acs-3 which has a Multi-ASIC. First I manually stopped a container by running the command sudo systemctl stop <container_name>, then I checked whether there was an alerting message in the syslog.

- Which release branch to backport (provide reason below if selected)

201811
201911
[ x] 202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

yozhao101 added 2 commits

December 18, 2020 14:08


          [Monit] Use Monit to monitor the running status of each container and

2d963ee

write an alerting message into syslog if a container which was expected
to run but was not running.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Remove trailing space.

75b924f

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes

View reviewed changes

Contributor

jleveque left a comment

As comments.

Also, I suggest renaming the monitoring_containers script to container_checker

files/image_config/monit/monitoring_containers Outdated Show resolved Hide resolved

files/image_config/monit/monitoring_containers Outdated Show resolved Hide resolved

files/image_config/monit/monitoring_containers Outdated Show resolved Hide resolved

lguohan force-pushed the master branch from 3690c1a to 512eb6b Compare

December 25, 2020 18:34

yozhao101 marked this pull request as ready for review

December 25, 2020 21:04

yozhao101 requested review from abdosi and lguohan

December 29, 2020 05:24

lguohan added the Request for 202012 Branch label

yozhao101 added 2 commits

January 4, 2021 13:36


          [Monit] Use python3 to run this script and change the script name to

88b7e0f

'container_checker'.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Use the updated script name in Monit configuration file.

a4aa60c

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes

View reviewed changes

files/image_config/monit/container_checker Outdated Show resolved Hide resolved


          [Monit] Organize the 'import's alphabetically.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes

View reviewed changes

files/image_config/monit/container_checker Show resolved Hide resolved

yozhao101 added 4 commits

January 4, 2021 17:11


          [Monit] Simplify the logic to generate the alerting message.

3b461fb

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Delete an extra blank line and add the 'import' statement.

5db9a8e

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Declare the logger to be a global variable.

b895ced

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Use 'print(...)' instead of Logger class to write the alerting

a9df984

message into syslog.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes

View reviewed changes

files/image_config/monit/container_checker Show resolved Hide resolved

abdosi reviewed

View reviewed changes

files/image_config/monit/container_checker Show resolved Hide resolved

yozhao101 added 2 commits

January 5, 2021 13:00


          [Monit] Add docstring for this script.

fe21185

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Add the missing keywords of Monit example in docstring.

90e5e5e

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes

View reviewed changes

files/image_config/monit/container_checker Show resolved Hide resolved


          [Monit] Add blank lines above and below the docstring.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes

View reviewed changes

files/image_config/monit/container_checker Outdated Show resolved Hide resolved

files/image_config/monit/container_checker Outdated Show resolved Hide resolved

yozhao101 added 2 commits

January 5, 2021 15:11


          [Monit] Fix the typo and write an log message showing unexpected running

e63ad78

containers.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Delete the extra trailing space.

d256748

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 commented

View reviewed changes

files/image_config/monit/container_checker Show resolved Hide resolved

jleveque suggested changes

View reviewed changes

files/image_config/monit/container_checker Outdated Show resolved Hide resolved


          [Monit] Simplify the logic to get current running containers.

3175f1c

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque reviewed

View reviewed changes

files/image_config/monit/container_checker Outdated Show resolved Hide resolved

yozhao101 added 2 commits

January 5, 2021 18:36


          [Monit] Use .rstrip() to remove "\n" before doing the split.

3eae57e

Signed-off-by: Yong Zhao <yozhao@microsoft.com>


          [Monit] Skip the header line when parsing the command output.

0ba3ea6

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

abdosi reviewed

View reviewed changes

files/image_config/monit/container_checker Outdated Show resolved Hide resolved

abdosi reviewed

View reviewed changes

files/image_config/monit/container_checker Show resolved Hide resolved


          [Monit] Chnage the syslog message for containers which were expected to

623001d

run but running.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

abdosi approved these changes

View reviewed changes

Contributor

abdosi left a comment

LGTM

jleveque approved these changes

View reviewed changes

lguohan merged commit 04cd1d6 into sonic-net:master

lguohan pushed a commit that referenced this pull request


          [Monit] Monitoring the running status of containers. (#6251)

bfec282

**- Why I did it**
This PR aims to monitor the running status of each container. Currently the auto-restart feature was enabled. If a critical process exited unexpected, the container will be restarted. If the container was restarted 3 times during 20 minutes, then it will not run anymore unless we cleared the flag using the command `sudo systemctl reset-failed <container_name>` manually. 

**- How I did it**
We will employ Monit to monitor a script. This script will generate the expected running container list and compare it with the current running containers. If there are containers which were expected to run but were not running, then an alerting message will be written into syslog.

**- How to verify it**
I tested this feature on a lab device `str-a7050-acs-3` which has single ASIC and `str2-n3164-acs-3` which has a Multi-ASIC. First I manually stopped a container by running the command `sudo systemctl stop <container_name>`, then I checked whether there was an alerting message in the syslog.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

daall added the Included in 202012 Branch label

yozhao101 mentioned this pull request

[pytest] Test the feature of container checker. sonic-net/sonic-mgmt#2852

Merged

2 tasks

yozhao101 added a commit to sonic-net/sonic-mgmt that referenced this pull request


          [pytest] Test the feature of container checker. (#2852)

e8bfd88

What is the motivation for this PR?
This PR aims to test the feature of container checker and PR link of container checker is sonic-net/sonic-buildimage#6251.

The script of container_checker was run periodically by Monit and aims to monitor the running status of each container. Currently the auto-restart feature was enabled. If a critical process exited unexpected, the container will be restarted. If the container was restarted 3 times during 20 minutes, then it will not run anymore unless we cleared the flag using the command sudo systemctl reset-failed <container_name> manually.

How did you do it?
This pytest script will test the script container_checker in the following steps:

Stop the containers explicitly.
Check whether the names of stopped containers appear in the Monit alerting message.
Restart the corresponding stopped containers.
Post-check all the critical processes are running and BGP sessions are established.
How did you verify/test it?
I tested this pytest script on a virtual testbed.

Any platform specific information?
N/A

Supported testbed topology if it's a new test case?
N/A

yozhao101 mentioned this pull request

[pytest] Test the feature of container checker. sonic-net/sonic-mgmt#2890

Merged

2 tasks

yozhao101 added a commit to sonic-net/sonic-mgmt that referenced this pull request


          [pytest] Test the feature of container checker. (#2890)

b96a54c

Signed-off-by: Yong Zhao yozhao@microsoft.com

Description of PR
Summary:
This PR aims to test the feature of container checker and PR link is sonic-net/sonic-buildimage#6251.

Fixes # (issue)

Type of change
 Bug fix
 Testbed and Framework(new/improvement)
[ x] Test case(new/improvement)

Approach

What is the motivation for this PR?
This PR aims to test the feature of container checker and PR link of container checker is sonic-net/sonic-buildimage#6251.

The script of container_checker was run periodically by Monit and aims to monitor the running status of each container. Currently the auto-restart feature was enabled. If a critical process exited unexpected, the container will be restarted. If the container was restarted 3 times during 20 minutes, then it will not run anymore unless we cleared the flag using the command sudo systemctl reset-failed <container_name> manually.

How did you do it?
This pytest script will test the script container_checker in the following steps:

Stop the containers explicitly.
Check whether the names of stopped containers appear in the Monit alerting message.
Restart the containers by the config_reload(...).
Post-check all the critical processes are running and BGP sessions are established.

How did you verify/test it?
I tested the PR against the physical testbed (str-dx010-acs-1) which was installed image built from public master branch.

vmittal-msft pushed a commit to vmittal-msft/sonic-mgmt that referenced this pull request


          [pytest] Test the feature of container checker. (sonic-net#2890)

6fd3555

Signed-off-by: Yong Zhao yozhao@microsoft.com

Description of PR
Summary:
This PR aims to test the feature of container checker and PR link is sonic-net/sonic-buildimage#6251.

Fixes # (issue)

Type of change
 Bug fix
 Testbed and Framework(new/improvement)
[ x] Test case(new/improvement)

Approach

What is the motivation for this PR?
This PR aims to test the feature of container checker and PR link of container checker is sonic-net/sonic-buildimage#6251.

The script of container_checker was run periodically by Monit and aims to monitor the running status of each container. Currently the auto-restart feature was enabled. If a critical process exited unexpected, the container will be restarted. If the container was restarted 3 times during 20 minutes, then it will not run anymore unless we cleared the flag using the command sudo systemctl reset-failed <container_name> manually.

How did you do it?
This pytest script will test the script container_checker in the following steps:

Stop the containers explicitly.
Check whether the names of stopped containers appear in the Monit alerting message.
Restart the containers by the config_reload(...).
Post-check all the critical processes are running and BGP sessions are established.

How did you verify/test it?
I tested the PR against the physical testbed (str-dx010-acs-1) which was installed image built from public master branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Included in 202012 Branch Request for 202012 Branch