Description
The error message ('telemetry' process is not running) failed many test cases (portstat, config, leg_2, ...) because of loganalyzer.
Steps to reproduce the issue:
- sudo service telemetry restart
- sudo tail -F /var/log/syslog
Describe the results you received:
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:28,609 INFO stopped: telemetry (terminated by SIGTERM)
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:28,610 INFO stopped: rsyslogd (exit status 0)
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:29,614 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,037 CRIT Set uid to user 0
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,037 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,045 INFO RPC interface 'supervisor' initialized
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,045 INFO supervisord started with pid 1
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:32,048 INFO spawned: 'supervisor-proc-exit-listener' with pid 8
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:32,051 INFO spawned: 'start.sh' with pid 9
Jul 28 05:44:32.547862 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:32,259 INFO spawned: 'rsyslogd' with pid 14
Jul 28 05:44:33.265617 lab-esc601-32-MAX INFO telemetry#supervisord: start.sh rsyslogd: started
Jul 28 05:44:34.515893 lab-esc601-32-MAX INFO telemetry#supervisord: start.sh telemetry: started
Jul 28 05:44:36.532508 lab-esc601-32-MAX INFO telemetry#supervisord: start.sh dialout: started
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,261 INFO success: supervisor-proc-exit-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,262 INFO success: start.sh entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,262 INFO success: rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,509 INFO spawned: 'telemetry' with pid 19
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:34,512 INFO success: telemetry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:34,771 INFO spawned: 'dialout' with pid 39
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:36,529 INFO success: dialout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:36,551 INFO exited: start.sh (exit status 0; expected)
Jul 28 05:44:50.272379 lab-esc601-32-MAX ERR monit[648]: 'telemetry' process is not running
Jul 28 05:45:50.811250 lab-esc601-32-MAX ERR monit[648]: 'telemetry' process is not running
Jul 28 05:46:50.864305 lab-esc601-32-MAX ERR monit[648]: 'telemetry' process is not running
Additional information you deem important:
admin@lab-esc601-32-MAX:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16797ec69d4f docker-sonic-telemetry:latest "/usr/bin/supervisord" About an hour ago Up 6 minutes telemetry
09e521b20425 docker-snmp-sv2:latest "/usr/bin/supervisord" 11 days ago Up 5 hours snmp
10ec62936578 docker-sflow:latest "/usr/bin/supervisord" 11 days ago Up 5 hours sflow
541418bf08eb docker-nat:latest "/usr/bin/supervisord" 11 days ago Up 5 hours nat
70a19cf01587 docker-lldp-sv2:latest "/usr/bin/docker-lld…" 11 days ago Up 5 hours lldp
cf850887cb80 docker-dhcp-relay:latest "/usr/bin/docker_ini…" 11 days ago Up 2 hours dhcp_relay
e258ba607974 docker-sonic-mgmt-framework:latest "/usr/bin/supervisord" 11 days ago Up 5 hours mgmt-framework
91c0d168f4ee docker-router-advertiser:latest "/usr/bin/supervisord" 11 days ago Up 5 hours radv
017fb73fd8d7 docker-syncd-invm-rpc:latest "/usr/bin/supervisord" 11 days ago Up 5 hours syncd
fcda9466132f docker-teamd:latest "/usr/bin/supervisord" 11 days ago Up 5 hours teamd
221dd7a77fd0 docker-orchagent:latest "/usr/bin/supervisord" 11 days ago Up 5 hours swss
d33e7f70ef32 docker-platform-monitor:latest "/usr/bin/docker_ini…" 11 days ago Up 5 hours pmon
8383297a8db7 docker-fpm-frr:latest "/usr/bin/supervisord" 11 days ago Up 5 hours bgp
f724e939cc43 docker-database:latest "/usr/local/bin/dock…" 11 days ago Up 5 hours database
Description
The error message ('telemetry' process is not running) failed many test cases (portstat, config, leg_2, ...) because of loganalyzer.
Steps to reproduce the issue:
Describe the results you received:
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:28,609 INFO stopped: telemetry (terminated by SIGTERM)
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:28,610 INFO stopped: rsyslogd (exit status 0)
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:29,614 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,037 CRIT Set uid to user 0
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,037 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,045 INFO RPC interface 'supervisor' initialized
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:31,045 INFO supervisord started with pid 1
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:32,048 INFO spawned: 'supervisor-proc-exit-listener' with pid 8
Jul 28 05:44:32.547797 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:32,051 INFO spawned: 'start.sh' with pid 9
Jul 28 05:44:32.547862 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:32,259 INFO spawned: 'rsyslogd' with pid 14
Jul 28 05:44:33.265617 lab-esc601-32-MAX INFO telemetry#supervisord: start.sh rsyslogd: started
Jul 28 05:44:34.515893 lab-esc601-32-MAX INFO telemetry#supervisord: start.sh telemetry: started
Jul 28 05:44:36.532508 lab-esc601-32-MAX INFO telemetry#supervisord: start.sh dialout: started
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,261 INFO success: supervisor-proc-exit-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,262 INFO success: start.sh entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,262 INFO success: rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:33,509 INFO spawned: 'telemetry' with pid 19
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:34,512 INFO success: telemetry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:34,771 INFO spawned: 'dialout' with pid 39
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:36,529 INFO success: dialout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 28 05:44:43.091208 lab-esc601-32-MAX INFO telemetry#supervisord 2020-07-28 05:44:36,551 INFO exited: start.sh (exit status 0; expected)
Jul 28 05:44:50.272379 lab-esc601-32-MAX ERR monit[648]: 'telemetry' process is not running
Jul 28 05:45:50.811250 lab-esc601-32-MAX ERR monit[648]: 'telemetry' process is not running
Jul 28 05:46:50.864305 lab-esc601-32-MAX ERR monit[648]: 'telemetry' process is not running
Additional information you deem important:
admin@lab-esc601-32-MAX:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16797ec69d4f docker-sonic-telemetry:latest "/usr/bin/supervisord" About an hour ago Up 6 minutes telemetry
09e521b20425 docker-snmp-sv2:latest "/usr/bin/supervisord" 11 days ago Up 5 hours snmp
10ec62936578 docker-sflow:latest "/usr/bin/supervisord" 11 days ago Up 5 hours sflow
541418bf08eb docker-nat:latest "/usr/bin/supervisord" 11 days ago Up 5 hours nat
70a19cf01587 docker-lldp-sv2:latest "/usr/bin/docker-lld…" 11 days ago Up 5 hours lldp
cf850887cb80 docker-dhcp-relay:latest "/usr/bin/docker_ini…" 11 days ago Up 2 hours dhcp_relay
e258ba607974 docker-sonic-mgmt-framework:latest "/usr/bin/supervisord" 11 days ago Up 5 hours mgmt-framework
91c0d168f4ee docker-router-advertiser:latest "/usr/bin/supervisord" 11 days ago Up 5 hours radv
017fb73fd8d7 docker-syncd-invm-rpc:latest "/usr/bin/supervisord" 11 days ago Up 5 hours syncd
fcda9466132f docker-teamd:latest "/usr/bin/supervisord" 11 days ago Up 5 hours teamd
221dd7a77fd0 docker-orchagent:latest "/usr/bin/supervisord" 11 days ago Up 5 hours swss
d33e7f70ef32 docker-platform-monitor:latest "/usr/bin/docker_ini…" 11 days ago Up 5 hours pmon
8383297a8db7 docker-fpm-frr:latest "/usr/bin/supervisord" 11 days ago Up 5 hours bgp
f724e939cc43 docker-database:latest "/usr/local/bin/dock…" 11 days ago Up 5 hours database