#2149 (comment)

https://circle-production-customer-artifacts.s3.amazonaws.com/picard/forks/5b21c78bcfe8df0013110d01/241124988/6018ecd34b6ad53423a303d8-0-build/artifacts/opt/overlay_ws/log/test/nav2_system_tests/stdout.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210202T235537Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20210202%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=c284ed4d2c93c5334be3730e829b934924a1832a73297d568bf1f271416d9717

I also just skimmed one of the tests, 13, and I never see that gazebo crash, the system just fails to ever initialize. I actually don't see either of the lines you posted on the crash / exception in that log for system tests. Instead I see

14: [lifecycle_manager-6] [ERROR] [1612246927.316509864] [lifecycle_manager_localization]: Failed to change state for node: amcl
14: [lifecycle_manager-6] [ERROR] [1612246927.316589436] [lifecycle_manager_localization]: Failed to bring up all requested nodes. Aborting bringup.

Most of them have some version of "Failed to change state for node" like the lifecycle manager services are failing. Maybe that gazebo thing you ran into was transient unless you've seen it multiple times? All 4 of the tests that are failing had "Failed to change state for node" issue for random nodes. The other snippets I showed could be caused by a particular node failing to come up (amcl, bt nav, etc).

sigh is this another service regression issue (only major use of services we have are lifecycle state transitions and clear costmap calls)? Something lifecycle specific? DDS vendor related?

https://app.circleci.com/pipelines/github/ros-planning/navigation2?branch=main interestingly, look at the nightly jobs. They're passing all with Cyclone. But with Fast DDS its occasionally failing with this issue. @EduPonz @JaimeMartin has there been a recent Fast-DDS regression? I'm not sure why the nightlies arent failing as often, maybe more bandwidth on resources late at night in the cloud servers? I think they should have the same resource allocations (CC @ruffsl )

But we changed absolutely nothing and went from 100% test success to like 40% essentially overnight, it has to be something upstream of us.

CI is newly flaky #2158

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions