Feature request

Feature description

As of today, release testing is a manual, time consuming process that demands the ROS2 team and willing external contributors a significant share of time during the weeks that precede a release. An attempt is made to cover a testing matrix that spans tutorials, demos, CLI across Tier ~1 platforms and RMW implementations, which can only get worse as the available feature or the supported platform set grow over time.

There's, however, a fair amount of tests that could be automated right away, and some redundancy within the matrix itself that could be reduced, increasing overall code quality and likely catching regressions earlier in the cycle.

Edit: As a first attempt, full test coverage for all ROS 2 CLI commands is to be attained.

Implementation considerations

Here's a sample of the release testing matrix. It has been color coded to separate what can be automated from what not, and what should be automated and what not. It may also be used as a TODO list and discussion site.

Most tests, if not all, require further specification to be coded. This task, though initially delegated to the tests author and reviewer, should ensure basic coverage e.g.:

• Validate correct output like stdout, messages and logs, both in terms of content and latency.
• Validate correct behavior when run in unusual ways e.g. repeatedly re-running it or running things out of (the usual) order.
• Validate graceful signal handling e.g. SIGINT and SIGTERM.

As of Dashing, ros_testing, a launch based testing framework, can be leveraged when automating many of the process-level tests that can be found in the release testing matrix. It may also be worth increasing test coverage for the framework itself as we rely more and more on it.