Enable launch test discovery in pytest by hidmic · Pull Request #312 · ros2/launch

hidmic · 2019-08-16T15:22:10Z

Closes #237. This pull request allows for launch tests discovery and execution via pytest. It also sets up the scene for downstream packages i.e. launch_testing_ros to customize this.

Note this is not a tight integration between launch_testing and pytest but simply a mechanism for the latter to be able to execute launch tests.

Depends on #308.

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-08-20T17:57:26Z

CI up to launch_testing, launch_testing_ros and test_communication for extra safety:

Linux
Linux-aarch64
macOS
Windows

ivanpauno

I left some minimal comments.
I surfed in pytest plugin documentation and I don't fully understand the patch. But I don't have much more to add if it's working.

ivanpauno · 2019-08-21T12:55:56Z

launch_testing/launch_testing/pytest/hooks.py

+def find_launch_test_entrypoint(path):
+    try:
+        return getattr(path.pyimport(), 'generate_test_description', None)
+    except SyntaxError:


I'm not sure if that's the only possible exception, I would catch everything.

A SyntaxError is the only error I'm expecting. I'd rather not catch (and hide) unexpected errors e.g. some exception as a result of code being executed at the module level.

This is old code, but just as reference:
https://github.com/ros2/launch_ros/blob/bf3d800a53ea53d9465da9298c104998a7461dcf/ros2launch/ros2launch/command/launch.py#L125-L141

Couldn't value error happen here too?

launch_testing/test/launch_testing/examples/README.md

launch_testing/test/launch_testing/test_examples.py

ivanpauno · 2019-08-21T13:43:23Z

Extra comment, instead of rostest files needing a pytest mark and launchtest files just needing a generate_test_description function in the file, why not requiring a mark in them too? Something like, @pytest.mark.launchtest.
I prefer the difference being more explicit (than one explicit and the other implicit).

Also, aren't files with @pytest.mark.rostest mark being collected by both hooks now?

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

…unch_testing plugin. Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-08-21T18:21:25Z

Extra comment, instead of rostest files needing a pytest mark and launchtest files just needing a generate_test_description function in the file, why not requiring a mark in them too? Something like, @pytest.mark.launchtest.
I prefer the difference being more explicit (than one explicit and the other implicit).

Right now, any module in the search path that has a generate_test_description() is taken as a launch test and thus any mark was redundant in the pure launch test case. But I see the value in being explicit in both cases. Check 4491759.

Also, aren't files with @pytest.mark.rostest mark being collected by both hooks now?

Nope, this hook collects them appropriately once.

hidmic · 2019-08-21T18:22:20Z

I had to fix other failing tests, due to examples not being installed and pytest compulsively re-writing asserts within launch_testing. @ivanpauno PTAL.

ivanpauno

LGTM with green CI

ivanpauno · 2019-08-21T20:34:42Z

launch_testing/launch_testing/parametrize.py

-        def __iter__(self):
-            return iter(self.__calls)
+                yield partial, partial_args
+        setattr(_wrapped, '__parametrized__', True)


Couldn't be:

_wrapped.__parametrized__ = True

?

Yes, it can be. Done in 0705777.

ivanpauno · 2019-08-21T20:48:23Z

launch_testing/launch_testing/asserts/assert_exit_codes.py

+"""
+A module providing exit code assertions.
+
+PYTEST_DONT_REWRITE


Maybe, add a comment/link explaining why this is needed (e.g.: https://docs.pytest.org/en/latest/assert.html#assertion-introspection-details).

Sure, done in c7e6f80.

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-08-23T16:24:16Z

Alright, all green! Going in.

launch_testing/setup.py

hidmic mentioned this pull request Aug 16, 2019

Support launch_ros test runner in pytest ros2/launch_ros#54

Merged

hidmic added 3 commits August 20, 2019 14:53

Enable launch test discovery in pytest.

c177332

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Test examples using pytest.

67d53fe

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Enable downstream customization of launch tests execution.

94922bc

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic force-pushed the hidmic/launch-tests-in-python branch from 1af785b to 94922bc Compare August 20, 2019 17:54

hidmic requested review from dirk-thomas and ivanpauno August 20, 2019 17:57

hidmic mentioned this pull request Aug 20, 2019

Unit tests for composition actions ros2/launch_ros#15

Open

ivanpauno reviewed Aug 21, 2019

View reviewed changes

hidmic added 7 commits August 21, 2019 12:29

Move launch testing examples README content to the root README.

8014b4c

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Mark launch tests explicitly.

4491759

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Refactor parameterization support to play nice with pytest markers.

5be3345

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Improve pytest output on launch test failure.

e7cc2b9

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Please flake8

b658591

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Fix failing launch_testing tests.

d15a47c

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Prevent pytest from performing any assertion rewriting outside the la…

b4b56a2

…unch_testing plugin. Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

ivanpauno approved these changes Aug 21, 2019

View reviewed changes

hidmic added 2 commits August 22, 2019 15:14

Change explicit setattr for regular assignment.

0705777

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Better document PYTEST_DONT_REWRITE docstring.

c7e6f80

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

ivanpauno approved these changes Aug 22, 2019

View reviewed changes

Please flake8

dfa47c4

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic merged commit 5d6a87b into master Aug 23, 2019

delete-merged-branch bot deleted the hidmic/launch-tests-in-python branch August 23, 2019 16:24

dirk-thomas reviewed Aug 23, 2019

View reviewed changes

launch_testing/setup.py Show resolved Hide resolved

hidmic mentioned this pull request Aug 26, 2019

Update launch_testing example name in launch_testing_ament_cmake #320

Merged

hidmic mentioned this pull request Nov 19, 2019

Release Testing Automation - Iteration #1 ros2/ros2#739

Closed

Conversation

hidmic commented Aug 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hidmic commented Aug 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivanpauno left a comment

Choose a reason for hiding this comment

Uh oh!

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

hidmic Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivanpauno commented Aug 21, 2019

Uh oh!

hidmic commented Aug 21, 2019

Uh oh!

hidmic commented Aug 21, 2019

Uh oh!

ivanpauno left a comment

Choose a reason for hiding this comment

Uh oh!

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

hidmic Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

hidmic Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

hidmic commented Aug 23, 2019

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hidmic commented Aug 16, 2019 •

edited

Loading

hidmic commented Aug 20, 2019 •

edited

Loading