When processing a MutuallyExclusive callback_group, the multi-threaded executor ends up getting into a race condition with the can_be_taken_from_ atomic boolean. The problem is that the Executor::execute_any_executable resets the MutuallyExclusive callback_group can_be_taken_from_ atomic boolean and the AnyExecutable destructor also does reset the calllback group flag as well.
These two action cause the can_be_taken_from_ exclusion flag to get out of sync with reality causing a concurrency issue.

The resulting observable behaviour could be a re-ordering of messages within the same topic subscription (ie: older messages being delivered after newer messages).

Bug report

Required Info:

• Operating System:
  • Ubuntu 16.04
• Installation type:
  • From source
• Version or commit hash:
  -0.6.4
• DDS implementation:
  • Fast-RTPS
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

Subscriber node (where the issue will be seen):

• Create a ROS2 rclcpp node with the multi-threaded executor (default arguments are fine)
• Register to a topic caring some basic ordered data (a topic of Header type with a properly field timestamp would be fine)
• In the topic callback, verify that the message timestamps are in the proper order (newer after older)
  Publisher Node:
• Create a ROS2 node with the topic the subscriber node subscribed to (as stated earlier)
• Publish Header messages with properly filled timestamps (as fast as possible)

Expected behavior

Every Header messages are received in order

Actual behavior

Header messages are randomly re-ordered (older message arriving last)