Skip to content

EventsExecutor may miss some timers after timer reset #2889

New issue
New issue
Closed
Closed
EventsExecutor may miss some timers after timer reset#2889
Labels
bugSomething isn't working
@almaslov

Description

@almaslov
almaslov
opened on Jul 4, 2025

Generated by Generative AI

No response

Operating System:

Linux my-host 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

ROS version or commit hash:

jazzy

RMW implementation (if applicable):

No response

RMW Configuration (if applicable):

No response

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

ros2 doctor --report 
   NETWORK CONFIGURATION
inet         : 127.0.0.1
inet4        : ['127.0.0.1']
netmask      : 255.0.0.0
device       : lo
flags        : 73<UP,RUNNING,LOOPBACK>
mtu          : 65536
inet         : 10.20.0.109
inet4        : ['10.20.0.109']
ether        : fa:16:3e:10:54:50
netmask      : 255.255.254.0
device       : ens3
flags        : 4163<BROADCAST,UP,RUNNING,MULTICAST>
mtu          : 1500
broadcast    : 10.20.1.255
inet         : 172.17.0.1
inet4        : ['172.17.0.1']
ether        : 02:42:3a:db:e4:1a
netmask      : 255.255.0.0
device       : docker0
flags        : 4099<BROADCAST,UP,MULTICAST>
mtu          : 1500
broadcast    : 172.17.255.255

   PACKAGE VERSIONS
action_msgs                               : latest=2.0.2, local=2.0.2
actionlib_msgs                            : latest=5.3.6, local=5.3.6
ament_cmake                               : latest=2.5.4, local=2.5.4
ament_cmake_auto                          : latest=2.5.4, local=2.5.4
ament_cmake_copyright                     : latest=0.17.2, local=0.17.2
ament_cmake_core                          : latest=2.5.4, local=2.5.4
ament_cmake_cppcheck                      : latest=0.17.2, local=0.17.2
ament_cmake_cpplint                       : latest=0.17.2, local=0.17.2
ament_cmake_export_definitions            : latest=2.5.4, local=2.5.4
ament_cmake_export_dependencies           : latest=2.5.4, local=2.5.4
ament_cmake_export_include_directories    : latest=2.5.4, local=2.5.4
ament_cmake_export_interfaces             : latest=2.5.4, local=2.5.4
ament_cmake_export_libraries              : latest=2.5.4, local=2.5.4
ament_cmake_export_link_flags             : latest=2.5.4, local=2.5.4
ament_cmake_export_targets                : latest=2.5.4, local=2.5.4
ament_cmake_flake8                        : latest=0.17.2, local=0.17.2
ament_cmake_gen_version_h                 : latest=2.5.4, local=2.5.4
ament_cmake_gmock                         : latest=2.5.4, local=2.5.4
ament_cmake_gtest                         : latest=2.5.4, local=2.5.4
ament_cmake_include_directories           : latest=2.5.4, local=2.5.4
ament_cmake_libraries                     : latest=2.5.4, local=2.5.4
ament_cmake_lint_cmake                    : latest=0.17.2, local=0.17.2
ament_cmake_pep257                        : latest=0.17.2, local=0.17.2
ament_cmake_pytest                        : latest=2.5.4, local=2.5.4
ament_cmake_python                        : latest=2.5.4, local=2.5.4
ament_cmake_ros                           : latest=0.12.0, local=0.12.0
ament_cmake_target_dependencies           : latest=2.5.4, local=2.5.4
ament_cmake_test                          : latest=2.5.4, local=2.5.4
ament_cmake_uncrustify                    : latest=0.17.2, local=0.17.2
ament_cmake_version                       : latest=2.5.4, local=2.5.4
ament_cmake_xmllint                       : latest=0.17.2, local=0.17.2
ament_copyright                           : latest=0.17.2, local=0.17.2
ament_cppcheck                            : latest=0.17.2, local=0.17.2
ament_cpplint                             : latest=0.17.2, local=0.17.2
ament_flake8                              : latest=0.17.2, local=0.17.2
ament_index_cpp                           : latest=1.8.1, local=1.8.1
ament_index_python                        : latest=1.8.1, local=1.8.1
ament_lint                                : latest=0.17.2, local=0.17.2
ament_lint_auto                           : latest=0.17.2, local=0.17.2
ament_lint_cmake                          : latest=0.17.2, local=0.17.2
ament_lint_common                         : latest=0.17.2, local=0.17.2
ament_package                             : latest=0.16.4, local=0.16.4
ament_pep257                              : latest=0.17.2, local=0.17.2
ament_uncrustify                          : latest=0.17.2, local=0.17.2
ament_xmllint                             : latest=0.17.2, local=0.17.2
builtin_interfaces                        : latest=2.0.2, local=2.0.2
class_loader                              : latest=2.7.0, local=2.7.0
common_interfaces                         : latest=5.3.6, local=5.3.6
composition_interfaces                    : latest=2.0.2, local=2.0.2
console_bridge_vendor                     : latest=1.7.1, local=1.7.1
diagnostic_msgs                           : latest=5.3.6, local=5.3.6
domain_coordinator                        : latest=0.12.0, local=0.12.0
fastrtps_cmake_module                     : latest=3.6.1, local=3.6.1
geometry_msgs                             : latest=5.3.6, local=5.3.6
launch                                    : latest=3.4.5, local=3.4.5
launch_ros                                : latest=0.26.8, local=0.26.8
launch_testing                            : latest=3.4.5, local=3.4.5
launch_testing_ament_cmake                : latest=3.4.5, local=3.4.5
launch_testing_ros                        : latest=0.26.8, local=0.26.8
launch_xml                                : latest=3.4.5, local=3.4.5
launch_yaml                               : latest=3.4.5, local=3.4.5
libstatistics_collector                   : latest=1.7.4, local=1.7.4
libyaml_vendor                            : latest=1.6.3, local=1.6.3
lifecycle_msgs                            : latest=2.0.2, local=2.0.2
nav_msgs                                  : latest=5.3.6, local=5.3.6
osrf_pycommon                             : latest=2.1.6, local=2.1.6
pluginlib                                 : latest=5.4.2, local=5.4.2
python_cmake_module                       : latest=0.11.1, local=0.11.1
rcl                                       : latest=9.2.7, local=9.2.7
rcl_action                                : latest=9.2.7, local=9.2.7
rcl_interfaces                            : latest=2.0.2, local=2.0.2
rcl_lifecycle                             : latest=9.2.7, local=9.2.7
rcl_logging_interface                     : latest=3.1.1, local=3.1.1
rcl_logging_spdlog                        : latest=3.1.1, local=3.1.1
rcl_yaml_param_parser                     : latest=9.2.7, local=9.2.7
rclcpp                                    : latest=28.1.10, local=28.1.10
rclcpp_action                             : latest=28.1.10, local=28.1.10
rclcpp_components                         : latest=28.1.10, local=28.1.10
rclcpp_lifecycle                          : latest=28.1.10, local=28.1.10
rclpy                                     : latest=7.1.4, local=7.1.4
rcpputils                                 : latest=2.11.2, local=2.11.2
rcutils                                   : latest=6.7.2, local=6.7.2
rmw                                       : latest=7.3.2, local=7.3.2
rmw_dds_common                            : latest=3.1.0, local=3.1.0
rmw_fastrtps_cpp                          : latest=8.4.2, local=8.4.2
rmw_fastrtps_shared_cpp                   : latest=8.4.2, local=8.4.2
rmw_implementation                        : latest=2.15.5, local=2.15.5
rmw_implementation_cmake                  : latest=7.3.2, local=7.3.2
ros2action                                : latest=0.32.4, local=0.32.4
ros2cli                                   : latest=0.32.4, local=0.32.4
ros2cli_common_extensions                 : latest=0.3.0, local=0.3.0
ros2component                             : latest=0.32.4, local=0.32.4
ros2doctor                                : latest=0.32.4, local=0.32.4
ros2interface                             : latest=0.32.4, local=0.32.4
ros2launch                                : latest=0.26.8, local=0.26.8
ros2lifecycle                             : latest=0.32.4, local=0.32.4
ros2multicast                             : latest=0.32.4, local=0.32.4
ros2node                                  : latest=0.32.4, local=0.32.4
ros2param                                 : latest=0.32.4, local=0.32.4
ros2pkg                                   : latest=0.32.4, local=0.32.4
ros2run                                   : latest=0.32.4, local=0.32.4
ros2service                               : latest=0.32.4, local=0.32.4
ros2topic                                 : latest=0.32.4, local=0.32.4
ros_core                                  : latest=0.11.0, local=0.11.0
ros_environment                           : latest=4.2.1, local=4.2.1
ros_workspace                             : latest=1.0.3, local=1.0.3
rosgraph_msgs                             : latest=2.0.2, local=2.0.2
rosidl_adapter                            : latest=4.6.5, local=4.6.5
rosidl_cli                                : latest=4.6.5, local=4.6.5
rosidl_cmake                              : latest=4.6.5, local=4.6.5
rosidl_core_generators                    : latest=0.2.0, local=0.2.0
rosidl_core_runtime                       : latest=0.2.0, local=0.2.0
rosidl_default_generators                 : latest=1.6.0, local=1.6.0
rosidl_default_runtime                    : latest=1.6.0, local=1.6.0
rosidl_dynamic_typesupport                : latest=0.1.2, local=0.1.2
rosidl_dynamic_typesupport_fastrtps       : latest=0.1.0, local=0.1.0
rosidl_generator_c                        : latest=4.6.5, local=4.6.5
rosidl_generator_cpp                      : latest=4.6.5, local=4.6.5
rosidl_generator_py                       : latest=0.22.1, local=0.22.1
rosidl_generator_type_description         : latest=4.6.5, local=4.6.5
rosidl_parser                             : latest=4.6.5, local=4.6.5
rosidl_pycommon                           : latest=4.6.5, local=4.6.5
rosidl_runtime_c                          : latest=4.6.5, local=4.6.5
rosidl_runtime_cpp                        : latest=4.6.5, local=4.6.5
rosidl_runtime_py                         : latest=0.13.1, local=0.13.1
rosidl_typesupport_c                      : latest=3.2.2, local=3.2.2
rosidl_typesupport_cpp                    : latest=3.2.2, local=3.2.2
rosidl_typesupport_fastrtps_c             : latest=3.6.1, local=3.6.1
rosidl_typesupport_fastrtps_cpp           : latest=3.6.1, local=3.6.1
rosidl_typesupport_interface              : latest=4.6.5, local=4.6.5
rosidl_typesupport_introspection_c        : latest=4.6.5, local=4.6.5
rosidl_typesupport_introspection_cpp      : latest=4.6.5, local=4.6.5
rpyutils                                  : latest=0.4.1, local=0.4.1
sensor_msgs                               : latest=5.3.6, local=5.3.6
service_msgs                              : latest=2.0.2, local=2.0.2
shape_msgs                                : latest=5.3.6, local=5.3.6
spdlog_vendor                             : latest=1.6.1, local=1.6.1
sros2                                     : latest=0.13.3, local=0.13.3
sros2_cmake                               : latest=0.13.3, local=0.13.3
statistics_msgs                           : latest=2.0.2, local=2.0.2
std_msgs                                  : latest=5.3.6, local=5.3.6
std_srvs                                  : latest=5.3.6, local=5.3.6
stereo_msgs                               : latest=5.3.6, local=5.3.6
tinyxml2_vendor                           : latest=0.9.1, local=0.9.1
tracetools                                : latest=8.2.3, local=8.2.3
trajectory_msgs                           : latest=5.3.6, local=5.3.6
type_description_interfaces               : latest=2.0.2, local=2.0.2
uncrustify_vendor                         : latest=3.0.1, local=3.0.1
unique_identifier_msgs                    : latest=2.5.0, local=2.5.0
visualization_msgs                        : latest=5.3.6, local=5.3.6
eigen3_cmake_module                       : latest=0.3.0, local=0.3.0
geometry2                                 : latest=0.36.12, local=0.36.12
kdl_parser                                : latest=2.11.0, local=2.11.0
keyboard_handler                          : latest=0.3.1, local=0.3.1
liblz4_vendor                             : latest=0.26.7, local=0.26.7
mcap_vendor                               : latest=0.26.7, local=0.26.7
message_filters                           : latest=4.11.6, local=4.11.6
orocos_kdl_vendor                         : latest=0.5.1, local=0.5.1
pybind11_vendor                           : latest=3.1.3, local=3.1.3
python_orocos_kdl_vendor                  : latest=0.5.1, local=0.5.1
robot_state_publisher                     : latest=3.3.3, local=3.3.3
ros2bag                                   : latest=0.26.7, local=0.26.7
ros_base                                  : latest=0.11.0, local=0.11.0
rosbag2                                   : latest=0.26.7, local=0.26.7
rosbag2_compression                       : latest=0.26.7, local=0.26.7
rosbag2_compression_zstd                  : latest=0.26.7, local=0.26.7
rosbag2_cpp                               : latest=0.26.7, local=0.26.7
rosbag2_interfaces                        : latest=0.26.7, local=0.26.7
rosbag2_py                                : latest=0.26.7, local=0.26.7
rosbag2_storage                           : latest=0.26.7, local=0.26.7
rosbag2_storage_default_plugins           : latest=0.26.7, local=0.26.7
rosbag2_storage_mcap                      : latest=0.26.7, local=0.26.7
rosbag2_storage_sqlite3                   : latest=0.26.7, local=0.26.7
rosbag2_transport                         : latest=0.26.7, local=0.26.7
sensor_msgs_py                            : latest=5.3.6, local=5.3.6
sqlite3_vendor                            : latest=0.26.7, local=0.26.7
tf2                                       : latest=0.36.12, local=0.36.12
tf2_bullet                                : latest=0.36.12, local=0.36.12
tf2_eigen                                 : latest=0.36.12, local=0.36.12
tf2_eigen_kdl                             : latest=0.36.12, local=0.36.12
tf2_geometry_msgs                         : latest=0.36.12, local=0.36.12
tf2_kdl                                   : latest=0.36.12, local=0.36.12
tf2_msgs                                  : latest=0.36.12, local=0.36.12
tf2_py                                    : latest=0.36.12, local=0.36.12
tf2_ros                                   : latest=0.36.12, local=0.36.12
tf2_ros_py                                : latest=0.36.12, local=0.36.12
tf2_sensor_msgs                           : latest=0.36.12, local=0.36.12
tf2_tools                                 : latest=0.36.12, local=0.36.12
urdf                                      : latest=2.10.0, local=2.10.0
urdf_parser_plugin                        : latest=2.10.0, local=2.10.0
yaml_cpp_vendor                           : latest=9.0.1, local=9.0.1
zstd_vendor                               : latest=0.26.7, local=0.26.7

   PLATFORM INFORMATION
system           : Linux
platform info    : Linux-5.15.0-113-generic-x86_64-with-glibc2.39
release          : 5.15.0-113-generic
processor        : x86_64

   QOS COMPATIBILITY LIST
compatibility status    : No publisher/subscriber pairs found

   RMW MIDDLEWARE
middleware name    : rmw_fastrtps_cpp

   ROS 2 INFORMATION
distribution name      : jazzy
distribution type      : ros2
distribution status    : active
release platforms      : {'debian': ['bookworm'], 'rhel': ['9'], 'ubuntu': ['noble']}

   TOPIC LIST
topic               : none
publisher count     : 0
subscriber count    : 0

Steps to reproduce issue

  1. Run ros:jazzy container docker run -it --rm --net=host ros:jazzy bash
  2. Create package with test node ros2 pkg create timers_example --destination-directory /ros2/src/ --build-type ament_cmake --node-name timers_example_node --dependencies rclcpp std_msgs
  3. Take timers_example_node.cpp from code block bellow.
  4. cd ros2 && colcon build
  5. source install/setup.sh
  6. Run node ros2 run timers_example timers_example_node
  7. Wait 5-10 seconds
  8. Send message to reset timer ros2 topic pub -w 1 --once /topic std_msgs/msg/Empty
  9. What for timers to be triggered.
  10. Fail

timers_example_node.cpp contents:

#include "rclcpp/rclcpp.hpp"
#include "rclcpp/experimental/executors/events_executor/events_executor.hpp"
#include "std_msgs/msg/empty.hpp"

using namespace std::chrono_literals;

int main(int argc, char ** argv)
{
  rclcpp::init(argc, argv);

  auto node = std::make_shared<rclcpp::Node>("timers_example_node");
  auto logger = node->get_logger();

  auto t0 = node->create_wall_timer(20000ms, [&logger](){ RCLCPP_INFO(logger, "Timer 0 triggered"); });
  auto t1 = node->create_wall_timer(20001ms, [&logger](){ RCLCPP_INFO(logger, "Timer 1 triggered"); });
  auto t2 = node->create_wall_timer(20002ms, [&logger](){ RCLCPP_INFO(logger, "Timer 2 triggered"); });
  auto t3 = node->create_wall_timer(20003ms, [&logger](){ RCLCPP_INFO(logger, "Timer 3 triggered"); });

  auto sub = node->create_subscription<std_msgs::msg::Empty>("topic", 10,
    [logger, t1](std_msgs::msg::Empty::ConstSharedPtr /*unused*/){
      t1->reset();
      RCLCPP_INFO(logger, "Timer 1 reset");
    });

  auto executor = std::make_shared<rclcpp::experimental::executors::EventsExecutor>();
  // auto executor = std::make_shared<rclcpp::executors::SingleThreadedExecutor>();
  executor->add_node(node->get_node_base_interface());

  RCLCPP_INFO(logger, "Spinning");
  executor->spin();

  return 0;
}

Expected behavior

All timers except for the one which is reset, triggered in time, nearly simultaneously. Desired behaviour in case of SingleThreadExecutor gives next log:

[INFO] [1751627525.583] [timers_example_node]: Spinning
[INFO] [1751627532.188] [timers_example_node]: Timer 1 reset
[INFO] [1751627545.581] [timers_example_node]: Timer 0 triggered
[INFO] [1751627545.583] [timers_example_node]: Timer 2 triggered
[INFO] [1751627545.584] [timers_example_node]: Timer 3 triggered
[INFO] [1751627552.189] [timers_example_node]: Timer 1 triggered

Actual behavior

Only two of three timers triggered in time. Timer 3 is skipped and triggered only after timer 1.

[INFO] [1751628182.411] [timers_example_node]: Spinning
[INFO] [1751628187.836] [timers_example_node]: Timer 1 reset
[INFO] [1751628202.410] [timers_example_node]: Timer 0 triggered
[INFO] [1751628202.412] [timers_example_node]: Timer 2 triggered
[INFO] [1751628207.837] [timers_example_node]: Timer 1 triggered
[INFO] [1751628207.837] [timers_example_node]: Timer 3 triggered

Additional information

Not sure that it's the simplest way to reproduce, but it definitely works. It is originally found in ros-humble branch of https://github.com/irobot-ros/events-executor repo.
As far as I understand timers heap becomes broken after timer reset occurs. I hot fixed it by force rehipify after any timers update, but may be there is more efficient way to do it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions