tedge-mapper-c8y built-in bridge stopped sending mqtt messages to Cumulocity #3713
Description
Describe the bug
A incident was detected on the OSADL QA farm device, osadl-rackfslot2, where the tedge-mapper-c8y using the built-in bridge stopped sending data back to Cumulocity.
The device could still receive operations from Cumulocity (so the device could be "rescued" by restarting the tedge-mapper-c8y service), however the operation status and metrics stopped being sent back to Cumulocity.
Symptoms
- Operation status is not updated, but operations were being processed (e.g. could get tedge-configuration-plugin and even connect using remote access)
- Cumulocity Availability status displayed as RED/GREEN.
- Alarm in Cumulocity, "No data received from device within required interval" since May 28, 2025.May 28, 2025, 10:03:35 AM CEST, but since the required interval is set to 60mins, it's likely that the connected stopped up to 60mins before the alarm date.
- tedge-mapper-c8y log entries shows that there are waiting messages that don't seem to change (e.g. "waiting=15")
- A remote access session (via SSH) could be established successfully
- Restarting the "tedge-mapper-c8y" service resumed "normal" function
tedge connect c8y --teststill passses (e.g. exit code 0)- A jwt can still be successfully requested from Cumulocity (e.g. received on the
c8y/s/uattopic)
Log Snippet
Below shows a log snippet taken before the "tedge-mapper-c8y" service was restarted
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.342187594Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge cloud connection: received=16 forwarded=16 published=51880 waiting=15 acknowledged=51857 finalized=51857
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.353437447Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (cloud) Incoming(Publish(Topic = s/dat, Qos = AtLeastOnce, Retain = false, Pkid = 1, Payload Size = 810))
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.353505261Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge cloud connection: received=16 forwarded=16 published=51881 waiting=15 acknowledged=51857 finalized=51857
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.353716222Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (local) Outgoing(Publish(15))
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.353743722Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge local connection: received=51886 forwarded=51886 published=16 waiting=0 acknowledged=16 finalized=16
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.354495825Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (local) Incoming(PubAck(PubAck { pkid: 15 }))
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.354562139Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge local connection: received=51886 forwarded=51886 published=17 waiting=1 acknowledged=16 finalized=16
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.354782878Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (cloud) Outgoing(PubAck(1))
Jun 27 06:16:10 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:16:10.354822989Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge cloud connection: received=17 forwarded=17 published=51881 waiting=15 acknowledged=51857 finalized=51857
Jun 27 06:17:08 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:08.193373611Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (local) Outgoing(PingReq)
Jun 27 06:17:08 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:08.19349511Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge local connection: received=51886 forwarded=51886 published=17 waiting=0 acknowledged=17 finalized=17
Jun 27 06:17:08 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:08.193933958Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (local) Incoming(PingResp)
Jun 27 06:17:08 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:08.194001161Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge local connection: received=51886 forwarded=51886 published=17 waiting=0 acknowledged=17 finalized=17
Jun 27 06:17:09 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:09.803175771Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (cloud) Outgoing(PingReq)
Jun 27 06:17:09 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:09.803303399Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge cloud connection: received=17 forwarded=17 published=51881 waiting=15 acknowledged=51857 finalized=51857
Jun 27 06:17:09 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:09.811120249Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:474: Received notification (cloud) Incoming(PingResp)
Jun 27 06:17:09 rackfslot1s tedge-mapper[2973910]: 2025-06-27T04:17:09.811210007Z DEBUG tedge_mqtt_bridge: crates/extensions/tedge_mqtt_bridge/src/lib.rs:475: Bridge cloud connection: received=17 forwarded=17 published=51881 waiting=15 acknowledged=51857 finalized=51857
To Reproduce
It is unclear how to reproduce the issue.
Expected behavior
Screenshots
Environment (please complete the following information):
| Property | Value |
|---|---|
| OS [incl. version] | Debian GNU/Linux 12 (bookworm) |
| Hardware [incl. revision] | Raspberry Pi 4 Model B Rev 1.5 |
| System-Architecture | Linux rackfslot1s 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux |
| thin-edge.io version | tedge 1.5.0 |
Additional context
Logs from around the time where the problem started
| Date | Time |
|---|---|
| 2025-05-28 | 08:00 to 12:00 (CEST) |
tedge-mapper-c8y_2025-05-28T08_00_12_00.log
Logs from the day also showing the service restart
| Date | Time |
|---|---|
| 2025-06-27 | 00:00 to 06:20 (CEST) |
- Service was restarted at "06:17:53"