When receiving a lot of BMP messages, we have a giant lock protection the routing table. This makes Lookup() wait on the lock. See:
BART has lock-free methods, but this is only one part of the solution, as we have a map and the intern values, both of them requiring a lock. The intern values could be replaced by Go's unique package (or just stealing its way of keeping references). For the map, sync.Map could be used.
Also see #254, #255, #278, and #1461 for some of the previous attempts. Prioritizing readers is not a good solution either, as the writer may be starved and declared dead by the remote BMP sender (this is #1461).
The discussion below contains more profiling graphs.
Discussed in #2090
Originally posted by bogi788 November 11, 2025
Since updating our Akvorado setup to 2.0 we're having issues with the outlet not processing incoming flows fast enough (and the Kafka consumer group lag increasing) with too many BMP messages arriving. Temporarily blocking BMP (e.g. using an iptables rule) allows the outlet to catch up.
Are there any workaround we can apply (besides using two orchestrators and outlets writing to the same clickhouse database)?
When receiving a lot of BMP messages, we have a giant lock protection the routing table. This makes
Lookup()wait on the lock. See:BART has lock-free methods, but this is only one part of the solution, as we have a map and the intern values, both of them requiring a lock. The intern values could be replaced by Go's
uniquepackage (or just stealing its way of keeping references). For the map,sync.Mapcould be used.Also see #254, #255, #278, and #1461 for some of the previous attempts. Prioritizing readers is not a good solution either, as the writer may be starved and declared dead by the remote BMP sender (this is #1461).
The discussion below contains more profiling graphs.
Discussed in #2090
Originally posted by bogi788 November 11, 2025
Since updating our Akvorado setup to 2.0 we're having issues with the outlet not processing incoming flows fast enough (and the Kafka consumer group lag increasing) with too many BMP messages arriving. Temporarily blocking BMP (e.g. using an iptables rule) allows the outlet to catch up.
Are there any workaround we can apply (besides using two orchestrators and outlets writing to the same clickhouse database)?