Recommendations for production #241

kostik2022 · 2022-11-07T11:03:25Z

kostik2022
Nov 7, 2022

Hello!
Thank you for outstanding tool!

Are there some recommendations for buffer and queue sizes? or the best way is to achive correct values by practice?
Right now we set queue-size at inlet to 40k, we have about 13-15k flows/s, and in case of BGP "flap"/re-initiate/switchover/something_like_this we got drops at inlet (queue oversize). This is the reason of the question ;)

With best regards,
Konstantin

kostik2022 · 2022-11-08T06:58:21Z

kostik2022
Nov 8, 2022
Author

After any BGP session flaps, the following situation is visible in the logs:

BMP module removing the peer
Error 503
~10x error "dropping flow due to queue full (size 250000)"

{"level":"info","time":"2022-11-07T17:33:20Z","caller":"akvorado/inlet/bmp/events.go:104","module":"akvorado/inlet/bmp","message":"remove peer x.x.x.x for exporter x.x.x.x"}
{"level":"info","handler":"/api/","method":"GET","url":"/api/v0/healthcheck","ip":"127.0.0.1:51396","user-agent":"Go-http-client/1.1","status":503,"size":233,"duration":5000.73073,"time":"2022-11-07T17:33:55Z","caller":"akvorado/common/http/root.go:113","module":"akvorado/common/http","message":"HTTP request"}
{"level":"warn","worker":"12","listen":"0.0.0.0:2055","time":"2022-11-07T17:34:10Z","caller":"akvorado/inlet/flow/input/udp/root.go:196","module":"akvorado/inlet/flow/input/udp","message":"dropping flow due to queue full (size 250000)"}

It seems that this happens if we have more than 10k flows/s

4 replies

vincentbernat Nov 8, 2022
Maintainer

I am not using the BMP feature myself yet. Could you share the metrics related to BMP to understand how big your setup is?

kostik2022 Nov 8, 2022
Author

Something like this?

# TYPE akvorado_inlet_bmp_closed_connections_total counter
akvorado_inlet_bmp_closed_connections_total{exporter="10.0.0.1"} 6
akvorado_inlet_bmp_closed_connections_total{exporter="10.0.0.2"} 6
# HELP akvorado_inlet_bmp_errors_total Number of errors while processing BMP messages.
# TYPE akvorado_inlet_bmp_errors_total counter
akvorado_inlet_bmp_errors_total{error="cannot read BMP body",exporter="10.0.0.2"} 1
akvorado_inlet_bmp_errors_total{error="cannot read BMP header",exporter="10.0.0.1"} 6
akvorado_inlet_bmp_errors_total{error="cannot read BMP header",exporter="10.0.0.2"} 5
# HELP akvorado_inlet_bmp_locked_duration_seconds Duration during which the RIB is locked.
# TYPE akvorado_inlet_bmp_locked_duration_seconds summary
akvorado_inlet_bmp_locked_duration_seconds{reason="stale",quantile="0.5"} NaN
akvorado_inlet_bmp_locked_duration_seconds{reason="stale",quantile="0.9"} NaN
akvorado_inlet_bmp_locked_duration_seconds{reason="stale",quantile="0.99"} NaN
akvorado_inlet_bmp_locked_duration_seconds_sum{reason="stale"} 40.208234342
akvorado_inlet_bmp_locked_duration_seconds_count{reason="stale"} 3
# HELP akvorado_inlet_bmp_messages_received_total Number of BMP messages received.
# TYPE akvorado_inlet_bmp_messages_received_total counter
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.3",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.3",type="peer-down-notification"} 250
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.3",type="peer-up-notification"} 281
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.3",type="route-monitoring"} 4.262422e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.3",type="statistics-report"} 527
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.4",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.4",type="peer-down-notification"} 227
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.4",type="peer-up-notification"} 251
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.4",type="route-monitoring"} 3.660789e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.4",type="statistics-report"} 408
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.5",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.5",type="peer-down-notification"} 222
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.5",type="peer-up-notification"} 251
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.5",type="route-monitoring"} 3.709176e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.5",type="statistics-report"} 493
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.6",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.6",type="peer-down-notification"} 254
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.6",type="peer-up-notification"} 326
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.6",type="route-monitoring"} 5.393591e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.6",type="statistics-report"} 1224
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.7",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.7",type="peer-down-notification"} 19
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.7",type="peer-up-notification"} 87
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.7",type="route-monitoring"} 3.824077e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.7",type="statistics-report"} 1156
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.8",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.8",type="peer-down-notification"} 16
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.8",type="peer-up-notification"} 85
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.8",type="route-monitoring"} 4.495299e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.8",type="statistics-report"} 1173
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.1",type="initiation"} 7
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.1",type="peer-up-notification"} 144
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.1",type="route-monitoring"} 3.033494e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.1",type="statistics-report"} 1016
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.2",type="initiation"} 7
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.2",type="peer-up-notification"} 126
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.2",type="route-monitoring"} 2.363158e+06
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.2",type="statistics-report"} 894
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.9",type="initiation"} 1
akvorado_inlet_bmp_messages_received_total{exporter="10.0.0.10",type="initiation"} 1
# HELP akvorado_inlet_bmp_opened_connections_total Number of opened connections.
# TYPE akvorado_inlet_bmp_opened_connections_total counter
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.3"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.4"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.5"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.6"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.7"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.8"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.1"} 7
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.2"} 7
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.9"} 1
akvorado_inlet_bmp_opened_connections_total{exporter="10.0.0.10"} 1
# HELP akvorado_inlet_bmp_peers_total Number of peers up.
# TYPE akvorado_inlet_bmp_peers_total gauge
akvorado_inlet_bmp_peers_total{exporter="10.0.0.3"} 31
akvorado_inlet_bmp_peers_total{exporter="10.0.0.4"} 24
akvorado_inlet_bmp_peers_total{exporter="10.0.0.5"} 29
akvorado_inlet_bmp_peers_total{exporter="10.0.0.6"} 73
akvorado_inlet_bmp_peers_total{exporter="10.0.0.7"} 68
akvorado_inlet_bmp_peers_total{exporter="10.0.0.8"} 69
akvorado_inlet_bmp_peers_total{exporter="10.0.0.1"} 72
akvorado_inlet_bmp_peers_total{exporter="10.0.0.2"} 63
akvorado_inlet_bmp_peers_total{exporter="10.0.0.9"} 0
akvorado_inlet_bmp_peers_total{exporter="10.0.0.10"} 0
# HELP akvorado_inlet_bmp_routes_total Number of routes up.
# TYPE akvorado_inlet_bmp_routes_total gauge
akvorado_inlet_bmp_routes_total{exporter="10.0.0.3"} 5.668835e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.4"} 5.330792e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.5"} 5.256664e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.6"} 6.83122e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.7"} 6.087558e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.8"} 6.863959e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.1"} 3.730034e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.2"} 2.552681e+06
akvorado_inlet_bmp_routes_total{exporter="10.0.0.9"} 0
akvorado_inlet_bmp_routes_total{exporter="10.0.0.10"} 0```

vincentbernat Nov 8, 2022
Maintainer

Yes, that's helpful. As a workaround until I find some time to optimize stuff, you can:

try to use locrib instead of adjrib-in if your equipments support that (what vendor is it?)
only have one exporter using BMP: there is little gain in having many of them if the routes are mostly the same

Also, it may help if you could profile Akvorado while a peer is down. You need to enable the profile (inlet → http → profiler: true in your configuration). Restart the orchestrator, then the inlet. Use docker container inspect akvorado_akvorado-inlet_1 | grep IPAddress to get the IP address of the inlet. Then, from a machine with Go installed, ssh -L 6060:172.18.1.4:8080 hostrunningdocker. You need to run go tool pprof http://localhost:6060/debug/pprof/profile as soon you see the first log message clearing a peer. It will run for 30 seconds, then you can get a profile graph by typing "web".

vincentbernat Nov 8, 2022
Maintainer

Also, please switch the discussion to #253.

Recommendations for production #241

Uh oh!

Uh oh!

kostik2022 Nov 7, 2022

Replies: 1 comment · 4 replies

Uh oh!

Uh oh!

kostik2022 Nov 8, 2022 Author

Uh oh!

Uh oh!

vincentbernat Nov 8, 2022 Maintainer

Uh oh!

Uh oh!

kostik2022 Nov 8, 2022 Author

Uh oh!

vincentbernat Nov 8, 2022 Maintainer

Uh oh!

vincentbernat Nov 8, 2022 Maintainer

kostik2022
Nov 7, 2022

Replies: 1 comment 4 replies

kostik2022
Nov 8, 2022
Author

vincentbernat Nov 8, 2022
Maintainer

kostik2022 Nov 8, 2022
Author

vincentbernat Nov 8, 2022
Maintainer

vincentbernat Nov 8, 2022
Maintainer