Every time a user requests a unique URL that returns 404, the Prometheus middleware generates new metrics. This can easily generate millions of timeseries, overloading a Prometheus server or triggering other limits.
Minimal reproducer: https://gist.github.com/brietaylor/c5aa76df6a2f3fa3a8e14e4f8250cab1
Debugging:
I was able to trace this back through the code somewhat. The code that generates these URLs ultimately traces back to router.go:Find() in Echo. That code initially sets Context.path to the request path, but on all success paths, it overrides Context.path to the route path. I couldn't find any way for middlewares to detect which was done, though, so this might need to be an upstream fix.
https://github.com/labstack/echo/blob/master/router.go#L527
Every time a user requests a unique URL that returns 404, the Prometheus middleware generates new metrics. This can easily generate millions of timeseries, overloading a Prometheus server or triggering other limits.
Minimal reproducer: https://gist.github.com/brietaylor/c5aa76df6a2f3fa3a8e14e4f8250cab1
Debugging:
I was able to trace this back through the code somewhat. The code that generates these URLs ultimately traces back to
router.go:Find()in Echo. That code initially setsContext.pathto the request path, but on all success paths, it overridesContext.pathto the route path. I couldn't find any way for middlewares to detect which was done, though, so this might need to be an upstream fix.https://github.com/labstack/echo/blob/master/router.go#L527