pprof: support mutex contention and blocked goroutine profiling#41154
pprof: support mutex contention and blocked goroutine profiling#41154pchaigno merged 1 commit intocilium:mainfrom
Conversation
bimmlerd
left a comment
There was a problem hiding this comment.
I think that's fine. For the agent at least I think gops already provides this capability; possibly even with starting it at runtime instead of having to have it configured at startup - would that also work for you or is this needed?
d6944fa to
e7b8894
Compare
antonipp
left a comment
There was a problem hiding this comment.
For the agent at least I think gops already provides this capability; possibly even with starting it at runtime instead of having to have it configured at startup - would that also work for you or is this needed?
Unless I'm missing something, I don't think that gops provides this functionality. It does have an integration with some pprof profile types, namely CPU (with gops pprof-cpu) and memory (with gops pprof-heap) but I don't really see any way for it to extract mutex or blocked goroutine profiles. And even if it could, it wouldn't get any data, because as far as I can tell, the functionality needs to be enabled in the code by calling the appropriate functions in the runtime package. Here's a similar example from the Istio project: istio/istio#44688 (comment)
|
/test |
bimmlerd
left a comment
There was a problem hiding this comment.
Yep you're right, I confused the memory profiling things of gops with this; LGTM!
|
Hmm I realized that I also need to add the options to the |
Signed-off-by: Anton Ippolitov <anton.ippolitov@datadoghq.com>
e7b8894 to
8f763b3
Compare
|
/test |
Description
We support exporting profiling data using
pprof, however, at the moment, it’s limited to CPU, memory, and goroutine profiles. This PR adds support for two more types of profiles:This data is not exposed by default. In order to enable it, we need to explicitly set these runtime configuration values:
This PR adds two new flags to enable these profilers. The blocked goroutine profiler comes with performance overhead so I added some callouts about that as well.
Testing
Tested on the Operator. Enabled :
Then enabled
mutexandblockpprofscraping with Datadog (these profiles are of course compatible with any monitoring tool, I just used DD for convenience):and validated that the profiles were properly collected now:
