Add Basic Out-of-Process EventPipe Control and Heap Snapshot Capture#19720
Add Basic Out-of-Process EventPipe Control and Heap Snapshot Capture#19720brianrob merged 21 commits intodotnet:masterfrom
Conversation
| > export COMPlus_EnableEventPipe=4 | ||
| > ``` | ||
|
|
||
| 2. Specify the output file path: |
There was a problem hiding this comment.
Indicate that this is optional and what the file is if you don't specify it.
|
|
||
| # Collection Examples # | ||
|
|
||
| ## Start/Stop Collection at Any Time with Default Collection Parameters ## |
There was a problem hiding this comment.
Tell them what the defaults are (CPU sampling).
|
|
||
| ## Start/Stop Collection at Any Time with Default Collection Parameters ## | ||
|
|
||
| 1. Tell the runtime to listen for enable/disable commands: |
There was a problem hiding this comment.
Indicate that this has to be done before the program starts, but has no perf impact until the control file exists.
| { | ||
| if (s_controllerInstance == null) | ||
| { | ||
| string strEnabledValue = CompatibilitySwitch.GetValueInternal("EnableEventPipe"); |
There was a problem hiding this comment.
Presumably the logic that turns things on if the value is 1 is in native code. Any chance we can plumb things so that we move that here? Ideally all 'high level' control logic is managed. Certainly it is strange that it is split.
There was a problem hiding this comment.
Yes, this can be done, and is a good thing to do. I've fixed this.
src/vm/eventpipe.cpp
Outdated
| outputPath.Printf("Process-%d.netperf", GetCurrentProcessId()); | ||
|
|
||
| // Get the circular buffer size in megabytes. | ||
| unsigned int circularMB = CLRConfig::GetConfigValue(CLRConfig::INTERNAL_EventPipeCircularMB); |
There was a problem hiding this comment.
Ideally all of this just gets pushed to managed code and it is there where config values are fetched and set as needed. Native code should not know about config at all.
adamsitnik
left a comment
There was a problem hiding this comment.
LGTM, modulo comments.
The only thing I don't like is the hacky nature of the EventPipe configuration (env vars + files). I hope that we will be able to come up with something more flexible for 3.0
| string strEnabledValue = CompatibilitySwitch.GetValueInternal("EnableEventPipe"); | ||
| if (strEnabledValue != null) | ||
| { | ||
| int enabledValue = Convert.ToInt32(strEnabledValue); |
There was a problem hiding this comment.
Perhaps int.TryParse would be better than Convert.ToInt32? I know that we have a catch block here, but we could avoid the first chance exception here
There was a problem hiding this comment.
It turns out that Convert.ToInt32 will return 0 if the input string is null.
There was a problem hiding this comment.
But it's going to throw when the string will be incorrect, example: Convert.ToInt32("not a number");
There was a problem hiding this comment.
This is true. I've switched to TryParse. Thanks for pointing this out.
| } | ||
| } | ||
| catch { } | ||
| } |
There was a problem hiding this comment.
I would simplify the method and reduce nesting to:
internal static void Initialize()
{
if (s_controllerInstance != null)
return;
try
{
if(int.TryParse(CompatibilitySwitch.GetValueInternal("EnableEventPipe"), out int flag) && flag == Constants.EventPipeInitiallyDisabled)
s_controllerInstance = new EventPipeController();
}
catch {}
}Also I would check if CompatibilitySwitch.GetValueInternal can throw. If not, I would remove the catch block.
There was a problem hiding this comment.
I want to leave the try/catch block here to be explicit and to be a reminder in the future that this must not throw.
| } | ||
|
|
||
| // Wait for the polling interval. | ||
| Thread.Sleep(PollingIntervalMilliseconds); |
There was a problem hiding this comment.
Thread.Sleep(Timespan.FromSeconds(10)) would be more self-describing
There was a problem hiding this comment.
You're right - the reason I didn't do this is because TimeSpan.FromSeconds has a higher cost than just using a const int, and then it just has to be converted to an integer.
| RETAIL_CONFIG_STRING_INFO(INTERNAL_EventPipeOutputFile, W("EventPipeOutputFile"), "The full path including file name for the trace file that will be written when COMPlus_EnableEventPipe&=1") | ||
| RETAIL_CONFIG_STRING_INFO(INTERNAL_EventPipeConfig, W("EventPipeConfig"), "Configuration for EventPipe.") | ||
| RETAIL_CONFIG_DWORD_INFO(INTERNAL_EventPipeRundown, W("EventPipeRundown"), 1, "Enable/disable eventpipe rundown.") | ||
| RETAIL_CONFIG_DWORD_INFO(INTERNAL_EventPipeCircularMB, W("EventPipeCircularMB"), 1024, "The EventPipe circular buffer size in megabytes.") |
There was a problem hiding this comment.
👍 I like that you mention the unit in both variable name and the description!
| } | ||
|
|
||
| // Wait for the polling interval. | ||
| Thread.Sleep(PollingIntervalMilliseconds); |
There was a problem hiding this comment.
While polling every 10 seconds to enable makes sense to keep the overhead very low, it does not make sense for disable . It would be useful to change to to 1 sec when enabled like this.
Thread.Sleep(PollingIntervalMilliseconds);
to
Thread.Sleep(isEnabled ? 1 : PollingIntervalMilliseconds);
| 3. Specify the following collection configuration, which contains the default configuration + heap dump collection: | ||
|
|
||
| > ```cmd | ||
| > export COMPlus_EventPipeConfig=Microsoft-Windows-DotNETRuntime:0x4C1DFCCBD:5,Microsoft-Windows-DotNETRuntimePrivate:0x4002000B:5,Microsoft-DotNETCore-SampleProfiler:0x0:5 |
There was a problem hiding this comment.
You should not need private provider or the sample profiling provider to do a heap dump.
You SHOULD just need the keyword 0x1180109 at informational (4) level. (Please test it). See https://github.com/Microsoft/perfview/blob/master/src/EtwHeapDump/DotNetHeapDumper.cs#L47
We should consider making shorcuts for common configurations. For example we could make $ be special config name marker and
export COMPlus_EventPipeConfig=$HeapDump
means
export COMPlus_EventPipeConfig=Microsoft-Windows-DotNETRuntime:0x1180109
What I want is for it to be at least conceivable that you could do this without a script or cut/paste.
There was a problem hiding this comment.
Alternatively you could make the number used for COMPlus_EnableEventPipe indicate the default config.
There was a problem hiding this comment.
Let me think about how best to do this. I suspect COMPlus_EventPipeConfig=$HeapDump would probably be the right way. We can also have $GCCollect and $GCCollectOnly as well.
|
|
||
| > ```cmd | ||
| > export COMPlus_EventPipeOutputFile=/path/to/trace.netperf | ||
| > ``` |
There was a problem hiding this comment.
Note that there is a bit of an ugly problem if you end up running more than one .NET Process because this environment variable will be used by each process. At best you lose the ability to collect data on the other processes (somebody wins).
I would recommend changing things so that if the target file exists, you morph the name to .PID.netperf (roughly what you do by default). In the 'normal' case they get what they expected, but in the unusual case data is not lost.
There was a problem hiding this comment.
Fixed. The control file name won't change - just the trace file itself (and only if the requested name already exists).
| > - 4: Initially disabled, but can be enabled via control file. | ||
|
|
||
| 2. COMPlus_EventPipeConfig: Describes the set of providers/keywords/levels to be collected. Uses xperf-style format: provider:keywords:level,provider:keywords:level... | ||
| > - Optional. If not specified, the default configuration is used. |
There was a problem hiding this comment.
Indicate at least loosely what is in the default (includes CPU/THreadtime sampling, and .NET runtime events including GC, JIT, Exception and ThreadPool)
| { | ||
| // If the output file path is not specified then use | ||
| // path/to/current/working/directory/Process-<pid>.netperf.ctl | ||
| m_traceFilePath = EventPipeInternal.GetDefaultTraceFileName(); |
There was a problem hiding this comment.
There is a way to get the processID in managed code in system.private.corelib
Win32Native.GetCurrentProcessId()
EventSource already uses this. You should use this and generate the file name here, rather than the pain of creating an FCALL to do it on the native side.
There was a problem hiding this comment.
Win32Native.GetCurrentProcessId() is a Windows-specific pinvoke:
[DllImport(Interop.Libraries.Kernel32, CharSet = CharSet.Auto, SetLastError = true)]
internal static extern uint GetCurrentProcessId();
If I remember correctly, this is used for some ETW-specific features.
If desired, we can expose the process ID via FCALL and do the work in managed code.
There was a problem hiding this comment.
I think we'll need to just plumb the PID to managed code to use it to resolve filename conflicts.
There was a problem hiding this comment.
Hmm, I saw that that definition was windows specific, but what I also know is that the call site for Win32Native.GetCurrentProcessId() is unconditional (EnsureDescriptorsInitialized has to be called even on Linux (or eventListeners don't work) and there are no ifdefs around the call to GetCurrentProcessId.
I was assuming on Linux we created look-alikes, but frankly I don't know how it works, so I could be wrong. Still just try it and testing will show that it works on Linux (or you have to plumb through the FCALL).
|
|
||
| Win32EventWrite(Microsoft_Windows_DotNETRuntimeHandle, &BulkType, iDesc, EventData); | ||
|
|
||
| #else // FEATURE_PAL |
There was a problem hiding this comment.
I agree this is windows-specific code is no longer needed (we can do something that is naturally cross-platform).
Have we tested that the new heap dump works the same as the old (if we have a heap dump test, that would do it)
There was a problem hiding this comment.
I have not written any specific tests for this, but I have done some adhoc validation that looks good so far.
…ld the default trace file name in managed code.
… file name with PID if requested file already exists.
|
I believe I have addressed all of the feedback except for @vancem's comment on having config aliases such as $HeapDump. I will address this in a subsequent commit. |
| m_markerFileExists = false; | ||
|
|
||
| // Start a new thread to listen for tracing commands. | ||
| Task.Factory.StartNew(WaitForTracingCommand, TaskCreationOptions.LongRunning); |
There was a problem hiding this comment.
We can avoid burning the thread by
m_timer = new Timer(PollTracingCommand, null, 10000, 10000);
Which achieves the same effect but does not uses threadpool threads (and only for very short durations).
PollTracingCommand is now one iteration of the loop you have below.
|
Brian, don't bother making an alias like $HeapDump. I will tell you more in e-mail |
| #endif // FEATURE_WIN32_REGISTRY | ||
|
|
||
| [DllImport(JitHelpers.QCall, CharSet = CharSet.Unicode)] | ||
| private static extern int GetProcessId(); |
There was a problem hiding this comment.
I did check and on Linux Win32Native.GetCurrentProcessId() doe exist. What happens is on linux Interop.Libraries.Kernel32 is redirected to "libcoreclr", and these methods are implemented in libcoreclr.so.
Here is a list of methods that are exported in libcoreclr.so (nm -D libcoreclr.so)
00000000004e2c10 T CloseHandle
00000000000ccdf0 T coreclr_create_delegate
00000000000ccf00 T coreclr_execute_assembly
00000000000cc8f0 T coreclr_initialize
00000000000cccf0 T coreclr_shutdown
00000000000ccd70 T coreclr_shutdown_2
00000000000cc180 T CoreDllMain
0000000000524fa0 T CoTaskMemAlloc
0000000000524fc0 T CoTaskMemFree
0000000000524fb0 T CoTaskMemRealloc
00000000004ffb20 T CreateEventExW
00000000004ffa90 T CreateEventW
00000000005008b0 T CreateMutexExW
00000000005007b0 T CreateMutexW
00000000005001c0 T CreateSemaphoreExW
0000000000500250 T CreateSemaphoreW
00000000000cc280 T DllMain
00000000004e2960 T DuplicateHandle
00000000004ee9c0 T FormatMessageW
00000000004ee250 T FreeEnvironmentStringsW
00000000004e5c80 T GetACP
00000000000f4210 T GetCLRRuntimeHost
0000000000509f80 T GetCurrentProcessId
000000000050e780 T GetCurrentThreadId
00000000004edfe0 T GetEnvironmentStringsW
00000000004edbf0 T GetEnvironmentVariableW
00000000000cc3e0 T GetMetaDataInternalInterface
00000000000cc3f0 T GetMetaDataInternalInterfaceFromPublic
00000000000cc400 T GetMetaDataPublicInterfaceFromInternal
00000000004e4640 T GetProcAddress
00000000004dccd0 T GetStdHandle
00000000004f0190 T GetSystemInfo
00000000004f01f0 T GlobalMemoryStatusEx
00000000004ebff0 T LocalAlloc
00000000004ec080 T LocalFree
00000000004ec040 T LocalReAlloc
00000000004ce000 T lstrlenA
00000000004ce020 T lstrlenW
00000000000cc2e0 T MetaDataGetDispenser
00000000004e5e90 T MultiByteToWideChar
00000000004ffe70 T OpenEventW
0000000000500d80 T OpenMutexW
0000000000500470 T OpenSemaphoreW
00000000004d4dc0 T OutputDebugStringW
00000000004e5230 T PAL_RegisterModule
00000000004e53f0 T PAL_UnregisterModule
00000000004f06b0 T QueryPerformanceCounter
00000000004f0710 T QueryPerformanceFrequency
00000000004d5da0 T RaiseException
00000000005008e0 T ReleaseMutex
00000000005002e0 T ReleaseSemaphore
00000000004ffd80 T ResetEvent
00000000004edd70 T SetEnvironmentVariableW
00000000004ffbc0 T SetEvent
00000000005243c0 T SysAllocStringLen
0000000000524570 T SysFreeString
00000000005245b0 T SysStringLen
00000000004ea840 T VirtualAlloc
00000000004eb3f0 T VirtualFree
00000000004eb9d0 T VirtualQuery
00000000004e5f20 T WideCharToMultiByte
00000000004dc9f0 T WriteFile
Notice that GetCurrentProcessId is one of them.
Thus I don't believe you need create the QCALL for it, just use Win32Native.GetCurrentProcessId.
There was a problem hiding this comment.
I have switched to this and will confirm that it works on Linux.
There was a problem hiding this comment.
I have confirmed that you are correct, and this does work on Linux.
|
|
||
| // Start a new thread to listen for tracing commands. | ||
| Task.Factory.StartNew(WaitForTracingCommand, TaskCreationOptions.LongRunning); | ||
| } |
There was a problem hiding this comment.
I am including this again, because it may have been lost inadvertantly.
We can avoid burning the thread by
m_timer = new Timer(PollTracingCommand, null, 10000, 10000);
WHere PollForTracingCommand is now one iteration of the loop in WaitForTracingCommand.
This achieves the same effect but does not tie up a thread for the duration of the process.
There was a problem hiding this comment.
Yes, this was lost inadvertently. I have switched to a timer. Please take a look at the implementation. I've set it up to avoid concurrent polling operations by scheduling the next poll operation during the callback. I've done some measurement and have not seen this show up, but of course, this is with a sampling profiler. Let me know if you have any concerns with this.
There was a problem hiding this comment.
I have no concern with scheduling the next timer in the callback. A 10 second interval insures that the cost will be << .1%.
|
@dotnet-bot test this please |
3 similar comments
|
@dotnet-bot test this please |
|
@dotnet-bot test this please |
|
@dotnet-bot test this please |
|
Re-triggering runs due to infrastructure failures. @dotnet-bot test OSX10.12 x64 Checked Innerloop Build and Test |
|
@dotnet-bot help |
|
Welcome to the dotnet/coreclr Perf help The following is a list of valid commands on this PR. To invoke a command, comment the indicated phrase on the PR The following commands are valid for all PRs and repositories. Click to expand
The following jobs are launched by default for each PR against dotnet/coreclr:master. Click to expand
The following optional jobs are available in PRs against dotnet/coreclr:master. Click to expand
Have a nice day! |
|
Welcome to the dotnet/coreclr Repository The following is a list of valid commands on this PR. To invoke a command, comment the indicated phrase on the PR The following commands are valid for all PRs and repositories. Click to expand
The following jobs are launched by default for each PR against dotnet/coreclr:master. Click to expand
The following optional jobs are available in PRs against dotnet/coreclr:master. Click to expand
Have a nice day! |
|
Welcome to the dotnet/coreclr Repository The following is a list of valid commands on this PR. To invoke a command, comment the indicated phrase on the PR The following commands are valid for all PRs and repositories. Click to expand
The following jobs are launched by default for each PR against dotnet/coreclr:master. Click to expand
The following optional jobs are available in PRs against dotnet/coreclr:master. Click to expand
Have a nice day! |
|
@dotnet-bot test CROSS Check |
|
@dotnet-bot test Windows_NT x64 Checked Innerloop Build and Test (Jit - TieredCompilation=0) |
This PR implements very basic out-of-process control for EventPipe. This is to unblock investigations that can't be done by tracing for the entire life of the application (such as heap snapshot capture).
How it works:
Instructions on how to use this feature are included in Documentation/project-docs/eventpipe.md.
This PR also fixes additional issues that prevented collection of heap dumps. When tracing is enabled, if requested via the heap snapshot collection keyword, a single heap snapshot will be collected when EventPipe is enabled. If there is a desire to collect additional heap snapshots tracing must be disabled and re-enabled to capture another heap snapshot.
Reviewers should note the following areas for additional scrutiny: