In #40400 we added the ability for metricsets to report their status using the Elastic Agent control protocol to make errors more visible. One consequence of this has been surfacing previously hidden problems collecting information about PIDs.
It appears that by default we fail to get complete metrics for some PIDs on both Windows and Linux, resulting in the Elastic Agent being permanently unhealthy in the Fleet UI where no quick fix:
While this functionality is useful and has helped us find bugs, we do not want users to immediately experience unhealthy agents they can't obviously fix. As a stop gap while we fix the underlying problems, let's keep reporting the error message but have the system process metricsets report as healthy. This will put the errors messages in the UI without making the agent unhealthy, which most users treat as something to quickly fix.
For example we see something like the following in our leak detection tests regularly:
- id: system/metrics-default
state:
message: 'Healthy: communicating with pid ''1556'''
pid: 0
state: 2
units:
input-system/metrics-default-system/metrics-system-5f5e65eb-2fd6-41e1-8c29-f24d57e66509:
state: DEGRADED
message: |-
Error fetching data for metricset system.process_summary: Not enough privileges to fetch information: Not enough privileges to fetch information: GetInfoForPid: could not get all information for PID 0: error fetching name: OpenProcess failed for pid=0: The parameter is incorrect.
error fetching status: OpenProcess failed for pid=0: The parameter is incorrect.
GetInfoForPid: could not get all information for PID 4: error fetching name: GetProcessImageFileName failed for pid=4: GetProcessImageFileName failed: invalid argument
payload:
streams:
system/metrics-system.process.summary-5f5e65eb-2fd6-41e1-8c29-f24d57e66509:
error: |-
Error fetching data for metricset system.process_summary: Not enough privileges to fetch information: Not enough privileges to fetch information: GetInfoForPid: could not get all information for PID 0: error fetching name: OpenProcess failed for pid=0: The parameter is incorrect.
error fetching status: OpenProcess failed for pid=0: The parameter is incorrect.
GetInfoForPid: could not get all information for PID 4: error fetching name: GetProcessImageFileName failed for pid=4: GetProcessImageFileName failed: invalid argument
status: DEGRADEDThe proposal is that we keep the error messages, but report the state as healthy.
- id: system/metrics-default
state:
message: 'Healthy: communicating with pid ''1556'''
pid: 0
state: 2
units:
input-system/metrics-default-system/metrics-system-5f5e65eb-2fd6-41e1-8c29-f24d57e66509:
state: HEALTHY
message: |-
Error fetching data for metricset system.process_summary: Not enough privileges to fetch information: Not enough privileges to fetch information: GetInfoForPid: could not get all information for PID 0: error fetching name: OpenProcess failed for pid=0: The parameter is incorrect.
error fetching status: OpenProcess failed for pid=0: The parameter is incorrect.
GetInfoForPid: could not get all information for PID 4: error fetching name: GetProcessImageFileName failed for pid=4: GetProcessImageFileName failed: invalid argument
payload:
streams:
system/metrics-system.process.summary-5f5e65eb-2fd6-41e1-8c29-f24d57e66509:
error: |-
Error fetching data for metricset system.process_summary: Not enough privileges to fetch information: Not enough privileges to fetch information: GetInfoForPid: could not get all information for PID 0: error fetching name: OpenProcess failed for pid=0: The parameter is incorrect.
error fetching status: OpenProcess failed for pid=0: The parameter is incorrect.
GetInfoForPid: could not get all information for PID 4: error fetching name: GetProcessImageFileName failed for pid=4: GetProcessImageFileName failed: invalid argument
status: HEALTHYThis will keep the information accessible for bug reports but should hopefully reduce the perceived urgency of the problem and the level of support cases. The control protocol always allows both a state and a message regardless of what the state is, see here.
There will be a follow up issue to make switching between these two.
In #40400 we added the ability for metricsets to report their status using the Elastic Agent control protocol to make errors more visible. One consequence of this has been surfacing previously hidden problems collecting information about PIDs.
It appears that by default we fail to get complete metrics for some PIDs on both Windows and Linux, resulting in the Elastic Agent being permanently unhealthy in the Fleet UI where no quick fix:
While this functionality is useful and has helped us find bugs, we do not want users to immediately experience unhealthy agents they can't obviously fix. As a stop gap while we fix the underlying problems, let's keep reporting the error message but have the system process metricsets report as healthy. This will put the errors messages in the UI without making the agent unhealthy, which most users treat as something to quickly fix.
For example we see something like the following in our leak detection tests regularly:
The proposal is that we keep the error messages, but report the state as healthy.
This will keep the information accessible for bug reports but should hopefully reduce the perceived urgency of the problem and the level of support cases. The control protocol always allows both a state and a message regardless of what the state is, see here.
There will be a follow up issue to make switching between these two.