Sometime metricbeat get empty metric result of EC2 instance. There is an issue in aws.GetStartTimeEndTime
- Version: 8.3.3
- Steps to Reproduce:
get aws ec2 instance metrics with the python script.
import boto3
import datetime
client = boto3.client("cloudwatch", region_name="ap-northeast-2")
period = 300
end = datetime.datetime.utcnow().replace(second=0)
start = end - datetime.timedelta(seconds=period * 1)
response = client.get_metric_data(
MetricDataQueries=[
{
"Id": "id1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/EC2",
"MetricName": "CPUUtilization",
"Dimensions": [
{"Name": "InstanceId", "Value": "your instance id"},
],
},
"Period": period,
"Stat": "Average",
},
"Label": "label1",
},
],
StartTime=start,
EndTime=end,
)
print('now: ', datetime.datetime.utcnow())
print('end: ', end)
print(response["MetricDataResults"])If end mod period is between 60s and 120s , there may be empty metric result.
now: 2022-08-09 13:51:12.032382
end: 2022-08-09 13:51:00.150606
[{'Id': 'id1', 'Label': 'label1', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]
now: 2022-08-09 13:51:03.090935
end: 2022-08-09 13:51:00.091199
[{'Id': 'id1', 'Label': 'label1', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]
- CloudWatch metricset use this function to get start and end timestamp.
|
func GetStartTimeEndTime(period time.Duration, latency time.Duration) (time.Time, time.Time) { |
|
endTime := time.Now() |
|
if latency != 0 { |
|
// add latency if config is not 0 |
|
endTime = endTime.Add(latency * -1) |
|
} |
|
|
|
// Set startTime to be one period earlier than the endTime. If metrics are |
|
// not being collected, use latency config parameter to offset the startTime |
|
// and endTime. |
|
startTime := endTime.Add(period * -1) |
|
// Defining duration |
|
d := 60 * time.Second |
|
// Calling Round() method |
|
return startTime.Round(d), endTime.Round(d) |
|
} |
- According to the AWS API doc, we should make
StartTime and EndTime align with period.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html#API_GetMetricData_RequestParameters
For better performance, specify StartTime and EndTime values that align with the value of the metric's Period and sync up with the beginning and end of an hour. For example, if the Period of a metric is 5 minutes, specifying 12:05 or 12:30 as StartTime can get a faster response from CloudWatch than setting 12:07 or 12:29 as the StartTime.
- endTime := time.Now()
+ endTime := time.Now().Truncate(period)
Sometime metricbeat get empty metric result of EC2 instance. There is an issue in
aws.GetStartTimeEndTimeget aws ec2 instance metrics with the python script.
If
end mod periodis between 60s and 120s , there may be empty metric result.beats/x-pack/metricbeat/module/aws/utils.go
Lines 22 to 37 in 62ddcc4
StartTimeandEndTimealign with period.https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html#API_GetMetricData_RequestParameters