Skip to content

Describe EC2 availability zone IDs at most once per refresh (#9142)#9143

Merged
roidelapluie merged 4 commits intoprometheus:release-2.29from
gebn:ec2-sd
Aug 3, 2021
Merged

Describe EC2 availability zone IDs at most once per refresh (#9142)#9143
roidelapluie merged 4 commits intoprometheus:release-2.29from
gebn:ec2-sd

Conversation

@gebn
Copy link
Contributor

@gebn gebn commented Aug 2, 2021

This was an oversight on my part. If Prometheus is missing ec2:DescribeAvailabilityZones, the SD will attempt to refresh for every target, resulting in lots of 403s:

level=warn ts=2021-08-02T12:17:48.880Z caller=ec2.go:245 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2a err="could not describe availability zones: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: b9c37a4b-789a-4963-8655-db4671bf8cf4"
level=warn ts=2021-08-02T12:17:48.884Z caller=ec2.go:245 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2c err="could not describe availability zones: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: efc89d50-fbf9-43a2-9b63-c527dcffccf5"
level=warn ts=2021-08-02T12:17:48.889Z caller=ec2.go:245 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2a err="could not describe availability zones: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 1d55608a-def0-4757-b401-be726630031d"
level=warn ts=2021-08-02T12:17:48.893Z caller=ec2.go:245 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2c err="could not describe availability zones: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: b197e921-af02-4320-b950-c98f43c2d542"

This PR caps this to a single request per refresh(), turning the above into this:

level=warn ts=2021-08-02T12:54:44.102Z caller=ec2.go:256 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2a err="could not describe availability zones: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: e1086f51-d8fd-4383-b1e7-814e167c5caf"                                                                                                                                                         
level=warn ts=2021-08-02T12:54:44.102Z caller=ec2.go:256 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2c err="no availability zone ID mapping found for eu-west-2c"        
level=warn ts=2021-08-02T12:54:44.102Z caller=ec2.go:256 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2a err="no availability zone ID mapping found for eu-west-2a"
level=warn ts=2021-08-02T12:54:44.102Z caller=ec2.go:256 component="discovery manager scrape" discovery=ec2 msg="Unable to determine availability zone ID" az=eu-west-2c err="no availability zone ID mapping found for eu-west-2c"

It would be good to integrate this before v0.29.0 final.

@gebn gebn mentioned this pull request Aug 2, 2021
Merged
roidelapluie
roidelapluie reviewed Aug 2, 2021
View reviewed changes
@gebn
Describe EC2 availability zones at most once per SD load
2d8f9c6 
Closes prometheus#9142.

Signed-off-by: George Brighton <george@gebn.co.uk>
roidelapluie
roidelapluie reviewed Aug 2, 2021
View reviewed changes
roidelapluie
roidelapluie reviewed Aug 2, 2021
View reviewed changes
@gebn
Incorporate feedback
1f752b6 
Signed-off-by: George Brighton <george@gebn.co.uk>
roidelapluie
roidelapluie reviewed Aug 3, 2021
View reviewed changes
@roidelapluie
Copy link
Member

roidelapluie commented Aug 3, 2021

Thanks, one last small thing :)

@gebn
Integrate feedback
e6e7b08 
Signed-off-by: George Brighton <george@gebn.co.uk>
@roidelapluie
Copy link
Member

roidelapluie commented Aug 3, 2021

Thanks, I will merge on green!

@roidelapluie roidelapluie merged commit ba4911a into prometheus:release-2.29 Aug 3, 2021
@roidelapluie roidelapluie mentioned this pull request Aug 4, 2021
Closed
@gebn gebn deleted the ec2-sd branch February 11, 2024 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@roidelapluie roidelapluie roidelapluie approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gebn @roidelapluie