Skip to content

[AWS Config] - Added rate limiting and retry configuration for AWS Config data stream.#16579

Merged
ShourieG merged 5 commits intoelastic:mainfrom
ShourieG:bugfix/aws_15764
Dec 18, 2025
Merged

[AWS Config] - Added rate limiting and retry configuration for AWS Config data stream.#16579
ShourieG merged 5 commits intoelastic:mainfrom
ShourieG:bugfix/aws_15764

Conversation

@ShourieG
Copy link
Copy Markdown
Contributor

@ShourieG ShourieG commented Dec 16, 2025

Type of change

  • Enhancement

Proposed commit message

aws: add rate limiting and retry configuration to config data stream

The AWS Config integration was failing to collect all compliance data
due to ThrottlingException errors from AWS APIs. The integration makes
calls to DescribeConfigRules and GetComplianceDetailsByConfigRule APIs
in rapid succession without any rate limiting, exceeding AWS rate limits.

Added resource.rate_limit and resource.retry configurations to the CEL
input to prevent throttling and handle transient errors gracefully:
- rate_limit (default 1/s): limits API request rate to stay within AWS limits
- rate_limit_burst (default 1): prevents request bursting
- retry_max_attempts (default 5): retries failed requests with exponential backoff
- retry_wait_min/max (1s-60s): configures backoff timing

Fixes: https://github.com/elastic/integrations/issues/15764

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@ShourieG ShourieG self-assigned this Dec 16, 2025
@ShourieG ShourieG added enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 16, 2025
@ShourieG ShourieG marked this pull request as ready for review December 16, 2025 12:38
@ShourieG ShourieG requested review from a team as code owners December 16, 2025 12:38
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 16, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

chrisberkhout
chrisberkhout approved these changes Dec 16, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@chrisberkhout chrisberkhout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just needs the version number fix in the changelog.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 18, 2025

💚 Build Succeeded

History

cc @ShourieG

@ShourieG ShourieG merged commit ba39970 into elastic:main Dec 18, 2025
8 checks passed
@ShourieG ShourieG deleted the bugfix/aws_15764 branch December 18, 2025 10:53
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 18, 2025

Package aws - 5.3.0 containing this change is available at https://epr.elastic.co/package/aws/5.3.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrisberkhout chrisberkhout chrisberkhout approved these changes

Assignees

@ShourieG ShourieG

Labels

enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[aws.config] fails to collect all compliance data due to ThrottlingException

3 participants

@ShourieG @elasticmachine @chrisberkhout