Situation

All major CSPs provide SDKs for accessing their API. Beats use the cloud provider API to collect logs, metrics, and metadata.

Over time, CSPs release new versions of SDK. Most of the time, new versions are minor or patch releases. Major releases with breaking changes usually happen every few years.

Usually, we upgrade the cloud provider SDK in two circumstances:

• we need a new feature
• we need a bugfix

That's only available in a new SDK version.

Our attitude is primarily reactive.

Problem statement

The current reactive posture has a few downsides:

• On average, our SDK modules are outdated to various degrees (missing fixes)
• Upgrades happen not so often, sometimes taking big jumps in versions (increasing risks)
• On average, if we need a bugfix in one of our dependencies, we can add it now, but the subsequent stack releases may be weeks away; we only backport to the previous release.

Solutions

Manage AWS/Azure/GCP SDK version incrementally using Dependabot.

Pros:

• It is more manageable to upgrade 1-2 dependencies at a time instead of doing a big batch occasionally.
• SDKs are up to date with fixes and improvements
• We integrate updates in the next release to improve the change or avoid bug reports and support requests.

Cons:

• We are making more changes; we need to mitigate this risk by improving our test suite (we'll address this in a dedicated issue).

### Tasks
- [ ] https://github.com/elastic/beats/pull/39495
- [ ] https://github.com/elastic/beats/pull/39739
- [ ] https://github.com/elastic/beats/pull/40150
- [ ] https://github.com/elastic/beats/pull/39738

Related

Here are a few related issues and PRs:

• Fix the AWS SDK dependencies issue causing the "not found, ResolveEndpointV2" error #39454