Skip to content

[Metricbeat] Add Data Granularity config option for AWS Cloudwatch metrics#33166

Merged
girodav merged 14 commits intoelastic:mainfrom
legoguy1000:33133-cloudwatch-granularity
Oct 25, 2022
Merged

[Metricbeat] Add Data Granularity config option for AWS Cloudwatch metrics#33166
girodav merged 14 commits intoelastic:mainfrom
legoguy1000:33133-cloudwatch-granularity

Conversation

@legoguy1000
Copy link
Copy Markdown
Contributor

@legoguy1000 legoguy1000 commented Sep 22, 2022

…trics

What does this PR do?

Add Data Granularity config option for AWS Cloudwatch metrics

Why is it important?

Allows for fewer API calls with the same level of details in the metrics

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@legoguy1000 legoguy1000 requested a review from a team as a code owner September 22, 2022 14:06
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 22, 2022
@legoguy1000
Copy link
Copy Markdown
Contributor Author

legoguy1000 commented Sep 22, 2022

@kaiyan-sheng let me know what u think??

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 22, 2022

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @legoguy1000? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-10-24T17:24:52.381+0000

  • Duration: 113 min 30 sec

Test stats 🧪

Test Results
Failed 0
Passed 4200
Skipped 1039
Total 5239

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@girodav girodav added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Sep 22, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 22, 2022
@girodav girodav added the aws Enable builds in the CI for aws cloud testing label Sep 22, 2022
@girodav girodav requested a review from a team September 22, 2022 14:19
@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Sep 22, 2022

@legoguy1000 Thank you so much for adding this!!
Quick question: (I have not tested this locally) Since with the data_granularity config, we are getting more data points, should timestamp := aws.FindTimestamp(metricDataResults) also be adjusted so we actually report all data points instead of only one?

kaiyan-sheng
kaiyan-sheng reviewed Sep 22, 2022
View reviewed changes
@legoguy1000
Update CHANGELOG.next.asciidoc
5646580 
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
@legoguy1000
Copy link
Copy Markdown
Contributor Author

legoguy1000 commented Sep 22, 2022

@legoguy1000 Thank you so much for adding this!! Quick question: (I have not tested this locally) Since with the data_granularity config, we are getting more data points, should timestamp := aws.FindTimestamp(metricDataResults) also be adjusted so we actually report all data points instead of only one?

I'd have to test it but after just looking at it, we'd have to iterate over the values list for each metrics and create an event for each one for each metrics with its associated timestamp. Is that what you're thinking?

@legoguy1000
Copy link
Copy Markdown
Contributor Author

legoguy1000 commented Sep 22, 2022 •
edited
Loading

@kaiyan-sheng Just updated. I don't even know if the FindTimestamp function is needed. The 2 loops within the function appear to return the same result so IDK why they are separated like that. Also with my change, i can only assume there will be a timestamp per value.

girodav
girodav reviewed Sep 23, 2022
View reviewed changes
@girodav
Copy link
Copy Markdown
Contributor

girodav commented Sep 23, 2022

cc @ravikesarwani I think it would be good to have your input on the docs-related changes

@girodav
Copy link
Copy Markdown
Contributor

girodav commented Sep 30, 2022

/test

@girodav
Copy link
Copy Markdown
Contributor

girodav commented Oct 6, 2022

Thanks @legoguy1000 for adding this :) . I am currently keeping this PR on hold as we might have some refactoring to do, as follow-up of #33105 (cc @kaiyan-sheng )

@girodav girodav self-assigned this Oct 12, 2022
kaiyan-sheng
kaiyan-sheng reviewed Oct 12, 2022
View reviewed changes
sayden
sayden reviewed Oct 18, 2022
View reviewed changes
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Oct 18, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 33133-cloudwatch-granularity upstream/33133-cloudwatch-granularity
git merge upstream/main
git push upstream 33133-cloudwatch-granularity

kaiyan-sheng
kaiyan-sheng approved these changes Oct 20, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

kaiyan-sheng
kaiyan-sheng reviewed Oct 24, 2022
View reviewed changes
@girodav girodav merged commit 4cac5d8 into elastic:main Oct 25, 2022
@legoguy1000 legoguy1000 deleted the 33133-cloudwatch-granularity branch October 25, 2022 11:27
@francescayeye francescayeye mentioned this pull request Feb 6, 2023
Merged
4 tasks
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@legoguy1000 @girodav @chrisberkhout
[Metricbeat] Add Data Granularity config option for AWS Cloudwatch me…
f10fcc7 
…trics (#33166)

* [Metricbeat] Add Data Granularity config option for AWS Cloudwatch metrics

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: girodav <1390902+girodav@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@girodav girodav girodav left review comments

+2 more reviewers

@sayden sayden sayden left review comments

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@girodav girodav

@legoguy1000 legoguy1000

Labels

aws Enable builds in the CI for aws cloud testing Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[AWS] Add a new "Data Granularity" parameter in Cloudwatch metricbeat module

5 participants

@legoguy1000 @elasticmachine @kaiyan-sheng @girodav @sayden