Skip to content

Change max query size for GetMetricData API to 500 and add RecentlyActive config#33105

Merged
kaiyan-sheng merged 17 commits intoelastic:mainfrom
kaiyan-sheng:change_max_query_size
Oct 11, 2022
Merged

Change max query size for GetMetricData API to 500 and add RecentlyActive config#33105
kaiyan-sheng merged 17 commits intoelastic:mainfrom
kaiyan-sheng:change_max_query_size

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Sep 15, 2022 •
edited
Loading

What does this PR do?

This PR is to change max query size for GetMetricData API from 100 to 500.

500 is the maximum metrics for a single API request: A GetMetricData operation that does not include a query can retrieve as many as 500 different metrics in a single request, with a total of as many as 100,800 data points.

AWS documentation link: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

This PR also adds RecentlyActive config parameter into ListMetrics API call to get recent metrics instead of all.

This is the first step to help improve performance for the CloudWatch metricset, please see #33195 for more details.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 15, 2022
@kaiyan-sheng kaiyan-sheng self-assigned this Sep 15, 2022
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 15, 2022

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@kaiyan-sheng kaiyan-sheng added bug backport-7.17 Automated backport to the 7.17 branch with mergify Team:Cloud-Monitoring Label for the Cloud Monitoring team backport-v8.4.0 Automated backport with mergify labels Sep 15, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 15, 2022
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review September 15, 2022 22:16
@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner September 15, 2022 22:16
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 15, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-10-10T20:00:26.293+0000

  • Duration: 120 min 18 sec

Test stats 🧪

Test Results
Failed 0
Passed 1914
Skipped 153
Total 2067

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

girodav
girodav reviewed Sep 19, 2022
View reviewed changes
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 19, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b change_max_query_size upstream/change_max_query_size
git merge upstream/main
git push upstream change_max_query_size

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Sep 20, 2022 •
edited
Loading

CI error:

[2022-09-20T04:31:02.411Z] │ Error: Error creating S3 bucket: TooManyBuckets: You have attempted to create more buckets than allowed

This is caused by By default, you can create up to 100 buckets in each of your AWS accounts. I removed several S3 buckets manually and now we are getting a new CI error:


[2022-09-20T22:52:15.990Z] │ Error: Could not load plugin

[2022-09-20T22:52:15.990Z] │ 

[2022-09-20T22:52:15.990Z] │ 

[2022-09-20T22:52:15.990Z] │ Plugin reinitialization required. Please run "terraform init".

[2022-09-20T22:52:15.990Z] │ 

[2022-09-20T22:52:15.990Z] │ Plugins are external binaries that Terraform uses to access and manipulate

[2022-09-20T22:52:15.990Z] │ resources. The configuration provided requires plugins which can't be

[2022-09-20T22:52:15.990Z] │ located,

[2022-09-20T22:52:15.990Z] │ don't satisfy the version constraints, or are otherwise incompatible.

[2022-09-20T22:52:15.990Z] │ 

[2022-09-20T22:52:15.990Z] │ Terraform automatically discovers provider requirements from your

[2022-09-20T22:52:15.990Z] │ configuration, including providers used in child modules. To see the

[2022-09-20T22:52:15.990Z] │ requirements and constraints, run "terraform providers".

[2022-09-20T22:52:15.990Z] │ 

[2022-09-20T22:52:15.990Z] │ 2 problems:

[2022-09-20T22:52:15.990Z] │ 

[2022-09-20T22:52:15.990Z] │ - failed to instantiate provider "registry.terraform.io/hashicorp/aws" to

[2022-09-20T22:52:15.990Z] │ obtain schema: unknown provider "registry.terraform.io/hashicorp/aws"

[2022-09-20T22:52:15.990Z] │ - failed to instantiate provider "registry.terraform.io/hashicorp/random"

[2022-09-20T22:52:15.990Z] │ to obtain schema: unknown provider "registry.terraform.io/hashicorp/random"

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner September 22, 2022 18:11
@kaiyan-sheng kaiyan-sheng requested review from belimawr and rdner and removed request for a team September 22, 2022 18:11
@botelastic botelastic bot added the Team:Automation Label for the Observability productivity team label Sep 22, 2022
@kaiyan-sheng kaiyan-sheng added the backport-v8.5.0 Automated backport with mergify label Sep 22, 2022
@botelastic botelastic bot added the Team:Automation Label for the Observability productivity team label Sep 26, 2022
@kaiyan-sheng kaiyan-sheng requested a review from rdner September 27, 2022 14:56
@kaiyan-sheng kaiyan-sheng mentioned this pull request Sep 28, 2022
Merged
francescayeye
francescayeye reviewed Sep 29, 2022
View reviewed changes
francescayeye
francescayeye reviewed Sep 30, 2022
View reviewed changes
@girodav girodav mentioned this pull request Oct 6, 2022
Merged
6 tasks
@kaiyan-sheng kaiyan-sheng merged commit 35cab79 into elastic:main Oct 11, 2022
@kaiyan-sheng kaiyan-sheng deleted the change_max_query_size branch October 11, 2022 03:29
mergify bot pushed a commit that referenced this pull request Oct 11, 2022
@mergify
Change max query size for GetMetricData API to 500 and add RecentlyAc…
b63a42d 
…tive config (#33105)

(cherry picked from commit 35cab79)
@mergify mergify bot mentioned this pull request Oct 11, 2022
Merged
mergify bot pushed a commit that referenced this pull request Oct 11, 2022
@mergify
Change max query size for GetMetricData API to 500 and add RecentlyAc…
d83ef93 
…tive config (#33105)

(cherry picked from commit 35cab79)
@mergify mergify bot mentioned this pull request Oct 11, 2022
Merged
kaiyan-sheng pushed a commit that referenced this pull request Oct 11, 2022
@mergify
Change max query size for GetMetricData API to 500 and add RecentlyAc…
b1c177d 
…tive config (#33105) (#33301)

(cherry picked from commit 35cab79)

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
kaiyan-sheng pushed a commit that referenced this pull request Oct 11, 2022
@mergify
[8.4](backport #33105) Change max query size for GetMetricData API to…
47864b3 
… 500 and add RecentlyActive config (#33300)

* Change max query size for GetMetricData API to 500 and add RecentlyActive config (#33105)

(cherry picked from commit 35cab79)

* Update CHANGELOG.next.asciidoc

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Oct 12, 2022

RecentlyActive configuration parameter is unfortunately not available for 0.24.0(previous version we were using).

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@chrisberkhout
Change max query size for GetMetricData API to 500 and add RecentlyAc…
2379035 
…tive config (#33105)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rdner rdner rdner approved these changes

@girodav girodav girodav approved these changes

@tommyers-elastic tommyers-elastic Awaiting requested review from tommyers-elastic

+1 more reviewer

@francescayeye francescayeye francescayeye approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport-v8.4.0 Automated backport with mergify backport-v8.5.0 Automated backport with mergify bug Team:Automation Label for the Observability productivity team Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaiyan-sheng @elasticmachine @francescayeye @rdner @girodav @tommyers-elastic