Skip to content

Add latency config option for aws-cloudwatch input#28509

Merged
kaiyan-sheng merged 5 commits intoelastic:masterfrom
kaiyan-sheng:latency_cloudwatch_input
Oct 19, 2021
Merged

Add latency config option for aws-cloudwatch input#28509
kaiyan-sheng merged 5 commits intoelastic:masterfrom
kaiyan-sheng:latency_cloudwatch_input

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Oct 18, 2021 •
edited
Loading

What does this PR do?

This PR is to add latency config parameter into Filebeat aws-cloudwatch input, similar to what we did in cloudwatch metricbeat metricset.
Some AWS services send logs to CloudWatch with a latency to process larger than aws-cloudwatch input scan_frequency. In this case, we need to use the latency config to shift the collection start time and end time in order to collect the logs.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 18, 2021
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Oct 18, 2021

This pull request does not have a backport label. Could you fix it @kaiyan-sheng? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 18, 2021
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 18, 2021
@kaiyan-sheng kaiyan-sheng added backport-v7.16.0 Automated backport with mergify Team:Integrations Label for the Integrations team labels Oct 18, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 18, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 18, 2021

Pinging @elastic/integrations (Team:Integrations)

@kaiyan-sheng kaiyan-sheng removed the backport-skip Skip notification from the automated backport with mergify label Oct 18, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 18, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-10-18T21:31:19.350+0000

  • Duration: 108 min 35 sec

  • Commit: 34cfa08

Test stats 🧪

Test Results
Failed 0
Passed 7753
Skipped 1201
Total 8954

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

@kaiyan-sheng kaiyan-sheng merged commit 0e19e9e into elastic:master Oct 19, 2021
@kaiyan-sheng kaiyan-sheng deleted the latency_cloudwatch_input branch October 19, 2021 13:43
mergify bot pushed a commit that referenced this pull request Oct 19, 2021
Add latency config option for aws-cloudwatch input (#28509)
468137a 
(cherry picked from commit 0e19e9e)
@mergify mergify bot mentioned this pull request Oct 19, 2021
Merged
kaiyan-sheng pushed a commit that referenced this pull request Oct 19, 2021
@mergify
Add latency config option for aws-cloudwatch input (#28509) (#28531)
3d448c2 
(cherry picked from commit 0e19e9e)

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
Add latency config option for aws-cloudwatch input (elastic#28509)
6987cc8
@kaiyan-sheng kaiyan-sheng mentioned this pull request Dec 15, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@francescayeye francescayeye francescayeye approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport-v7.16.0 Automated backport with mergify Team:Integrations Label for the Integrations team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Filebeat CloudWatch input misses events

3 participants

@kaiyan-sheng @elasticmachine @francescayeye