Skip to content

Add log_group_name_prefix config option for aws-cloudwatch input#26187

Merged
kaiyan-sheng merged 5 commits intoelastic:masterfrom
kaiyan-sheng:wildcard_cloudwatch
Jun 28, 2021
Merged

Add log_group_name_prefix config option for aws-cloudwatch input#26187
kaiyan-sheng merged 5 commits intoelastic:masterfrom
kaiyan-sheng:wildcard_cloudwatch

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Jun 7, 2021 •
edited
Loading

What does this PR do?

This PR is to add log_group_name_prefix config option into aws-cloudwatch input to equivalently support wildcard for log_group_name.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

With filebeat input below, you should be able to see Filebeat starts aws-cloudwatch input to collect logs from all log groups that has names start with /aws/ecs.

filebeat.inputs:
- type: aws-cloudwatch
  credential_profile_name: elastic-beats
  log_group_name_prefix: /aws/ecs
  region_name: us-east-1

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 7, 2021
@kaiyan-sheng kaiyan-sheng self-assigned this Jun 7, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 7, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26187 updated

  • Start Time: 2021-06-24T14:33:19.645+0000

  • Duration: 109 min 9 sec

  • Commit: f2a4019

Test stats 🧪

Test Results
Failed 0
Passed 7395
Skipped 1201
Total 8596

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7395
Skipped 1201
Total 8596

@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review June 23, 2021 11:46
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Jun 23, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b wildcard_cloudwatch upstream/wildcard_cloudwatch
git merge upstream/master
git push upstream wildcard_cloudwatch

@kaiyan-sheng kaiyan-sheng requested a review from ChrsMark June 23, 2021 12:05
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. needs_integration_sync Changes in this PR need synced to elastic/integrations. Team:Integrations Label for the Integrations team labels Jun 23, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 23, 2021

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 23, 2021
ChrsMark
ChrsMark approved these changes Jun 28, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks!

@kaiyan-sheng kaiyan-sheng merged commit 1c9a488 into elastic:master Jun 28, 2021
@kaiyan-sheng kaiyan-sheng deleted the wildcard_cloudwatch branch June 28, 2021 13:57
@kaiyan-sheng kaiyan-sheng mentioned this pull request Jun 28, 2021
Merged
6 tasks
@kaiyan-sheng kaiyan-sheng added v7.14.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jun 28, 2021
kaiyan-sheng pushed a commit that referenced this pull request Jun 28, 2021
Add log_group_name_prefix config option for aws-cloudwatch input (#26187
c6ab0c1 
) (#26527)

(cherry picked from commit 1c9a488)
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 29, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
d4a87cb 
* master:
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  Add inttests for the x-pack/metricbeat on a PR/branches basis (elastic#26526)
  Suppress too many errors (elastic#26224)
  Fix master's linting issue (elastic#26517)
  [libbeat] Fix encoding and file offset issues in the disk queue (elastic#26484)
  Add log_group_name_prefix config option for aws-cloudwatch input (elastic#26187)
  Update shared-deduplication.asciidoc (elastic#26492)
  Add Recorded Future support to threatintel module (elastic#26481)
@kaiyan-sheng kaiyan-sheng removed the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jul 19, 2021
@plasalas
Copy link
Copy Markdown

plasalas commented Aug 12, 2021

can this feature be added to Filebeat 7.10?

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Aug 17, 2021

can this feature be added to Filebeat 7.10?

Sorry since this is a feature, we can't backport it to older releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

Team:Integrations Label for the Integrations team v7.14.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Filebeat] Adding wildcard support for log_group_name in cloudwatch input

4 participants

@kaiyan-sheng @elasticmachine @plasalas @ChrsMark