Skip to content

[aws custom logs] Updating file selector field to receive mutline configurations#10789

Merged
gizas merged 2 commits intomainfrom
aws_filedescriptor
Aug 27, 2024
Merged

[aws custom logs] Updating file selector field to receive mutline configurations#10789
gizas merged 2 commits intomainfrom
aws_filedescriptor

Conversation

@gizas
Copy link
Copy Markdown
Contributor

@gizas gizas commented Aug 14, 2024

  • Enhancement

Proposed commit message

WHAT: Update the file selector field of Collect Logs from S3 Bucket datastream to be able to receive multiline configurations
WHY: It is needed in order to be able to provide specific parsers confguration to specific file selectors. Otherwise the parsers is not applied to the files selected

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  1. Clone this PR
  2. Navigate to /integrations/packages/aws_logs
  3. Run elastic-package build
  4. Run elastic-package stack up -d -v --version=8.15.0 within the same folder. This will create a local elastic stack
  5. Create an agent policy with aws_custom_logs integration
  6. In the policy enable Collect Logs from S3 Bucket datastream
  7. Under the advanced options update File Selectors field
- regex: .*windows-events-.*
  parsers:
    - multiline:
        pattern: ^[0-9]{4}-[0-9]{2}-[0-9]
        negate:  true
        match:   after
  1. Update the credentials to connect to an s3 bucket
    9 Upload a multiline file to your s3 bucket aws s3 cp test.log s3://gizas-se-test2/windows-events-5.log

Related issues

Screenshots

Screenshot 2024-08-14 at 12 32 29 PM

@gizas gizas requested a review from a team as a code owner August 14, 2024 09:42
@gizas gizas requested review from kaiyan-sheng and zmoog August 14, 2024 09:44
@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

@elastic-sonarqube
Copy link
Copy Markdown

@andrewkroh andrewkroh added Integration:aws_logs Custom AWS Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] enhancement New feature or request labels Aug 14, 2024
Copy link
Copy Markdown

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! I think aws_bedrock also has the multi:true configuration. @agithomas it might worth some investigation there.

@gizas gizas merged commit ddbe733 into main Aug 27, 2024
@gizas gizas deleted the aws_filedescriptor branch August 27, 2024 06:59
@elasticmachine
Copy link
Copy Markdown

Package aws_logs - 1.4.0 containing this change is available at https://epr.elastic.co/search?package=aws_logs

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
…figurations (elastic#10789)

* udpating file selector field to be mutliline

* udpating cahngelog
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
…figurations (elastic#10789)

* udpating file selector field to be mutliline

* udpating cahngelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:aws_logs Custom AWS Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants