Skip to content

Fix adding processors in cloudfront logs#4395

Merged
tommyers-elastic merged 2 commits intoelastic:mainfrom
kaiyan-sheng:fix_processors
Oct 6, 2022
Merged

Fix adding processors in cloudfront logs#4395
tommyers-elastic merged 2 commits intoelastic:mainfrom
kaiyan-sheng:fix_processors

Conversation

@kaiyan-sheng
Copy link
Copy Markdown

@kaiyan-sheng kaiyan-sheng commented Oct 5, 2022 •
edited
Loading

What does this PR do?

This PR is to fix adding processors for cloudfront logs in AWS package. With the change, we can see processors in agent policy is in the right format now:

inputs:
  - id: aws-s3-cloudfront-096288f1-4c9d-4f1b-a6b1-7e73ecc7a849
    name: aws-1
    revision: 1
    type: aws-s3
    use_output: default
    meta:
      package:
        name: aws
        version: 1.24.3
    data_stream:
      namespace: default
    streams:
      - id: aws-s3-aws.cloudfront_logs-096288f1-4c9d-4f1b-a6b1-7e73ecc7a849
        data_stream:
          dataset: aws.cloudfront_logs
          type: logs
        queue_url: 'https://sqs.us-east-1.amazonaws.com/123/test'
        max_number_of_messages: 5
        access_key_id: a
        secret_access_key: b
        session_token: c
        tags:
          - forwarded
          - aws-cloudfront
        publisher_pipeline.disable_host: true
        processors:
          - drop_event:
              when:
                regexp:
                  message: ^#.*
          - drop_event:
              when:
                not:
                  regexp:
                    message: .*text/html.*

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner October 5, 2022 20:23
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 5, 2022
@kaiyan-sheng kaiyan-sheng added bug Something isn't working, use only for issues Integration:aws AWS labels Oct 5, 2022
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 5, 2022
Merged
4 tasks
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 5, 2022 •
edited
Loading

🚀 Benchmarks report

Package aws 👍(7) 💚(4) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
cloudwatch_logs 500000 333333.33 -166666.67 (-33.33%) 💔
route53_resolver_logs 6211.18 4587.16 -1624.02 (-26.15%) 💔

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 5, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-10-05T20:25:13.172+0000

  • Duration: 33 min 16 sec

Test stats 🧪

Test Results
Failed 0
Passed 162
Skipped 2
Total 164

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 5, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (13/13) 💚
Files 92.857% (13/14) 👎 -4.635
Classes 92.857% (13/14) 👎 -4.635
Methods 84.232% (203/241) 👎 -6.057
Lines 95.697% (5204/5438) 👍 4.062
Conditionals 100.0% (0/0) 💚

tommyers-elastic
tommyers-elastic approved these changes Oct 6, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

it's not ideal that indentation errors can affect behavior so severely! i guess the same is true of this type of YAML config in general. 😬

@tommyers-elastic tommyers-elastic merged commit 0939fd3 into elastic:main Oct 6, 2022
@kaiyan-sheng kaiyan-sheng deleted the fix_processors branch October 6, 2022 12:44
@kaiyan-sheng
Copy link
Copy Markdown
Author

kaiyan-sheng commented Oct 6, 2022

@tommyers-elastic I think the main issue is missing - in front of {{processors}} :) Thanks for the review!!

@elasticsatch
Copy link
Copy Markdown

elasticsatch commented Oct 11, 2022

Thanks for the fix. So this change requires input NOT to have dash in front of each processor? Could you show me some examples (or maybe a test case?)

@kaiyan-sheng
Copy link
Copy Markdown
Author

kaiyan-sheng commented Oct 11, 2022 •
edited
Loading

@elasticsatch You mean requires/or not to have a dash in front of each processor on the kibana UI? You can see an example in the screenshot for the Kibana UI when adding a processor here: #4394

@elasticsatch
Copy link
Copy Markdown

elasticsatch commented Oct 11, 2022

Oh thanks, I overlooked the screenshot. So if I want multiple processors we should put something like this, right?

dissect:
  ...

drop_event:
  ..

And we can put blank lines in between, correct?

@kaiyan-sheng
Copy link
Copy Markdown
Author

kaiyan-sheng commented Oct 11, 2022

@elasticsatch OMG good call...... Nope right now it only support adding one processor! I will look into it tomorrow and fix it for multiple processors. Thanks!

@elasticsatch
Copy link
Copy Markdown

elasticsatch commented Oct 11, 2022

Good to hear and no pressure. Please take your time.

I also suggest to put the instruction somewhere in a document so users can see how to write the processors correctly. There is no good ones as of this moment so it took a little bit of time to figure it out on my own.

@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 11, 2022
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tommyers-elastic tommyers-elastic tommyers-elastic approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

bug Something isn't working, use only for issues Integration:aws AWS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[AWS] cloudfront log integration failed to add processors

4 participants

@kaiyan-sheng @elasticmachine @elasticsatch @tommyers-elastic