Skip to content

Cherry-pick #22940 to 7.x: [Processors] Mime-Type Detection#22972

Merged
andrewstucki merged 4 commits intoelastic:7.xfrom
andrewstucki:backport_22940_7.x
Dec 8, 2020
Merged

Cherry-pick #22940 to 7.x: [Processors] Mime-Type Detection#22972
andrewstucki merged 4 commits intoelastic:7.xfrom
andrewstucki:backport_22940_7.x

Conversation

@andrewstucki
Copy link
Copy Markdown

@andrewstucki andrewstucki commented Dec 8, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #22940 to 7.x branch. Original message:

What does this PR do?

Adds a basic mime type sniffer beats processor and uses it in packetbeat. This allows us to implement the new ECS 1.7 http.*.mime_type fields.

Basically we do the following:

  1. Run a portion of the whatever data we want to run detection on through detection via h2non/filetype
  2. If that fails, run through the net/http sniffer
  3. If the net/http sniffer says this is plain text (no binary encoding/html detected), attempt to determine if we have some sort of "specially encoded" text (i.e. json, xml, etc.)
  4. If all else fails and we get back a generic mime type (application/octet-stream) return without filling in the field

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

[Processors] Mime-Type Detection (elastic#22940)
d42fb7c 
* Add mimetype processor

* Add mimetype detection for packetbeat

* Update changelog

* Rev go.sum

* Refactor for reusability and rename to detect_mime_type

* reformat imports

* update docs

* Update maxHeaderSize name and add comment on the fallback behavior

(cherry picked from commit 5f52979)
@andrewstucki andrewstucki requested a review from a team as a code owner December 8, 2020 01:05
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 8, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 8, 2020

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 8, 2020
@botelastic
Copy link
Copy Markdown

botelastic bot commented Dec 8, 2020

This pull request doesn't have a Team:<team> label.

@andrewstucki andrewstucki requested a review from a team December 8, 2020 01:06
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 8, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #22972 updated

  • Start Time: 2020-12-08T17:42:52.192+0000

  • Duration: 96 min 33 sec

Test stats 🧪

Test Results
Failed 0
Passed 16876
Skipped 1398
Total 18274

Steps errors 2

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 14 sec . View more details on here
Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 16 sec . View more details on here

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 16876
Skipped 1398
Total 18274

@andrewstucki andrewstucki merged commit a5ab280 into elastic:7.x Dec 8, 2020
@andrewstucki andrewstucki deleted the backport_22940_7.x branch December 8, 2020 21:12
@zube zube bot removed the [zube]: Done label Mar 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

No one assigned

Labels

backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewstucki @elasticmachine @andrewkroh