Skip to content

Network direction processor#66644

Merged
danhermann merged 11 commits intoelastic:masterfrom
andrewstucki:network-direction-processor
Jan 25, 2021
Merged

Network direction processor#66644
danhermann merged 11 commits intoelastic:masterfrom
andrewstucki:network-direction-processor

Conversation

@andrewstucki
Copy link
Copy Markdown

@andrewstucki andrewstucki commented Dec 18, 2020

This PR computes the ECS perimeter-based network direction of a document given a source and destination ip and a list of ip addresses that specify the internal network boundary. It's essentially a port of the beats processor along with named range support.

➜  ~ curl -H "Content-Type: application/json" -X POST -u elastic:password http://localhost:9200/_ingest/pipeline/_simulate\?verbose --data-binary @- << EOF
{
  "pipeline": {
    "processors": [
      {
        "network_direction": {
          "internal_networks": [ "private" ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "source": {
          "ip": "128.232.110.120"
        },
        "destination": {
          "ip": "192.168.1.1"
        }
      }
    }
  ]
}
EOF
{"docs":[{"processor_results":[{"processor_type":"network_direction","status":"success","doc":{"_index":"_index","_id":"_id","_source":{"destination":{"ip":"192.168.1.1"},"source":{"ip":"128.232.110.120"},"network":{"direction":"inbound"}},"_ingest":{"pipeline":"_simulate_pipeline","timestamp":"2020-12-18T20:23:46.589327Z"}}}]}]}%

See "network":{"direction":"inbound"}

@andrewstucki andrewstucki added :Distributed/Ingest Node Execution or management of Ingest Pipelines >feature Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v7.12.0 v8.0.0 labels Dec 18, 2020
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Dec 18, 2020

Pinging @elastic/es-core-features (Team:Core/Features)

@danhermann danhermann self-requested a review December 18, 2020 20:43
@danhermann
Copy link
Copy Markdown
Contributor

danhermann commented Dec 18, 2020

@andrewstucki, thanks for submitting this. I'll review it from the ES Core Features side, but it might wait until early January since our team has limited bandwidth until then.

@andrewstucki
Copy link
Copy Markdown
Author

andrewstucki commented Dec 18, 2020

@danhermann that's perfectly fine, I'm hoping it can get in by 7.12 so that it can unblock some ECS upgrade work we postponed on the beats ingest side of things, but no rush. Just whipped this up to gain a bit more familiarity with ES.

@danhermann
Copy link
Copy Markdown
Contributor

danhermann commented Jan 7, 2021

@elasticmachine update branch

@danhermann
Copy link
Copy Markdown
Contributor

danhermann commented Jan 7, 2021

@andrewstucki, there are some checkstyle violations here that need to be addressed. You should be able to see them either in the output of the elasticsearch-ci/2 build or by running ./gradlew :x-pack:plugin:ingest:checkstyleTest locally.

@andrewstucki
Copy link
Copy Markdown
Author

andrewstucki commented Jan 13, 2021

@danhermann sorry about the late response. I merged latest master in so this wasn't stale anymore. The checkstyle issues seemed to be outdated and related to a .* import that I initially had. For some reason I don't think CI picked up the change when I fixed it. Hopefully we'll see the build pass this time around.

danhermann
danhermann reviewed Jan 19, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@danhermann danhermann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewstucki, this looks good. I added two very minor comments below and once those are addressed, I can get this merged.

@danhermann
Copy link
Copy Markdown
Contributor

danhermann commented Jan 25, 2021

Thanks, @andrewstucki. This looks good now and I'll get it merged into the appropriate branches.

@danhermann danhermann merged commit eb2f344 into elastic:master Jan 25, 2021
@danhermann danhermann mentioned this pull request Jan 25, 2021
Merged
@andrewstucki andrewstucki mentioned this pull request Feb 8, 2021
Closed
89 tasks
@danhermann
Copy link
Copy Markdown
Contributor

danhermann commented Feb 8, 2021

cc: @elastic/es-ui in case Kibana auto-complete needs to be updated with this new processor.

@cjcenizal cjcenizal mentioned this pull request Feb 8, 2021
Closed
danhermann pushed a commit to danhermann/elasticsearch that referenced this pull request Feb 16, 2021
@danhermann
Network direction processor (elastic#66644)
db21ea6
@danhermann danhermann mentioned this pull request Feb 16, 2021
Merged
@stevejgordon stevejgordon mentioned this pull request Feb 22, 2021
Closed
34 tasks
This was referenced Mar 15, 2021
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

@danhermann danhermann Awaiting requested review from danhermann

Assignees

No one assigned

Labels

:Distributed/Ingest Node Execution or management of Ingest Pipelines >feature Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v7.12.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewstucki @elasticmachine @danhermann @jakelandis