URI parts ingest processor by danhermann · Pull Request #65150 · elastic/elasticsearch

danhermann · 2020-11-17T16:30:11Z

Adds a new uri_parts processor that decomposes a URI into its constituent parts. E.g.:

POST _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "url_parts": {
          "field": "uri_field",
          "target_field": "url",
          "keep_original": true,
          "remove_if_successful": true
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "uri_field": "http://user:pw@www.google.com:80/blarg.gif#ref"
      }
    }
  ]
}

results in:

"processor_type" : "uri_parts",
"status" : "success",
"doc" : {
  "_index" : "_index",
  "_id" : "_id",
  "_source" : {
    "url" : {
      "path" : "/blarg.gif",
      "fragment" : "ref",
      "extension" : "gif",
      "password" : "pw",
      "original" : "http://user:pw@www.google.com:80/blarg.gif#ref",
      "scheme" : "http",
      "port" : 80,
      "user_info" : "user:pw",
      "domain" : "www.google.com",
      "username" : "user"
    }
  }

The processor relies on the java.net.URI class to parse the URI and attempts to map the parts into ECS fields. Some ECS fields are not part of the URI spec, so see the table below for how those are handled:

URL Parts Processor	ECS	java.net.URI	Comments
domain	url.domain	getHost()
extension	url.extension		This is not part of the URI spec and is manually parsed out of the path element on a best-effort basis if a `.` exists in the path
fragment	url.fragment	getFragment()
	url.full
original	url.original		The processor includes an option to retain the original URL
password	url.password		The URI spec defines an "authority" field but does not define either `username` or `password` though they are commonly presented with the `username:password` convention. The `username` and `password` fields are parsed out of the `user_info` field on a best-effort basis if a `:` exists.
path	url.path	getPath()
port	url.port	getPort()
query	url.query	getQuery()
	url.registered_domain
scheme	url.scheme	getProtocol()
	url.top_level_domain
username	url.username		See comment on `password` above
user_info		getUserInfo()	Corresponds to the "authority" field of the URI spec without the domain

Also introduces a new module for ingest processors.

Closes #57481

elasticmachine · 2020-11-17T16:30:14Z

Pinging @elastic/es-core-features (Team:Core/Features)

danhermann · 2020-11-17T16:32:39Z

@leehinman and @andrewkroh, I would be interested in your feedback (and/or others, as appropriate) on whether this provides the functionality you described in #57481.

andrewkroh

Nice, this will make writing pipelines simpler for us.

andrewkroh · 2020-11-18T01:18:32Z

x-pack/plugin/ingest/src/main/java/org/elasticsearch/xpack/ingest/UrlPartsProcessor.java

+
+        URL url;
+        try {
+            url = new URL(value);


Are there limitations on which schemes that this can parse? Would these parse?

ftp://ftp.is.co.za/rfc/rfc1808.txt

ldap://[2001:db8::7]/c=GB?objectClass?one

telnet://192.0.2.16:80/

Perhaps using java.net.URI would be more forgiving and not require a URLStreamHandler to be loaded.

Similarly parts of the URL are required for parsing to work?

@andrewkroh, thanks for looking it over and commenting. I switched over to java.net.URI which does support more schemes including all three of the examples you list above.

Right now, no parts of a URI are required beyond what java.net.URI needs to construct an instance. Is that what you would prefer?

andrewkroh · 2020-11-18T01:28:40Z

x-pack/plugin/ingest/src/main/java/org/elasticsearch/xpack/ingest/UrlPartsProcessor.java

+            throw new IllegalArgumentException("unable to parse URL [" + value + "]");
+        }
+        var urlParts = new HashMap<String, Object>();
+        urlParts.put("domain", url.getHost());


ECS isn't clear on what's correct here, but what does getHost return for bracked IPv6 addresses?

@elastic/ecs Should url.domain include the brackets that are required when using IPv6 addresses in URLs? https://www.ietf.org/rfc/rfc2732.txt

Per this source, getHost() will return the IPv6 address enclosed in the brackets.

Since brackets are required with a literal IPv6 address, url.domain should include the brackets. We can improve the description of url.domain in the ECS docs to clarify.

andrewkroh

LGTM. 👍

webmat · 2020-11-18T18:55:14Z

This is great @danhermann 👍

Note that I think it's fine if this processor doesn't populate the domain breakdown fields url.registered_domain and url.top_level_domain. Properly populating those to consider all the edge case effective TLDs requires lookups in a DB like Mozilla's. Which I think should be a distinct processor (like Beat's), and is therefore out of scope here.

andrewkroh · 2020-11-18T19:01:27Z

There's a request for a registered domain processor (#57476). Perhaps if that gets implemented this one could internally use it.

danhermann · 2020-11-18T19:34:34Z

There's a request for a registered domain processor (#57476). Perhaps if that gets implemented this one could internally use it.

I can add that on this processor as an option once the registered domain processor is completed.

danhermann · 2020-11-18T19:48:40Z

Last functional question on this one -- should the processor be renamed to uri_parts since it's parsing schemes that aren't supported by the URL spec?

Edit: Though I do see that the ECS field names are all prefixed with url rather than uri. 😦

leehinman · 2020-11-18T19:58:22Z

This is awesome. Thank You.

webmat · 2020-11-18T20:25:03Z

should the processor be renamed to uri_parts

I don't really mind either way, but I think it'll indeed be clearer to users that they can use it on all sorts of URIs if it's named uri_parts. So I'd be in favor of that :-)

danhermann · 2020-11-18T21:43:22Z

@elasticmachine run elasticsearch-ci/2

danhermann · 2020-11-19T16:33:20Z

cc: @elastic/es-ui in case Kibana auto-complete needs to be updated with this new processor.

andreidan

LGTM, thanks for adding this processor @danhermann

Left a minor comment

andreidan · 2020-11-20T12:35:54Z

x-pack/plugin/ingest/src/main/java/org/elasticsearch/xpack/ingest/UriPartsProcessor.java

+            if (userInfo.contains(":")) {
+                int colonIndex = userInfo.indexOf(":");
+                uriParts.put("username", userInfo.substring(0, colonIndex));
+                uriParts.put("password", colonIndex < userInfo.length() ? userInfo.substring(colonIndex + 1) : "");


would this fail with IndexOutOfBounds for http://user:@www.google.com:80/blarg.gif#ref ? (no password)
Shall we add a test for this?

In that case, the password is set to an empty string. I'll add another test case to make that clear.

Thanks for the review, @andreidan!

alisonelizabeth · 2020-11-20T13:51:48Z

cc: @elastic/es-ui in case Kibana auto-complete needs to be updated with this new processor.

Thanks for the heads up @danhermann! I've opened elastic/kibana#83915 to add support for this in the UI.

URL parts processor with new ingest module

df786f8

danhermann added :Distributed/Ingest Node Execution or management of Ingest Pipelines v8.0.0 v7.11.0 labels Nov 17, 2020

elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Nov 17, 2020

danhermann added 3 commits November 17, 2020 10:36

spotless making code less readable :(

3d76502

forbidden APIs

828914e

add tests, don't remove target field

caf2715

andrewkroh reviewed Nov 18, 2020

View reviewed changes

switch to java.net.URI and add additional test cases

0c9163f

ebeahan mentioned this pull request Nov 18, 2020

Usage of brackets for a URL containing IPv6 address elastic/ecs#1131

Merged

andrewkroh approved these changes Nov 18, 2020

View reviewed changes

danhermann added 2 commits November 18, 2020 14:11

disable testing conventions check

30b9eff

spotless. again.

44bb394

rename URL to URI

6b052cb

danhermann changed the title ~~URL parts ingest processor~~ URI parts ingest processor Nov 18, 2020

andreidan approved these changes Nov 20, 2020

View reviewed changes

alisonelizabeth mentioned this pull request Nov 20, 2020

[Ingest Node Pipelines] Add support for uri_parts processor elastic/kibana#83915

Closed

test case for blank password in user info

c128522

danhermann added the >feature label Nov 20, 2020

danhermann merged commit ccf4b63 into elastic:master Nov 20, 2020

danhermann deleted the url_parts_processor branch November 20, 2020 14:43

danhermann added the backport pending label Nov 20, 2020

danhermann mentioned this pull request Dec 1, 2020

[DOCS] URI parts processor #65695

Merged

jrodewig mentioned this pull request Dec 2, 2020

URL parse ingest processor #65772

Closed

danhermann added a commit to danhermann/elasticsearch that referenced this pull request Dec 10, 2020

URI parts ingest processor (elastic#65150)

7cd146b

danhermann mentioned this pull request Dec 10, 2020

[7.x] URI parts ingest processor #66191

Merged

danhermann removed the backport pending label Dec 10, 2020

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

This was referenced Mar 12, 2021

[7.x] [DOCS] URI parts processor (#65695) #70352

Merged

[7.11] [DOCS] URI parts processor (#65695) #70353

Merged

[7.12] [DOCS] URI parts processor (#65695) #70354

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Conversation

danhermann commented Nov 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Nov 17, 2020

Uh oh!

danhermann commented Nov 17, 2020

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

andrewkroh Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danhermann Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

andrewkroh Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

ebeahan Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

webmat commented Nov 18, 2020

Uh oh!

andrewkroh commented Nov 18, 2020

Uh oh!

danhermann commented Nov 18, 2020

Uh oh!

danhermann commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leehinman commented Nov 18, 2020

Uh oh!

webmat commented Nov 18, 2020

Uh oh!

danhermann commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danhermann commented Nov 19, 2020

Uh oh!

andreidan left a comment

Choose a reason for hiding this comment

Uh oh!

andreidan Nov 20, 2020

Choose a reason for hiding this comment

Uh oh!

danhermann Nov 20, 2020

Choose a reason for hiding this comment

Uh oh!

danhermann Nov 20, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth commented Nov 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

danhermann commented Nov 17, 2020 •

edited

Loading

andrewkroh Nov 18, 2020 •

edited

Loading

danhermann commented Nov 18, 2020 •

edited

Loading

danhermann commented Nov 18, 2020 •

edited

Loading