Skip to content

Commit a2c162f

Browse files
tsgexekias
tsg
authored and
exekias
committed
Nginx module: use first not private IP address as remote_ip (elastic#4417)
A common customization to the nginx logs is to add the contents of the X-Forwarded-For header in front of the remote IPs. This typically results in a list of remote IPs. This adds a new field `remote_ip_list` which is an array, and uses a Painless script to automatically select the first non-private IP for the `remote_ip` field, which is the field on which GeoIP is applied. Fixes elastic#4322.
1 parent a79e780 commit a2c162f

8 files changed

Lines changed: 428 additions & 113 deletions

File tree

‎filebeat/docs/fields.asciidoc‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,6 +233,22 @@ type: geo_point
233233
The longitude and latitude.
234234
235235
236+
[float]
237+
=== apache2.access.geoip.region_name
238+
239+
type: keyword
240+
241+
The region name.
242+
243+
244+
[float]
245+
=== apache2.access.geoip.city_name
246+
247+
type: keyword
248+
249+
The city name.
250+
251+
236252
[float]
237253
== error Fields
238254
@@ -959,12 +975,20 @@ Contains fields for the Nginx access logs.
959975
960976
961977
978+
[float]
979+
=== nginx.access.remote_ip_list
980+
981+
type: array
982+
983+
An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like `X-Forwarded-For`. See also the `remote_ip` field.
984+
985+
962986
[float]
963987
=== nginx.access.remote_ip
964988
965989
type: keyword
966990
967-
Client IP address.
991+
Client IP address. The first public IP address from the `remote_ip_list` array. If no public IP addresses are present, this field contains the first private IP address from the `remote_ip_list` array.
968992
969993
970994
[float]
@@ -1147,6 +1171,22 @@ type: geo_point
11471171
The longitude and latitude.
11481172
11491173
1174+
[float]
1175+
=== nginx.access.geoip.region_name
1176+
1177+
type: keyword
1178+
1179+
The region name.
1180+
1181+
1182+
[float]
1183+
=== nginx.access.geoip.city_name
1184+
1185+
type: keyword
1186+
1187+
The city name.
1188+
1189+
11501190
[float]
11511191
== error Fields
11521192

‎filebeat/module/apache2/access/_meta/fields.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,4 +104,12 @@
104104
type: geo_point
105105
description: >
106106
The longitude and latitude.
107+
- name: region_name
108+
type: keyword
109+
description: >
110+
The region name.
111+
- name: city_name
112+
type: keyword
113+
description: >
114+
The city name.
107115

‎filebeat/module/nginx/access/_meta/fields.yml‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,17 @@
33
description: >
44
Contains fields for the Nginx access logs.
55
fields:
6+
- name: remote_ip_list
7+
type: array
8+
description: >
9+
An array of remote IP addresses. It is a list because it is common to include, besides the client
10+
IP address, IP addresses from headers like `X-Forwarded-For`. See also the `remote_ip` field.
611
- name: remote_ip
712
type: keyword
813
description: >
9-
Client IP address.
14+
Client IP address. The first public IP address from the `remote_ip_list` array. If no public IP
15+
addresses are present, this field contains the first private IP address from the `remote_ip_list`
16+
array.
1017
- name: user_name
1118
type: keyword
1219
description: >
@@ -104,4 +111,12 @@
104111
type: geo_point
105112
description: >
106113
The longitude and latitude.
114+
- name: region_name
115+
type: keyword
116+
description: >
117+
The region name.
118+
- name: city_name
119+
type: keyword
120+
description: >
121+
The city name.
107122

‎filebeat/module/nginx/access/ingest/default.json‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,24 @@
44
"grok": {
55
"field": "message",
66
"patterns":[
7-
"%{IPORHOST:nginx.access.remote_ip}(,\\s%{IPORHOST})* - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\""
7+
"\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\""
88
],
9+
"pattern_definitions": {
10+
"IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
11+
},
912
"ignore_missing": true
1013
}
11-
},{
14+
}, {
15+
"split": {
16+
"field": "nginx.access.remote_ip_list",
17+
"separator": "\"?,?\\s+"
18+
}
19+
}, {
20+
"script": {
21+
"lang": "painless",
22+
"inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { return true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return false; } catch (Exception e) { return false; } } def found = false; for (def item : ctx.nginx.access.remote_ip_list) { if (!isPrivate(item)) { ctx.nginx.access.remote_ip = item; found = true; break; } } if (!found) { ctx.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
23+
}
24+
}, {
1225
"remove":{
1326
"field": "message"
1427
}

‎filebeat/module/nginx/access/test/test.log‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,6 @@
11
10.0.0.2, 10.0.0.1, 127.0.0.1 - - [07/Dec/2016:11:05:07 +0100] "GET /ocelot HTTP/1.1" 200 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0"
22
172.17.0.1 - - [29/May/2017:19:02:48 +0000] "GET /stringpatch HTTP/1.1" 404 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" "-"
3+
10.0.0.2, 10.0.0.1, 85.181.35.98 - - [07/Dec/2016:11:05:07 +0100] "GET /ocelot HTTP/1.1" 200 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0"
4+
85.181.35.98 - - [07/Dec/2016:11:05:07 +0100] "GET /ocelot HTTP/1.1" 200 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0"
5+
"10.5.102.222, 199.96.1.1, 204.246.1.1" 10.2.1.185 - - [22/Jan/2016:13:18:29 +0000] "GET /assets/xxxx?q=100 HTTP/1.1" 200 25507 "-" "Amazon CloudFront"
6+
2a03:0000:10ff:f00f:0000:0000:0:8000, 10.225.192.17 10.2.2.121 - - [30/Dec/2016:06:47:09 +0000] "GET /test.html HTTP/1.1" 404 8571 "-" "Mozilla/5.0 (compatible; Facebot 1.0; https://developers.facebook.com/docs/sharing/webmasters/crawler)"

0 commit comments

Comments
 (0)