Skip to content

Change text fields to keyword for Metricbeat #10318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ruflin merged 6 commits into from
Jan 29, 2019
Merged

Change text fields to keyword for Metricbeat #10318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c5a65ef
Change text fields to keyword
ruflin Jan 24, 2019
e4edad5
change children to array and update data.json files
ruflin Jan 25, 2019
f855e19
change leftover field of children and update changelog
ruflin Jan 28, 2019
a19d4ce
add missing changelog entry
ruflin Jan 29, 2019
d9a7c5d
update fields
ruflin Jan 29, 2019
37d12a4
fix changelog
ruflin Jan 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Docker and Kubernetes labels/annotations will be "dedoted" by default. {pull}10338[10338]

*Auditbeat*

- Rename `process.exe` to `process.executable` in auditd module to align with ECS. {pull}9949[9949]
- Rename `process.cwd` to `process.working_directory` in auditd module to align with ECS. {pull}10195[10195]
- Change data type of `process.pid` and `process.ppid` to number in JSON output
Expand Down Expand Up @@ -80,6 +81,15 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Rename http.request.body field to http.request.body.content. {pull}10315[10315]
- Adjust php_fpm.process metricset fields to ECS. {pull}10366[10366]
- Adjust mongodb.status metricset to to ECS. {pull}10368[10368]
- Change the following fields from type text to keyword: {pull}10318[10318]
- ceph.osd_df.name
- ceph.osd_tree.name
Comment thread
ruflin marked this conversation as resolved.
Outdated
- ceph.osd_tree.children
- kafka.consumergroup.meta
- kibana.stats.name
- mongodb.metrics.replication.executor.network_interface
- php_fpm.process.request_uri
- php_fpm.process.script

*Packetbeat*

Expand Down
14 changes: 7 additions & 7 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1587,7 +1587,7 @@ osd node id
*`ceph.osd_df.name`*::
+
--
type: text
type: keyword

osd node name

Expand Down Expand Up @@ -1682,7 +1682,7 @@ osd or bucket node id
*`ceph.osd_tree.name`*::
+
--
type: text
type: keyword

osd or bucket node name

Expand Down Expand Up @@ -1712,7 +1712,7 @@ osd or bucket node typeID
*`ceph.osd_tree.children`*::
+
--
type: text
type: keyword

bucket children list, separated by comma.

Expand Down Expand Up @@ -10779,7 +10779,7 @@ consumer offset into partition being read
*`kafka.consumergroup.meta`*::
+
--
type: text
type: keyword

custom consumer meta data string

Expand Down Expand Up @@ -11028,7 +11028,7 @@ Kibana instance UUID
*`kibana.stats.name`*::
+
--
type: text
type: keyword
Copy link
Copy Markdown
Contributor

@ycombinator ycombinator Jan 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to be text. This value comes from the server.name setting in kibana.yml. The setting is meant for display purposes, according to it's documentation:

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

So it could be a string like "Marketing's Fantastic Analytics UI". In that case I can see the benefit of letting ES analyze the string. Thoughts?

Copy link
Copy Markdown
Contributor

@webmat webmat Jan 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If searching for specific words (e.g. "marketing", to follow your example), then we simply need server.name.text for these searches.

I agree it's really useful in situations where these resources are not always well tagged & so on.

But the canonical field -- server.name -- should really be keyword. Defaulting to keyword all the time, then only adding text as a multi-field when needed ensures this progression over time (adding text) is never a breaking change.

Copy link
Copy Markdown
Contributor

@ycombinator ycombinator Jan 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds good to me but I have a tangential question:

IIRC if the server.name field is mapped as keyword in ES, it wouldn't also get a server.name.text multi-field automatically. Wouldn't we need to explicitly specify that somehow in the fields.yml so ES knows to create the multi-field? Or do all keyword fields in fields.yml automatically get a multi-field mapping added to them in the template via dynamic templates or something?

Copy link
Copy Markdown
Contributor

@webmat webmat Jan 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another point: having server.name as keyword will enable users to aggregate per kibana server. It's the reason in ECS, we've decided to push for keyword across the board, by default

Copy link
Copy Markdown
Contributor

@webmat webmat Jan 24, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha I missed your answer when posting my previous message.

The template will have to create the multi-field explicitly for the .text.

Going for keyword across the board is also a small performance gain, so the default is kw only. Then we add text strategically, where it makes sense.

I'm not sure the exact incantation to make the multi-field in Beats fields.yml.

Copy link
Copy Markdown
Contributor

@ycombinator ycombinator Jan 24, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, I understand now. Thanks. I'm good with changing this to keyword for now and adding a text multi-field later if we need.


Kibana instance name

Expand Down Expand Up @@ -14736,7 +14736,7 @@ type: boolean
*`mongodb.metrics.replication.executor.network_interface`*::
+
--
type: text
type: keyword

--

Expand Down Expand Up @@ -18967,7 +18967,7 @@ The user (PHP_AUTH_USER) (or - if not set) (for the current request)
*`php_fpm.process.script`*::
+
--
type: text
type: keyword

The main script called (or - if not set) (for the current request)

Expand Down
22 changes: 14 additions & 8 deletions metricbeat/module/ceph/cluster_disk/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,32 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"ceph": {
"cluster_disk": {
"available": {
"bytes": 50847313920
"bytes": 0
},
"total": {
"bytes": 67371577344
"bytes": 0
},
"used": {
"bytes": 13071564800
"bytes": 0
}
}
},
"event": {
"dataset": "ceph.cluster_disk",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"host": "ceph:5000",
"module": "ceph",
"name": "cluster_disk",
"rtt": 115
"name": "cluster_disk"
},
"service": {
"address": "127.0.0.1:5000",
"type": "ceph"
}
}
18 changes: 12 additions & 6 deletions metricbeat/module/ceph/cluster_health/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"ceph": {
"cluster_health": {
"overall_status": "HEALTH_WARN",
"overall_status": "HEALTH_OK",
"timechecks": {
"epoch": 3,
"round": {
Expand All @@ -16,10 +16,16 @@
}
}
},
"event": {
"dataset": "ceph.cluster_health",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"host": "ceph:5000",
"module": "ceph",
"name": "cluster_health",
"rtt": 115
"name": "cluster_health"
},
"service": {
"address": "127.0.0.1:5000",
"type": "ceph"
}
}
32 changes: 19 additions & 13 deletions metricbeat/module/ceph/cluster_status/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
Expand All @@ -17,33 +17,39 @@
"total": 0
},
"osd": {
"epoch": 18,
"epoch": 3,
"full": false,
"in_osd_count": 1,
"in_osd_count": 0,
"nearfull": false,
"osd_count": 1,
"remapped_pg_count": 0,
"up_osd_count": 1
"up_osd_count": 0
},
"pg": {
"avail_bytes": 50847313920,
"data_bytes": 3656,
"total_bytes": 67371577344,
"used_bytes": 13071564800
"avail_bytes": 0,
"data_bytes": 0,
"total_bytes": 0,
"used_bytes": 0
},
"traffic": {
"read_bytes": 0,
"read_op_per_sec": 0,
"write_bytes": 0,
"write_op_per_sec": 0
},
"version": 32
"version": 4
}
},
"event": {
"dataset": "ceph.cluster_status",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"host": "ceph:5000",
"module": "ceph",
"name": "cluster_status",
"rtt": 115
"name": "cluster_status"
},
"service": {
"address": "127.0.0.1:5000",
"type": "ceph"
}
}
2 changes: 1 addition & 1 deletion metricbeat/module/ceph/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 19 additions & 13 deletions metricbeat/module/ceph/monitor_health/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"ceph": {
"monitor_health": {
"available": {
"kb": 49653688,
"pct": 75
"kb": 46814916,
"pct": 76
},
"health": "HEALTH_OK",
"last_updated": "2017-12-07T07:26:33.344939Z",
"name": "93c038136d94",
"last_updated": "2019-01-25T12:37:24.921587Z",
"name": "f0e2841bb524",
"store_stats": {
"last_updated": "0.000000",
"log": {
Expand All @@ -22,24 +22,30 @@
"bytes": 65552
},
"sst": {
"bytes": 726
"bytes": 1087
},
"total": {
"bytes": 2097894
"bytes": 2098255
}
},
"total": {
"kb": 65792556
"kb": 61255492
},
"used": {
"kb": 12767092
"kb": 11299252
}
}
},
"event": {
"dataset": "ceph.monitor_health",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"host": "ceph:5000",
"module": "ceph",
"name": "monitor_health",
"rtt": 115
"name": "monitor_health"
},
"service": {
"address": "127.0.0.1:5000",
"type": "ceph"
}
}
25 changes: 15 additions & 10 deletions metricbeat/module/ceph/osd_df/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,30 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"ceph": {
"osd_df": {
"available.byte": 49655580,
"available.byte": 0,
"device_class": "",
"id": 0,
"name": "osd.0",
"pg_num": 120,
"total.byte": 65792556,
"used.byte": 12765200,
"used.pct": 0.1940219498388237
"pg_num": 0,
"total.byte": 0,
"used.byte": 0
}
},
"event": {
"dataset": "ceph.osd_df",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"host": "ceph:5000",
"module": "ceph",
"name": "osd_df",
"rtt": 115
"name": "osd_df"
},
"service": {
"address": "127.0.0.1:5000",
"type": "ceph"
}
}
2 changes: 1 addition & 1 deletion metricbeat/module/ceph/osd_df/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
description: >
osd node id
- name: name
type: text
type: keyword
description: >
osd node name
- name: device_class
Expand Down
20 changes: 14 additions & 6 deletions metricbeat/module/ceph/osd_tree/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,31 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"ceph": {
"osd_tree": {
"children": "-2",
"children": [
"-2"
],
"father": "",
"id": -1,
"name": "default",
"type": "root",
"type_id": 10
}
},
"event": {
"dataset": "ceph.osd_tree",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"host": "ceph:5000",
"module": "ceph",
"name": "osd_tree",
"rtt": 115
"name": "osd_tree"
},
"service": {
"address": "127.0.0.1:5000",
"type": "ceph"
}
}
Loading