Suggest fields as keywords by ruflin · Pull Request #10518 · elastic/beats

ruflin · 2019-02-04T12:00:52Z

Most field in ECS are keyword and made optional .text if needed. Search for type: text revealed a few fields which we should potentially convert.

journalbeat/_meta/fields.common.yml

ruflin · 2019-02-04T12:01:44Z

packetbeat/_meta/fields.common.yml

        something like `GET /users/_search?name=test`. For MySQL, it is
        something like `SELECT id from users where name=test`.

+    #keyword(s)?


@andrewkroh Could you take a look at the fields below. Potentially these were discussed in previous PR's.

Good catch. These should both be set to keyword, to enable aggregations and fast indexing.

If full text search is deemed necessary by us or customers, text should be added as a multi-field.

ruflin · 2019-02-04T12:01:58Z

x-pack/auditbeat/module/system/user/_meta/fields.yml

@@ -24,6 +24,7 @@
      type: keyword


@cwurm you might know more here.

I think you mean user.user_information? I don't have a strong opinion I think. It's filled with strings like systemd Network Management,,,, Gnats Bug-Reporting System (admin). I don't expect too many users searching directly on it rather than something more unique like user.name.

yes, not sure how the comment added up here :-(

Is the number of different strings limited? Do you expect users to group/aggregate/filter on these?

When in doubt, use keyword. text should be added as a multi-field if deemed necessary.

text should be added as a multi-field if deemed necessary.

Can Beats generate multi-fields?

yes:

- name: thread type: keyword description: > Information about the running thread where the log originate. multi_fields: - name: text type: text

ruflin · 2019-02-04T12:02:21Z

auditbeat/_meta/fields.common.yml

      description: Set if the file has the `setgid` bit set. Omitted otherwise.

    - name: origin
+      # Should this be keyword?


@webmat @cwurm Thoughts? Also check below.

file.origin is only filled on macOS. It can be a URL or a computer name, maybe more. We treat URLs as keywords in other places, so I think keyword would make sense here, too.

/cc @adriansr who wrote the code for this

This should be a keyword

@adriansr Could you open a PR for this change?

Strings should always be keyword. text should be added as a multi-field if deemed necessary.

When in doubt, keyword

andrewkroh · 2019-02-04T23:20:47Z

auditbeat/module/auditd/_meta/fields.yml

          type: keyword
          description: This is the path associated with a unix socket.
    - name: messages
+      # As it's the raw message should it be keyword. Should it even be indexed?


It's rarely used (it requires a config change to enable), but when it is used it's really useful to have as text so that you can search it.

Now that we have event.original, I'm thinking this would make sense to put there (even if it is a keyword). Likewise the warnings field below could go in as error.message.

This field should be keyword. If we want to enable full text search, text should be added as a multi-field.

@webmat If it's called message it should be text by default. So far this convention holds in ECS I think.

@andrewkroh I assume your team will tackle these changes?

webmat

Almost all entries should move to keyword, IMO. Only exception was DHCP error message.

webmat · 2019-02-05T05:03:35Z

auditbeat/_meta/fields.common.yml

      description: Set if the file has the `setgid` bit set. Omitted otherwise.

    - name: origin
+      # Should this be keyword?


Strings should always be keyword. text should be added as a multi-field if deemed necessary.

When in doubt, keyword

webmat · 2019-02-05T05:04:55Z

auditbeat/module/auditd/_meta/fields.yml

          type: keyword
          description: This is the path associated with a unix socket.
    - name: messages
+      # As it's the raw message should it be keyword. Should it even be indexed?


This field should be keyword. If we want to enable full text search, text should be added as a multi-field.

filebeat/module/haproxy/_meta/fields.yml

webmat · 2019-02-05T05:10:45Z

packetbeat/_meta/fields.common.yml

        something like `GET /users/_search?name=test`. For MySQL, it is
        something like `SELECT id from users where name=test`.

+    #keyword(s)?


Good catch. These should both be set to keyword, to enable aggregations and fast indexing.

If full text search is deemed necessary by us or customers, text should be added as a multi-field.

webmat · 2019-02-05T05:11:51Z

packetbeat/protos/dhcpv4/_meta/fields.yml

                IP address.  In a server reply (DHCPOFFER), a DHCP server uses
                this option to specify the lease time it is willing to offer.

+            # error.message? or just message?


Yeah I vote for error.message, although if there's no time, not the end of the world.

webmat · 2019-02-05T05:13:24Z

x-pack/auditbeat/module/system/user/_meta/fields.yml

@@ -24,6 +24,7 @@
      type: keyword


When in doubt, use keyword. text should be added as a multi-field if deemed necessary.

ruflin · 2019-02-05T12:45:56Z

Closing this PR as this will not be merged. @cwurm @andrewkroh I assume your team will follow up with these if needed. Even if it's closed, we can keep the discussion going.

andrewkroh · 2019-02-05T16:51:46Z

I created a PR for these changes in #10577.

Suggest fields as keywords

e229e9f

Most field in ECS are keyword and made optional .text if needed. Search for type: text revealed a few fields which we should potentially convert.

ruflin added in progress Pull request is currently in progress. discuss Issue needs further discussion. ecs labels Feb 4, 2019

ruflin requested review from andrewkroh, kvch and webmat February 4, 2019 12:00

ruflin requested review from a team as code owners February 4, 2019 12:00

ruflin commented Feb 4, 2019

View reviewed changes

journalbeat/_meta/fields.common.yml Show resolved Hide resolved

ruflin commented Feb 4, 2019

View reviewed changes

ruflin requested a review from cwurm February 4, 2019 12:02

ruflin commented Feb 4, 2019

View reviewed changes

ruflin mentioned this pull request Feb 4, 2019

Transition Beats to ECS #8655

Closed

andrewkroh reviewed Feb 4, 2019

View reviewed changes

webmat reviewed Feb 5, 2019

View reviewed changes

ruflin added 2 commits February 5, 2019 10:07

remove journald changes

c6f75e8

revert haproxy

2aa1c86

ruflin closed this Feb 5, 2019

Conversation

ruflin commented Feb 4, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cwurm Feb 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin Feb 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webmat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin commented Feb 5, 2019

Uh oh!

andrewkroh commented Feb 5, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cwurm Feb 4, 2019 •

edited

Loading

ruflin Feb 4, 2019 •

edited

Loading