Skip to content

Remove event.dataset from logp ecs attributes#27433

Merged
michel-laterman merged 1 commit intoelastic:masterfrom
michel-laterman:remove-libbeat-dataset
Aug 17, 2021
Merged

Remove event.dataset from logp ecs attributes#27433
michel-laterman merged 1 commit intoelastic:masterfrom
michel-laterman:remove-libbeat-dataset

Conversation

@michel-laterman
Copy link
Copy Markdown
Contributor

@michel-laterman michel-laterman commented Aug 17, 2021

What does this PR do?

Remove the event.dataset ECS attribute from libbeat.logp.

Why is it important?

event.dataset key is appearing twice in documents sent by filebeat running under the elastic-agent. Once is the correct elastic_agent.X value, the other is an array that contains the log name.

Checklist

  • My code follows the style guidelines of this project
  • [] I have commented my code, particularly in hard-to-understand areas
  • [] I have made corresponding changes to the documentation
  • [] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Build + run agent and ensure that log entries only have a single event.dataset key set to elastic_agent.X

Related issues

@michel-laterman michel-laterman added bug Team:Elastic-Agent Label for the Agent team v7.15.0 backport-v7.15.0 Automated backport with mergify labels Aug 17, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 17, 2021

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 17, 2021
blakerouse
blakerouse approved these changes Aug 17, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I think its best that its not added, based on the comments from the issue.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 17, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-17T18:06:32.489+0000

  • Duration: 166 min 47 sec

  • Commit: 97840d3

Test stats 🧪

Test Results
Failed 0
Passed 53350
Skipped 5320
Total 58670

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 53350
Skipped 5320
Total 58670

@michel-laterman michel-laterman merged commit d9fbcef into elastic:master Aug 17, 2021
@michel-laterman michel-laterman deleted the remove-libbeat-dataset branch August 17, 2021 20:58
mergify bot pushed a commit that referenced this pull request Aug 17, 2021
@michel-laterman
Remove event.dataset from logp ecs attributes (#27433)
09bc7ee 
(cherry picked from commit d9fbcef)
@mergify mergify bot mentioned this pull request Aug 17, 2021
Merged
michel-laterman added a commit that referenced this pull request Aug 18, 2021
@mergify @michel-laterman
Remove event.dataset from logp ecs attributes (#27433) (#27437)
6a38878 
(cherry picked from commit d9fbcef)

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
@felixbarny
Copy link
Copy Markdown
Member

felixbarny commented Aug 18, 2021

LGTM.

Longer-term I think the dataset could be a part of a beat's config in ${beat.name}.yaml. That would then override any defaults set in the Filebeat config via a generic ECS parsing pipeline that gives precedence to the fields set in the log line itself. See also elastic/integrations#1454.

@michel-laterman michel-laterman added the backport-v7.14.0 Automated backport with mergify label Aug 19, 2021
mergify bot pushed a commit that referenced this pull request Aug 19, 2021
@michel-laterman
Remove event.dataset from logp ecs attributes (#27433)
09349fd 
(cherry picked from commit d9fbcef)
@mergify mergify bot mentioned this pull request Aug 19, 2021
Merged
mergify bot pushed a commit that referenced this pull request Aug 19, 2021
@michel-laterman
Remove event.dataset from logp ecs attributes (#27433)
40ff706 
(cherry picked from commit d9fbcef)
@mergify mergify bot mentioned this pull request Aug 19, 2021
Closed
michel-laterman added a commit that referenced this pull request Aug 19, 2021
@mergify @michel-laterman
[7.14](backport #27433) Remove event.dataset from logp ecs attributes (…
d9f786d 
…#27506)

* Remove event.dataset from logp ecs attributes (#27433)

(cherry picked from commit d9fbcef)

* Remove extra CHANGELOG entry

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
@amolnater-qasource
Copy link
Copy Markdown

amolnater-qasource commented Aug 25, 2021

Hi @EricDavisX
We have revalidated this issue on 7.15.0 BC-2 Kibana Cloud environment and found it fixed now.

  • event.datasets field shows datasets with logs under Logs tab.
  • The dataset selected from Datasets dropdown shows selected dataset under event.datasets field.

Build details:

Build: 43742
Commit: 113d5f2a745ab9be7d567d13ff906dfd0f5b7a9a

Screenshot:
15 - Copy

Thanks
QAS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blakerouse blakerouse blakerouse approved these changes

@ruflin ruflin Awaiting requested review from ruflin

@felixbarny felixbarny Awaiting requested review from felixbarny

Assignees

No one assigned

Labels

backport-v7.14.0 Automated backport with mergify backport-v7.15.0 Automated backport with mergify bug Team:Elastic-Agent Label for the Agent team v7.15.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Fleet]: event.datasets field is blank with logs under Logs tab.

5 participants

@michel-laterman @elasticmachine @felixbarny @amolnater-qasource @blakerouse