Skip to content

Change renaming @timestamp to copying values in Ingest pipelines#29425

Merged
kvch merged 2 commits intoelastic:masterfrom
kvch:fix-filebeat-keep-timestamp-in-modules
Dec 17, 2021
Merged

Change renaming @timestamp to copying values in Ingest pipelines#29425
kvch merged 2 commits intoelastic:masterfrom
kvch:fix-filebeat-keep-timestamp-in-modules

Conversation

@kvch
Copy link
Copy Markdown
Contributor

@kvch kvch commented Dec 14, 2021

What does this PR do?

This PR changes renaming @timestamp fields in Ingest pipelines to copying them.

Why is it important?

Filebeat sends events to data streams now. However, data streams require @timestamp fields in every event. Some Ingest pipelines either get rid of or on failure @timestamp is missing from events. This is a problem and the event cannot be indexed.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 14, 2021
@botelastic
Copy link
Copy Markdown

botelastic bot commented Dec 14, 2021

This pull request doesn't have a Team:<team> label.

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Dec 14, 2021

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Dec 14, 2021
andrewkroh
andrewkroh reviewed Dec 14, 2021
View reviewed changes
x-pack/filebeat/module/checkpoint/firewall/ingest/pipeline.yml Outdated
- set:
copy_from: "@timestamp"
field: "event.created"
ignore_empty_value: true
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh Dec 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore_empty_value only has an effect when using value. With copy_from you'd need an if to achieve a similar behavior. Or you can ignore_failure to continue if @timestamp does not exist.

Copy link
Copy Markdown
Contributor Author

@kvch kvch Dec 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After sleeping on it, I realized that we should not ignore empty values. If @timestamp is missing that is an error.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 14, 2021 •
edited by jenkins-beats-ci bot
Loading

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Start Time: 2021-12-15T13:40:29.668+0000

  • Duration: 363 min 49 sec

  • Commit: 4ed2e4b

Test stats 🧪

Test Results
Failed 0
Passed 9616
Skipped 1275
Total 10891

Steps errors 11

Expand to view the steps failures

Show only the first 10 steps failures

Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details here
  • Description: Required context class hudson.Launcher is missing Perhaps you forgot to surround the code with a ste
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details here
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details here
Verify if file exists in workspace
  • Took 0 min 0 sec . View more details here
  • Description: /var/lib/jenkins/workspace/PR-29425-2-cde30fc8-e681-43a7-a589-dbbb15e68078/src/github.com/elastic/beats@tmp/google-cloud-credentials.json
Determine current directory
  • Took 0 min 0 sec . View more details here
  • Description: [2021-12-15T19:42:50.524Z] beats-ci-immutable-ubuntu-1804-aarch64-1639580570486106765 was marked off
Determine current directory
  • Took 0 min 0 sec . View more details here
  • Description: [2021-12-15T19:43:00.597Z] beats-ci-immutable-ubuntu-1804-aarch64-1639580570486106765 was marked off
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details here
  • Description: Required context class hudson.Launcher is missing Perhaps you forgot to surround the code with a ste
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details here
  • Description: Required context class hudson.Launcher is missing Perhaps you forgot to surround the code with a ste
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details here
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error 'java.io.IOException: Unable to create live FilePath for beats-ci-immutable-ubuntu-1804-aarch64-1639580570486106765'

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@kvch kvch merged commit b52ad7f into elastic:master Dec 17, 2021
@kvch kvch added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.0.0 Automated backport with mergify and removed backport-7.17 Automated backport to the 7.17 branch with mergify labels Dec 17, 2021
mergify bot pushed a commit that referenced this pull request Dec 17, 2021
@kvch
Change renaming @timestamp to copying values in Ingest pipelines (#…
67b6c65 
…29425)

(cherry picked from commit b52ad7f)
@mergify mergify bot mentioned this pull request Dec 17, 2021
Merged
mtojek pushed a commit that referenced this pull request Dec 21, 2021
@mergify @kvch
Change renaming @timestamp to copying values in Ingest pipelines (#…
0f4ad0a 
…29425) (#29502)

(cherry picked from commit b52ad7f)

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
@kvch kvch mentioned this pull request Jan 26, 2022
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

No one assigned

Labels

backport-skip Skip notification from the automated backport with mergify backport-v8.0.0 Automated backport with mergify needs_team Indicates that the issue/PR needs a Team:* label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kvch @elasticmachine @andrewkroh