Skip to content

[Automatic Import] Do not remove message field for unstructured logs#193678

Merged
bhapas merged 2 commits intoelastic:mainfrom
bhapas:automatic_import_template_fix
Sep 23, 2024
Merged

[Automatic Import] Do not remove message field for unstructured logs#193678
bhapas merged 2 commits intoelastic:mainfrom
bhapas:automatic_import_template_fix

Conversation

@bhapas
Copy link
Copy Markdown
Contributor

@bhapas bhapas commented Sep 23, 2024 •
edited by kibanamachine
Loading

Summary

The pipeline template by default removes the message field. But in case of unstructured logs the grok processor adds the unstructured message part into a message field and it should be part of the final pipeline.

Hence, the remove processor is not executed in case the log_format is unstructured.

Before this PR

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "hostname": "mymachine",
          "process": "su",
          "ecs": {
            "version": "8.11.0"
          },
          "related": {
            "hosts": [
              "mymachine"
            ]
          },
          "priority": "34",
          "event": {
            "category": [
              "authentication",
              "process"
            ],
            "type": [
              "start"
            ]
          },
          "timestamp": "Oct 11 00:14:05"
        },
        "_ingest": {
          "timestamp": "2024-09-20T16:10:01.656597092Z"
        }
      }
    }
  ]
}

After this PR

image

@bhapas bhapas added bug Fixes for quality problems that affect the customer experience backport:prev-major Team:Security-Scalability Security Integrations Scalability Team Feature:AutomaticImport labels Sep 23, 2024
@bhapas bhapas self-assigned this Sep 23, 2024
@bhapas bhapas requested a review from a team as a code owner September 23, 2024 08:33
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 23, 2024

Pinging @elastic/security-scalability (Team:Security-Scalability)

@kibana-ci
Copy link
Copy Markdown

kibana-ci commented Sep 23, 2024

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @bhapas

@bhapas bhapas merged commit 2106df3 into elastic:main Sep 23, 2024
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 23, 2024
@bhapas
[Automatic Import] Do not remove message field for unstructured logs (e…
adf5fba 
…lastic#193678)

(cherry picked from commit 2106df3)
@kibanamachine kibanamachine mentioned this pull request Sep 23, 2024
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 23, 2024
@bhapas
[Automatic Import] Do not remove message field for unstructured logs (e…
e65b574 
…lastic#193678)

(cherry picked from commit 2106df3)
@kibanamachine kibanamachine mentioned this pull request Sep 23, 2024
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Sep 23, 2024

💚 All backports created successfully

Status Branch Result
✅ 8.15
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Sep 23, 2024
@kibanamachine @bhapas
[8.x] [Automatic Import] Do not remove message field for unstructured…
b3311a9 
… logs (#193678) (#193704)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Automatic Import] Do not remove message field for unstructured logs
(#193678)](#193678)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-23T11:08:59Z","message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Do not remove message field for unstructured
logs","number":193678,"url":"https://github.com/elastic/kibana/pull/193678","mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193678","number":193678,"mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
weizijun added a commit to weizijun/kibana that referenced this pull request Sep 23, 2024
@weizijun
Merge branch 'main' into add-alibaba-inference
2100828 
* main: (176 commits)
  [ML][Rules] Fixes deletion in Check interval input for anomaly detection rule (elastic#193420)
  Bump maximum supported package spec version to 3.2 (elastic#193574)
  [ES|QL] new pattern for `SORT` autocomplete (elastic#193595)
  [Inventory][ECO] Entities page search bar (elastic#193546)
  [Synthetics] Remove extra overview route (elastic#192449)
  [Obs Alerts table] Fix error on clicking alert reason message (elastic#193693)
  [Migrations] Remove tests that are not applicable in 9.x (elastic#193699)
  [EDR Workflows] Set Agent Tamper Protection to false on policy unassignment (elastic#193017)
  [Inventory][ECO] Enable elastic entity model from inventory (elastic#193557)
  [EDR Workflows] The host isolation exception tab is hidden on the basic license if no artifacts (elastic#192562)
  [Entity Analytics] Ensuring definition transforms are managed (elastic#193408)
  [Automatic Import] Do not remove message field for unstructured logs (elastic#193678)
  [Fleet] Add missing permissions for connector package (elastic#193573)
  [Fleet] using @kbn/config-schema part 2 (outputs and other apis)  (elastic#193326)
  [Migrations] Provide testing archives + tooling for migrations integration tests (elastic#193328)
  [ES|QL] Renames the textbased editor to esql editor (elastic#193521)
  [ES|QL] Update function metadata (elastic#193662)
  [Security Solution][Entity Analytics] Scoping the entity store to spaces (elastic#193303)
  [Docs] Update Sharing docs (elastic#190318)
  [ML] AIOps: Move Log Rate Analysis results callout to help popover. (elastic#192243)
  ...

# Conflicts:
#	x-pack/plugins/search_inference_endpoints/public/components/all_inference_endpoints/render_table_columns/render_endpoint/endpoint_info.test.tsx
#	x-pack/plugins/search_inference_endpoints/public/components/all_inference_endpoints/render_table_columns/render_endpoint/endpoint_info.tsx
kibanamachine added a commit that referenced this pull request Sep 23, 2024
@kibanamachine @bhapas @elasticmachine
[8.15] [Automatic Import] Do not remove message field for unstructure…
2d82be0 
…d logs (#193678) (#193702)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Automatic Import] Do not remove message field for unstructured logs
(#193678)](#193678)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-23T11:08:59Z","message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Do not remove message field for unstructured
logs","number":193678,"url":"https://github.com/elastic/kibana/pull/193678","mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193678","number":193678,"mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@P1llus P1llus P1llus approved these changes

Assignees

@bhapas bhapas

Labels

bug Fixes for quality problems that affect the customer experience Feature:AutomaticImport release_note:fix Team:Security-Scalability Security Integrations Scalability Team v8.15.2 v8.16.0 v9.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bhapas @elasticmachine @kibana-ci @kibanamachine @P1llus