Limit usage of the deprecated `log` and `container` inputs by rdner · Pull Request #42295 · elastic/beats

rdner · 2025-01-13T11:25:26Z

The log input has been deprecated since 7.16, it's time to disable it. The container input is just an alias/preset for the log input.

They still can be used under the following circumstances:

When running under Elastic Agent (by integrations)
When running as a part of a module
When allow_deprecated_use is set as a part of the the input configuration

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Users who are still using the deprecated log and container inputs in their hand-crafted configuration will see the error message on Filebeat's startup.

How to test this PR locally

Running log input in the user-crafted configuration fails

filebeat.yml

filebeat.inputs:
  - type: log
    paths:
      - "/logs/log*.log"
      - "/logs/log*.json"
path.data: "/data"
logging:
  level: debug
output.console:
  enabled: true

This configuration produces the following error message:

Similar for the container input:

filebeat.inputs:
  - type: container
    paths:
      - "/logs/log*.log"
      - "/logs/log*.json"
path.data: "/data"
logging:
  level: debug
output.console:
  enabled: true

Setting `allow_deprecated_use: true` enables the `log` input

filebeat.yml

filebeat.inputs:
  - type: log
    allow_deprecated_use: true
    paths:
      - "/logs/log*.log"
      - "/logs/log*.json"
path.data: "/data"
logging:
  level: debug
output.console:
  enabled: true

filebeat.inputs:
  - type: container
    allow_deprecated_use: true
    paths:
      - "/logs/log*.log"
      - "/logs/log*.json"
path.data: "/data"
logging:
  level: debug
output.console:
  enabled: true

These configurations run without the error.

`log` input still runs as a part of a module

For example, the postgresql module is using the log input:

beats/filebeat/module/postgresql/log/config/log.yml

Line 1 in ef3bd69

type: log

If we try to start this module via the configuration, Filebeat produces no error:

filebeat.yml

filebeat.modules:
  - module: postgresql
    log:
      enabled: true
path.data: "/data"
logging:
  level: debug
output.console:
  enabled: true

`log` input still runs as a part of an integration (Elastic Agent)

I ran the agent packaging with the Beats being on this branch:

DEV=true EXTERNAL=false SNAPSHOT=true PLATFORMS=darwin/arm64 PACKAGES=tar.gz mage -v package

Verified that the packaged agentbeat contains the code from this branch:

filebeat.yml

filebeat.inputs:
  - type: log
    allow_deprecated_use: true
    paths:
      - "/logs/log*.log"
      - "/logs/log*.json"
path.data: "/data"
logging:
  level: debug
output.console:
  enabled: true

#!/bin/bash

set -e

AGENT_ROOT=~/Projects/elastic-agent/build/distributions/elastic-agent-9.0.0-SNAPSHOT-darwin-aarch64/data/elastic-agent-c7c5ba/components

$AGENT_ROOT/agentbeat filebeat run -e --path.config "/deprecate_log/config/" #2> >(tee output.json | jq .)

and it failed as expected:

I use "Custom Logs" integration for the next test because it's based on the log input:

https://github.com/elastic/integrations/blob/a957144c41043e28e7effa343d4619f00f4deef3/packages/log/manifest.yml#L19

I created a cloud deployment and enrolled my custom agent to the Fleet with the "Custom Logs" integration.

./elastic-agent status --output json

{
    "info": {
        "id": "bbc5aff6-81db-421c-b674-5dcdf6e67d16",
        "version": "9.0.0",
        "commit": "c7c5ba4132605f9bdef4b1bd63ab6337fef4abba",
        "build_time": "2025-01-17 12:36:46 +0000 UTC",
        "snapshot": true,
        "pid": 85790,
        "unprivileged": false,
        "is_managed": true
    },
    "state": 2,
    "message": "Running",
    "components": [
        {
            "id": "log-default",
            "name": "log",
            "state": 2,
            "message": "Healthy: communicating with pid '85831'",
            "units": [
                {
                    "unit_id": "log-default",
                    "unit_type": 1,
                    "state": 2,
                    "message": "Healthy"
                },
                {
                    "unit_id": "log-default-logfile-logs-e9751ac9-bb83-4573-a41a-3824f423aa37",
                    "unit_type": 0,
                    "state": 2,
                    "message": "Healthy",
                    "payload": {
                        "streams": {
                            "logfile-log.logs-e9751ac9-bb83-4573-a41a-3824f423aa37": {
                                "error": "",
                                "status": "HEALTHY"
                            }
                        }
                    }
                }
            ],
            "version_info": {
                "name": "beat-v2-client",
                "meta": {
                    "build_time": "2025-01-17 12:32:44 +0000 UTC",
                    "commit": "265dda891e5bc6f9b7dbe7e417917ab2ba11d4e7"
                }
            }
        }
    ],
    "FleetState": 2,
    "FleetMessage": "Connected"
}

I saw in the logs that the log input was running

{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:38.405Z",
  "message": "Run input",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0",
  "log.logger": "input",
  "log.origin": {
    "file.line": 149,
    "file.name": "input/input.go",
    "function": "github.com/elastic/beats/v7/filebeat/input.(*Runner).Run"
  }
}
{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:38.405Z",
  "message": "Start next scan",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "log.logger": "input",
  "log.origin": {
    "file.line": 246,
    "file.name": "log/input.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run"
  },
  "service.name": "filebeat",
  "input_id": "fd29f73b-9e51-4c3e-8133-1d7a6f807732",
  "ecs.version": "1.6.0"
}
{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:38.418Z",
  "message": "Check file for harvesting: /Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "input_id": "fd29f73b-9e51-4c3e-8133-1d7a6f807732",
  "ecs.version": "1.6.0",
  "log.logger": "input",
  "log.origin": {
    "file.line": 494,
    "file.name": "log/input.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/log.getFileState"
  },
  "service.name": "filebeat"
}
{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:38.418Z",
  "message": "Update existing file for harvesting: /Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log, offset: 0",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "state_id": "native::210784429-16777233",
  "os_id": "210784429-16777233",
  "source_file": "/Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "old_finished": false,
  "ecs.version": "1.6.0",
  "input_id": "fd29f73b-9e51-4c3e-8133-1d7a6f807732",
  "log.logger": "input",
  "log.origin": {
    "file.line": 593,
    "file.name": "log/input.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile"
  },
  "old_source": "/Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "service.name": "filebeat",
  "finished": false,
  "old_os_id": "210784429-16777233"
}
{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:38.418Z",
  "message": "Harvester for file is still running: /Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "service.name": "filebeat",
  "input_id": "fd29f73b-9e51-4c3e-8133-1d7a6f807732",
  "os_id": "210784429-16777233",
  "source_file": "/Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "state_id": "native::210784429-16777233",
  "old_finished": false,
  "finished": false,
  "old_source": "/Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "ecs.version": "1.6.0",
  "log.logger": "input",
  "log.origin": {
    "file.line": 648,
    "file.name": "log/input.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile"
  },
  "old_os_id": "210784429-16777233"
}
{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:38.418Z",
  "message": "input states cleaned up. Before: 1, After: 1, Pending: 0",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "log.logger": "input",
  "log.origin": {
    "file.line": 310,
    "file.name": "log/input.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates"
  },
  "service.name": "filebeat",
  "input_id": "fd29f73b-9e51-4c3e-8133-1d7a6f807732",
  "ecs.version": "1.6.0"
}
{
  "log.level": "debug",
  "@timestamp": "2025-01-17T13:15:43.378Z",
  "message": "End of file reached: /Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log; Backoff now.",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "log-default",
    "type": "log"
  },
  "log": {
    "source": "log-default"
  },
  "input_id": "fd29f73b-9e51-4c3e-8133-1d7a6f807732",
  "finished": false,
  "harvester_id": "ce7d846b-bddc-4fa0-8af5-1b3af845775f",
  "log.origin": {
    "file.line": 111,
    "file.name": "log/log.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Log).Read"
  },
  "service.name": "filebeat",
  "source_file": "/Users/rdner/Projects/es_confs/deprecate_log/logs/log1.log",
  "state_id": "native::210784429-16777233",
  "os_id": "210784429-16777233",
  "ecs.version": "1.6.0",
  "log.logger": "input.harvester"
}

I saw the incoming data in Kibana too:

The log input has been depredated since 7.16, it's time to disable it. The container input is just an alias/preset for the log input. They still can be used under the following circumstances: * When running under Elastic Agent (by integrations) * When running as a part of a module * When `allow_deprecated_use` is set as a part of the the input configuration

rdner · 2025-01-16T18:33:25Z

filebeat/tests/system/test_reload_modules.py

  test:
    enabled: true
    wrong_field: error
+    var.paths: []


This test was broken, this was missing and the test was expecting an error for empty paths.

elasticmachine · 2025-01-17T13:24:35Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

belimawr

I'm just missing integration tests for both cases: standalone and under Elastic-Agent.

filebeat/_meta/test/module/foo/multi/config/multi.yml

filebeat/_meta/test/module/foo/multibad/config/multi.yml

filebeat/docs/how-filebeat-works.asciidoc

filebeat/docs/inputs/input-container.asciidoc

CHANGELOG.next.asciidoc

filebeat/docs/inputs/input-container.asciidoc

filebeat/input/log/input.go

Co-authored-by: Tiago Queiroz <me@tiago.life>

rdner · 2025-01-20T16:40:00Z

@belimawr

I'm just missing integration tests for both cases: standalone and under Elastic-Agent.

Do we have any integration tests in Beats that run Elastic Agent and integrations?

I'll work on the integration test for the standalone cases but I have doubts adding a test for Elastic Agent here would be the right place.

filebeat/docs/inputs/input-container.asciidoc

cmacknz · 2025-01-20T20:07:11Z

Do we have any integration tests in Beats that run Elastic Agent and integrations?

There are bunch in

beats/x-pack/filebeat/tests/integration/managerV2_test.go

Lines 110 to 121 in 0b5cfa9

    
           Config: &proto.UnitExpectedConfig{ 
        
           	Id:   "log-input", 
        
           	Type: "log", 
        
           	Name: "log", 
        
           	Streams: []*proto.Stream{ 
        
           		{ 
        
           			Id: "log-input-1", 
        
           			Source: integration.RequireNewStruct(t, map[string]interface{}{ 
        
           				"enabled": true, 
        
           				"type":    "log", 
        
           				"paths":   []interface{}{logFilePath}, 
        
           			}),

.

That you haven't had to change any of them might be enough, if you removed the fleetmode.Enabled check I'd expect some of those to fail.

belimawr · 2025-01-21T14:31:03Z

Do we have any integration tests in Beats that run Elastic Agent and integrations?

I'll work on the integration test for the standalone cases but I have doubts adding a test for Elastic Agent here would be the right place.

Yes, we do. Well, they don't run the Elastic-Agent, they mock the Elastic-Agent (via https://github.com/elastic/elastic-agent-client/tree/main/pkg/client/mock) and run Filebeat in a Elastic-Agent managed mode.

https://github.com/elastic/beats/blob/main/x-pack/filebeat/tests/integration/managerV2_test.go is a good starting point, the tests there have some extra complexity to assert state, but sending some configuration and asserting if Filebeat is behaving as expected is simple.

You need to instantiate the server and implement CheckinV2Impl to return the units you want Filebeat running.

To assess Filebeat's behaviour, looking at the logs or data shipped to the output usually is the easiest way.

Ping me if you have any questions, need any help.

mergify · 2025-01-22T12:20:28Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b deprecate-log-input upstream/deprecate-log-input
git merge upstream/main
git push upstream deprecate-log-input

rdner · 2025-01-27T08:34:32Z

@cmacknz @belimawr I think it's safe to merge this PR in the current state since I tested everything manually. I'm still finalizing the integration tests and will open a separate follow up PR with them.

belimawr · 2025-01-27T13:35:59Z

@cmacknz @belimawr I think it's safe to merge this PR in the current state since I tested everything manually. I'm still finalizing the integration tests and will open a separate follow up PR with them.

If you need to make the FF, that's fine for me.

rdner added Filebeat Filebeat breaking change backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team deprecation labels Jan 13, 2025

rdner self-assigned this Jan 13, 2025

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 13, 2025

rdner added 2 commits January 13, 2025 14:26

Add changelog entry

f1a1aa3

rdner force-pushed the deprecate-log-input branch from 6401032 to f1a1aa3 Compare January 13, 2025 13:29

Allow use of log input in tests

9cc2b16

rdner force-pushed the deprecate-log-input branch from 99d4d76 to 1f497a9 Compare January 15, 2025 10:14

Fix typos and linter

ba5df37

rdner force-pushed the deprecate-log-input branch from 1f497a9 to ba5df37 Compare January 15, 2025 10:23

Fix unit tests for the inputs

41e6772

rdner force-pushed the deprecate-log-input branch from 4fb2853 to 41e6772 Compare January 15, 2025 12:34

Update docs and tests

6de94ca

rdner force-pushed the deprecate-log-input branch from 508685e to 6de94ca Compare January 15, 2025 15:48

Fix flaky test on Windows

5c3bd44

rdner force-pushed the deprecate-log-input branch 10 times, most recently from a9aa39c to 6aa00a4 Compare January 16, 2025 14:50

rdner commented Jan 16, 2025

View reviewed changes

rdner requested review from belimawr, cmacknz and leehinman January 17, 2025 13:24

rdner marked this pull request as ready for review January 17, 2025 13:24

rdner requested a review from a team as a code owner January 17, 2025 13:24

belimawr reviewed Jan 17, 2025

View reviewed changes

cmacknz reviewed Jan 17, 2025

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

filebeat/docs/inputs/input-container.asciidoc Show resolved Hide resolved

filebeat/docs/inputs/input-container.asciidoc Outdated Show resolved Hide resolved

filebeat/input/log/input.go Show resolved Hide resolved

rdner and others added 3 commits January 20, 2025 09:39

Apply suggestions from code review

124ec68

Co-authored-by: Tiago Queiroz <me@tiago.life>

Improve the changelog entry

6714ce1

Improve container input docs

e82ee21

cmacknz reviewed Jan 20, 2025

View reviewed changes

filebeat/docs/inputs/input-container.asciidoc Outdated Show resolved Hide resolved

rdner added 4 commits January 24, 2025 13:13

Remove mention of take-over in the container docs

6037e7e

Revert change in the systemlogs input

dfe3113

Make integration tests run on Apple M4

87e7884

Comment the M4 workaround so the tests pass on the CI

144a0ef

rdner requested review from belimawr and cmacknz January 27, 2025 08:34

rdner mentioned this pull request Jan 27, 2025

Migrate container input to use filestream instead of log input #34393

Closed

belimawr approved these changes Jan 27, 2025

View reviewed changes

rdner merged commit 298a776 into elastic:main Jan 27, 2025

rdner deleted the deprecate-log-input branch January 27, 2025 17:21

jlind23 mentioned this pull request Jan 28, 2025

Jlind23/deprecate loginput elastic/integrations#12503

Merged

Conversation

rdner commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Disruptive User Impact

How to test this PR locally

Running log input in the user-crafted configuration fails

Setting allow_deprecated_use: true enables the log input

log input still runs as a part of a module

log input still runs as a part of an integration (Elastic Agent)

Uh oh!

rdner Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 17, 2025

Uh oh!

belimawr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdner commented Jan 20, 2025

Uh oh!

Uh oh!

cmacknz commented Jan 20, 2025

Uh oh!

belimawr commented Jan 21, 2025

Uh oh!

mergify bot commented Jan 22, 2025

Uh oh!

rdner commented Jan 27, 2025

Uh oh!

belimawr commented Jan 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rdner commented Jan 13, 2025 •

edited

Loading

Setting `allow_deprecated_use: true` enables the `log` input

`log` input still runs as a part of a module

`log` input still runs as a part of an integration (Elastic Agent)