Beats started by agent do not respect the allow_older_versions: true flag added to the output configuration. Beats by default will not connect to older versions of Elasticsearch, unless allow_older_versions: true appears in the output configuration.
Fleet managed agents will also refuse to enroll with older versions of Elasticsearch, but it is possible to configure a standalone agent to write to an older version. In this case the agent will run but the Beats will refuse to connect with the following error:
{"log.level":"error","@timestamp":"2023-01-10T17:54:57.248Z","message":"Failed to connect to backoff(elasticsearch(https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.5.3, Beat=8.6.0","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}Setting allow_older_versions: true in the output section of the agent policy should resolve this but it does not. This may be because the flag isn't available on this code path when the callback is first registered:
|
// checkElasticsearchVersion registers a global callback to make sure ES instance we are connecting |
|
// to is at least on the same version as the Beat. |
|
// If the check is disabled or the output is not Elasticsearch, nothing happens. |
|
func (b *Beat) checkElasticsearchVersion() { |
|
if b.isConnectionToOlderVersionAllowed() { |
|
return |
|
} |
|
|
|
_, _ = elasticsearch.RegisterGlobalCallback(func(conn *eslegclient.Connection) error { |
|
esVersion := conn.GetVersion() |
|
beatVersion, err := libversion.New(b.Info.Version) |
|
if err != nil { |
|
return err |
|
} |
|
if esVersion.LessThanMajorMinor(beatVersion) { |
|
return fmt.Errorf("%w ES=%s, Beat=%s", elasticsearch.ErrTooOld, esVersion.String(), b.Info.Version) |
|
} |
|
return nil |
|
}) |
|
} |
|
|
|
func (b *Beat) isConnectionToOlderVersionAllowed() bool { |
|
config := struct { |
|
AllowOlder bool `config:"allow_older_versions"` |
|
}{false} |
|
|
|
_ = b.Config.Output.Config().Unpack(&config) |
|
|
|
return config.AllowOlder |
|
} |
To reproduce this:
- Build or obtain the latest version of the agent, e.g. 8.6.0.
- Create an Elasticsearch deployment for the previous minor version e.g. 8.5.3
- Configure a standalone agent to write to to the 8.5.3 Elasticsearch endpoint, with the
allow_older_versions: true flag set on the output. For example:
outputs:
default:
type: elasticsearch
hosts: [https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443]
username: "elastic"
password: <REDACTED>
allow_older_versions: true
- Observe that the Beats refuse to connect to Elasticsearch and the
allow_older_versions: true flag is configured on the generated Beat configuration in the components directory of the elastic agent diagnostics bundle. Example:
outputs:
elasticsearch:
allow_older_versions: true
bulk_max_size: 50
hosts:
- https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443
password: <REDACTED>
type: elasticsearch
username: elasticFor example configuring
Beats started by agent do not respect the
allow_older_versions: trueflag added to the output configuration. Beats by default will not connect to older versions of Elasticsearch, unlessallow_older_versions: trueappears in the output configuration.Fleet managed agents will also refuse to enroll with older versions of Elasticsearch, but it is possible to configure a standalone agent to write to an older version. In this case the agent will run but the Beats will refuse to connect with the following error:
{"log.level":"error","@timestamp":"2023-01-10T17:54:57.248Z","message":"Failed to connect to backoff(elasticsearch(https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.5.3, Beat=8.6.0","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}Setting
allow_older_versions: truein the output section of the agent policy should resolve this but it does not. This may be because the flag isn't available on this code path when the callback is first registered:beats/libbeat/cmd/instance/beat.go
Lines 910 to 939 in b2a1310
To reproduce this:
allow_older_versions: trueflag set on the output. For example:allow_older_versions: trueflag is configured on the generated Beat configuration in thecomponentsdirectory of the elastic agent diagnostics bundle. Example:For example configuring