Logstash information:
We are using the Docker image of Logstash on version 8.15.1.
JVM (e.g. java -version):
The one bundled in the Docker image.
OS version (uname -a if on a Unix-like system):
We have tested this on Docker standalone 26.1.4, standalone 27.1.2, and swarm 26.1.4. The actual machines that run the different Docker instances:
Linux 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Linux 5.4.0-193-generic #213-Ubuntu SMP Fri Aug 2 19:14:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
We use the Logstash container to ingest through Beats and output to ElasticSearch. However, when upgrading from version 8.15.0 to 8.15.1 our pipeline configuration became invalid when supplying it through the CONFIG_STRING environment variable in our docker-compose.yml.
Our configuration (works on 8.15.0):
input {
stdin { }
beats { port => 5040 }
}
output {
elasticsearch {
hosts => ["https://es:9200"]
ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca.crt"]
data_stream => "true"
data_stream_type => "logs"
user => "removed"
password => "removed"
}
stdout { }
}But on 8.15.1 we get the following error:
Failed to execute action {
:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError",
:message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 7, column 20 (byte 94) after output {\n elasticsearch {\n hosts => [http",
:backtrace=>[
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'",
"org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'",
"org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'",
"org/jruby/RubyClass.java:949:in `new'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"
]
}The character at column 20 is the first : in our hosts => ["https://es:9200"]. Eliminating the https:// does not work, as it then fails on the : specifying the port. Completely removing the port breaks on the first / in the ssl_certificate_authorities.
To ensure that this is not an issue with logstash-output-elasticsearch we also tested with the standard HTTP output plugin using the following config:
input {
stdin { }
beats { port => 5040 }
}
output {
http {
url => "https://otherhost:9200"
http_method => "post"
}
stdout { }
}We get a similar error for the : in the url:
Failed to execute action {
:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError",
:message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \"}\" at line 7, column 17 (byte 82) after output {\n http {\n url => https",
:backtrace=>[
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'",
"org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'",
"org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'",
"org/jruby/RubyClass.java:949:in `new'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"
]
}
Providing either pipeline config through logstash.conf (or another file in /usr/share/logstash/pipeline) does work as expected. As such, we believe there is a change in the parsing of the environment variable between 8.15.0 and 8.15.1.
Logstash information:
We are using the Docker image of Logstash on version 8.15.1.
JVM (e.g.
java -version):The one bundled in the Docker image.
OS version (
uname -aif on a Unix-like system):We have tested this on Docker standalone 26.1.4, standalone 27.1.2, and swarm 26.1.4. The actual machines that run the different Docker instances:
Description of the problem including expected versus actual behavior:
We use the Logstash container to ingest through Beats and output to ElasticSearch. However, when upgrading from version 8.15.0 to 8.15.1 our pipeline configuration became invalid when supplying it through the
CONFIG_STRINGenvironment variable in ourdocker-compose.yml.Our configuration (works on 8.15.0):
input { stdin { } beats { port => 5040 } } output { elasticsearch { hosts => ["https://es:9200"] ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca.crt"] data_stream => "true" data_stream_type => "logs" user => "removed" password => "removed" } stdout { } }But on 8.15.1 we get the following error:
Failed to execute action { :action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 7, column 20 (byte 94) after output {\n elasticsearch {\n hosts => [http", :backtrace=>[ "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:949:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'" ] }The character at column 20 is the first
:in ourhosts => ["https://es:9200"]. Eliminating thehttps://does not work, as it then fails on the:specifying the port. Completely removing the port breaks on the first/in thessl_certificate_authorities.To ensure that this is not an issue with
logstash-output-elasticsearchwe also tested with the standard HTTP output plugin using the following config:input { stdin { } beats { port => 5040 } } output { http { url => "https://otherhost:9200" http_method => "post" } stdout { } }We get a similar error for the
:in theurl:Failed to execute action { :action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \"}\" at line 7, column 17 (byte 82) after output {\n http {\n url => https", :backtrace=>[ "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:949:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'" ] }Providing either pipeline config through
logstash.conf(or another file in/usr/share/logstash/pipeline) does work as expected. As such, we believe there is a change in the parsing of the environment variable between 8.15.0 and 8.15.1.