Fixes the issue where LS wipes out all quotes from docker env variables. by mashhurs · Pull Request #16456 · elastic/logstash

mashhurs · 2024-09-16T18:07:57Z

Release notes

Fixes the regression introduced in 8.15. The regression was Logstash wiped out single and double quotes from CONFIG_STRING. The issue happens when running Logstash on docker with CONFIG_STRING and the CONFIG_STRING contains quotes (for example hosts=>"my.es.host")

What does this PR do?

Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.

Why is it important/What is the impact to the user?

Unblocks the user who faced upgrade issue from 8.15.0 or 8.15.1

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

run ./gradlew clean bootstrap assemble installDefaultGems && rake artifact:docker command to build the artifact on your local. The artifact will be 9.0.0-SNAPSHOT
make sure docker engine is running

create docker-compose.yml file in your testing path and fill it with following config:

services:
logstash:
  image: docker.elastic.co/logstash/logstash:9.0.0-SNAPSHOT
  container_name: logstash
  ports:
    - "9600:9600"
  environment:
    - MONITORING_ENABLED=false
    - XPACK_MONITORING_ENABLED=false
    - CONFIG_STRING='input {
        stdin { }
        beats { port => 5040 }
      }
      output {
        elasticsearch {
          hosts => ["https://es:9200"]
          data_stream => "true"
          data_stream_type => "logs"
          user => "removed"
          password => "removed"
        }
        stdout { }
      }'

run docker compose up to check if Logstash spins up without issue

Note that, you can replace CONFIG_STRING='' single quote with double or remove it. You can also place ${VAR} in the CONFIG_STRING (example ${USER} will result in setting user). This PR makes the same user experience with 8.13.0 where we started adding env2yaml/substitution ${VAR} changes.
Without this change, we get following error:

logstash  | [2024-09-16T21:01:29,703][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 1, column 84 (byte 84) after output { elasticsearch { hosts => [https", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:949:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}

You can validate this issue by changing defined docker-compose.yml image to image: docker.elastic.co/logstash/logstash:8.15.1

How to run docker acceptance tests?

run ./gradlew clean bootstrap assemble installDefaultGems && rake artifact:docker command to build the artifact on your local. The artifact will be 9.0.0-SNAPSHOT
make sure docker engine is running
go to qa folder and run run bundle install && bundle exec rspec docker/spec/full/container_spec.rb

# logs
Failed (
expected true
     got false
). Sleeping for 2.0
    has an Ubuntu 20.04 base image

Finished in 2 minutes 14.2 seconds (files took 0.13976 seconds to load)
8 examples, 0 failures

Related issues

Use cases

Users who are deploying LS using docker-compose.yml
User who are using config file with ${VAR} defined in it

Screenshots

Logs

➜  docker-tests docker compose up
[+] Running 1/0
 ✔ Container logstash  Recreated                                                                                                                                                    0.0s 
Attaching to logstash
logstash  | 2024/09/16 21:03:01 Setting 'config.string' from environment.
logstash  | 2024/09/16 21:03:01 Setting 'xpack.monitoring.enabled' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk
logstash  | Replacing `${CONFIG_STRING}` with actual value
logstash  | Placeholder value: "input { stdin { } beats { port => 5040 } } output { elasticsearch { hosts => [\"https://es:9200\"] data_stream => \"true\" data_stream_type => \"logs\" user => \"removed\" password => \"removed\" } stdout { } }"
logstash  | refined_value-1: "input { stdin { } beats { port => 5040 } } output { elasticsearch { hosts => [\"https://es:9200\"] data_stream => \"true\" data_stream_type => \"logs\" user => \"removed\" password => \"removed\" } stdout { } }"
logstash  | refined_value-2: "input { stdin { } beats { port => 5040 } } output { elasticsearch { hosts => [\"https://es:9200\"] data_stream => \"true\" data_stream_type => \"logs\" user => \"removed\" password => \"removed\" } stdout { } }"
logstash  | Replacing `${XPACK_MONITORING_ENABLED}` with actual value
logstash  | Placeholder value: "false"
logstash  | refined_value-1: "false"
logstash  | refined_value-2: "false"
logstash  | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash  | [2024-09-16T21:03:07,387][WARN ][deprecation.logstash.settings] The setting `http.host` is a deprecated alias for `api.http.host` and will be removed in a future release of Logstash. Please use api.http.host instead
logstash  | [2024-09-16T21:03:07,393][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash  | [2024-09-16T21:03:07,394][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"9.0.0", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [aarch64-linux]"}
logstash  | [2024-09-16T21:03:07,395][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
logstash  | [2024-09-16T21:03:07,396][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
logstash  | [2024-09-16T21:03:07,396][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
logstash  | [2024-09-16T21:03:07,399][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash  | [2024-09-16T21:03:07,400][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash  | [2024-09-16T21:03:07,471][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
logstash  | [2024-09-16T21:03:07,479][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"be7ba1fa-b92d-48d4-af13-eb628303d572", :path=>"/usr/share/logstash/data/uuid"}
logstash  | [2024-09-16T21:03:07,694][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash  | [2024-09-16T21:03:07,802][INFO ][org.reflections.Reflections] Reflections took 54 ms to scan 1 urls, producing 139 keys and 484 values
logstash  | [2024-09-16T21:03:08,005][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
logstash  | [2024-09-16T21:03:08,010][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://es:9200"]}
logstash  | [2024-09-16T21:03:08,051][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://removed:xxxxxx@es:9200/]}}
logstash  | [2024-09-16T21:03:08,079][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"es: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: es: Name or service not known>}
logstash  | [2024-09-16T21:03:08,080][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://removed:xxxxxx@es:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es:9200/][Manticore::ResolutionFailure] es: Name or service not known"}
logstash  | [2024-09-16T21:03:08,088][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x55ae05b9 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
logstash  | [2024-09-16T21:03:08,479][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.39}
logstash  | [2024-09-16T21:03:08,482][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5040"}
logstash  | [2024-09-16T21:03:08,496][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
logstash  | [2024-09-16T21:03:08,509][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash  | [2024-09-16T21:03:08,530][INFO ][org.logstash.beats.Server][main][cf137c1a24220ee2df85a28ed886d5d3a188863995701d0bb62ab4c1159bfc30] Starting server on port: 5040

…es. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.

mashhurs · 2024-09-16T21:06:01Z

@@ -35,7 +35,7 @@
        ]


review note: please see the issue how-to-test section which solves the linked issues. I couldn't make it work CONFIG_STRING option with in the docker acceptance tests in a given short time. I will make it by adding more scenario tests with https://github.com/elastic/ingest-dev/issues/3995
Added docker acceptance test as well: 0ee0bf7
Will continue adding more unit tests with ingest-dev/issues/3995

jsvd

I tested with a few use cases, namely:

Ensuring a string isn't mutated just because it contains a reference: ✅

echo "input { heartbeat {} }\nfilter { ruby { code => \"event.set('result', '\${STRING}'.length)\" } }\noutput { stdout {} }" > cfg; cat cfg ; export STRING=hey; bin/logstash -f cfg

Checking a single element array works without brackets: ✅

❯ echo "xpack.management.enabled: true\nxpack.management.pipeline.id: \${XPACK_MANAGEMENT_PIPELINE_ID}" > config/logstash.yml ; cat config/logstash.yml; XPACK_MANAGEMENT_ENABLED=true XPACK_MANAGEMENT_PIPELINE_ID='"*"' bin/logstash --debug --config.debug | grep -e xpack.management -e "config store "
xpack.management.enabled: true
xpack.management.pipeline.id: ${XPACK_MANAGEMENT_PIPELINE_ID}
[2024-09-16T23:32:07,010][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["*"], :poll_interval=>"TimeValue{duration=5, timeUnit=SECONDS}ns"}

Check if single element works with square brackets: 🟥

❯ echo "xpack.management.enabled: true\nxpack.management.pipeline.id: \${XPACK_MANAGEMENT_PIPELINE_ID}" > config/logstash.yml ; cat config/logstash.yml; XPACK_MANAGEMENT_ENABLED=true XPACK_MANAGEMENT_PIPELINE_ID='["*"]' bin/logstash --debug --config.debug | grep -e xpack.management -e "config store "
xpack.management.enabled: true
xpack.management.pipeline.id: ${XPACK_MANAGEMENT_PIPELINE_ID}
ERROR: Pipeline id in `xpack.management.pipeline.id` must begin with a letter or underscore and contain only letters, underscores, dashes, and numbers. The asterisk wildcard `*` can also be used. Invalid ids: "*"

Multi element array: 🟥

❯ echo "xpack.management.enabled: true\nxpack.management.pipeline.id: \${XPACK_MANAGEMENT_PIPELINE_ID}" > config/logstash.yml ; cat config/logstash.yml; XPACK_MANAGEMENT_ENABLED=true XPACK_MANAGEMENT_PIPELINE_ID='["a", "b"]' bin/logstash --debug --config.debug | grep -e xpack.management -e "config store "
xpack.management.enabled: true
xpack.management.pipeline.id: ${XPACK_MANAGEMENT_PIPELINE_ID}
ERROR: Pipeline id in `xpack.management.pipeline.id` must begin with a letter or underscore and contain only letters, underscores, dashes, and numbers. The asterisk wildcard `*` can also be used. Invalid ids: "a", "b"

jsvd

The failure cases weren't working before (8.12 or 8.13), so LGTM

Let’s merge, backport to 8.x and 8.15 and then kick off a staging build for 8.15

elastic-sonarqube · 2024-09-17T05:45:12Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

elasticmachine · 2024-09-17T05:53:48Z

💚 Build Succeeded

Buildkite Build
Commit: 715ab0e

History

💚 Build #1482 succeeded 0ee0bf7
💛 Build #1481 was flaky ff9741a
💚 Build #1480 succeeded 83a788c

cc @mashhurs

jsvd · 2024-09-17T12:15:31Z

The last couple of scenarios work correctly now:

Check if single element works with square brackets: ✅

echo "xpack.management.enabled: true\nxpack.management.pipeline.id: \${XPACK_MANAGEMENT_PIPELINE_ID}" > config/logstash.yml ; cat config/logstash.yml; XPACK_MANAGEMENT_ENABLED=true XPACK_MANAGEMENT_PIPELINE_ID='["*"]' bin/logstash --debug --config.debug | grep -e xpack.management -e "config store "
xpack.management.enabled: true
xpack.management.pipeline.id: ${XPACK_MANAGEMENT_PIPELINE_ID}
[2024-09-17T13:13:35,055][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["*"], :poll_interval=>"TimeValue{duration=5, timeUnit=SECONDS}ns"}

Multi element array: ✅

echo "xpack.management.enabled: true\nxpack.management.pipeline.id: \${XPACK_MANAGEMENT_PIPELINE_ID}" > config/logstash.yml ; cat config/logstash.yml; XPACK_MANAGEMENT_ENABLED=true XPACK_MANAGEMENT_PIPELINE_ID='["a", "b"]' bin/logstash --debug --config.debug | grep -e xpack.management -e "config store "
xpack.management.enabled: true
xpack.management.pipeline.id: ${XPACK_MANAGEMENT_PIPELINE_ID}
[2024-09-17T13:13:58,785][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["a", "b"], :poll_interval=>"TimeValue{duration=5, timeUnit=SECONDS}ns"}

@mashhurs we're good to merge, backport and trigger staging DRA 🚢

mashhurs · 2024-09-17T13:46:40Z

@logstashmachine backport 8.15

…es. (#16456) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73)

mashhurs · 2024-09-17T14:02:33Z

@logstashmachine backport 8.x

…es. (#16456) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73)

…es. (#16456) (#16458) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>

…es. (#16456) (#16459) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>

* Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) * Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408) Fix: #16406 * Generate Dataset code with meaningful fields names (#16386) This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields. Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`. Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`. Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields. * Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322) This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it. This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch. Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node. Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event. --------- Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16426) * Release notes for 8.15.1 (#16405) (#16427) * Update release notes for 8.15.1 * update release note --------- Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com> Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> (cherry picked from commit 2fca7e3) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430) Make inner field of ConditionalEvaluationError transient to be avoided during serialization. (cherry picked from commit bb7ecc2) Co-authored-by: Andrea Selva <selva.andre@gmail.com> * use gnu tar compatible minitar to generate tar artifact (#16432) (#16434) Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters. For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem. (cherry picked from commit 69f0fa5) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * account for the 8.x in DRA publishing task (#16436) (#16440) the current DRA publishing task computes the branch from the version contained in the version.yml This is done by taking the major.minor and confirming that a branch exists with that name. However this pattern won't be applicable for 8.x, as that branch currently points to 8.16.0 and there is no 8.16 branch. This commit falls back to reading the buildkite injected BUILDKITE_BRANCH variable. (cherry picked from commit 17dba9f) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Known issue for 8.15.1 related to env vars references (#16455) (#16469) (cherry picked from commit b54caf3) Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> * bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480) (cherry picked from commit 51cca73) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Release notes for 8.15.2 (#16471) (#16478) Co-authored-by: andsel <selva.andre@gmail.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> (cherry picked from commit 01dc76f) * Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488) (cherry picked from commit 8368c00) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491) (cherry picked from commit fd1de39) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Don't use an older JRuby with oraclelinux-7 (#16499) (#16501) A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images and removed JRuby 9.4.5.0 and some older versions. This ended up breaking exhaustive test on Oracle Linux 7 that hard coded JRuby 9.4.5.0. PR #16489 worked around the problem by pinning to the new JRuby, but actually we don't need the conditional anymore since the original issue jruby/jruby#7579 (comment) has been resolved and none of our releasable branches (apart from 7.17 which uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`. Therefore, this commit removes conditional setting of JRuby for OracleLinux 7 agents in exhaustive tests (and relies on whatever `/.ruby-version` defines). (cherry picked from commit 07c01f8) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> * Improve pipeline bootstrap error logs (#16495) (#16504) This PR adds the cause errors details on the pipeline converge state error logs (cherry picked from commit e84fb45) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511) (cherry picked from commit 5195332) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Make health report test runner script executable. (#16446) (#16512) (cherry picked from commit 2ebf265) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493) * DLQ-ing events that trigger an conditional evaluation error. (#16423) When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline. This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it. (cherry picked from commit b69d993) * Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * add deprecation log for `--event_api.tags.illegal` (#16507) (#16515) - move `--event_api.tags.illegal` from option to deprecated_option - add deprecation log when the flag is explicitly used relates: #16356 Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> (cherry picked from commit a4eddb8) Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

* [health] bootstrap HealthObserver from agent to API (#16141) * [health] bootstrap HealthObserver from agent to API * specs: mocked agent needs health observer * add license headers * Merge `main` into `feature/health-report-api` (#16397) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Squashed merge from 8.x * Failure injector plugin implementation. (#16466) * Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes. * Fix the degrate method name typo. Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Add explanation to the config params and rebuild plugin gem. --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Health report integration tests bootstrapper and initial tests implementation (#16467) * Health Report integration tests bootstrapper and initial slow start scenario implementation. * Apply suggestions from code review Renaming expectation check method name. Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * Changed to branch concept, YAML structure simplified as changed to Dict. * Apply suggestions from code review Reflect `help_url` to the integration test. --------- Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * health api: expose `GET /_health_report` with pipelines/*/status probe (#16398) Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the resulting report status into the other API responses, replacing their hard-coded `green` with a meaningful status indication. --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * docs: health report API, and diagnosis links (feature-targeted) (#16518) * docs: health report API, and diagnosis links * Remove plus-for-passthrough markers Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * merge 8.x into feature branch... (#16519) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) * Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408) Fix: #16406 * Generate Dataset code with meaningful fields names (#16386) This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields. Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`. Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`. Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields. * Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322) This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it. This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch. Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node. Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event. --------- Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16426) * Release notes for 8.15.1 (#16405) (#16427) * Update release notes for 8.15.1 * update release note --------- Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com> Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> (cherry picked from commit 2fca7e3) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430) Make inner field of ConditionalEvaluationError transient to be avoided during serialization. (cherry picked from commit bb7ecc2) Co-authored-by: Andrea Selva <selva.andre@gmail.com> * use gnu tar compatible minitar to generate tar artifact (#16432) (#16434) Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters. For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem. (cherry picked from commit 69f0fa5) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * account for the 8.x in DRA publishing task (#16436) (#16440) the current DRA publishing task computes the branch from the version contained in the version.yml This is done by taking the major.minor and confirming that a branch exists with that name. However this pattern won't be applicable for 8.x, as that branch currently points to 8.16.0 and there is no 8.16 branch. This commit falls back to reading the buildkite injected BUILDKITE_BRANCH variable. (cherry picked from commit 17dba9f) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Known issue for 8.15.1 related to env vars references (#16455) (#16469) (cherry picked from commit b54caf3) Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> * bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480) (cherry picked from commit 51cca73) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Release notes for 8.15.2 (#16471) (#16478) Co-authored-by: andsel <selva.andre@gmail.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> (cherry picked from commit 01dc76f) * Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488) (cherry picked from commit 8368c00) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491) (cherry picked from commit fd1de39) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Don't use an older JRuby with oraclelinux-7 (#16499) (#16501) A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images and removed JRuby 9.4.5.0 and some older versions. This ended up breaking exhaustive test on Oracle Linux 7 that hard coded JRuby 9.4.5.0. PR #16489 worked around the problem by pinning to the new JRuby, but actually we don't need the conditional anymore since the original issue jruby/jruby#7579 (comment) has been resolved and none of our releasable branches (apart from 7.17 which uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`. Therefore, this commit removes conditional setting of JRuby for OracleLinux 7 agents in exhaustive tests (and relies on whatever `/.ruby-version` defines). (cherry picked from commit 07c01f8) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> * Improve pipeline bootstrap error logs (#16495) (#16504) This PR adds the cause errors details on the pipeline converge state error logs (cherry picked from commit e84fb45) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511) (cherry picked from commit 5195332) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Make health report test runner script executable. (#16446) (#16512) (cherry picked from commit 2ebf265) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493) * DLQ-ing events that trigger an conditional evaluation error. (#16423) When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline. This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it. (cherry picked from commit b69d993) * Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * add deprecation log for `--event_api.tags.illegal` (#16507) (#16515) - move `--event_api.tags.illegal` from option to deprecated_option - add deprecation log when the flag is explicitly used relates: #16356 Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> (cherry picked from commit a4eddb8) Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

* [health] bootstrap HealthObserver from agent to API (#16141) * [health] bootstrap HealthObserver from agent to API * specs: mocked agent needs health observer * add license headers * Merge `main` into `feature/health-report-api` (#16397) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Squashed merge from 8.x * Failure injector plugin implementation. (#16466) * Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes. * Fix the degrate method name typo. Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Add explanation to the config params and rebuild plugin gem. --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Health report integration tests bootstrapper and initial tests implementation (#16467) * Health Report integration tests bootstrapper and initial slow start scenario implementation. * Apply suggestions from code review Renaming expectation check method name. Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * Changed to branch concept, YAML structure simplified as changed to Dict. * Apply suggestions from code review Reflect `help_url` to the integration test. --------- Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * health api: expose `GET /_health_report` with pipelines/*/status probe (#16398) Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the resulting report status into the other API responses, replacing their hard-coded `green` with a meaningful status indication. --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * docs: health report API, and diagnosis links (feature-targeted) (#16518) * docs: health report API, and diagnosis links * Remove plus-for-passthrough markers Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * merge 8.x into feature branch... (#16519) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) * Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408) Fix: #16406 * Generate Dataset code with meaningful fields names (#16386) This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields. Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`. Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`. Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields. * Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322) This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it. This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch. Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node. Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event. --------- Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16426) * Release notes for 8.15.1 (#16405) (#16427) * Update release notes for 8.15.1 * update release note --------- Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com> Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> (cherry picked from commit 2fca7e3) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430) Make inner field of ConditionalEvaluationError transient to be avoided during serialization. (cherry picked from commit bb7ecc2) Co-authored-by: Andrea Selva <selva.andre@gmail.com> * use gnu tar compatible minitar to generate tar artifact (#16432) (#16434) Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters. For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem. (cherry picked from commit 69f0fa5) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * account for the 8.x in DRA publishing task (#16436) (#16440) the current DRA publishing task computes the branch from the version contained in the version.yml This is done by taking the major.minor and confirming that a branch exists with that name. However this pattern won't be applicable for 8.x, as that branch currently points to 8.16.0 and there is no 8.16 branch. This commit falls back to reading the buildkite injected BUILDKITE_BRANCH variable. (cherry picked from commit 17dba9f) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Known issue for 8.15.1 related to env vars references (#16455) (#16469) (cherry picked from commit b54caf3) Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> * bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480) (cherry picked from commit 51cca73) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Release notes for 8.15.2 (#16471) (#16478) Co-authored-by: andsel <selva.andre@gmail.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> (cherry picked from commit 01dc76f) * Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488) (cherry picked from commit 8368c00) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491) (cherry picked from commit fd1de39) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Don't use an older JRuby with oraclelinux-7 (#16499) (#16501) A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images and removed JRuby 9.4.5.0 and some older versions. This ended up breaking exhaustive test on Oracle Linux 7 that hard coded JRuby 9.4.5.0. PR #16489 worked around the problem by pinning to the new JRuby, but actually we don't need the conditional anymore since the original issue jruby/jruby#7579 (comment) has been resolved and none of our releasable branches (apart from 7.17 which uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`. Therefore, this commit removes conditional setting of JRuby for OracleLinux 7 agents in exhaustive tests (and relies on whatever `/.ruby-version` defines). (cherry picked from commit 07c01f8) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> * Improve pipeline bootstrap error logs (#16495) (#16504) This PR adds the cause errors details on the pipeline converge state error logs (cherry picked from commit e84fb45) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511) (cherry picked from commit 5195332) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Make health report test runner script executable. (#16446) (#16512) (cherry picked from commit 2ebf265) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493) * DLQ-ing events that trigger an conditional evaluation error. (#16423) When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline. This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it. (cherry picked from commit b69d993) * Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * add deprecation log for `--event_api.tags.illegal` (#16507) (#16515) - move `--event_api.tags.illegal` from option to deprecated_option - add deprecation log when the flag is explicitly used relates: #16356 Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> (cherry picked from commit a4eddb8) Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> (cherry picked from commit 7eb5185)

* [health] bootstrap HealthObserver from agent to API (#16141) * [health] bootstrap HealthObserver from agent to API * specs: mocked agent needs health observer * add license headers * Merge `main` into `feature/health-report-api` (#16397) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Squashed merge from 8.x * Failure injector plugin implementation. (#16466) * Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes. * Fix the degrate method name typo. Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Add explanation to the config params and rebuild plugin gem. --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Health report integration tests bootstrapper and initial tests implementation (#16467) * Health Report integration tests bootstrapper and initial slow start scenario implementation. * Apply suggestions from code review Renaming expectation check method name. Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * Changed to branch concept, YAML structure simplified as changed to Dict. * Apply suggestions from code review Reflect `help_url` to the integration test. --------- Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * health api: expose `GET /_health_report` with pipelines/*/status probe (#16398) Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the resulting report status into the other API responses, replacing their hard-coded `green` with a meaningful status indication. --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * docs: health report API, and diagnosis links (feature-targeted) (#16518) * docs: health report API, and diagnosis links * Remove plus-for-passthrough markers Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * merge 8.x into feature branch... (#16519) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: #16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) * Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408) Fix: #16406 * Generate Dataset code with meaningful fields names (#16386) This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields. Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`. Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`. Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields. * Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322) This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it. This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch. Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node. Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event. --------- Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16426) * Release notes for 8.15.1 (#16405) (#16427) * Update release notes for 8.15.1 * update release note --------- Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com> Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> (cherry picked from commit 2fca7e3) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430) Make inner field of ConditionalEvaluationError transient to be avoided during serialization. (cherry picked from commit bb7ecc2) Co-authored-by: Andrea Selva <selva.andre@gmail.com> * use gnu tar compatible minitar to generate tar artifact (#16432) (#16434) Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters. For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem. (cherry picked from commit 69f0fa5) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * account for the 8.x in DRA publishing task (#16436) (#16440) the current DRA publishing task computes the branch from the version contained in the version.yml This is done by taking the major.minor and confirming that a branch exists with that name. However this pattern won't be applicable for 8.x, as that branch currently points to 8.16.0 and there is no 8.16 branch. This commit falls back to reading the buildkite injected BUILDKITE_BRANCH variable. (cherry picked from commit 17dba9f) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459) * Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable. * Add a docker acceptance integration test. (cherry picked from commit 7c64c73) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Known issue for 8.15.1 related to env vars references (#16455) (#16469) (cherry picked from commit b54caf3) Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> * bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480) (cherry picked from commit 51cca73) Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Release notes for 8.15.2 (#16471) (#16478) Co-authored-by: andsel <selva.andre@gmail.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> (cherry picked from commit 01dc76f) * Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488) (cherry picked from commit 8368c00) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491) (cherry picked from commit fd1de39) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Don't use an older JRuby with oraclelinux-7 (#16499) (#16501) A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images and removed JRuby 9.4.5.0 and some older versions. This ended up breaking exhaustive test on Oracle Linux 7 that hard coded JRuby 9.4.5.0. PR #16489 worked around the problem by pinning to the new JRuby, but actually we don't need the conditional anymore since the original issue jruby/jruby#7579 (comment) has been resolved and none of our releasable branches (apart from 7.17 which uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`. Therefore, this commit removes conditional setting of JRuby for OracleLinux 7 agents in exhaustive tests (and relies on whatever `/.ruby-version` defines). (cherry picked from commit 07c01f8) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> * Improve pipeline bootstrap error logs (#16495) (#16504) This PR adds the cause errors details on the pipeline converge state error logs (cherry picked from commit e84fb45) Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> * Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511) (cherry picked from commit 5195332) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Make health report test runner script executable. (#16446) (#16512) (cherry picked from commit 2ebf265) Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493) * DLQ-ing events that trigger an conditional evaluation error. (#16423) When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline. This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it. (cherry picked from commit b69d993) * Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * add deprecation log for `--event_api.tags.illegal` (#16507) (#16515) - move `--event_api.tags.illegal` from option to deprecated_option - add deprecation log when the flag is explicitly used relates: #16356 Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> (cherry picked from commit a4eddb8) Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> (cherry picked from commit 7eb5185) Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>

Fixes the issue where LS wipes out all quotes from docker env variabl…

83a788c

…es. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.

mashhurs added the bug label Sep 16, 2024

mashhurs self-assigned this Sep 16, 2024

jsvd added the status:work-in-progress label Sep 16, 2024

Revert back docker acceptance test changes.

ff9741a

mashhurs marked this pull request as ready for review September 16, 2024 21:03

mashhurs requested a review from jsvd September 16, 2024 21:03

mashhurs commented Sep 16, 2024

View reviewed changes

jsvd added status:needs-triage and removed status:work-in-progress labels Sep 16, 2024

Add a docker acceptance integration test.

0ee0bf7

jsvd reviewed Sep 16, 2024

View reviewed changes

jsvd approved these changes Sep 16, 2024

View reviewed changes

jsvd added status:approved and removed status:needs-triage labels Sep 16, 2024

Apply stripe enclosing char on array values.

715ab0e

mashhurs merged commit 7c64c73 into elastic:main Sep 17, 2024

github-actions bot added the v8.15.2 label Sep 17, 2024

github-actions bot mentioned this pull request Sep 17, 2024

Backport PR #16456 to 8.15: Fixes the issue where LS wipes out all quotes from docker env variables. #16458

Merged

5 tasks

jsvd removed the status:approved label Sep 17, 2024

github-actions bot mentioned this pull request Sep 17, 2024

Backport PR #16456 to 8.x: Fixes the issue where LS wipes out all quotes from docker env variables. #16459

Merged

5 tasks

github-actions bot added the v8.16.0 label Sep 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes the issue where LS wipes out all quotes from docker env variables.#16456

Fixes the issue where LS wipes out all quotes from docker env variables.#16456
mashhurs merged 4 commits intoelastic:mainfrom
mashhurs:docker-with-config-string-issue-fix

mashhurs commented Sep 16, 2024 •

edited

Loading

Uh oh!

mashhurs Sep 16, 2024 •

edited

Loading

Uh oh!

jsvd left a comment

Uh oh!

jsvd left a comment

Uh oh!

elastic-sonarqube bot commented Sep 17, 2024

Uh oh!

elasticmachine commented Sep 17, 2024

Uh oh!

jsvd commented Sep 17, 2024

Uh oh!

mashhurs commented Sep 17, 2024

Uh oh!

mashhurs commented Sep 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mashhurs commented Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

How to run docker acceptance tests?

Related issues

Use cases

Screenshots

Logs

Uh oh!

mashhurs Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsvd left a comment

Choose a reason for hiding this comment

Uh oh!

jsvd left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Sep 17, 2024

Quality Gate passed

Uh oh!

elasticmachine commented Sep 17, 2024

💚 Build Succeeded

History

Uh oh!

jsvd commented Sep 17, 2024

Uh oh!

mashhurs commented Sep 17, 2024

Uh oh!

mashhurs commented Sep 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mashhurs commented Sep 16, 2024 •

edited

Loading

mashhurs Sep 16, 2024 •

edited

Loading