Skip to content

Commit 181cf69

Browse files
adriansradriansr
authored
Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (#26737)
Improve the pipeline/compatibility code so that all processors in a pipeline are scanned and acted-upon to ensure compatibility. This means: - Scan processors in on_failure section (both the pipeline's and each individual processor on-failure handler). - Scan the inner processor in a foreach processor. Add a new CI stage, module-compat-7.11, to filebeat/Jenkinsfile.yml and x-pack/filebeat/Jenkinsfile.yml, in order to run Filebeat modules tests under ES 7.11.2 to ensure that all pipelines are functional. This test uses a new flag, TESTING_FILEBEAT_SKIP_DIFF, to instruct the integration test to skip the comparison between the generated documents and the golden/expected files. The test will ensure that the pipeline loads, there are no ingest errors, the fields in the generated documents are valid and the number of returned documents matches the expected. This is intended to avoid having to maintain multiple versions of the golden files due to differences between ES versions and available processors. Also fixes the fortinet and threatintel modules pipelines so that they pass the new test, as some fields were left behind due to the uri_parts processor being removed.
1 parent a5f73d5 commit 181cf69

9 files changed

Lines changed: 552 additions & 101 deletions

File tree

‎filebeat/Jenkinsfile.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,14 @@ stages:
3939
mage: "mage build test"
4040
withModule: true ## run the ITs only if the changeset affects a specific module.
4141
stage: mandatory
42+
module-compat-7.11:
43+
mage: >- ## Run module integration tests under ES 7.11 to ensure ingest pipeline compatibility.
44+
STACK_ENVIRONMENT=7.11
45+
TESTING_FILEBEAT_SKIP_DIFF=1
46+
PYTEST_ADDOPTS='-k test_modules'
47+
mage pythonIntegTest
48+
withModule: true
49+
stage: mandatory
4250
macos:
4351
mage: "mage build unitTest"
4452
platforms: ## override default label in this specific stage.

0 commit comments

Comments
 (0)