What does this PR do?

This PR adds a mage command to package the system tests results. This command is invoked in the CI in the same conditions that in the past.

The command is composed by:

• a mage packageSystemTests command, plugged into filebeat and metricbeat modules (both OSS and xpack flavours).
• a Tar method under the commons package, so that it's possible to compress a source directory using TAR+GZIP algorithms, and writing it to the build dir in the root directory of the Beats repository.

Besides that, we are removing a python script that detected the path to the Python system-tests (used in the pipeline to store the folder name in a variable), and we are updating the Jenkins pipeline to call the new build system command instead of the previous script. To keep existing behaviour, that we could consider it wrong and would possible discuss about it in a follow-up issue, we are passing a new argument to the groovy method in the pipeline, representing the directory in which the Beat live (i.e. filebeat, x-pack/filebeat), so that it's possible to run the mage command in the right context.

Why is it important?

In the past, the collection of the system tests results was done at the pipeline level, so developers were not able to reproduce them. With this new command now it's possible to collect them directly from the build system.

For debugging purpose, it will be possible to debug a CI worker, capturing it and executing this command, to verify if the files to be uploaded are there.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

$ cd filebeat
$ mage pythonIntegTest
$ mage packageSystemTests
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_ignore_symlink/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_symlink_and_file/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_symlink_rotated/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_symlinks_enabled/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_truncate/log/symlink.log
>> skipping symlink: build/system-tests/run/test_input.Test.test_skip_symlinks/log/test.log

# checking file system contents
$ find build/system-tests/run -name "*" | wc -l
    3940
    
# checking TAR contents
$ tar -tf ../build/system-tests-filebeat.tar.gz | wc -l
    3934

Expected behaviour: a TAR file in the root's build directory named as the Beat will exist (i.e. ./build/system-tests-filebeat.tar.gz)

Related issues

• Relates [Build System] Add a command to package system tests files into a TAR file #26208

Use cases

Screenshots

Logs