Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files by kvch · Pull Request #25118 · elastic/beats

kvch · 2021-04-15T17:33:03Z

Cherry-pick of PR #24425 to 7.x branch. Original message:

What does this PR do?

Add support for better support truncated files with symlinks and adds migrates related tests from test_harvester.py.

Why is it important?

Increase coverage.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

…tic#24425) (cherry picked from commit a9279cd)

elasticmachine · 2021-04-15T17:33:06Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-04-15T18:17:21Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25118 opened
Start Time: 2021-04-15T17:33:52.968+0000
Duration: 43 min 13 sec
Commit: 92b81be

Test stats 🧪

Test	Results
Failed	0
Passed	9546
Skipped	1972
Total	11518

Trends 🧪

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 1 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 9 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`untar gsutil`

Took 0 min 0 sec . View more details on here
Description: tar -xpf gsutil.tar.gz --strip-components=1

`Docker login`

Took 0 min 2 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 2 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

`Docker login`

Took 0 min 9 sec . View more details on here
Description: set +x if command -v host 2>&1 > /dev/null; then host docker.io 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.io 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.io" 2>/dev/null

Log output

Expand to view the last 100 lines of log output

[2021-04-15T18:11:09.724Z] C:\Users\jenkins\workspace\PR-25118-1-c043c572-ce7f-40c7-8ae2-5d46e04137b9\src\github.com\elastic\beats>go clean -modcache 
[2021-04-15T18:11:44.974Z] warn: failed to upgrade pip (ignoring): running "null\build\ve\windows\Scripts\pip install -U pip" failed with exit code 1============================= test session starts =============================
[2021-04-15T18:11:44.974Z] platform win32 -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
[2021-04-15T18:11:44.974Z] rootdir: C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats, configfile: pytest.ini
[2021-04-15T18:11:44.974Z] plugins: rerunfailures-9.0, timeout-1.3.4
[2021-04-15T18:11:44.974Z] timeout: 90.0s
[2021-04-15T18:11:44.974Z] timeout method: thread
[2021-04-15T18:11:44.974Z] timeout func_only: True
[2021-04-15T18:11:44.974Z] collected 348 items
[2021-04-15T18:11:44.974Z] 
[2021-04-15T18:11:44.974Z] tests\system\test_autodiscover.py ss                                     [  0%]
[2021-04-15T18:11:44.974Z] tests\system\test_base.py ......ss                                       [  2%]
[2021-04-15T18:11:44.974Z] tests\system\test_cmd.py ...                                             [  3%]
[2021-04-15T18:11:46.786Z] tests\system\test_container.py ...                                       [  4%]
[2021-04-15T18:12:08.834Z] tests\system\test_crawler.py ......s..........                           [  9%]
[2021-04-15T18:12:09.714Z] tests\system\test_deprecated.py ..                                       [ 10%]
[2021-04-15T18:12:11.485Z] tests\system\test_fields.py ...                                          [ 10%]
[2021-04-15T18:12:13.330Z] tests\system\test_generate.py ...                                        [ 11%]
[2021-04-15T18:12:33.195Z] + gsutil --version
[2021-04-15T18:12:34.635Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-15T18:12:35.163Z] + gcloud auth activate-service-account --key-file ****
[2021-04-15T18:12:35.755Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-15T18:12:36.143Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMDkyYjgxYmVlOTFjNTQ3Nzg3Y2E3Nzc0NmEzY2VjODNmNjhiNGY0Mzc gs://beats-ci-temp/ci/cache/
[2021-04-15T18:12:43.402Z] tests\system\test_harvester.py ......s..............                     [ 17%]
[2021-04-15T18:12:44.581Z] tests\system\test_index_pattern.py ..                                    [ 18%]
[2021-04-15T18:13:11.374Z] tests\system\test_input.py .........ss...........                        [ 24%]
[2021-04-15T18:13:19.030Z] tests\system\test_json.py ..............                                 [ 28%]
[2021-04-15T18:13:20.251Z] tests\system\test_keystore.py ..                                         [ 29%]
[2021-04-15T18:13:20.251Z] tests\system\test_load.py sss                                            [ 30%]
[2021-04-15T18:13:20.251Z] tests\system\test_ml.py ss                                               [ 30%]
[2021-04-15T18:13:20.251Z] tests\system\test_modules.py sssssssssssssssssssssssssssssssssssssssssss [ 43%]
[2021-04-15T18:13:20.524Z] ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 63%]
[2021-04-15T18:13:20.784Z] ssssssssssssssssssssssssss                                               [ 71%]
[2021-04-15T18:13:37.837Z] tests\system\test_multiline.py .........                                 [ 73%]
[2021-04-15T18:13:37.837Z] tests\system\test_pipeline.py s                                          [ 74%]
[2021-04-15T18:13:44.999Z] tests\system\test_processors.py .............                            [ 77%]
[2021-04-15T18:13:45.949Z] tests\system\test_publisher.py .                                         [ 78%]
[2021-04-15T18:13:45.949Z] tests\system\test_redis.py s                                             [ 78%]
[2021-04-15T18:14:15.110Z] tests\system\test_registrar.py sss........ssss......ss..                 [ 85%]
[2021-04-15T18:14:15.110Z] tests\system\test_registrar_upgrade.py sssss                             [ 87%]
[2021-04-15T18:14:37.333Z] tests\system\test_reload_inputs.py .......                               [ 89%]
[2021-04-15T18:14:46.327Z] tests\system\test_reload_modules.py ...s..                               [ 90%]
[2021-04-15T18:14:46.327Z] tests\system\test_setup.py s                                             [ 91%]
[2021-04-15T18:14:57.614Z] tests\system\test_shutdown.py ..s..                                      [ 92%]
[2021-04-15T18:14:58.838Z] tests\system\test_stdin.py ...                                           [ 93%]
[2021-04-15T18:15:00.588Z] tests\system\test_syslog.py ...ssss                                      [ 95%]
[2021-04-15T18:15:03.544Z] tests\system\test_tcp.py .....                                           [ 96%]
[2021-04-15T18:15:06.484Z] tests\system\test_tcp_tls.py ......                                      [ 98%]
[2021-04-15T18:15:06.484Z] tests\system\test_udp.py s                                               [ 98%]
[2021-04-15T18:15:06.484Z] tests\system\test_unix.py ssss                                           [100%]
[2021-04-15T18:15:06.484Z] 
[2021-04-15T18:15:06.484Z] - generated xml file: C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats\filebeat\build\TEST-python-unit.xml -
[2021-04-15T18:15:06.484Z] ============================ slowest 20 durations =============================
[2021-04-15T18:15:06.484Z] 15.85s call     filebeat/tests/system/test_harvester.py::Test::test_close_timeout
[2021-04-15T18:15:06.484Z] 6.08s call     filebeat/tests/system/test_registrar.py::Test::test_restart_state
[2021-04-15T18:15:06.484Z] 5.27s call     filebeat/tests/system/test_shutdown.py::Test::test_shutdown_wait_timeout
[2021-04-15T18:15:06.484Z] 4.60s call     filebeat/tests/system/test_crawler.py::Test::test_encodings
[2021-04-15T18:15:06.484Z] 4.56s call     filebeat/tests/system/test_reload_inputs.py::Test::test_reload_add
[2021-04-15T18:15:06.484Z] 4.37s call     filebeat/tests/system/test_crawler.py::Test::test_multiple_appends
[2021-04-15T18:15:06.484Z] 3.85s call     filebeat/tests/system/test_input.py::Test::test_rotating_close_inactive_low_write_rate
[2021-04-15T18:15:06.484Z] 3.78s call     filebeat/tests/system/test_input.py::Test::test_close_inactive_file_rotation_and_removal2
[2021-04-15T18:15:06.484Z] 3.75s call     filebeat/tests/system/test_multiline.py::Test::test_close_timeout_with_multiline
[2021-04-15T18:15:06.484Z] 3.68s call     filebeat/tests/system/test_input.py::Test::test_harvester_limit
[2021-04-15T18:15:06.484Z] 3.63s call     filebeat/tests/system/test_reload_inputs.py::Test::test_reload_same_config
[2021-04-15T18:15:06.484Z] 3.58s call     filebeat/tests/system/test_reload_inputs.py::Test::test_start_stop_replace
[2021-04-15T18:15:06.484Z] 3.57s call     filebeat/tests/system/test_reload_inputs.py::Test::test_reload_same_input
[2021-04-15T18:15:06.484Z] 3.54s call     filebeat/tests/system/test_registrar.py::Test::test_state_after_rotation_ignore_older
[2021-04-15T18:15:06.484Z] 3.52s call     filebeat/tests/system/test_registrar.py::Test::test_state_after_rotation
[2021-04-15T18:15:06.484Z] 3.49s call     filebeat/tests/system/test_reload_modules.py::Test::test_start_stop
[2021-04-15T18:15:06.484Z] 3.48s call     filebeat/tests/system/test_reload_inputs.py::Test::test_start_stop
[2021-04-15T18:15:06.484Z] 3.28s call     filebeat/tests/system/test_shutdown.py::Test::test_shutdown
[2021-04-15T18:15:06.484Z] 2.88s call     filebeat/tests/system/test_multiline.py::Test::test_timeout
[2021-04-15T18:15:06.484Z] 2.50s call     filebeat/tests/system/test_registrar.py::Test::test_ignore_older_state_clean_inactive
[2021-04-15T18:15:06.484Z] ================ 166 passed, 182 skipped in 209.57s (0:03:29) =================
[2021-04-15T18:15:06.745Z] >> python test: Unit Testing Complete
[2021-04-15T18:15:07.277Z] 
[2021-04-15T18:15:07.277Z] C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-04-15T18:15:10.183Z] 
[2021-04-15T18:15:10.183Z] C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-04-15T18:15:15.469Z] Copy .\filebeat\build into build\filebeat\build
[2021-04-15T18:15:15.469Z] Copy .\filebeat\null\build into build\filebeat\null\build
[2021-04-15T18:15:15.511Z] Running in C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats\build
[2021-04-15T18:15:15.581Z] Recording test results
[2021-04-15T18:15:17.234Z] [Checks API] No suitable checks publisher found.
[2021-04-15T18:15:17.787Z] 
[2021-04-15T18:15:17.787Z] C:\Users\jenkins\workspace\PR-25118-1-9d149756-7a32-4d24-ba28-5aaec6d84074\src\github.com\elastic\beats>go clean -modcache 
[2021-04-15T18:16:56.602Z] + gsutil --version
[2021-04-15T18:16:58.126Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-15T18:16:58.498Z] + gcloud auth activate-service-account --key-file ****
[2021-04-15T18:16:59.078Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-15T18:16:59.682Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwOTJiODFiZWU5MWM1NDc3ODdjYTc3NzQ2YTNjZWM4M2Y2OGI0ZjQzNw gs://beats-ci-temp/ci/cache/
[2021-04-15T18:17:02.738Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-15T18:17:02.899Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25118/src/github.com/elastic/beats
[2021-04-15T18:17:04.442Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25118
[2021-04-15T18:17:04.550Z] [INFO] getVaultSecret: Getting secrets
[2021-04-15T18:17:04.696Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-15T18:17:05.860Z] + chmod 755 generate-build-data.sh
[2021-04-15T18:17:05.861Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/runs/1 FAILURE 2592603
[2021-04-15T18:17:06.411Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/runs/1/steps/?limit=10000 -o steps-info.json
[2021-04-15T18:17:07.755Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25118/runs/1/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9546
Skipped	1972
Total	11518

Add tests for truncated and symlinked files in filestream input (elas…

92b81be

…tic#24425) (cherry picked from commit a9279cd)

kvch added [zube]: In Review backport Team:Elastic-Agent Label for the Agent team labels Apr 15, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021

kvch merged commit e748e45 into elastic:7.x Apr 16, 2021

zube bot added [zube]: Done and removed [zube]: In Review labels Apr 16, 2021

zube bot removed the [zube]: Done label Jul 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files#25118

Cherry-pick #24425 to 7.x: Add tests for truncated and symlinked files#25118
kvch merged 1 commit intoelastic:7.xfrom
kvch:backport_24425_7.x

kvch commented Apr 15, 2021 •

edited by zube bot

Loading

Uh oh!

elasticmachine commented Apr 15, 2021

Uh oh!

elasticmachine commented Apr 15, 2021

Build stats

Test stats 🧪

Trends 🧪

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`untar gsutil`

`Docker login`

`Docker login`

`Docker login`

Test stats 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kvch commented Apr 15, 2021 • edited by zube bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Uh oh!

elasticmachine commented Apr 15, 2021

Uh oh!

elasticmachine commented Apr 15, 2021

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Steps errors

Docker login

Docker login

Docker login

Docker login

Docker login

Docker login

untar gsutil

Docker login

Docker login

Docker login

Log output

💚 Flaky test report

Test stats 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kvch commented Apr 15, 2021 •

edited by zube bot

Loading

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`Docker login`

`untar gsutil`

`Docker login`

`Docker login`

`Docker login`