Skip to content

Do not overwrite content type if it is multipart in Kibana loader#27424

Merged
kvch merged 3 commits intoelastic:masterfrom
kvch:fix-libbeat-saved-objects-api-dashboard
Aug 18, 2021
Merged

Do not overwrite content type if it is multipart in Kibana loader#27424
kvch merged 3 commits intoelastic:masterfrom
kvch:fix-libbeat-saved-objects-api-dashboard

Conversation

@kvch
Copy link
Copy Markdown
Contributor

@kvch kvch commented Aug 17, 2021

What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.

Why is it important?

Index pattern and dashboard loading was not working.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Closes #27418
Closes #27417

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 17, 2021
@kvch kvch requested review from andrewkroh and jsoriano August 17, 2021 15:42
@kvch kvch added backport-v7.15.0 Automated backport with mergify Team:Elastic-Agent Label for the Agent team labels Aug 17, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 17, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 17, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-18T10:17:14.402+0000

  • Duration: 101 min 11 sec

  • Commit: 823527c

Test stats 🧪

Test Results
Failed 0
Passed 53258
Skipped 5320
Total 58578

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 2

Expand to view the steps failures

filebeat-goIntegTest - mage goIntegTest
  • Took 15 min 37 sec . View more details on here
  • Description: mage goIntegTest
x-pack/metricbeat-unitTest - mage build unitTest
  • Took 4 min 45 sec . View more details on here
  • Description: mage build unitTest

Log output

Expand to view the last 100 lines of log output

[2021-08-18T11:53:18.847Z] + gsutil -m -q cp ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwODIzNTI3YzdiOWVkY2RiNjQxMGJlZjQ1N2Q0MTlmNjEwZjAwYjJjNw gs://beats-ci-temp/ci/cache/
[2021-08-18T11:53:59.091Z] + gsutil --version
[2021-08-18T11:54:00.531Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-08-18T11:54:00.860Z] + gcloud auth activate-service-account --key-file ****
[2021-08-18T11:54:01.435Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-08-18T11:54:02.023Z] + gsutil -m -q cp cGFja2V0YmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTA4MjM1MjdjN2I5ZWRjZGI2NDEwYmVmNDU3ZDQxOWY2MTBmMDBiMmM3 gs://beats-ci-temp/ci/cache/
[2021-08-18T11:54:04.018Z] tests\system\test_processors.py ........                                 [ 13%]
[2021-08-18T11:54:32.555Z] tests\system\test_reload.py ...                                          [ 14%]
[2021-08-18T11:54:32.817Z] tests\system\test_template.py s                                          [ 15%]
[2021-08-18T11:54:42.829Z] tests\system\test_timeseries.py .                                        [ 15%]
[2021-08-18T11:54:42.829Z] module\aerospike\test_aerospike.py s                                     [ 16%]
[2021-08-18T11:54:42.830Z] module\apache\test_apache.py ss                                          [ 17%]
[2021-08-18T11:54:42.830Z] module\ceph\test_ceph.py ssssssssssssssssssss                            [ 25%]
[2021-08-18T11:54:42.830Z] module\consul\test_consul.py ss                                          [ 25%]
[2021-08-18T11:54:42.830Z] module\couchbase\test_couchbase.py sss                                   [ 27%]
[2021-08-18T11:54:42.830Z] module\couchdb\test_couchdb.py s                                         [ 27%]
[2021-08-18T11:54:42.830Z] module\docker\test_docker.py sssssssss                                   [ 31%]
[2021-08-18T11:54:42.830Z] module\dropwizard\test_dropwizard.py sss                                 [ 32%]
[2021-08-18T11:54:42.830Z] module\elasticsearch\test_elasticsearch.py sssssssssss                   [ 36%]
[2021-08-18T11:54:42.830Z] module\envoyproxy\test_envoyproxy.py ss                                  [ 37%]
[2021-08-18T11:54:42.830Z] module\etcd\test_etcd.py ssssssss                                        [ 40%]
[2021-08-18T11:54:42.830Z] module\golang\test_golang.py s                                           [ 41%]
[2021-08-18T11:54:42.830Z] module\haproxy\test_haproxy.py sssssssssssssssssssssssss                 [ 51%]
[2021-08-18T11:54:50.981Z] module\http\test_http.py s.                                              [ 52%]
[2021-08-18T11:54:50.981Z] module\jolokia\test_jolokia.py ssss                                      [ 53%]
[2021-08-18T11:54:50.981Z] module\kafka\test_kafka.py ssssssssssssssssssss                          [ 61%]
[2021-08-18T11:54:50.981Z] module\kibana\test_kibana.py ss                                          [ 62%]
[2021-08-18T11:54:50.982Z] module\logstash\test_logstash.py sss                                     [ 63%]
[2021-08-18T11:54:50.982Z] module\memcached\test_memcached.py s                                     [ 64%]
[2021-08-18T11:54:50.982Z] module\mongodb\test_mongodb.py s                                         [ 64%]
[2021-08-18T11:54:50.982Z] module\munin\test_munin.py s                                             [ 65%]
[2021-08-18T11:54:50.982Z] module\mysql\test_mysql.py sssssss                                       [ 68%]
[2021-08-18T11:54:50.982Z] module\nats\test_nats.py ssssssssssssssssss                              [ 75%]
[2021-08-18T11:54:50.982Z] module\openmetrics\test_openmetrics.py s                                 [ 75%]
[2021-08-18T11:54:50.982Z] module\php_fpm\test_phpfpm.py s                                          [ 76%]
[2021-08-18T11:54:51.252Z] module\postgresql\test_postgresql.py ssssssssssssssssssss                [ 84%]
[2021-08-18T11:54:51.252Z] module\prometheus\test_prometheus.py sss                                 [ 85%]
[2021-08-18T11:54:51.252Z] module\redis\test_redis.py ssssssssssss                                  [ 90%]
[2021-08-18T11:56:46.880Z] module\system\test_system.py .....s..s...s.s.                            [ 96%]
[2021-08-18T11:56:46.880Z] module\traefik\test_traefik.py s                                         [ 97%]
[2021-08-18T11:56:46.880Z] module\uwsgi\test_uwsgi.py ss                                            [ 97%]
[2021-08-18T11:56:46.880Z] module\vsphere\test_vsphere.py sss                                       [ 99%]
[2021-08-18T11:56:46.880Z] module\zookeeper\test_zookeeper.py ss                                    [100%]
[2021-08-18T11:56:46.880Z] 
[2021-08-18T11:56:46.880Z] ============================== warnings summary ===============================
[2021-08-18T11:56:46.880Z] tests\system\test_lightmodules.py:57
[2021-08-18T11:56:46.880Z]   C:\Users\jenkins\workspace\PR-27424-4-aa40aab3-a1df-47a5-a21b-20525004187b\src\github.com\elastic\beats\metricbeat\tests\system\test_lightmodules.py:57: PytestCollectionWarning: cannot collect test class 'TestHTTPHandler' because it has a __init__ constructor (from: metricbeat/tests/system/test_lightmodules.py)
[2021-08-18T11:56:46.880Z]     class TestHTTPHandler(http.server.BaseHTTPRequestHandler):
[2021-08-18T11:56:46.880Z] 
[2021-08-18T11:56:46.880Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-08-18T11:56:46.880Z] - generated xml file: C:\Users\jenkins\workspace\PR-27424-4-aa40aab3-a1df-47a5-a21b-20525004187b\src\github.com\elastic\beats\metricbeat\build\TEST-python-unit.xml -
[2021-08-18T11:56:46.880Z] ============================ slowest 20 durations =============================
[2021-08-18T11:56:46.880Z] 17.00s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_list
[2021-08-18T11:56:46.880Z] 16.97s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_enable
[2021-08-18T11:56:46.880Z] 16.91s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_disable
[2021-08-18T11:56:46.880Z] 12.57s call     metricbeat/tests/system/test_reload.py::Test::test_start_stop
[2021-08-18T11:56:46.880Z] 12.47s call     metricbeat/tests/system/test_config.py::ConfigTest::test_service_name
[2021-08-18T11:56:46.880Z] 10.89s call     metricbeat/module/system/test_system.py::Test::test_process
[2021-08-18T11:56:46.880Z] 10.81s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_test_error
[2021-08-18T11:56:46.880Z] 10.65s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_condition
[2021-08-18T11:56:46.880Z] 9.60s call     metricbeat/tests/system/test_reload.py::Test::test_reload
[2021-08-18T11:56:46.880Z] 9.30s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_complex_condition
[2021-08-18T11:56:46.880Z] 9.18s call     metricbeat/tests/system/test_processors.py::Test::test_include_fields
[2021-08-18T11:56:46.880Z] 9.08s call     metricbeat/tests/system/test_processors.py::Test::test_multiple_actions
[2021-08-18T11:56:46.880Z] 9.08s call     metricbeat/tests/system/test_processors.py::Test::test_contradictory_multiple_actions
[2021-08-18T11:56:46.880Z] 9.07s call     metricbeat/tests/system/test_processors.py::Test::test_dropfields_with_condition
[2021-08-18T11:56:46.880Z] 9.07s call     metricbeat/tests/system/test_timeseries.py::TestTimeseries::test_enable_timeseries
[2021-08-18T11:56:46.880Z] 8.99s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-08-18T11:56:46.880Z] 8.91s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-08-18T11:56:46.880Z] 8.90s call     metricbeat/tests/system/test_base.py::Test::test_export_config
[2021-08-18T11:56:46.880Z] 8.88s call     metricbeat/module/http/test_http.py::Test::test_server
[2021-08-18T11:56:46.881Z] 8.88s call     metricbeat/tests/system/test_base.py::Test::test_export_template
[2021-08-18T11:56:46.881Z] =========== 40 passed, 207 skipped, 1 warning in 401.68s (0:06:41) ============
[2021-08-18T11:56:47.141Z] >> python test: Unit Testing Complete
[2021-08-18T11:56:47.536Z] 
[2021-08-18T11:56:47.537Z] C:\Users\jenkins\workspace\PR-27424-4-aa40aab3-a1df-47a5-a21b-20525004187b\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-08-18T11:56:49.757Z] 
[2021-08-18T11:56:49.757Z] C:\Users\jenkins\workspace\PR-27424-4-aa40aab3-a1df-47a5-a21b-20525004187b\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-08-18T11:56:51.141Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-08-18T11:56:51.141Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-08-18T11:56:51.160Z] Running in C:\Users\jenkins\workspace\PR-27424-4-aa40aab3-a1df-47a5-a21b-20525004187b\src\github.com\elastic\beats\build
[2021-08-18T11:56:51.180Z] Recording test results
[2021-08-18T11:56:55.036Z] [Checks API] No suitable checks publisher found.
[2021-08-18T11:56:55.417Z] 
[2021-08-18T11:56:55.417Z] C:\Users\jenkins\workspace\PR-27424-4-aa40aab3-a1df-47a5-a21b-20525004187b\src\github.com\elastic\beats>go clean -modcache 
[2021-08-18T11:58:18.476Z] + gsutil --version
[2021-08-18T11:58:19.912Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-08-18T11:58:20.235Z] + gcloud auth activate-service-account --key-file ****
[2021-08-18T11:58:20.809Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-08-18T11:58:21.402Z] + gsutil -m -q cp bWV0cmljYmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTA4MjM1MjdjN2I5ZWRjZGI2NDEwYmVmNDU3ZDQxOWY2MTBmMDBiMmM3 gs://beats-ci-temp/ci/cache/
[2021-08-18T11:58:23.512Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-08-18T11:58:23.560Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-08-18T11:58:23.643Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27424/src/github.com/elastic/beats
[2021-08-18T11:58:24.986Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-27424
[2021-08-18T11:58:25.049Z] [INFO] getVaultSecret: Getting secrets
[2021-08-18T11:58:25.107Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-08-18T11:58:25.860Z] + chmod 755 generate-build-data.sh
[2021-08-18T11:58:25.861Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27424/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27424/runs/4 FAILURE 6071190
[2021-08-18T11:58:25.861Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27424/runs/4/steps/?limit=10000 -o steps-info.json
[2021-08-18T11:58:37.958Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27424/runs/4/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 53258
Skipped 5320
Total 58578

Copy link
Copy Markdown
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@kvch
Copy link
Copy Markdown
Contributor Author

kvch commented Aug 18, 2021

jenkins run tests

Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mention in the PR description index templates? Should this be index pattern?


}

func TestNewKibanaClientWithMultipartData(t *testing.T) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ on having tests 👍

@kvch
Copy link
Copy Markdown
Contributor Author

kvch commented Aug 18, 2021

I fixed the PR description. I indeed meant index pattern.

@kvch kvch added the backport-v7.16.0 Automated backport with mergify label Aug 18, 2021
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@kvch kvch merged commit 47c0c72 into elastic:master Aug 18, 2021
mergify bot pushed a commit that referenced this pull request Aug 18, 2021
…7424)

## What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.

(cherry picked from commit 47c0c72)
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Aug 18, 2021

@kvch 7.14 branch has been created, please make sure it is also backported there.

@ruflin ruflin added the backport-v7.14.0 Automated backport with mergify label Aug 18, 2021
mergify bot pushed a commit that referenced this pull request Aug 18, 2021
…7424)

## What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.

(cherry picked from commit 47c0c72)

# Conflicts:
#	libbeat/kibana/client.go
mergify bot pushed a commit that referenced this pull request Aug 18, 2021
…7424)

## What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.

(cherry picked from commit 47c0c72)
@ruflin ruflin removed the backport-v7.14.0 Automated backport with mergify label Aug 18, 2021
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Aug 18, 2021

@kvch Confusion on my end. I created the 7.15 branch and please double check it also went there and not only into 7.x

kvch added a commit that referenced this pull request Aug 18, 2021
…7424) (#27461)

## What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.

(cherry picked from commit 47c0c72)

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
kvch added a commit that referenced this pull request Aug 19, 2021
…7424) (#27464)

## What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.

(cherry picked from commit 47c0c72)

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
@bhavyarm
Copy link
Copy Markdown

This didn't make BC1 for 7.15.0 - https://staging.elastic.co/7.15.0-d9929120/summary-7.15.0.html thanks!

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
…astic#27424)

## What does this PR do?

This PR fixes dashboard loading. Kibana loader overwrites every content type to application/json. However, for multipart file transfer it has to be set to multipart/formdata and application/ndjson. Otherwise, loading the assets fails.

Also, a module file has not been updated with new dashboard names.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify Team:Elastic-Agent Label for the Agent team

Projects

None yet

6 participants