Skip to content

Cherry-pick #21215 to 7.x: Handle multiple upstreams in ingress-controller#21373

Merged
ChrsMark merged 1 commit intoelastic:7.xfrom
ChrsMark:backport_21215_7.x
Sep 29, 2020
Merged

Cherry-pick #21215 to 7.x: Handle multiple upstreams in ingress-controller#21373
ChrsMark merged 1 commit intoelastic:7.xfrom
ChrsMark:backport_21215_7.x

Conversation

@ChrsMark
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark commented Sep 29, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #21215 to 7.x branch. Original message:

What does this PR do?

This PR tunes ingress-controller's pipeline so as to handle multiple upstreams. We put the multiple lengths, times and status_codes in upstream.response.length_list, upstream.response.time_list and upstream.response.status_code_list fields while we sum time under single field upstream.response.time and we only store last response length in upstream.response.length (with this we also preserve the existing dashboards

beats/filebeat/module/nginx/_meta/kibana/7/dashboard/Filebeat-nginx-ingress-overview.json

Line 1007 in 54ea284

"field": "nginx.ingress_controller.upstream.response.length",
).

In addition, we preserve the multiple upstreams at upstream_ip_list and we only store the last upstream.ip and upstream.port as well as the last status_code.

Sample log:
192.168.64.14 - - [07/Feb/2020:12:02:42 +0000] "GET /v2/some HTTP/1.1" 200 61 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:72.0) Gecko/20100101 Firefox/72.0" 348 0.001 [default-web2-8080] [] 172.17.0.6:8080, 172.17.0.7:8080 61, 100 0.100, 0.004 200, 203 835136ae24486dbb4156dcbe21f5d402
Sample event:

{
        "nginx.ingress_controller.http.request.id": "835136ae24486dbb4156dcbe21f5d402",
        "nginx.ingress_controller.http.request.length": 348,
        "nginx.ingress_controller.http.request.time": 0.001,
        "nginx.ingress_controller.remote_ip_list": [
            "192.168.64.14"
        ],
        "nginx.ingress_controller.upstream.alternative_name": "",
        "nginx.ingress_controller.upstream.ip": "172.17.0.7",
        "nginx.ingress_controller.upstream.name": "default-web2-8080",
        "nginx.ingress_controller.upstream.port": 8080,
        "nginx.ingress_controller.upstream.response.length": 100,
        "nginx.ingress_controller.upstream.response.length_list": [
            "61",
            "100"
        ],
        "nginx.ingress_controller.upstream.response.status_code": 203,
        "nginx.ingress_controller.upstream.response.status_code_list": [
            "200",
            "203"
        ],
        "nginx.ingress_controller.upstream.response.time": 0.104,
        "nginx.ingress_controller.upstream.response.time_list": [
            "0.100",
            "0.004"
        ],
        "nginx.ingress_controller.upstream_address_list": [
            "172.17.0.6:8080",
            "172.17.0.7:8080"
        ],
}

Why is it important?

So as to handle cases where nginx-ingress retries multiple upstreams.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@ChrsMark ChrsMark added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Sep 29, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 29, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 29, 2020
@ChrsMark ChrsMark self-assigned this Sep 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 29, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21373 opened]

  • Start Time: 2020-09-29T09:23:09.336+0000

  • Duration: 120 min 59 sec

Test stats 🧪

Test Results
Failed 7
Passed 20092
Skipped 1877
Total 21976

Test errors

Expand to view the tests failures

  • Name: Build&Test / x-pack/elastic-agent-macos-macosx / TestFleetGateway – application

    • Age: 1
    • Duration: 0
    • Error Details: Failed

  • Name: Build&Test / x-pack/elastic-agent-macos-macosx / TestFleetGateway/send_no_event_and_receive_no_action – application

    • Age: 1
    • Duration: 0
    • Error Details: Failed

  • Name: Build&Test / x-pack/elastic-agent-macos-macosx / TestFleetGateway/Successfully_connects_and_receives_a_series_of_actions – application

    • Age: 1
    • Duration: 0
    • Error Details: Failed

  • Name: Build&Test / x-pack/elastic-agent-macos-macosx / TestFleetGateway/Periodically_communicates_with_Fleet – application

    • Age: 1
    • Duration: 0
    • Error Details: Failed

  • Name: Build&Test / x-pack/metricbeat-build / TestFetch – ec2

    • Age: 1
    • Duration: 19.53
    • Error Details: Failed

  • Name: Build&Test / libbeat-build / TestOutputReload – pipeline

    • Age: 1
    • Duration: 70.89
    • Error Details: Failed

  • Name: Build&Test / libbeat-build / TestOutputReload/client – pipeline

    • Age: 1
    • Duration: 36.69
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: mage build test

    • Description: mage build test

    • Duration: 24 min 51 sec

    • Start Time: 2020-09-29T09:50:08.926+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-29T10:14:15.420+0000

    • log

  • Name: mage build unitTest

    • Description: mage build unitTest

    • Duration: 11 min 55 sec

    • Start Time: 2020-09-29T11:08:04.555+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: FAILURE

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-29T11:19:13.482+0000

    • log

  • Name: mage build test

    • Description: mage build test

    • Duration: 24 min 39 sec

    • Start Time: 2020-09-29T09:59:10.575+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-29T10:23:01.853+0000

    • log

  • Name: Process JUnit reports with runbld

    • Description:

    • Duration: 0 min 18 sec

    • Start Time: 2020-09-29T11:22:49.426+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-09-29T11:22:31.071Z] + python .ci/scripts/pre_archive_test.py
[2020-09-29T11:22:31.240Z] Copy ./x-pack/functionbeat/build into build/x-pack/functionbeat/build
[2020-09-29T11:22:31.257Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/build
[2020-09-29T11:22:31.307Z] WARNING: Unknown parameter(s) found for class type 'hudson.tasks.junit.pipeline.JUnitResultsStep': id,stashedTestReports
[2020-09-29T11:22:31.364Z] Recording test results
[2020-09-29T11:22:32.019Z] Stashed 2 file(s)
[2020-09-29T11:22:32.094Z] Archiving artifacts
[2020-09-29T11:22:33.295Z] + python .ci/scripts/search_system_tests.py
[2020-09-29T11:22:33.538Z] [INFO] system-tests='build/x-pack/functionbeat/build/system-tests'. If no empty then let's create a tarball
[2020-09-29T11:22:34.378Z] + tar --version
[2020-09-29T11:22:35.429Z] + tar --exclude=x-pack-functionbeat--system-tests-darwin.tgz -czf x-pack-functionbeat--system-tests-darwin.tgz build/x-pack/functionbeat/build/system-tests
[2020-09-29T11:22:35.680Z] Archiving artifacts
[2020-09-29T11:22:36.862Z] Docker machine "default" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
[2020-09-29T11:22:36.862Z] Docker machine "default" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
[2020-09-29T11:22:36.862Z] Client: Docker Engine - Community
[2020-09-29T11:22:36.862Z]  Version:           19.03.1
[2020-09-29T11:22:36.862Z]  API version:       1.40
[2020-09-29T11:22:36.862Z]  Go version:        go1.12.5
[2020-09-29T11:22:36.862Z]  Git commit:        74b1e89
[2020-09-29T11:22:36.862Z]  Built:             Thu Jul 25 21:18:17 2019
[2020-09-29T11:22:36.862Z]  OS/Arch:           darwin/amd64
[2020-09-29T11:22:36.862Z]  Experimental:      false
[2020-09-29T11:22:36.862Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-09-29T11:22:36.862Z] It requires Docker daemon to be installed and running
[2020-09-29T11:22:44.085Z] Post stage
[2020-09-29T11:22:44.111Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats
[2020-09-29T11:22:44.151Z] Archiving artifacts
[2020-09-29T11:22:44.467Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats
[2020-09-29T11:22:44.503Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/uncategorized-1601372686354
[2020-09-29T11:22:44.606Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/libbeat-stress-tests-1601373023280
[2020-09-29T11:22:44.702Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/winlogbeat-crosscompile-1601373098446
[2020-09-29T11:22:44.797Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/auditbeat-crosscompile-1601373125732
[2020-09-29T11:22:44.896Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-elastic-agent-build-1601373165551
[2020-09-29T11:22:44.997Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-dockerlogbeat-build-1601373187434
[2020-09-29T11:22:45.106Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/journalbeat-unitTest-1601373202434
[2020-09-29T11:22:45.206Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-functionbeat-build-1601373285384
[2020-09-29T11:22:45.306Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/libbeat-crosscompile-1601373305278
[2020-09-29T11:22:45.405Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/packetbeat-build-1601373399817
[2020-09-29T11:22:45.502Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1601373428725
[2020-09-29T11:22:45.596Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/metricbeat-unitTest-1601373527086
[2020-09-29T11:22:45.693Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/auditbeat-windows-windows-2019-1601373583193
[2020-09-29T11:22:45.793Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/heartbeat-windows-windows-2019-1601373586533
[2020-09-29T11:22:45.890Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/heartbeat-build-1601373600272
[2020-09-29T11:22:45.991Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019-1601373663845
[2020-09-29T11:22:46.094Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-auditbeat-build-1601373673046
[2020-09-29T11:22:46.198Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/metricbeat-crosscompile-1601373673098
[2020-09-29T11:22:46.300Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-winlogbeat-build-windows-2019-1601373695097
[2020-09-29T11:22:46.409Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/generator-beat-test-1601373704867
[2020-09-29T11:22:46.503Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/auditbeat-build-1601373734559
[2020-09-29T11:22:46.598Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019-1601373789462
[2020-09-29T11:22:46.697Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/filebeat-windows-windows-2019-1601373792023
[2020-09-29T11:22:46.791Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-libbeat-build-1601373795525
[2020-09-29T11:22:46.884Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/generator-metricbeat-test-1601373842730
[2020-09-29T11:22:46.984Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/packetbeat-windows-windows-2019-1601373843155
[2020-09-29T11:22:47.084Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/winlogbeat-windows-windows-2019-1601373863124
[2020-09-29T11:22:47.185Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1601373874409
[2020-09-29T11:22:47.286Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/metricbeat-windows-windows-2019-1601374074421
[2020-09-29T11:22:47.389Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019-1601374087575
[2020-09-29T11:22:47.492Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/libbeat-build-1601374445741
[2020-09-29T11:22:47.599Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/filebeat-build-1601374772754
[2020-09-29T11:22:47.704Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/metricbeat-goIntegTest-1601374952039
[2020-09-29T11:22:47.810Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-metricbeat-build-1601374974730
[2020-09-29T11:22:47.913Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/metricbeat-pythonIntegTest-1601375035818
[2020-09-29T11:22:48.026Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-filebeat-build-1601375085800
[2020-09-29T11:22:48.139Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/generator-macos-metricbeat-macosx-1601377379896
[2020-09-29T11:22:48.255Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/generator-macos-beat-macosx-1601377551921
[2020-09-29T11:22:48.365Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/filebeat-macos-macosx-1601377599593
[2020-09-29T11:22:48.471Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/heartbeat-macos-macosx-1601377632087
[2020-09-29T11:22:48.576Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/auditbeat-macos-macosx-1601377734025
[2020-09-29T11:22:48.679Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-auditbeat-macos-macosx-1601377819584
[2020-09-29T11:22:48.781Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/metricbeat-macos-macosx-1601377854540
[2020-09-29T11:22:48.884Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-filebeat-macos-macosx-1601377949424
[2020-09-29T11:22:48.984Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-metricbeat-macos-macosx-1601378225009
[2020-09-29T11:22:49.083Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-elastic-agent-macos-macosx-1601378343996
[2020-09-29T11:22:49.195Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/packetbeat-macos-macosx-1601378353281
[2020-09-29T11:22:49.296Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats/x-pack-functionbeat-macos-macosx-1601378551945
[2020-09-29T11:22:49.721Z] + cat
[2020-09-29T11:22:49.721Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-09-29T11:22:49.721Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-29T11:22:56.309Z] runbld>>> runbld started
[2020-09-29T11:22:56.309Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-29T11:22:58.849Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-29T11:22:58.849Z] runbld>>> Matches in the system config:
[2020-09-29T11:22:58.849Z] runbld>>> - Matched ^elastic\+beats
[2020-09-29T11:22:58.849Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-29T11:23:00.231Z] runbld>>> Debug logging enabled.
[2020-09-29T11:23:00.231Z] runbld>>> Storing result
[2020-09-29T11:23:00.231Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-29T11:23:00.231Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200929112259-787819E7
[2020-09-29T11:23:00.231Z] runbld>>> Adding system facts.
[2020-09-29T11:23:01.168Z] runbld>>> Sending debug log to infra-root+runbld-debug@e***.co
[2020-09-29T11:23:02.106Z] runbld>>> Error: The source clone was not found in /var/lib/jenkins/workspace/Beats_beats_PR-21373/src/github.com/elastic/beats.  The most common cause is that Jenkins and runbld are configured with different working directories (referred to as 'basedir' in JJB and 'cwd' in runbld config).
[2020-09-29T11:23:07.410Z] ERROR: runbld post build action failed.
[2020-09-29T11:23:07.410Z] ERROR: script returned exit code 1
[2020-09-29T11:23:07.628Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21373
[2020-09-29T11:23:07.720Z] [INFO] getVaultSecret: Getting secrets
[2020-09-29T11:23:07.816Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-29T11:23:08.664Z] + chmod 755 generate-build-data.sh
[2020-09-29T11:23:08.665Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21373/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21373/runs/1 FAILURE 7199053
[2020-09-29T11:23:08.665Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21373/runs/1/steps/?limit=10000 -o steps-info.json

@ChrsMark ChrsMark merged commit a98d839 into elastic:7.x Sep 29, 2020
@zube zube bot removed the [zube]: Done label Dec 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ChrsMark ChrsMark

Labels

backport Team:Platforms Label for the Integrations - Platforms team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ChrsMark @elasticmachine @kaiyan-sheng