Skip to content

Dynatrace Output plugin not able to push metrics to 2 separate DT environments. #8328

Closed
#8329
Closed
Dynatrace Output plugin not able to push metrics to 2 separate DT environments.#8328
#8329
Labels
bugunexpected problem or unintended behavior
@PavanVasist

Description

@PavanVasist
PavanVasist
opened on Oct 28, 2020

Relevant telegraf.conf:

    [agent]
      collection_jitter = "0s"
      debug = true
      flush_interval = "10s"
      flush_jitter = "0s"
      hostname = "$HOSTNAME"
      interval = "30s"
      logfile = ""
      metric_batch_size = 10000
      metric_buffer_limit = 20000
      omit_hostname = false
      precision = ""
      quiet = false
      round_interval = true
    [[processors.enum]]
      [[processors.enum.mapping]]
        dest = "status_code"
    [[outputs.dynatrace]]
      api_token = "<api-token>"
      insecure_skip_verify = true
      prefix = "prefix1."
      url = "https://<host>-managed.com/e/<env-id>/api/v2/metrics/ingest/"
    [[outputs.dynatrace]]
      api_token = "<api-token>"
      insecure_skip_verify = true
      prefix = "prefix2."
      url = "https://<host>-managed.com/e/<env-id>/api/v2/metrics/ingest/"

    [[outputs.health]]
      service_address = "http://:8888"
      [[outputs.health.compares]]
        field = "buffer_size"
        lt = 5000.0
      [[outputs.health.contains]]
        field = "buffer_size"
    [[inputs.prometheus]]
      urls = [
        "http://istiod.istio-system.svc.cluster.local:15014/metrics",
        "http://prometheus-server.monitoring.svc.cluster.local:80/metrics"
      ]
    [[inputs.postgresql]]
      address = "postgres://<username>:<password>@postgres.postgres.svc.cluster.local:5432/postgresdb"

System info: Using https://github.com/influxdata/telegraf/releases/tag/v1.16.0

Docker - pavanvasisht/telegraf:1.16.1

Steps to reproduce:

  1. Configure one of the input plugin - prometheus.
  2. In the outputs, setup 2 DT cluster environments.
  3. Deploy the configuration to a K8s cluster using helm chart
  4. The values.yaml file used for deploying, PFB -
image:
  repo: "pavanvasisht/telegraf"
  tag: "1.16.1"
  pullPolicy: Always
resources:
  requests:
    memory: 512Mi
    cpu: 500m
  limits:
    memory: 1024Mi
    cpu: 1000m
config:
  agent:
    interval: "30s"
    round_interval: true
    metric_batch_size: 10000
    metric_buffer_limit: 20000
    collection_jitter: "0s"
    flush_interval: "10s"
    flush_jitter: "0s"
    precision: ""
    debug: true
    quiet: false
    logfile: ""
    hostname: "$HOSTNAME"
    omit_hostname: false
  outputs:
    - dynatrace:
        url: "https://<host>-managed.com/e/4<env-id>/api/v2/metrics/ingest/"
        api_token: "<api-token>"
        prefix: "prefix1."
        insecure_skip_verify: true
    - dynatrace:
        url: "https://<host>-managed.com/e/<env-id>/api/v2/metrics/ingest/"
        api_token: "<api-token>"
        prefix: "prefix2."
        insecure_skip_verify: true

  inputs:
    - prometheus:
        urls: 
          - "http://istiod.istio-system.svc.cluster.local:15014/metrics" 
          - "http://prometheus-server.monitoring.svc.cluster.local:80/metrics"     
        metric_version: 2
    - postgresql:
        address: "postgres://<username>:<password>@postgres.postgres.svc.cluster.local:5432/postgresdb"

Expected behavior:

The metrics scraped from prometheus end points needs to be pushed to 2 configured DT environments.

Actual behavior:

The metrics scraped not not being pushed to any of the configured environments.
We believe that it's happening because of huge volume of metrics coming from the scraped URLs.
Because when postgresql was configured as input to scrape, it was able to successfully push for both the configured environments

Additional info:

Log output -

2020-10-28T03:17:11Z I! Starting Telegraf 1.16.0
2020-10-28T03:17:11Z I! Using config file: /etc/telegraf/telegraf.conf
2020-10-28T03:17:11Z I! Loaded inputs: internal postgresql prometheus
2020-10-28T03:17:11Z I! Loaded aggregators:
2020-10-28T03:17:11Z I! Loaded processors: enum
2020-10-28T03:17:11Z I! Loaded outputs: dynatrace (2x) health
2020-10-28T03:17:11Z I! Tags enabled: host=telegraf-polling-service
2020-10-28T03:17:11Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"telegraf-polling-service", Flush Interval:10s
2020-10-28T03:17:11Z D! [agent] Initializing plugins
2020-10-28T03:17:11Z D! [agent] Connecting outputs
2020-10-28T03:17:11Z D! [agent] Attempting connection to [outputs.dynatrace]
2020-10-28T03:17:11Z D! [agent] Successfully connected to outputs.dynatrace
2020-10-28T03:17:11Z D! [agent] Attempting connection to [outputs.dynatrace]
2020-10-28T03:17:11Z D! [agent] Successfully connected to outputs.dynatrace
2020-10-28T03:17:11Z D! [agent] Attempting connection to [outputs.health]
2020-10-28T03:17:11Z I! [outputs.health] Listening on http://[::]:8888
2020-10-28T03:17:11Z D! [agent] Successfully connected to outputs.health
2020-10-28T03:17:11Z D! [agent] Starting service inputs
2020-10-28T03:17:21Z D! [outputs.health] Buffer fullness: 0 / 20000 metrics
2020-10-28T03:17:21Z D! [outputs.dynatrace] Buffer fullness: 0 / 20000 metrics
2020-10-28T03:17:21Z D! [outputs.dynatrace] Buffer fullness: 0 / 20000 metrics
2020-10-28T03:17:31Z D! [outputs.health] Wrote batch of 1137 metrics in 100.786µs
2020-10-28T03:17:31Z D! [outputs.health] Buffer fullness: 0 / 20000 metrics
fatal error: concurrent map read and map write

goroutine 150 [running]:
runtime.throw(0x2e26c5b, 0x21)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0009b1ab8 sp=0xc0009b1a88 pc=0x437832
runtime.mapaccess2_faststr(0x2807120, 0xc000400ed0, 0xc0003d43c0, 0x12d, 0xfb, 0xc0003d43c0)
	/usr/local/go/src/runtime/map_faststr.go:116 +0x4a5 fp=0xc0009b1b28 sp=0xc0009b1ab8 pc=0x4144e5
github.com/influxdata/telegraf/plugins/outputs/dynatrace.(*Dynatrace).Write(0xc00017a300, 0xc0001b0800, 0x471, 0x471, 0xc000458ed8, 0x822552)
	/go/src/github.com/influxdata/telegraf/plugins/outputs/dynatrace/dynatrace.go:201 +0x5ac fp=0xc0009b1e68 sp=0xc0009b1b28 pc=0x22446cc
github.com/influxdata/telegraf/models.(*RunningOutput).write(0xc000126200, 0xc0001b0800, 0x471, 0x471, 0x471, 0x2f31d00)
	/go/src/github.com/influxdata/telegraf/models/running_output.go:246 +0x99 fp=0xc0009b1ee8 sp=0xc0009b1e68 pc=0x828f59
github.com/influxdata/telegraf/models.(*RunningOutput).Write(0xc000126200, 0x41ae3000, 0x66c000)
	/go/src/github.com/influxdata/telegraf/models/running_output.go:203 +0x1fe fp=0xc0009b1f78 sp=0xc0009b1ee8 pc=0x828b5e
github.com/influxdata/telegraf/models.(*RunningOutput).Write-fm(0x4413d6, 0x2f35f98)
	/go/src/github.com/influxdata/telegraf/models/running_output.go:182 +0x2a fp=0xc0009b1fa0 sp=0xc0009b1f78 pc=0xa07aea
github.com/influxdata/telegraf/agent.(*Agent).flushOnce.func1(0xc0001507e0, 0xc000882960)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:847 +0x27 fp=0xc0009b1fd0 sp=0xc0009b1fa0 pc=0xa06487
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0009b1fd8 sp=0xc0009b1fd0 pc=0x470561
created by github.com/influxdata/telegraf/agent.(*Agent).flushOnce
	/go/src/github.com/influxdata/telegraf/agent/agent.go:846 +0x7a

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0001c90b8)
	/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0001c90b0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/influxdata/telegraf/agent.(*Agent).Run(0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x1, 0x1)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:198 +0x5b9
main.runAgent(0x32b5a00, 0xc0001f90c0, 0x4842350, 0x0, 0x0, 0x4842350, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:207 +0x926
main.reloadLoop(0x4842350, 0x0, 0x0, 0x4842350, 0x0, 0x0, 0xc000809d10, 0x0, 0x0, 0xc000809d10, ...)
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:110 +0x267
main.run(...)
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf_posix.go:7
main.main()
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:361 +0x71e

goroutine 70 [select]:
golang.org/x/net/http2.awaitRequestCancel(0xc000228100, 0xc0002363c0, 0x0, 0x2)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:309 +0x125
golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc0002246e0, 0xc000228100)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:335 +0x45
created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:2029 +0x768

goroutine 17 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000137130)
	/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
	/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x57

goroutine 8 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:150 +0x45

goroutine 9 [select]:
main.reloadLoop.func1(0xc00007ef60, 0xc00007d490, 0xc000353870)
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:97 +0xae
created by main.reloadLoop
	/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:96 +0x1fe

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0x7f9647fdfc18, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc000126398, 0x72, 0x0, 0x0, 0x2db5de7)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000126380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000126380, 0xc000616db8, 0xc00006ac00, 0x7f966ec3e7d0)
	/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc0001e76e0, 0xc000616df8, 0x40e438, 0x30)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0001e76e0, 0x2b7cca0, 0xc000350a80, 0x27e1460, 0x47c0550)
	/usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc000178460, 0x32a96c0, 0xc0001e76e0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2937 +0x266
github.com/influxdata/telegraf/plugins/outputs/health.(*Health).Connect.func1(0xc0003f4160, 0x32a96c0, 0xc0001e76e0)
	/go/src/github.com/influxdata/telegraf/plugins/outputs/health/health.go:161 +0x7f
created by github.com/influxdata/telegraf/plugins/outputs/health.(*Health).Connect
	/go/src/github.com/influxdata/telegraf/plugins/outputs/health/health.go:159 +0x26a

goroutine 11 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000224720, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000224710)
	/usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc000224708, 0xc000053d48, 0x4, 0x4, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc0002246e0, 0xc000053d48, 0x4, 0x4, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:2083 +0xaf
io.ReadAtLeast(0x7f9647dce9d8, 0xc0002246e0, 0xc000053d48, 0x4, 0x4, 0x4, 0x23, 0xee, 0xc00058dda8)
	/usr/local/go/src/io/io.go:314 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:333
github.com/ericchiang/k8s.(*watcherPB).next(0xc000277710, 0x325e520, 0xc000220b40, 0x329aa40, 0xc000220b40)
	/go/pkg/mod/github.com/ericchiang/k8s@v1.2.0/watch.go:100 +0xb5
github.com/ericchiang/k8s.(*watcherPB).Next(0xc000277710, 0x325e520, 0xc000220b40, 0xc0002b18a1, 0x9, 0x325e520, 0xc000220060)
	/go/pkg/mod/github.com/ericchiang/k8s@v1.2.0/watch.go:81 +0x72
github.com/ericchiang/k8s.(*Watcher).Next(...)
	/go/pkg/mod/github.com/ericchiang/k8s@v1.2.0/watch.go:37
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).watch(0xc00009e300, 0x32b5a00, 0xc0004663c0, 0xc000401230, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/prometheus/kubernetes.go:102 +0x1dd
github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).start.func1(0xc00009e300, 0x32b5a00, 0xc0004663c0, 0xc000401230)
	/go/src/github.com/influxdata/telegraf/plugins/inputs/prometheus/kubernetes.go:69 +0x175
created by github.com/influxdata/telegraf/plugins/inputs/prometheus.(*Prometheus).start
	/go/src/github.com/influxdata/telegraf/plugins/inputs/prometheus/kubernetes.go:62 +0xbb

goroutine 12 [select]:
database/sql.(*DB).connectionOpener(0xc0002d0dd0, 0x32b5a00, 0xc000466600)
	/usr/local/go/src/database/sql/sql.go:1126 +0xf5
created by database/sql.OpenDB
	/usr/local/go/src/database/sql/sql.go:740 +0x12a

goroutine 13 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runOutputs(0xc000208138, 0xc0001e76a0, 0x0, 0x404040202030302)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:774 +0x2b6
github.com/influxdata/telegraf/agent.(*Agent).Run.func1(0xc0001c90b0, 0xc000208138, 0xc0001e76a0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:152 +0x65
created by github.com/influxdata/telegraf/agent.(*Agent).Run
	/go/src/github.com/influxdata/telegraf/agent/agent.go:150 +0x4ec

goroutine 14 [semacquire]:
sync.runtime_Semacquire(0xc00057ea08)
	/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc00057ea00)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/influxdata/telegraf/agent.(*Agent).runProcessors(0xc000208138, 0xc000208150, 0x1, 0x1, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:569 +0xb2
github.com/influxdata/telegraf/agent.(*Agent).Run.func4(0xc0001c90b0, 0xc000208138, 0xc000208150, 0x1, 0x1)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:182 +0x7b
created by github.com/influxdata/telegraf/agent.(*Agent).Run
	/go/src/github.com/influxdata/telegraf/agent/agent.go:180 +0x67f

goroutine 15 [semacquire]:
sync.runtime_Semacquire(0xc0001c90c8)
	/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0001c90c0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/influxdata/telegraf/agent.(*Agent).runInputs(0xc000208138, 0x32b5a00, 0xc0001f90c0, 0xbfde568de2384c85, 0x41e8ed5, 0x48044c0, 0xc0001e77c0, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:330 +0x39f
github.com/influxdata/telegraf/agent.(*Agent).Run.func5(0xc0001c90b0, 0xc000208138, 0x32b5a00, 0xc0001f90c0, 0xbfde568de2384c85, 0x41e8ed5, 0x48044c0, 0xc0001e77c0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:192 +0xab
created by github.com/influxdata/telegraf/agent.(*Agent).Run
	/go/src/github.com/influxdata/telegraf/agent/agent.go:190 +0x5a8

goroutine 16 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000466680, 0x32b5a00, 0xc000466640, 0xc000137860)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:85 +0x134
github.com/influxdata/telegraf/agent.newAlignedTicker.func1(0xc000466680, 0x32b5a00, 0xc000466640, 0xc000137860)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x76
created by github.com/influxdata/telegraf/agent.newAlignedTicker
	/go/src/github.com/influxdata/telegraf/agent/tick.go:60 +0x1a7

goroutine 49 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x32d3380, 0xc0001ee100, 0xc000136a50, 0x3283b00, 0xc000466680, 0x6fc23ac00)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:462 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0xc0001c90c0, 0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x32d3380, 0xc0001ee100, 0x3283b00, 0xc000466680, 0x6fc23ac00, 0xc000136a50)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0xbf
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0x14a

goroutine 25 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc00057ea00, 0xc0001e77a0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:558 +0xfa
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
	/go/src/github.com/influxdata/telegraf/agent/agent.go:554 +0x91

goroutine 26 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushOnce(0xc000208138, 0xc000126100, 0x3283b40, 0xc000401320, 0xc000a3d940, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:851 +0x1bc
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc000208138, 0x32b5a00, 0xc0001f9140, 0xc000126100, 0x3283b40, 0xc000401320)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:823 +0x4c5
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0xc00057ea10, 0x2540be400, 0x0, 0xc000208138, 0x32b5a00, 0xc0001f9140, 0xc000126100)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:770 +0xe5
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:764 +0x1a5

goroutine 27 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushOnce(0xc000208138, 0xc000126200, 0x3283b40, 0xc0004012f0, 0xc000882960, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:851 +0x1bc
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc000208138, 0x32b5a00, 0xc0001f9140, 0xc000126200, 0x3283b40, 0xc0004012f0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:823 +0x4c5
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0xc00057ea10, 0x2540be400, 0x0, 0xc000208138, 0x32b5a00, 0xc0001f9140, 0xc000126200)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:770 +0xe5
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:764 +0x1a5

goroutine 28 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc000208138, 0x32b5a00, 0xc0001f9140, 0xc000126280, 0x3283b40, 0xc000350ab0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:818 +0x20e
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0xc00057ea10, 0x2540be400, 0x0, 0xc000208138, 0x32b5a00, 0xc0001f9140, 0xc000126280)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:770 +0xe5
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:764 +0x1a5

goroutine 29 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc000350ab0, 0x32b5a00, 0xc0001f9180, 0xc0001f6960)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:252 +0x128
github.com/influxdata/telegraf/agent.newRollingTicker.func1(0xc000350ab0, 0x32b5a00, 0xc0001f9180, 0xc0001f6960)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:240 +0x76
created by github.com/influxdata/telegraf/agent.newRollingTicker
	/go/src/github.com/influxdata/telegraf/agent/tick.go:238 +0x165

goroutine 50 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000466740, 0x32b5a00, 0xc0004666c0, 0xc0001379a0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:85 +0x134
github.com/influxdata/telegraf/agent.newAlignedTicker.func1(0xc000466740, 0x32b5a00, 0xc0004666c0, 0xc0001379a0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x76
created by github.com/influxdata/telegraf/agent.newAlignedTicker
	/go/src/github.com/influxdata/telegraf/agent/tick.go:60 +0x1a7

goroutine 51 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x32d3380, 0xc0001ee180, 0xc000136c30, 0x3283b00, 0xc000466740, 0x6fc23ac00)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:462 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0xc0001c90c0, 0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x32d3380, 0xc0001ee180, 0x3283b00, 0xc000466740, 0x6fc23ac00, 0xc000136c30)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0xbf
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0x14a

goroutine 52 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc0004667c0, 0x32b5a00, 0xc000466780, 0xc000137ae0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:85 +0x134
github.com/influxdata/telegraf/agent.newAlignedTicker.func1(0xc0004667c0, 0x32b5a00, 0xc000466780, 0xc000137ae0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x76
created by github.com/influxdata/telegraf/agent.newAlignedTicker
	/go/src/github.com/influxdata/telegraf/agent/tick.go:60 +0x1a7

goroutine 53 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x32d3380, 0xc0001ee240, 0xc000136d70, 0x3283b00, 0xc0004667c0, 0x6fc23ac00)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:462 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0xc0001c90c0, 0xc000208138, 0x32b5a00, 0xc0001f90c0, 0x32d3380, 0xc0001ee240, 0x3283b00, 0xc0004667c0, 0x6fc23ac00, 0xc000136d70)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0xbf
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
	/go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0x14a

goroutine 54 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc0004012f0, 0x32b5a00, 0xc000466800, 0xc000137bd0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:252 +0x128
github.com/influxdata/telegraf/agent.newRollingTicker.func1(0xc0004012f0, 0x32b5a00, 0xc000466800, 0xc000137bd0)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:240 +0x76
created by github.com/influxdata/telegraf/agent.newRollingTicker
	/go/src/github.com/influxdata/telegraf/agent/tick.go:238 +0x165

goroutine 55 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc000401320, 0x32b5a00, 0xc000466840, 0xc000137c70)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:252 +0x128
github.com/influxdata/telegraf/agent.newRollingTicker.func1(0xc000401320, 0x32b5a00, 0xc000466840, 0xc000137c70)
	/go/src/github.com/influxdata/telegraf/agent/tick.go:240 +0x76
created by github.com/influxdata/telegraf/agent.newRollingTicker
	/go/src/github.com/influxdata/telegraf/agent/tick.go:238 +0x165

goroutine 69 [IO wait]:
internal/poll.runtime_pollWait(0x7f9647fdfb38, 0x72, 0x32670c0)
	/usr/local/go/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc0000f8118, 0x72, 0xc0003c2a00, 0x9d5, 0x9d5)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0000f8100, 0xc0003c2a80, 0x9d5, 0x9d5, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc0000f8100, 0xc0003c2a80, 0x9d5, 0x9d5, 0x203000, 0x16000000fa, 0x1800000017)
	/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00011c178, 0xc0003c2a80, 0x9d5, 0x9d5, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0001f4cc0, 0xc0003c2a80, 0x9d5, 0x9d5, 0x1ffb000000fd, 0x1ffc0007fff0, 0xc00058f710)
	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc000154980, 0x325c9a0, 0xc0001f4cc0, 0x40b7e5, 0x2888080, 0x2cda7e0)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000154700, 0x3263e60, 0xc00011c178, 0x5, 0xc00011c178, 0x7fffde007fffdd)
	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000154700, 0x0, 0x0, 0x2dcf096)
	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc000154700, 0xc000261000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc00025e1e0, 0xc0002501f8, 0x9, 0x9, 0xc00009ea80, 0xc00058fd28, 0x405dee)
	/usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x325c6a0, 0xc00025e1e0, 0xc0002501f8, 0x9, 0x9, 0x9, 0x2db0b65, 0x4647a0, 0xc00025e240)
	/usr/local/go/src/io/io.go:314 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc0002501f8, 0x9, 0x9, 0x325c6a0, 0xc00025e1e0, 0x0, 0xc000000000, 0xc000220a20, 0xc00012e3f0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002501c0, 0xc000220a20, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00058ffa8, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00023c180)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200904194848-62affa334b73/http2/transport.go:695 +0x66e

goroutine 165 [runnable]:
github.com/influxdata/telegraf/metric.(*metric).Fields(0xc000c0a540, 0xc000ca0060)
	/go/src/github.com/influxdata/telegraf/metric/metric.go:112 +0xcb
github.com/influxdata/telegraf/plugins/outputs/dynatrace.(*Dynatrace).Write(0xc00017a180, 0xc000ca6000, 0x471, 0x471, 0xc000034ed8, 0x822552)
	/go/src/github.com/influxdata/telegraf/plugins/outputs/dynatrace/dynatrace.go:157 +0x12f
github.com/influxdata/telegraf/models.(*RunningOutput).write(0xc000126100, 0xc000ca6000, 0x471, 0x471, 0x471, 0x0)
	/go/src/github.com/influxdata/telegraf/models/running_output.go:246 +0x99
github.com/influxdata/telegraf/models.(*RunningOutput).Write(0xc000126100, 0x0, 0x0)
	/go/src/github.com/influxdata/telegraf/models/running_output.go:203 +0x1fe
github.com/influxdata/telegraf/agent.(*Agent).flushOnce.func1(0xc0005940c0, 0xc000a3d940)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:847 +0x27
created by github.com/influxdata/telegraf/agent.(*Agent).flushOnce
	/go/src/github.com/influxdata/telegraf/agent/agent.go:846 +0x7a

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugunexpected problem or unintended behavior

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions