Add WebSocket output plugin by FZambia · Pull Request #9188 · influxdata/telegraf

FZambia · 2021-04-26T08:20:17Z

Updated associated README.md.
Wrote appropriate unit tests.

resolves #

Hello, added a generic WebSocket output plugin. We need it for Grafana to stream data into panels from Telegraf (aiming to introduce basic streaming support in Grafana v8). A video that demonstrates our early work on this (this is a CPU data stream from Telegraf using output plugin code introduced here):

Screen.Recording.2021-04-26.at.10.46.58.mov

We considered going with HTTP output plugin first, but the latency overhead for each HTTP request was pretty big. Unfortunately optimizing HTTP path resulted in pretty "special" HTTP endpoint in Grafana with many useful middleware cut off. More details in this draft pull request.

WebSocket allows us to reuse the same port as HTTP server (so no need to open new ports), also message processing does not require going through all middleware stack - thus reducing overall latency and load on GC.

This plugin works somewhat similar to existing socket_writer output plugin.

One important thing that while Gorilla WebSocket required for plugin to work it already existed in indirect dependencies of Telegraf, Gorilla WebSocket itself is de-facto library for Websocket in Go, well-tested and has zero dependencies itself.

Unit tests cover most things, but to test manually one can do the following:

Run WebSocket echo server:

docker run --rm -it -p 8080:8080 inanimate/echo-server

Run Telegraf with config:

[agent]
  interval = "1s"
  flush_interval = "1s"

[[inputs.cpu]]
  totalcpu = true

[[outputs.websocket]]
  url = "ws://localhost:8080"
  data_format = "influx"
  use_text_frames = true
  write_timeout = "100ms"

After this requests should be seen in server log output.

telegraf-tiger · 2021-04-26T08:20:24Z

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

FZambia · 2021-04-26T09:06:03Z

!signed-cla

telegraf-tiger · 2021-04-26T12:59:47Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

ivorybilled

Code looks good to me, just a couple of linter issues to fix. Thanks!

telegraf-tiger · 2021-04-26T17:00:03Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

FZambia · 2021-04-26T17:44:10Z

@jagularr hello, thanks for looking! Just fixed linter issues

ryantxu · 2021-04-28T15:38:30Z

@jagularr is there anything we can do to make this easier to review?

asdfd2013 · 2021-05-10T10:34:18Z

I try to modify the configration data_format = "json" in [[outputs.websocket]] , but still output default influx format .
The version i used was windows_amd64.zip within Add WebSocket output plugin #9188 (comment)
Is there any other necessary configuration I didn't set up?
And hope i could use this output plugin to put telegraf client ip infomation by webSocketSession.getRemoteAddress()

thanks for your attention.

FZambia · 2021-05-10T11:15:47Z

I try to modify the configration data_format = "json" in [[outputs.websocket]] , but still output default influx format .
The version i used was windows_amd64.zip within #9188 (comment)
Is there any other necessary configuration I didn't set up?

Just tried to reproduce this - it works fine for me (I am using Telegraf built from latest commit in my branch). My config:

[agent]
  interval = "1000ms"
  flush_interval = "1000ms"

[[inputs.cpu]]
  totalcpu = true

[[outputs.websocket]]
  url = "ws://localhost:8080"
  data_format = "json"
  use_text_frames = true
  write_timeout = "100ms"

In server console I see sth like this:

{"metrics":[{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":80.99999999976717,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":10.000000000218279,"usage_user":8.999999999650754},"name":"cpu","tags":{"cpu":"cpu0","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu1","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":88.00000000046566,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":7.000000000334694,"usage_user":4.9999999995634425},"name":"cpu","tags":{"cpu":"cpu2","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu3","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":91.08910891368707,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":4.950495049672567,"usage_user":3.9603960398821316},"name":"cpu","tags":{"cpu":"cpu4","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu5","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":91.99999999837019,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":4.0000000000873115,"usage_user":4.0000000000873115},"name":"cpu","tags":{"cpu":"cpu6","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu7","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":94.99999999563443,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":1.9999999999272404,"usage_user":2.9999999997089617},"name":"cpu","tags":{"cpu":"cpu8","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu9","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":97.00000000285218,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":2.000000000160071,"usage_user":0.9999999998981366},"name":"cpu","tags":{"cpu":"cpu10","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu11","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745},{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":95.33333333286767,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":2.500000000024253,"usage_user":2.166666666501745},"name":"cpu","tags":{"cpu":"cpu-total","host":"MacBook-Pro-Alexander.local"},"timestamp":1620644745}]}

I.e. JSON format. I am on MacOS, but not sure this can make a difference. Make sure you reloaded Telegraf when changing data format.

And hope i could use this output plugin to put telegraf client ip infomation by webSocketSession.getRemoteAddress()

I believe that there is nothing to add here to achieve this since IP comes from TCP/IP level or if modified on load balancer level usually can be extracted from X-Real-Ip, X-Forwarded-For like headers. So just get client IP on your WebSocket server side.

asdfd2013 · 2021-05-11T01:31:59Z

Ok,i reboot and the configuration actually work.Sorry to bother you for my carelessness.

plugins/outputs/websocket/websocket.go

ssoroka · 2021-05-12T19:35:53Z

Thanks! looks slick; just have a few comments

FZambia · 2021-05-12T20:28:07Z

@ssoroka hello, thanks for reviewing - pushed some fixes and commented where I think things already work fine.

telegraf-tiger · 2021-05-12T20:37:21Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-05-14T16:44:45Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-05-15T14:06:27Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

plugins/outputs/websocket/websocket.go

srebhan · 2021-05-18T16:19:58Z

plugins/outputs/websocket/websocket.go

+	messageType := ws.BinaryMessage
+	if w.UseTextFrames {
+		messageType = ws.TextMessage
+	}


This can be moved to the Init() function I think.

That just a frame type detection based on config allocated on stack, moving it to WebSocket struct will only make code harder to follow in my opinion.

TBH I like it more if "one-time things" are not in here, as it clutters the code. But if @ssoroka aggrees we can leave it here.

plugins/outputs/websocket/websocket.go

srebhan

Thanks @FZambia for this nicely written plugin! I have some comments in the code. Mostly minor things, but please do not set the logging facility manually.

srebhan · 2021-05-18T16:30:51Z

Btw @FZambia as you are already on it... If you have some time left and are motivated, you might want to rework my input plugin (#8867) to use gorilla's implementation of websocket... :-P

FZambia · 2021-05-18T17:01:58Z

@srebhan to be fair lots of stuff to do outside of Telegraf these days, but I definitely suggest using Gorilla WebSocket - it's robust, widely used, receives security updates. And you already have it in go.sum.

telegraf-tiger · 2021-05-18T17:06:21Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

srebhan

Thanks for the very quick round! Amazing. The only really tiny bit of discussion is the one-time message-type thing but as I wrote if @ssoroka is ok with it I'm also. ;-)

srebhan · 2021-05-19T14:58:39Z

@srebhan to be fair lots of stuff to do outside of Telegraf these days, but I definitely suggest using Gorilla WebSocket - it's robust, widely used, receives security updates. And you already have it in go.sum.

Oh damn, thought I can get rid of it... :-P

plugins/outputs/websocket/websocket.go

FZambia · 2021-05-25T10:54:18Z

@ssoroka hello, I think I addressed all your review comments - could you please take a look?

ryantxu · 2021-05-27T23:10:42Z

For reference, here is a first draft on a tutorial using this PR to stream metrics to grafana:
https://grafana.com/tutorials/stream-metrics-from-telegraf-to-grafana/#stream-using-websocket-endpoint

reimda · 2021-06-03T17:02:35Z

@ssoroka Are you ok with this PR? I see all your review comments are resolved but your review still says you requested changes.

FZambia added 6 commits April 22, 2021 19:24

implement websocket output plugin

882c29c

use binary frames by default

10956dc

add tests

37f0af2

minor test cleanups

a71debd

update readme - add websocket output

6489224

better handling of read errors

2e0d372

telegraf-tiger bot added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin new plugin plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins labels Apr 26, 2021

sample config: correct headers, rm copy-paste

7f33db2

ivorybilled reviewed Apr 26, 2021

View reviewed changes

fix linter issues

8e7b1a2

ssoroka reviewed May 12, 2021

View reviewed changes

plugins/outputs/websocket/websocket.go Outdated Show resolved Hide resolved

plugins/outputs/websocket/websocket.go Show resolved Hide resolved

ssoroka suggested changes May 12, 2021

View reviewed changes

plugins/outputs/websocket/websocket.go Show resolved Hide resolved

plugins/outputs/websocket/websocket.go Show resolved Hide resolved

plugins/outputs/websocket/websocket.go Show resolved Hide resolved

plugins/outputs/websocket/websocket.go Show resolved Hide resolved

review fixes

180303a

Merge branch 'master' into websocket_output

1890e06

make connect and write timeout larger

b18c32f

FZambia requested a review from ssoroka May 16, 2021 06:45