fix: Graylog plugin TLS support and message format by alespour · Pull Request #9862 · influxdata/telegraf

alespour · 2021-10-05T13:30:32Z

Required for all PRs:

Updated associated README.md.
Wrote appropriate unit tests.
Pull request title or commits are in conventional commit format

Resolves #9772
Resolves #9773

This PR

fixes JSON output message where non-standard fields should be prefixed with _. New config option name_field_no_prefix can be used to use old format.
adds TLS support for the plugin. Plugin config is extended with "standard" TLS config section like many other plugins have.

  ## Endpoints for your graylog instances.
  servers = ["udp://127.0.0.1:12201"]

  ## Connection timeout.
  # timeout = "5s"

  ## The field to use as the GELF short_message, if unset the static string
  ## "telegraf" will be used.
  ##   example: short_message_field = "message"
  # short_message_field = ""

  ## According to GELF payload specification, additional fields names must be prefixed
  ## with an underscore. Previous versions did not prefix custom field 'name' with underscore.
  ## Set to true for backward compatibility.
  # name_field_no_prefix = false

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

telegraf-tiger · 2021-10-06T08:04:14Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

srebhan

@alespour thanks for caring about this output! The changes look reasonable, I've only some questions about the serialization where you change name to _name: Was this working before? Why do you require this change? Will it break existing setups?

A more general comment: The whole plugin looks similar to the socket_writer output with a special serialization format. If this is true, my question is, why not simply implement a serialzer plugin and use it with socket_writer?

plugins/outputs/graylog/graylog.go

srebhan · 2021-10-06T14:00:05Z

Regarding the serialization, I think you misunderstood my proposal. I'm proposing you add a new serializer under plugins/serializers/graylog and implement your serializer there. A serializer has to implement this interface and takes a metric (or a batch of metrics) and outputs []bytes. This way you can do exactly what you are currently doing in serialize() with the only difference that you output a byte stream. Would that be possible?

alespour · 2021-10-06T15:46:28Z

@srebhan Sorry I have misread your proposal.

Anyway, I do not think that socket_writer together with special serializer could work like graylog output plugin now. serialize function encodes the payload (ie. metric to JSON), but then there are 3 transport encoding scenarios that the plugin handles:

UDP non-chunked - OK
UDP chunked (proprietary) - NOT possible with serializer interface, it returns []byte, but [][]byte would be needed here
TCP - OK

srebhan · 2021-10-06T18:51:30Z

Hmmm we could extend the serializer interface. @reimda, @powersj or @sspaink do you think it's worth thinking about an extension of the parser interface? We would need SerializeChunked(metric []telegraf.Metrics) ([][]byte, error)...

srebhan

Looks good to me. Let's follow-up on the serializer part. If you find some time to write a PoC that would be helpful...

telegraf-tiger · 2021-10-18T12:51:34Z

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_armv7.tar.gz
armhf.deb	i386.rpm	freebsd_i386.tar.gz
i386.deb	ppc64le.rpm	linux_amd64.tar.gz
mips.deb	s390x.rpm	linux_arm64.tar.gz
mipsel.deb	x86_64.rpm	linux_armel.tar.gz
ppc64el.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

srebhan

Looks good to me.

* origin/master: (176 commits) fix: Linter fixes for plugins/inputs/[h-j]* (influxdata#9986) fix(inputs/kube_inventory): don't skip resources with zero s/ns timestamps (influxdata#9978) fix: update gjson to v1.10.2 (influxdata#9998) fix: procstat tags were not getting generated correctly (influxdata#9973) chore: create bug report form (influxdata#9976) fix: Allow for non x86 macs in Go install script (influxdata#9982) test: add sqlserver plugin integration tests (influxdata#9943) feat: plugins/common/tls/config.go: Filter client certificates by DNS names (influxdata#9910) feat: add option to skip table creation in azure data explorer output (influxdata#9942) docs: update nightlies links (influxdata#9989) fix: add s390x to nightlies (influxdata#9990) feat: Add more details to processors.ifname logmessages (influxdata#9984) docs: Create SECURITY.md (influxdata#9951) fix: set NIGHTLY=1 for correctly named nightly artifacts (influxdata#9987) feat: Kafka Add metadata full to config (influxdata#9833) chore: Update to AWS SDK v2 (influxdata#9647) chore: lint ignore fmt.Printf unhandled error (influxdata#9967) fix: starlark pop operation for non-existing keys (influxdata#9954) feat: update etc/telegraf.conf and etc/telegraf_windows.conf (influxdata#9876) fix: Check return code of zfs command for FreeBSD. (influxdata#9956) chore: update go to 1.17.2 (influxdata#9873) fix: Graylog plugin TLS support and message format (influxdata#9862) docs: update README with info on package repos (influxdata#9964) feat: Modbus connection settings (serial) (influxdata#9256) fix: segfault in ingress, persistentvolumeclaim, statefulset in kube_inventory (influxdata#9585) fix: add normalization of tags for ethtool input plugin (influxdata#9901) chore: remove empty build.py (influxdata#9958) fix: internet_speed input plugin not collecting/reporting latency (influxdata#9957) chore: reference db2 external plugin (influxdata#9952) chore: update readme go version from 1.14 to 1.17 (influxdata#9944) fix: decode Prometheus scrape path from Kuberentes labels (influxdata#9662) docs: fix broken link (influxdata#9812) fix: Correct conversion of int with specific bit size (influxdata#9933) fix: update golanci-lint to v1.42.1 (influxdata#9932) feat: Azure Event Hubs output plugin (influxdata#9346) feat: more fields for papertrail event webhook (influxdata#9940) fix: solve compatibility issue for mongodb inputs when using 5.x relicaset (influxdata#9892) docs: Add symlink to command documentation (influxdata#9926) docs: update contributing.md (influxdata#9914) chore: reference oracle external plugin (influxdata#9934) ...

1tft · 2021-10-27T17:27:48Z

Graylog output plugin does not handle default gelf field "full_message" as default field. It treates it as custom field and adds "_" at beginning. Thats not correct, like "short_message" field, also full_message field must not start with underscore (see https://docs.graylog.org/en/4.0/pages/gelf.html). Please can you fix this?

Hipska · 2021-10-28T08:25:28Z

@1tft can you create a separate issue for this? This comment will get lost in the haystack otherwise. Thanks!

1tft · 2021-10-28T17:59:30Z

Thank you Hipska for advice. Yesterday I reported this issue to InfluxDB support (@alespour), so I hope he will do that...

* origin/master: (176 commits) fix: Linter fixes for plugins/inputs/[h-j]* (influxdata#9986) fix(inputs/kube_inventory): don't skip resources with zero s/ns timestamps (influxdata#9978) fix: update gjson to v1.10.2 (influxdata#9998) fix: procstat tags were not getting generated correctly (influxdata#9973) chore: create bug report form (influxdata#9976) fix: Allow for non x86 macs in Go install script (influxdata#9982) test: add sqlserver plugin integration tests (influxdata#9943) feat: plugins/common/tls/config.go: Filter client certificates by DNS names (influxdata#9910) feat: add option to skip table creation in azure data explorer output (influxdata#9942) docs: update nightlies links (influxdata#9989) fix: add s390x to nightlies (influxdata#9990) feat: Add more details to processors.ifname logmessages (influxdata#9984) docs: Create SECURITY.md (influxdata#9951) fix: set NIGHTLY=1 for correctly named nightly artifacts (influxdata#9987) feat: Kafka Add metadata full to config (influxdata#9833) chore: Update to AWS SDK v2 (influxdata#9647) chore: lint ignore fmt.Printf unhandled error (influxdata#9967) fix: starlark pop operation for non-existing keys (influxdata#9954) feat: update etc/telegraf.conf and etc/telegraf_windows.conf (influxdata#9876) fix: Check return code of zfs command for FreeBSD. (influxdata#9956) chore: update go to 1.17.2 (influxdata#9873) fix: Graylog plugin TLS support and message format (influxdata#9862) docs: update README with info on package repos (influxdata#9964) feat: Modbus connection settings (serial) (influxdata#9256) fix: segfault in ingress, persistentvolumeclaim, statefulset in kube_inventory (influxdata#9585) fix: add normalization of tags for ethtool input plugin (influxdata#9901) chore: remove empty build.py (influxdata#9958) fix: internet_speed input plugin not collecting/reporting latency (influxdata#9957) chore: reference db2 external plugin (influxdata#9952) chore: update readme go version from 1.14 to 1.17 (influxdata#9944) fix: decode Prometheus scrape path from Kuberentes labels (influxdata#9662) docs: fix broken link (influxdata#9812) fix: Correct conversion of int with specific bit size (influxdata#9933) fix: update golanci-lint to v1.42.1 (influxdata#9932) feat: Azure Event Hubs output plugin (influxdata#9346) feat: more fields for papertrail event webhook (influxdata#9940) fix: solve compatibility issue for mongodb inputs when using 5.x relicaset (influxdata#9892) docs: Add symlink to command documentation (influxdata#9926) docs: update contributing.md (influxdata#9914) chore: reference oracle external plugin (influxdata#9934) ...

telegraf-tiger bot added the fix pr to fix corresponding bug label Oct 5, 2021

alespour force-pushed the fix/graylog-output branch from 16dea28 to d5e7d59 Compare October 6, 2021 07:47

alespour marked this pull request as ready for review October 6, 2021 08:04

alespour changed the title ~~fix: Graylog plugin~~ fix: Graylog plugin TLS support and message format Oct 6, 2021

srebhan reviewed Oct 6, 2021

View reviewed changes

plugins/outputs/graylog/graylog.go Outdated Show resolved Hide resolved

srebhan self-assigned this Oct 6, 2021

sspaink mentioned this pull request Oct 6, 2021

fix: duplicate line_protocol when using object and fields #9872

Merged

Hipska approved these changes Oct 7, 2021

View reviewed changes

srebhan approved these changes Oct 12, 2021

View reviewed changes

Hipska added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Oct 14, 2021

alespour added 7 commits October 18, 2021 14:32

fix: add missing TLS support

2825086

fix: prefix non-standard fields with underscore

ec988f6

docs: update README

0ce54fb

refactor: connect to servers in plugin's Connect method

2833c36

test: fix reconnect scenario

4d24d89

refactor: removed unused field

0b068ef

fix: added option for custom field 'name' serialization control

cb274fd

alespour force-pushed the fix/graylog-output branch from cace222 to cb274fd Compare October 18, 2021 12:33

srebhan approved these changes Oct 18, 2021

View reviewed changes

MyaLongmire approved these changes Oct 21, 2021

View reviewed changes

MyaLongmire merged commit e685f3b into influxdata:master Oct 21, 2021

alespour deleted the fix/graylog-output branch October 29, 2021 08:41

alespour mentioned this pull request Oct 29, 2021

Graylog output plugin: full_message is handled like a custom field #10024

Closed

VladislavSenkevich pushed a commit to gwos/telegraf that referenced this pull request Nov 23, 2021

fix: Graylog plugin TLS support and message format (influxdata#9862)

3ff100a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Graylog plugin TLS support and message format#9862

fix: Graylog plugin TLS support and message format#9862
MyaLongmire merged 7 commits intoinfluxdata:masterfrom
bonitoo-io:fix/graylog-output

alespour commented Oct 5, 2021 •

edited

Loading

Uh oh!

telegraf-tiger bot commented Oct 6, 2021

Artifact URLs

Uh oh!

srebhan left a comment

Uh oh!

Uh oh!

srebhan commented Oct 6, 2021

Uh oh!

alespour commented Oct 6, 2021 •

edited

Loading

Uh oh!

srebhan commented Oct 6, 2021

Uh oh!

srebhan left a comment

Uh oh!

telegraf-tiger bot commented Oct 18, 2021

Artifact URLs

Uh oh!

srebhan left a comment

Uh oh!

1tft commented Oct 27, 2021

Uh oh!

Hipska commented Oct 28, 2021

Uh oh!

1tft commented Oct 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

alespour commented Oct 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Required for all PRs:

Uh oh!

telegraf-tiger bot commented Oct 6, 2021

Artifact URLs

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

srebhan commented Oct 6, 2021

Uh oh!

alespour commented Oct 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srebhan commented Oct 6, 2021

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

telegraf-tiger bot commented Oct 18, 2021

Artifact URLs

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

1tft commented Oct 27, 2021

Uh oh!

Hipska commented Oct 28, 2021

Uh oh!

1tft commented Oct 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alespour commented Oct 5, 2021 •

edited

Loading

alespour commented Oct 6, 2021 •

edited

Loading