I'm still getting these intermittently on the data stream I'm testing with. The issue is that the upper bound is actually a negative offset so this still panics.

I added some quick instrumentation to see what's going on.

Here is what the relevant values look like that are used to calculate the upper bound for the slice. It seems that the cursor for lineStart ends up way beyond the error column for some reason:

2021-10-08T21:36:38Z I! Column: 236 lineStart: 75774
panic: runtime error: slice bounds out of range [:-75539]

I unfortunately did not capture the buffer at the time. I've set things up so that I get the data once this happens again.

📦 Looks like new artifacts were built from this PR.

Here is a case we caught in the data firehose where the parser is parsing a tag value as an integer and for some reason the error message output offset gets incremented too far:

2021-10-12T09:29:48Z I! http: panic serving 7.150.32.18:21520: out of bounds error in Decoder.syntaxErrorf; d.r: [0, 215]; offset: 216; d.line: 153:217; buf: [0:215]="zzzz_zzzzzz,application=XxxxXxx,author=Xxxxxxxx,description=Xxx\\ 0\\ In\\ 20,device_type=X00,host=zzz-yyyy,interface=xxxx,model_name=xxx-xx,val=0,port=20,serial_number=xxxx-xx-xxxxx-x,source=xxxx max_queue_time=0i "; error: cannot parse value for field key "\\ 0\\ In\\ 4,dev": invalid integer value syntax

@powersj any update on the problem of panicing?

No - you can check the status of that on influxdata/line-protocol#50

@powersj @srebhan
It looks like influxdata/line-protocol#51 has resolved the issue.

I built a version of Telegraf that incorporates that PR and have not been able to crash it.

@oplehto thanks for confirming! I will get this PR updated with the latest parser code and we can let it bake a bit more.

Looks good to me. Excellent work @akrantz01 and @powersj! @oplehto can you please confirm that this PR does not panic anymore?

I've had this running for a while now ingesting a very large amount of various metrics data and can confirm that I have not seen any issues anymore.

We're planning on this new parser ~a month to bake in cloud. We'll check in on merging this in to Telegraf mid January 2022.

Now we're into mid-Jan, we should re-visit this PR.
Are we confident this has been exercised enough elsewhere to land in Telegraf?

There have not been any issues with this new parser in InfluxDB Cloud. This will get added on our list to get reviewed and merged soon. Since this is new parser is a major feature implementation it won't officially be included until our next minor release in a couple months. However, once merged, you can use it and test with our nightlies.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Looks like new artifacts were built from this PR.

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

Hi,

Thanks to everyone for providing feedback on this feature. After talking through this change as a team, we decided to merge the feature as a flag of the parser. This means users can enable the use of the new parser backend by adding a new parser config option, "influx_parser_type" as shown below:

    data_format = "influx"
    influx_parser_type = "upstream"

The "upstream" value will enable the use of the new faster more memory efficient parser, while "internal", which is the default setting, will remain with the existing parser. This allows users to opt-in to the new parser and allow us to collect more confidence in ensuring existing configurations will not suddenly break. Our plan is to switch the default parser type in a future release.

The change was merged in as a part of #10749 and is now in master. Tonight's nightlies should have the change included them, and next weeks v1.22.0 release will be the first release with the new features. As such I am closing this PR.

Thanks again!