UDP prospector type with plain harvester#4452
Conversation
filebeat/prospector/udp/harvester.go
Outdated
There was a problem hiding this comment.
@hartfordfive Any recommendation on the value here?
There was a problem hiding this comment.
@ruflin I realize in theory you could have up to 65507 bytes of data in IPv4 (removing UDP and IP header overhead of 20 and 8 bytes respectively) although I know that you risk having fragmentation of this because of MTU (which I guess is generally between 1400 and 1500 - not taking into account jumbo frames). Considering this should only be done locally on the server, i figured this shouldn't happen, so thus why I chose a slightly larger 2048 or 4096 bytes buffer size.
I know many applications tend to limit to <= 1024 bytes for a UDP buffer (especially for cross-host communication), although from what I've tested so far, I haven't had messages that were cut off, but I'm aware this could happen, especially if you're going form one host to another even on the same rack. You could potentially stick with the 1024 byte buffer as a default but allow users to specify the port and the buffer size via the prospector config parameters.
I think it would maybe be nice to have someone else give some feedback on this, especially from the networking stack side to confirm if this is a reasonable assumption or not. What do you think?
There was a problem hiding this comment.
Thanks for sharing the thoughts. I set it 1024 for the moment and made it configurable. Time will tell and we can still change the default.
filebeat/prospector/udp/harvester.go
Outdated
There was a problem hiding this comment.
@tsg I'm asking myself if this should be code? Other options here could be json.
There was a problem hiding this comment.
Do we need a "harverster" key? Maybe we add it when we have more of them.
I'm thinking that, for example, JSON can be accomplished with the processor, so I'm not sure harvester support for it is needed.
There was a problem hiding this comment.
We don't need it at the moment an can still add it later.
ce9a97e to
f0ce934
Compare
filebeat/_meta/common.full.p2.yml
Outdated
There was a problem hiding this comment.
Two things here:
- I think it would be more clear if it would say "Maximum size of the message received over UDP".
- 1024 seems low, UDP can go up to 65K. How about 10K, which should be large enough for most usecases but still small enough to not be noticeable in the mem usage?
There was a problem hiding this comment.
I set it to 10240 now. 1024 was based on the discussion here.
42100f1 to
cfac655
Compare
exekias
left a comment
There was a problem hiding this comment.
Left a comment on default value, LGTM apart from that
filebeat/prospector/udp/config.go
Outdated
There was a problem hiding this comment.
changed, thanks for spotting.
…get about it :-)
TODO
* [x] Changelog
* [ ] Docs
* [x] Tests
* [x] Code cleanup
And even looks as following
```
{
"@timestamp": "2017-05-26T11:57:17.68711727+02:00",
"beat": {
"hostname": "ruflin",
"name": "ruflin",
"version": "6.0.0-alpha2"
},
"message": "Hello, World!99",
"prospector": {
"harvester": "plain",
"type": "udp"
}
}
```
|
WFG |
|
Can I remove the needs_docs tag? Is #4940 sufficient, or is additional doc required? |
|
@dedemorton I was hoping for more docs that also shows how to configure the host for example. But I don't know how this fits into our current doc model of prospectors. Perhaps worth to discuss quickly over zoom? |
|
I'm going to remove the needs_docs label here in favor of tracking the work in #5632 |
Opening this prospector mainly to show this is WIP and so I don't forget about it :-)
TODO
And even looks as following