Mentioning it mostly for awareness: if I'm not wrong, the Auditbeat report.Event()doesn't guarantee sending at least once. If Elasticsearch is down, for example, it will retry 3 times, and then drop the event. This means that we can potentially lose login events, despite saving the state. I think we could solve this after the Filebeat refactoring, but I suggest documenting this as a limitation for now.

Hm, that's a shame. It guess this affects all metricsets, and we might lose process starts, socket connections, password changes, audit events, FIM events, etc.

The internal queue does not keep events until they can be sent (or it overflows)? That would seem desirable.

It does, and one can also use spooling to disk to make the queue larger. But Filebeat offers an extra guarantee by waiting for the ACK and only afterwards update the state on disk. This is not currently possible in Auditbeat. That’s the difference I wanted to highlight.

Right, maybe if at some point report.Event() would return false if it couldn't write to the queue or otherwise cannot guarantee sending the event eventually - so we can retry later.

What does this mean for big-endian architectures? Will this code work?

Hm, good point and I don't know. Do we support any big-endian systems?

I'm surprised Go doesn't have a binary.MachineEndian.

Just revisiting this. I'm thinking that it might not be likely that this would be used on a big-endian system that uses UTMP login records. In the interest of keeping the code simple I'd leave it as is. If there's a need to change things we always can.

@adriansr getByteOrder() looks like a useful function that Go itself should have, indeed. Or maybe we should have it in a shared location within Beats so code like this here can use it easily.

I think it's likely that someone will use it on big-endian. We've had several contributions to go-libaudit to make auditbeat work on big-endian.

Ok, I've added support for it. Thanks @adriansr for the code snippet!

This is probably fine for low volumes. But we have seen from the auditd module that this call is expensive. Some caching could be a useful future enhancement.

Same, long name and a many exported symbols that we might not need somewhere else.

LoginRecord is encoded and decoded by gob so it needs to be exported, unfortunately.

In filebeat we also store the device id.

I think for now it's unlikely that UTMP files would be stored on separate file systems. I.e. it would be mean different parts of /var/log are on different devices. I'm not even sure that's possible.

How big do we expect these files to be? Looks like this function will load all available logs into memory on first run.

So far I've assumed they're usually quite small, but that might not be true in all cases. The /etc/logrotate.conf on my Ubuntu VM rotates it monthly. On that VM that's getting a number of daily logins/logouts, they're <100K. On a real-world machine with a lot of logins, I assume it could be more, I wonder how many records it would take to be significant.

Nevertheless, I guess we could use a channel between Fetch and ReadNew and output records as events as they come.

This is more or less what filebeat does as well. Have one go-routine per file and forward event by event. That is, the module/input is more or less active all the time. Not sure how well this fits the 'Fetch' semantics in the metricbeat framework. But a bounded channel can indeed help keeping memory usage in place. Given the complexity this introduces I wonder if that's really a problem.

I've changed to using channels now, and storing offsets of the files (instead of re-reading until the last read record). So hopefully that will make it scalable even if the files are very large.

Looks like we are ignoring all messages read so far if we end up here (e.g. we read while copytruncate like logrotation happens). We still want to publish known records? Alternatively we can just error and return here, but rely on fetch to reset the file offset and read new contents the next time Fetch is executed.

Good point, maybe I was a bit too overzealous to make sure we never miss an event here and just returning is better.

However, I'm curious about copytruncate: As I understand it, the logfile is copied (e.g. cp /var/log/wtmp /var/log/wtmp.1) and then truncated. If that happens, I suspect the metricset as it is at the moment (with the file pattern being /var/log/wtmp* by default) would read all of /var/log/wtmp.1 (because it'll have a different inode) - even though it has read all of those records already. Does Filebeat handle this in some way?

But can wtmp be copy-truncated? I thought it's guaranteed to be renamed with moves. I wouldn't want to bring all the complexity of Filebeat's corner cases in here :)

Rename with move should be ok. But them we don't get into this code path.

Instead of having a tail recursive call we could use a for loop and reset some state and use continue in order to jump to beginning of the reader and not drop events. But then it's an edge case that normally should not occur.

The code is now using offsets and if there is some problem in jumping to the stored offset it will jump back to the beginning and re-read the file. This is maybe a bit overzealous, but I think at least for now I'd prefer it if events were sent twice rather than not at all. If there are any problems we can always change.

Turns out gosigar has a GetEndian() method for this.

Oh nice, thanks!