See .go-version file for current go version in the repository.

Is there some way to access that from mage?

dev-tools/mage/settings.go exports GoVersion

Thanks!

Where did you get this line from?

Docker's example logging plugin does this, so every other plugin copied it and does the same thing. I'm guessing it's to 'reset' the reader in case of some undefined error, but I can't really make sense of it.

Implementation: https://github.com/gogo/protobuf/blob/master/io/uint32.go#L114

It does not reset the reader, but returns an ErrShortBuffer. The generated reader reads the complete event into memory and then unmarshals it. The 2e6 is the max read buffer size.

Once you get a message that is potentially bigger, you are stuck. The length field has already been read (read pointer has advanced), and you MUST NOT continue reading from the reader once you get ErrShortBuffer. This potentially leaves you with another problem.

I think it is fair to rule out old big messages (keep memory usage in check). Just creating a new reader (as done here) without consuming any more bytes will put you in the middle of a protobuf message, giving you an invalid read. If we want a reader that can drop large messages, then we need another implementation. But the current loop will happily produce errors and reinitialize the reader over and over again after until it did find a valid length field. The way protobuf is encoded, there are lengths fields for nested structures and strings all over the place. Unmarshaling will likely fail over and over again itself.

Ah, thanks. I'll keep reading into this.

Have you considered an alternative non-blocking mode?

...There's a non-blocking mode?

See: https://github.com/elastic/beats/blob/master/libbeat/beat/pipeline.go#L176

We run CloseClientWithFile in a separate go-routine (building an unbuffered 'channel'), because we don't want the API to block if CloseClientWithFile blocks. Due to the mutex here we do allow CreateClientWithConfig to block indirectly, though. Is this ok?

Yes. If CreateClientWithConfig bombs out while the HTTP handler is waiting for it, the container startup will fail and an error will be returned to the user via the CLI or whatever they're using to start. Otherwise a container might fail after a user a started it, or we get into some weird scenario where a container is running but the user doesn't know it's not logging. I'm less worried about the container stop passing its errors to the user.

To clarify: We are ok with CreateClientWithConfig and startLoggingHandler blocking?

The WaitClose setting (currently 0 and not exposed) could force startLoggingHandler to block for as many seconds as are configured on this setting (no matter which queue).

If you want similar behavior to other Beats, consider common.NewConfigFrom(...)

common.NewConfigFrom(...) is called after parseCfgKeys. I had to hack this in because NewConfigFrom doesn't like the map[string]string structures we get from docker. I had to coerce the string value into an an interface wrapper so it'll handle config keys like ["192.168.1.2","192.168.1.3"]

Hm, sounds like a bug in go-ucfg. It should happily accept map[string]string.

Oh, it'll accept map[string]string, but if you pass it something like

map[string]string{
    "output.elasticsearch.hosts" : `["192.168.1.2","192.168.1.3"]`
}

You get Elasticsearch url: http://['172.18.0.3:9200,'172.18.0.2']:9200

oh, I see. The -E flag is implemented in libbeat/cfgfile/cfgfile.go. It uses SettingFlag from libbeat/common/flags.go. The final flag parsing is implemented here: https://github.com/elastic/go-ucfg/blob/master/flag/value.go#L48

The value parser is an internal package, but maybe we can export it for use-cases like this: CLI settings getting passed indirectly.

to restart the reader after an error?

Yah, Steffen and I were discussing this. This was from docker's example, but it's almost certainly a terrible idea.

can we move this to an issue?

What exactly? How do debug?

Oh, the Issues? Yah, that's getting ported.