Skip to content

Shipper output running into gRPC marshalling errors on filebeat #34319

@fearful-symmetry

Description

@fearful-symmetry

cc @cmacknz
Currently, filebeat with the shipper output is producing a large number of gRPC marshalling errors:

{"log.level":"error","@timestamp":"2023-01-19T20:43:04.427Z","message":"18/249: \"failed to convert event fields to protobuf: proto: invalid type: []string\", dropped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"shipper","log.origin":{"file.line":210,"file.name":"shipper/shipper.go"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-19T20:43:04.427Z","message":"19/249: \"failed to convert event fields to protobuf: proto: invalid type: []string\", dropped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log.logger":"shipper","log.origin":{"file.line":210,"file.name":"shipper/shipper.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-19T20:43:04.427Z","message":"20/249: \"failed to convert event fields to protobuf: proto: invalid type: []string\", dropped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"shipper","log.origin":{"file.line":210,"file.name":"shipper/shipper.go"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-19T20:43:04.428Z","message":"21/249: \"failed to convert event fields to protobuf: proto: invalid type: []string\", dropped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log.logger":"shipper","log.origin":{"file.line":210,"file.name":"shipper/shipper.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-19T20:43:04.428Z","message":"22/249: \"failed to convert event fields to protobuf: proto: invalid type: []string\", dropped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"shipper","log.origin":{"file.line":210,"file.name":"shipper/shipper.go"},"ecs.version":"1.6.0"}

Looks like this is happening in convertMapStr() in shipper/shipper.go, presumably this switch block needs more cases so we can convert []string to []interface :

		switch v := value.(type) {
		case mapstr.M:
			protoValue, err = convertMapStr(v)
		default:
			protoValue, err = helpers.NewValue(v)
		}

Metadata

Metadata

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions