Log4j 2 supports the Gelf Layout. The Gelf spec mandates that lines be terminated with a null value. For reliability we would like to write our GELF formatted logs to files and then transfer them to Logstash using filebeat. We currently have Log4j 2 writing directly to Logstash via TCP but we experienced an issue last week where ElasticSearch ran out of space which caused Logstash to hang, which in turn caused all the apps to hang trying to write to Logstash. So rather than having everything go dead we would like to decouple the TCP connection to Logstash from the application.
Doing this means that filebeats basically just read the raw input from the files and send it to logstash unaltered. I've seen no options that allow filebeats to not parse the data so having it recognize that a null value terminates the line seems to be the next best option.
Log4j 2 supports the Gelf Layout. The Gelf spec mandates that lines be terminated with a null value. For reliability we would like to write our GELF formatted logs to files and then transfer them to Logstash using filebeat. We currently have Log4j 2 writing directly to Logstash via TCP but we experienced an issue last week where ElasticSearch ran out of space which caused Logstash to hang, which in turn caused all the apps to hang trying to write to Logstash. So rather than having everything go dead we would like to decouple the TCP connection to Logstash from the application.
Doing this means that filebeats basically just read the raw input from the files and send it to logstash unaltered. I've seen no options that allow filebeats to not parse the data so having it recognize that a null value terminates the line seems to be the next best option.