In #87939 we merged support for request/response logging to the KP. As that PR was already quite large, there are a few cleanup tasks that we wanted to address separately:

1. Clean up default pattern layout

The response logs contain a lot of meta which clutters up the console quite a bit and makes the logs difficult to scan, as the default log pattern places the meta before the message. Here's an example:

[2021-02-04T22:13:20.382Z][DEBUG][http.server.response] {"ecs":{"version":"1.7.0"},"client":{"ip":"127.0.0.1"},"http":{"request":{"method":"GET","mime_type":null,"referrer":"http://localhost:5601/ahl/app/dashboards","headers":{"connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36","dnt":"1","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"http://localhost:5601/ahl/app/dashboards","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,ja;q=0.8,gl;q=0.7,es;q=0.6","cookie":"[REDACTED]","x-forwarded-for":"127.0.0.1","x-forwarded-port":"60795","x-forwarded-proto":"http","x-forwarded-host":"localhost:5601","host":"localhost:5601"}},"response":{"body":{"bytes":56765},"status_code":200,"headers":{"content-type":"application/javascript; charset=utf-8","etag":"\"18832936a2c8ab58934beb031ec10bedc394f5fc-/ahl/9007199254740991/bundles/plugin/visTypeTimeseries/-gzip\"","cache-control":"must-revalidate","kbn-name":"kbn-blue","kbn-license-sig":"dbdfb6e0d6b82db37115e70770dcef6a759bdc7c764195552e83d98b76a846ab","vary":"accept-encoding","content-encoding":"gzip"},"responseTime":7}},"url":{"path":"/9007199254740991/bundles/plugin/visTypeTimeseries/visTypeTimeseries.chunk.8.js","query":""},"user_agent":{"original":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"}} GET /9007199254740991/bundles/plugin/visTypeTimeseries/visTypeTimeseries.chunk.8.js 200 7ms - 55.4KB 

We'd like to clean these up by default, and discussed a few options for how we might do this:

1. Remove meta property from default pattern layout
2. Relocate meta property to the end of default pattern layout (doesn't decrease clutter, but makes it easier to scan the logs as the message appears first)
3. Take an approach similar to Elasticsearch where context-specific defaults are applied via a dedicated config file
4. Provide some recommended defaults in the kibana.yml that we ship with Kibana (would require users to uncomment if they choose to use them)

Overall (3) seems ideal to me because it provides the most flexibility, however creating default appender configurations poses some challenges:

• We want to provide a set of predefined defaults only if the user hasn't already set defaults
• configSchema default settings feels like the natural place to do this
• However the default appenders value is never actually used because we inject the "default" legacy appender, so a value always exists there
• So any additional defaults would, AFAICT, need to be loaded in kbn-config's LegacyObjectToConfigAdapter as well (might be fine as a temporary solution until it is removed in 8.0?)

2. Improve get_payload_bytes

The way we are calculating payload bytes can be continually improved, but there are a few immediate things we should address:

• Ensure object responses that are arrays get counted (in [core.logging] Fix issue where logs fail to calculate size of gunzip streams. #90353 this was changed to fix a bug that was breaking Fleet on master, but we should go back and adjust it)
• Add support for zlib streams which are used in Fleet and SIEM

cc @restrry Any other thoughts?