We have lots of bulk write operations which will produce around 500-600GB audit log events and we think that is too much since most of them are related (90% of them):

{"type":"audit", "timestamp":"2020-08-10T10:02:18,036+0200", "node.id":"VflKv3IKT1iL5rre6PGOOw", "event.type":"transport", "event.action":"access_granted", "user.name":"elastic", "user.realm":"reserved", "user.roles":["superuser"], "origin.type":"rest", "origin.address":"127.0.0.1:54454", "request.id":"514DjCkJQfmnoxjQIZ30lw", "action":"indices:data/write/index:op_type/create", "request.name":"BulkItemRequest", "indices":["2020-08-07"]}"

We want to get rid of this logs from the audit systems (Which will help us save more 90% of the disks), I have read the corresponding audit filters and policy provided by Elastic, but I don't see there is an option we can filter out by the "action" field. It would be even better if it can filter by the combination of request.name, action, and index.

Original Report:

Elasticsearch version (bin/elasticsearch --version):
7.7.1

Plugins installed: []

JVM version (java -version):
1.8.0_232

OS version (uname -a if on a Unix-like system):
#56~18.04.1-Ubuntu

Description of the problem including expected versus actual behavior:
I would like to filter out some audit activities by the log4j regex filter, however, it seems there is some problem to filter out.

Here is the audit log produced:
{"type":"audit", "timestamp":"2020-08-10T10:30:41,681+0200", "node.id":"VflKv3IKT1iL5rre6PGOOw", "event.type":"transport", "event.action":"access_granted", "user.name":"elastic", "user.realm":"reserved", "user.roles":["superuser"], "origin.type":"rest", "origin.address":"127.0.0.1:55202", "request.id":"5so0C_m9SJSw3J1EbMUbMQ", "action":"indices:data/write/index:op_type/create", "request.name":"BulkItemRequest", "indices":["2020-08-07"]}
I can see the regex works well if I only want to filter by the content of any json filed, such as .indices:data/write/index:op_type/create. or .request.name.

However, if I want to filter it out with more precise regex, such as ."action":"indices:data/write/index:op_type/create".

Then nothing will happen.

It seems the double quote and colon are not identified properly.

Steps to reproduce:

1. Download fresh elastic 7.7.1 and unpack it
2. Update elaticsearch.yml to enable json based audit log
   xpack.security.enabled: true xpack.security.audit.enabled: true
3. Config audit log log4j regex filter: such as:

appender.audit_rolling.filter.regex.regex = .*"action".*
appender.audit_rolling.filter.regex.onMatch = DENY
appender.audit_rolling.filter.regex.onMismatch = ACCEPT

4. Run the elastic and produce some audit log:
   curl -u elastic:123456 -X POST "localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d' { "create" : { "_index" : "2020-08-07", "_id" : "1" } } { "field1" : "value1" } { "create" : { "_index" : "2020-08-07", "_id" : "2" } } { "field1" : "value3" } { "create" : {"_id" : "1", "_index" : "2020-08-07"} } { "doc" : {"field2" : "value2"} }
5. We can monitor from audit log file that none of audit log is filtered because the regex contains double quote.
6. If we try to update the regex like this:
   appender.audit_rolling.filter.regex.regex = .*action.*
   Then it works. The similar problem also happens for the colon.

If we define the regex like this:
appender.audit_rolling.filter.regex.regex = .*request\\.name\.*:.*BulkItemRequest.*

This will not work.

However, if I remove the colon, it works again.
appender.audit_rolling.filter.regex.regex = .*request\\.name\.*BulkItemRequest.*

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1.To run the following to create the audit log behavior
curl -u elastic:123456 -X POST "localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d' { "create" : { "_index" : "2020-08-07", "_id" : "1" } } { "field1" : "value1" } { "create" : { "_index" : "2020-08-07", "_id" : "2" } } { "field1" : "value3" } { "create" : {"_id" : "1", "_index" : "2020-08-07"} } { "doc" : {"field2" : "value2"} }

2.No mapping is needed
3.special setting for log4j:

appender.audit_rolling.filter.regex.regex = .*"action".*
appender.audit_rolling.filter.regex.onMatch = DENY
appender.audit_rolling.filter.regex.onMismatch = ACCEPT

4. special setting for elaticsearch.yml

xpack.security.enabled: true
xpack.security.audit.enabled: true

Provide logs (if relevant):