Integration Name

Google Threat Intelligence [ti_google_threat_intelligence]

Dataset Name

No response

Integration Version

0.6.0

Agent Version

9.1.3

Agent Output Type

elasticsearch

Elasticsearch Version

9.1.3

OS Version and Architecture

Red Hat Enterprise Linux 9.6 (Plow)

Software/API Version

No response

Error Message

Error while parsing document for index [logs-ti_google_threat_intelligence_latest.dest_file_ioc-1]: [1:1061] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id 'dDCEBV6A2eN_bjy9XeZDTUBTAAAAAAAA'. Preview of field's value: 'ti_google_threat_intelligence.cryptominer'

Error while parsing document for index [logs-ti_google_threat_intelligence_latest.dest_domain_ioc-1]: [1:944] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id 'dDBA7W4uhtWQgwLQ0gIwJQILAAAAAAAA'. Preview of field's value: 'ti_google_threat_intelligence.infostealer'

Error while parsing document for index [logs-ti_google_threat_intelligence_latest.dest_url_ioc-1]: [1:1214] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id 'dDCXNHbfpp6HlG5zefXxCBlIAAAAAAAA'. Preview of field's value: 'ti_google_threat_intelligence.first_stage_delivery_vectors'

Event Original

No response

What did you do?

What did you see?

Data was dropped due to the mapping conflict.

What did you expect to see?

Data to be ingested successfully.

Anything else?

We have observed the mapping errors with the following transforms/ associated indices:

• transform: logs-ti_google_threat_intelligence.domain_ioc-default-0.6.0
  • index: logs-ti_google_threat_intelligence_latest.dest_domain_ioc-1
• transform: logs-ti_google_threat_intelligence.file_ioc-default-0.6.0
  • index: logs-ti_google_threat_intelligence_latest.dest_file_ioc-1
• transform: logs-ti_google_threat_intelligence.url_ioc-default-0.6.0
  • index: logs-ti_google_threat_intelligence_latest.dest_url_ioc-1
• transform: logs-ti_google_threat_intelligence.ip_ioc-default-0.6.0
  • index: logs-ti_google_threat_intelligence_latest.dest_ip_ioc-1

For example the logs-ti_google_threat_intelligence_latest.dest_file_ioc-1 index can contain multiple data stream data sets such as:

• ti_google_threat_intelligence.ransomware
• ti_google_threat_intelligence.cryptominer

Checking the mapping for logs-ti_google_threat_intelligence_latest.dest_file_ioc-1 shows that the data_stream.dataset is populated with a constant_keyword of ti_google_threat_intelligence.ransomware:

However the related error message shows a document with the data_stream.dataset attempting to be ingested with ti_google_threat_intelligence.cryptominer. This prevents more than one data set from being ingested into the latest index.

Error while parsing document for index [logs-ti_google_threat_intelligence_latest.dest_file_ioc-1]: [1:1061] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id 'dDCEBV6A2eN_bjy9XeZDTUBTAAAAAAAA'. Preview of field's value: 'ti_google_threat_intelligence.cryptominer'