Hey @jamiehynds, @andrewkroh, let me know what you guys think on the analysis here:

Going through the MISP documentation about expiring indicators, following articles are found useful:
https://www.misp-project.org/2019/09/12/Decaying-Of-Indicators.html/
https://www.misp-project.org/misp-training/handout/a.5-decaying-indicators_handout.pdf

Both of the articles talk about the approach taken by MISP to expire the indicators using decay models. Users have option to even create their own decay models.

Among the 2 datastreams (threat_attributes and threat) we support, following is the analysis:

1. Threat Attributes:

We currently use the REST API: attributes/restSearch documented at: https://www.misp-project.org/openapi/#tag/Attributes/operation/restSearchAttributes to fetch the list of attributes (indicators as per MISP doc)

The articles also indicate that this API attributes/restSearch has option to include decay_score property in the response when includeDecayScore is set to true inside request body. The response now includes following:

"decay_score": [
    {
        "base_score": 80,
        "decayed": true,
        "decaying_model": {
            "id": "12345",
            "name": "Phishing model"
        },
        "score": 10.5
    }
],

It is an array containing objects with decay model info and score based on each model that users have enabled/selected on their end for their attributes. There are also default models supplied by MISP as default..
The decayed property shows if the attribute/indicator is no longer relevant and hence expired.
Since multiple models could be possible, some models might mark the same indicator as expired and some don't.
Finding consensus on expiration:
- Majority: We can find consensus based on the majority of the models marking the indicator as decayed or not. For example, if the array size is 3 (which means 3 models are used to calculate decay), if 2 models has "decayed": true, then we could agree that the indicator is probably expired.
- Atleast one: If single model marked the indicator as decayed, then we consider the indicator is expired.
- All: If all the models marked the indicator as decayed, then we consider the indicator is expired.
  (Let me know what you guys think should be the approach for agreeing on expiration. I would probably choose majority.)
Once we consider the indicator is expired, we can add a decayed_at timestamp inside the ingest pipeline. This is similar to Anomali's deleted_at timestamp.
The transform uses this decayed_at timestamp under retention_policy to delete the document from the destination index similar to Anomali: https://github.com/elastic/integrations/blob/main/packages/ti_anomali/elasticsearch/transform/latest_ioc/transform.yml#L26

2. Threat:

We currently use the REST API: events/restSearch documented at: https://www.misp-project.org/openapi/#tag/Events/operation/restSearchEvents to fetch the list of events. Events contains one or more attributes as per MISP Event and Create Event
Neither the article nor the REST API contains enough documentation regarding expiration of attributes/indicators pertaining to the event. The response from the API doesn't include the decay scores too.
Since expiration is attribute-related, we may not consider this datastream for IOC expiration.

[TI_MISP] Add support for IOC expiration #7326

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions