Skip to content

[Security Solution][Endpoint] Adds logic for checking and completing Sentinelone isolate and release actions#179013

Merged
paul-tavares merged 35 commits intoelastic:mainfrom
paul-tavares:task/olm-8895-sentinelone-isolate-release-async-complete
Apr 9, 2024
Merged

[Security Solution][Endpoint] Adds logic for checking and completing Sentinelone isolate and release actions#179013
paul-tavares merged 35 commits intoelastic:mainfrom
paul-tavares:task/olm-8895-sentinelone-isolate-release-async-complete

Conversation

@paul-tavares
Copy link
Copy Markdown
Contributor

@paul-tavares paul-tavares commented Mar 19, 2024

Summary

Adds checks for isolate and release response action for SentinelOne to see if they have completed. Changes include:

  • isolate and release are no longer marked "complete" once request is sent to SentinelOne. They are kept in pending state
  • Added logic to the SentinelOne Response Actions client to use the activity data from SentinelOne (index logs-sentinel_one.activity-default) and check if isolate and release action were complete
    • logic is invoked from the background task, which is not yet fully enabled
    • Note that the data in the SentinelOne index may be delayed. The frequency of how often that data is pulled in is defined via the SentinelOne Integration
  • Enhanced the SentinelOne host script so that it also automatically generates a SentinelOne alert when the host VM running the SentinelOne agent is created
  • Refactor: Movedserver/endpoint/mocks.ts file to its own directory (server/endpoint/mocks/**)
    • File is getting to large... having it in a directory will allows us to break this file down into smaller pieces while still exporting out of server/endpoint/mocks path.
    • Added new utils.mock.ts that includes 1 new generic utilty - applyEsClientSearchMock()
Sample output to kibana log (when running with `debug` mode)

Output showing completion of an Isolate action by checking SentinelOne's activity log:

[2024-04-05T08:50:08.357-04:00][DEBUG][plugins.securitySolution.CompleteExternalActionsTaskRunner.c29i02] Started: Checking status of external response actions
[2024-04-05T08:50:08.365-04:00][DEBUG][plugins.securitySolution.SentinelOneActionsClient] searching for isolate responses from [logs-sentinel_one.activity-default] index with:
{
  index: 'logs-sentinel_one.activity-default',
  query: {
    bool: {
      must: [
        { terms: { 'sentinel_one.activity.type': [ 1001, 2010 ] } }
      ],
      should: [
        {
          bool: {
            filter: [
              {
                term: {
                  'sentinel_one.activity.agent.id': '1913920934584665209'
                }
              },
              {
                range: {
                  'sentinel_one.activity.updated_at': { gt: '2024-04-05T12:49:12.113Z' }
                }
              }
            ]
          }
        }
      ],
      minimum_should_match: 1
    }
  },
  collapse: {
    field: 'sentinel_one.activity.agent.id',
    inner_hits: {
      name: 'first_found',
      size: 1,
      sort: [ { 'sentinel_one.activity.updated_at': 'asc' } ]
    }
  },
  _source: false,
  sort: [ { 'sentinel_one.activity.updated_at': { order: 'asc' } } ],
  size: 1000
}
[2024-04-05T08:50:08.421-04:00][DEBUG][plugins.securitySolution.SentinelOneActionsClient] Search results for SentinelOne isolate activity documents:
{
  took: 46,
  timed_out: false,
  _shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
  hits: {
    total: { value: 1, relation: 'eq' },
    max_score: null,
    hits: [
      {
        _index: '.ds-logs-sentinel_one.activity-default-2024.04.05-000001',
        _id: 'UP8uvXmyc0P6OBys00iGA2gzFOc=',
        _score: null,
        fields: { 'sentinel_one.activity.agent.id': [ '1913920934584665209' ] },
        sort: [ 1712321353320 ],
        inner_hits: {
          first_found: {
            hits: {
              total: { value: 1, relation: 'eq' },
              max_score: null,
              hits: [
                {
                  _index: '.ds-logs-sentinel_one.activity-default-2024.04.05-000001',
                  _id: 'UP8uvXmyc0P6OBys00iGA2gzFOc=',
                  _score: null,
                  _source: [Object],
                  sort: [Array]
                }
              ]
            }
          }
        }
      }
    ]
  }
}
[2024-04-05T08:50:08.422-04:00][DEBUG][plugins.securitySolution.SentinelOneActionsClient] 1 isolate action responses generated:
[
  {
    '@timestamp': '2024-04-05T12:50:08.422Z',
    agent: { id: 'c06d63d9-9fa2-046d-e91e-dc94cf6695d8' },
    EndpointActions: {
      action_id: '583b8a6c-0227-497e-9bbf-8af551e69ed6',
      input_type: 'sentinel_one',
      started_at: '2024-04-05T12:50:08.422Z',
      completed_at: '2024-04-05T12:50:08.422Z',
      data: { command: 'isolate' }
    },
    error: undefined,
    meta: {
      elasticDocId: 'UP8uvXmyc0P6OBys00iGA2gzFOc=',
      activityLogEntryId: '1921767625558733652',
      activityLogEntryType: 1001,
      activityLogEntryDescription: 'Agent ptavares-sentinelone-1371 was disconnected from network.'
    }
  }
]
[2024-04-05T08:50:08.426-04:00][DEBUG][plugins.securitySolution.CompleteExternalActionsTaskRunner.c29i02.QueueProcessor] Processing batch [1] with [1] items. Items remaining in queue: [0]
[2024-04-05T08:50:09.515-04:00][DEBUG][plugins.securitySolution.CompleteExternalActionsTaskRunner.c29i02.QueueProcessor] Processed [1] batches and a total of [1] items

How to Test

  • Bring up the stack
  • Setup SentinelOne host. you can can the following script to get it all setup: node x-pack/plugins/security_solution/scripts/endpoint/run_sentinelone_host.js . The credentials needed to setup the host can be retrieved from here.
  • isolate or release the host. The response actions should remain in pending until the action is complete in SentinelOne and data (sentinelone activity logs) is ingested back to elasticsearch. NOTE: this could take a bit of time depending on the frequency setup in the integration policy for sentinelone

Checklist

@paul-tavares paul-tavares added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.14.0 labels Mar 19, 2024
@paul-tavares paul-tavares self-assigned this Mar 19, 2024
@paul-tavares
Copy link
Copy Markdown
Contributor Author

/ci

@paul-tavares
Copy link
Copy Markdown
Contributor Author

/ci

…e-release-async-complete' into task/olm-8895-sentinelone-isolate-release-async-complete
@paul-tavares
Copy link
Copy Markdown
Contributor Author

/ci

1 similar comment
@paul-tavares
Copy link
Copy Markdown
Contributor Author

/ci

@paul-tavares
Copy link
Copy Markdown
Contributor Author

/ci

paul-tavares and others added 9 commits April 1, 2024 11:49
…e-release-async-complete' into task/olm-8895-sentinelone-isolate-release-async-complete
…nelone-isolate-release-async-complete

# Conflicts:
#	x-pack/plugins/security_solution/scripts/endpoint/sentinelone_host/index.ts
#	x-pack/plugins/security_solution/server/endpoint/mocks/mocks.ts
#	x-pack/plugins/security_solution/server/endpoint/services/actions/clients/sentinelone/sentinel_one_actions_client.ts
…nelone-isolate-release-async-complete

# Conflicts:
#	x-pack/plugins/security_solution/server/endpoint/services/actions/clients/sentinelone/sentinel_one_actions_client.test.ts
#	x-pack/plugins/security_solution/server/endpoint/services/actions/clients/sentinelone/sentinel_one_actions_client.ts
@paul-tavares paul-tavares marked this pull request as ready for review April 5, 2024 12:57
@paul-tavares paul-tavares requested review from a team as code owners April 5, 2024 12:57
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

@paul-tavares paul-tavares requested review from tomsonpl and removed request for parkiino April 5, 2024 12:57
@kibana-ci
Copy link
Copy Markdown

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 5296 5298 +2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 73.4KB 73.4KB +93.0B
Unknown metric groups

API count

id before after diff
securitySolution 195 196 +1

ESLint disabled line counts

id before after diff
securitySolution 496 497 +1

Total ESLint disabled count

id before after diff
securitySolution 573 574 +1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @paul-tavares

Copy link
Copy Markdown
Contributor

@tomsonpl tomsonpl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks for doing this. It helped me understand how pendingActions work so I could prepare similar for CS 👍
Also TILT: collapse search, I haven't tried it before 👍

I left basically two suggestions for you to consider (nothing blocking from merging) :

  • put activity codes in enum
  • refactor from using _source to fields in search

export interface SentinelOneActionRequestCommonMeta {
/** The S1 agent id */
agentId: string;
/** The S1 agent assigned UUID */
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have trouble understanding what is the difference between agentId and agentUUID - could you explain in more details?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess one is referencing Elastic Agent with the integration?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure its the "official" answer, but:

In SentinelOne, agents have both an id and a uuid. It seems to me that the uuid can be re-generated - I think I saw some action on the agent there to re-assign. new UUID. The id seems to be more in line with how we use "id" in elastic and this also what is used internally in SentinelOne to trigger several actions or run scripts.

Because we are using the UUID of the agent in kibana we need to (in this case) store the (what I am calling "internal") id of the agent in SentinelOne with the action so that we can use it later without having to call their API again.

index: SENTINEL_ONE_ACTIVITY_INDEX,
query,
// There may be many documents for each host/agent, so we collapse it and only get back the
// first one that came in after the isolate request was sent
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

command === 'isolate'
? [
// {
// "id": 1001
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would you consider putting keeping these IDs in enum ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of enums as they have some drawbacks. I did think about putting all of these codes in a const object, but at this time I am not seeing the need to use them outside of this code here, so I felt it was unnecessary.

sort: [{ 'sentinel_one.activity.updated_at': 'asc' }],
},
},
// we don't need the source. The document will be stored in `inner_hits.first_found`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I've heard fields is preferred above _source when possible. Does collapse work the same way as fields meaning it returns just the agent.id field, or the whole document?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. Collapse (in my understanding) takes the search results, groups them by a given field and then applies additional logic to it. Son in this case, it groups them by S1 agent ID, then than it sorts those in ascending order, and then it takes the first (most recent) record from that and it stores it in a property called first_found.

You can see what collapse does by running this in the dev console (just update the Agent ids below):

GET logs-sentinel_one.activity-default/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "sentinel_one.activity.type": [
              1001,
              2010
            ]
          }
        }
      ],
      
      "should": [
        {
          "bool": {
            "filter": [
              { "term": { "sentinel_one.activity.agent.id": "1913920934584665209" } },
              { "range": { "sentinel_one.activity.updated_at": { "gt": "2024-04-02T12:59:22.229Z" } } }
            ]
          }
        }
      ],
      "minimum_should_match": 1
    }
  },
  "_source": false,
  "collapse": {
    "field": "sentinel_one.activity.agent.id",
    "inner_hits": {
      "name": "first_found",
      "size": 1,
      "sort": [ { "sentinel_one.activity.updated_at": "asc" } ]
    }
  }, 
  "size": 1000,
  "sort": [
    {
      "sentinel_one.activity.updated_at": {
        "order": "desc"
      }
    }
  ]
}

you can also reference the docs here: https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.html

: actionRequest.agent.id,
data: { command },
error:
activityLogEntryType === 2010 && command === 'isolate'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, I think 2010 as an enum would help in readability

hit.inner_hits = {
first_found: {
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
hits: { hits: [this.toEsSearchHit(hit._source!, hit._index)] },
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider refactoring from using _source to fields.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you will need to provide real working examples for me to understand how fields would help here

Copy link
Copy Markdown
Member

@ashokaditya ashokaditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested it out and it works as expected. The agent isolation status is not consistent with the action log but that is being updated with the PR I'm working on.

Screenshot from my test
Screenshot 2024-04-09 at 2 09 56 PM

// due to use of `collapse
_source: false,
sort: [{ 'sentinel_one.activity.updated_at': { order: 'asc' } }],
size: 1000,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume this should be 10,000

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its very unlikely that we would even hit 1000 documents in this context... but sure, I'll update to 10k in my next PR.

inner_hits: {
name: 'first_found',
size: 1,
sort: [{ 'sentinel_one.activity.updated_at': 'asc' }],
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe, the sort order for sentinel_one.activity.updated_at in the inner_hits as well as the outer query should be desc so that the most recent item is first in the array/list.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no... it should be asc here based on the approach we need to take to determine if the action was complete. I can explain more if you like, but essentially: you want the first document that was received where the updated_at value is greater than when the response action was sent

@paul-tavares paul-tavares merged commit 6348ab3 into elastic:main Apr 9, 2024
@kibanamachine kibanamachine added the backport:skip This PR does not require backporting label Apr 9, 2024
@paul-tavares paul-tavares deleted the task/olm-8895-sentinelone-isolate-release-async-complete branch April 9, 2024 12:42
@paul-tavares
Copy link
Copy Markdown
Contributor Author

Thanks @ashokaditya .

Re: The agent isolation status is not consistent with the action log but that is being updated with the PR I'm working on.

Yes, that is expected for now. My guess is that you are seeing isolated must before the action shows completed, correct?

@ashokaditya
Copy link
Copy Markdown
Member

Thanks @ashokaditya .

Re: The agent isolation status is not consistent with the action log but that is being updated with the PR I'm working on.

Yes, that is expected for now. My guess is that you are seeing isolated must before the action shows completed, correct?

Correct. Same for release. The isolated badge disappears while the action is pending.

However, VM was inaccessible at the same instance when the status showed Isolated even though the action was pending. Likewise, for release, the VM accessible when the badge disappeared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.14.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants