Skip to content

Add action responses data stream#9661

Merged
aleksmaus merged 2 commits intoelastic:mainfrom
aleksmaus:feature/action_responses_datastream
May 28, 2024
Merged

Add action responses data stream#9661
aleksmaus merged 2 commits intoelastic:mainfrom
aleksmaus:feature/action_responses_datastream

Conversation

@aleksmaus
Copy link
Copy Markdown
Contributor

@aleksmaus aleksmaus commented Apr 22, 2024

Proposed commit message

Add action responses data stream: logs-osquery_manager.action.responses-default

This allows osquerybeat to post the actions responses directly to elasticsearch and fix the issues with the current the transform job based approach, where the actions results could be lost at scale, and presently there is no better solution to address this at the elasticsearch stack.
For more details check this ticket:
https://github.com/elastic/security-team/issues/8893

Since the results are now posted into the proper logs-osquery_manager.action.responses-default datastream, Kibana would need to be adjusted to use it instead of the currently used index .logs-osquery_manager.action.responses-default

There will be the osquerybeat PR with corresponding changes, related to this.

The package manifest updated constraint to 8.15 version of the stack.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Full regression testing.

Related issues

Screenshots

Screenshot 2024-04-22 at 11 22 31 AM

The new logs-osquery_manager.action.responses-default document example:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-osquery_manager.action.responses-default-2024.04.22-000001",
        "_id": "mmpnBo8BWSVhX2NCTq2S",
        "_score": 1,
        "_source": {
          "agent": {
            "name": "mi6",
            "id": "33a86aa3-4932-4846-8125-6ef996d8b2e7",
            "type": "osquerybeat",
            "ephemeral_id": "670ae42f-cbcb-49c2-8214-4d9bd401f55a",
            "version": "8.13.1"
          },
          "agent_id": "33a86aa3-4932-4846-8125-6ef996d8b2e7",
          "elastic_agent": {
            "id": "33a86aa3-4932-4846-8125-6ef996d8b2e7",
            "version": "8.13.1",
            "snapshot": false
          },
          "action_input_type": "osquery",
          "action_data": {
            "query": "select * from osquery_info",
            "id": "c98b474d-1635-45ce-8fd9-9d814ee1dfcc"
          },
          "completed_at": "2024-04-22T15:22:54.306025Z",
          "action_response": {
            "osquery": {
              "count": 1
            }
          },
          "@timestamp": "2024-04-22T15:22:54.306Z",
          "ecs": {
            "version": "8.0.0"
          },
          "action_id": "281f35b4-fffd-4c36-a9f1-bfed002b9f9a",
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "osquery_manager.action.responses"
          },
          "host": {
            "hostname": "mi6",
            "os": {
              "build": "23E224",
              "kernel": "23.4.0",
              "name": "macOS",
              "family": "darwin",
              "type": "macos",
              "version": "14.4.1",
              "platform": "darwin"
            },
            "ip": [
              "fe80::f4d4:88ff:fe6c:4520",
              "fe80::88:cf6:575c:32d5",
              "192.168.50.246",
              "fe80::10c9:d6ff:febc:c723",
              "fe80::10c9:d6ff:febc:c723",
              "fe80::1b7d:a2a5:2d5d:3265",
              "fe80::64a0:4808:f967:c032",
              "fe80::7166:a04c:47bb:2b29",
              "fe80::ce81:b1c:bd2c:69e"
            ],
            "name": "mi6",
            "id": "70A8C8D7-AF7E-5BCB-BF12-E2B381A5B1AC",
            "mac": [
              "12-C9-D6-BC-C7-23",
              "36-35-D5-EB-20-80",
              "36-35-D5-EB-20-84",
              "36-35-D5-EB-20-88",
              "CA-B1-D3-08-D0-0F",
              "CA-B1-D3-08-D0-10",
              "CA-B1-D3-08-D0-11",
              "CA-B1-D3-08-D0-2F",
              "CA-B1-D3-08-D0-30",
              "CA-B1-D3-08-D0-31",
              "F4-D4-88-6C-45-20",
              "F6-D4-88-6C-45-20"
            ],
            "architecture": "arm64"
          },
          "started_at": "2024-04-22T15:22:54.100066Z",
          "event": {
            "agent_id_status": "verified",
            "ingested": "2024-04-22T15:23:04Z",
            "dataset": "osquery_manager.action.responses"
          }
        }
      }
    ]
  }
}

@aleksmaus aleksmaus requested a review from a team as a code owner April 22, 2024 15:26
@aleksmaus aleksmaus mentioned this pull request Apr 22, 2024
Merged
6 tasks
@aleksmaus
Copy link
Copy Markdown
Contributor Author

aleksmaus commented Apr 22, 2024

This PR currently fails CI because 8.15 build specified in the manifest is not available at the moment.

@aleksmaus aleksmaus requested review from a team and andrewkroh April 24, 2024 15:41
@aleksmaus
Copy link
Copy Markdown
Contributor Author

aleksmaus commented May 13, 2024

/test

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented May 13, 2024

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
25.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 13, 2024

💚 Build Succeeded

History

@aleksmaus aleksmaus requested a review from mjwolf May 14, 2024 14:54
@szwarckonrad szwarckonrad mentioned this pull request May 21, 2024
Merged
szwarckonrad added a commit to elastic/kibana that referenced this pull request May 24, 2024
@szwarckonrad
[EDR Workflows][Osquery] Use newly added action responses data stream (…
9bafd06 
…#183892)

**Prerequisite**: elastic/elasticsearch#108849

**Follow-up**: elastic/integrations#9661

This PR introduces a new index
`logs-osquery_manager.action.responses-default` for action responses.
This index will be added in Osquery Manager integration version `1.12`
and will replace the existing
`.logs-osquery_manager.action.responses-default`, which is currently
populated by a transform from `.fleet-actions`.

Since most users will still be using the old integration package, we
ensured that the implementation checks the old index first and returns
the response from there unless the new index is available. If the new
index is available, the response will come from it. This change ensures
compatibility with all user scenarios.
szwarckonrad
szwarckonrad approved these changes May 27, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@szwarckonrad szwarckonrad left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the latest changes in Kibana, which expect the new index to exist. If it doesn't, it falls back to the current one. I confirmed that these changes work across all Osquery functionalities in Kibana:

  • Live query / Pack live query
  • Query history
  • Pack periodic runs
  • Automated response actions

In all these scenarios, I observed the logs-osquery_manager.action.responses-default index being properly populated and consumed by Kibana.

tomsonpl
tomsonpl approved these changes May 28, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@tomsonpl tomsonpl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@aleksmaus aleksmaus merged commit 65a0f10 into elastic:main May 28, 2024
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 28, 2024

Package osquery_manager - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=osquery_manager

@andrewkroh andrewkroh added the Integration:osquery_manager Osquery Manager label Jul 22, 2024
@richlv
Copy link
Copy Markdown

richlv commented Oct 7, 2024

Heya, the issue that's referenced in the description, https://github.com/elastic/security-team/issues/8893 , does not seem to be accessible.
Could it please be shared, or at least some additional detail shared on how this would manifest itself for users?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mjwolf mjwolf mjwolf approved these changes

@tomsonpl tomsonpl tomsonpl approved these changes

@szwarckonrad szwarckonrad szwarckonrad approved these changes

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

Assignees

No one assigned

Labels

Integration:osquery_manager Osquery Manager

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@aleksmaus @elasticmachine @richlv @mjwolf @tomsonpl @szwarckonrad @andrewkroh