Skip to content

[8.x] [UII] Add status tracking for agentless integrations (#199567)#201901

Merged
jen-huang merged 5 commits intoelastic:8.xfrom
jen-huang:backport/8.x/pr-199567
Nov 27, 2024
Merged

[8.x] [UII] Add status tracking for agentless integrations (#199567)#201901
jen-huang merged 5 commits intoelastic:8.xfrom
jen-huang:backport/8.x/pr-199567

Conversation

@jen-huang
Copy link
Copy Markdown
Contributor

Backport

This will backport the following commits from main to 8.x:

Questions ?

Please refer to the Backport tool documentation

## Summary

Resolves https://github.com/elastic/ingest-dev/issues/3933. For
deployments that support agentless, integrations with agentless
deployment mode enabled will allow the status of agentless integration
policies to be tracked.

### Key technical changes

- A new field `supports_agentless` was added to package policies. This
field already exists on agent policies. When an agentless integration is
created, `supports_agentless: true` is now added to both the package
policy and its parent agent policy.
- This allows easier filtering for agentless integrations as we avoid
having to retrieve & check against every parent agent policy.
- This also means existing agentless policies do not get this new status
tracking UI, only new ones created after this change. Since agentless is
not yet GA, I think this is okay.
- `/api/fleet/agent_status/data` now takes optional query params
`pkgName` and `pkgVersion`. When both are specified, the API will check
if agent(s) have ingested data for only that package's datastreams.

## UI walkthrough
<details>
<summary>🖼️ Click to show screenshots</summary>

1. **Integration policies** page now shows two tables for integrations
meeting the above condition, one for agentless policies and one for
agent-based policies:

![image](https://github.com/user-attachments/assets/58c6a932-9bda-4229-ba5f-d341bdbd539a)

2. Clicking the status badge in the agentless policies table opens a
flyout with two steps: confirm agentless enrollment and confirm incoming
data:

![image](https://github.com/user-attachments/assets/e19e6ba0-f40d-48a7-a524-0373934ac46a)

3. Confirm agentless enrollment polls for an agent enrolled into that
integration policy's agent policy. If that agent is reporting an
unhealthy status, the integration component UI is shown. This UI is the
same one used on Fleet > Agents > Agent details page and shows all
components reported by that agent:

![image](https://github.com/user-attachments/assets/ce214f7f-4bdd-48e5-a5eb-a1e8fcc7a512)

4. Once a healthy agentless enrollment is established, confirm incoming
data starts polling for data for that integration ingested by that agent
ID in the past 5 minutes:

![image](https://github.com/user-attachments/assets/7f3de40b-3418-4174-b529-e805407949b6)

5. If data could not be retrieved in 5 minutes, an error message shows
while polling continues in the background:

![image](https://github.com/user-attachments/assets/a3fd198e-1570-4357-9b7f-e541a769d33f)

6. If data is retrieved, a success message is shown:

![image](https://github.com/user-attachments/assets/f4e442af-ca60-4448-9bfb-3f244cd03c2d)
</details>

## Testing
Easiest way to test is use the Cloud deployment from this PR. Enable
Beta integrations and navigate to CSPM. Add a CSPM integration using
`Agentless` setup technology. Then you can track the status of the
agentless deployment on the Integrations policies tab.

For local testing, the following is required to simulate agentless
agent:
1. Add the following to kibana.dev.yml:
```
xpack.cloud.id: 'anything-to-pass-cloud-validation-checks'
xpack.fleet.agentless.enabled: true
xpack.fleet.agentless.api.url: 'https://localhost:8443'
xpack.fleet.agentless.api.tls.certificate: './config/certs/ess-client.crt'
xpack.fleet.agentless.api.tls.key: './config/certs/ess-client.key'
xpack.fleet.agentless.api.tls.ca: './config/certs/ca.crt'
```
2. Apply [this
patch](https://gist.github.com/jen-huang/dfc3e02ceb63976ad54bd1f50c524cb4)
to prevent attempt to create agentless pod
3. Enroll a Fleet Server as usual
4. Enable Beta integrations and navigate to CSPM. Add a CSPM integration
using `Agentless` setup technology.
5. Enroll a normal Elastic Agent to the agent policy for that CSPM
integration by using the token from Enrollment tokens

## To-do
- [x] API tests
- [x] Unit UI tests
- [x] Manual Cloud tests
- [x] File docs request
  - elastic/ingest-docs#1466
- [ ] Update troubleshooting guide link once available

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 3188cda)

# Conflicts:
#	oas_docs/bundle.json
#	oas_docs/bundle.serverless.json
#	oas_docs/output/kibana.serverless.yaml
#	oas_docs/output/kibana.yaml
#	src/core/server/integration_tests/ci_checks/saved_objects/check_registered_types.test.ts
#	x-pack/plugins/fleet/public/components/package_policy_actions_menu.test.tsx
#	x-pack/plugins/fleet/public/components/package_policy_actions_menu.tsx
#	x-pack/plugins/fleet/server/routes/agent/handlers.ts
#	x-pack/plugins/fleet/server/types/models/package_policy.ts
@jen-huang jen-huang added the backport This PR is a backport of another PR label Nov 27, 2024
@jen-huang jen-huang enabled auto-merge (squash) November 27, 2024 02:10
@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Nov 27, 2024
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

kibanamachine and others added 4 commits November 27, 2024 02:26
…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --update --no-serverless'
… src/core/server/integration_tests/ci_checks'
@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
fleet 1246 1251 +5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
fleet 1306 1308 +2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
fleet 1.8MB 1.9MB +16.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
fleet 173.7KB 172.8KB -933.0B
Unknown metric groups

API count

id before after diff
fleet 1433 1435 +2

History

@jen-huang jen-huang merged commit 2a508ef into elastic:8.x Nov 27, 2024
@jen-huang jen-huang deleted the backport/8.x/pr-199567 branch November 27, 2024 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR Team:Fleet Team label for Observability Data Collection Fleet team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants