Skip to content

rapid7_insightvm: use terminate function to stop pagination in HTTPJSON templates#15093

Closed
navnit-elastic wants to merge 1 commit intoelastic:mainfrom
navnit-elastic:rapid7_insightvm-2.3.0
Closed

rapid7_insightvm: use terminate function to stop pagination in HTTPJSON templates#15093
navnit-elastic wants to merge 1 commit intoelastic:mainfrom
navnit-elastic:rapid7_insightvm-2.3.0

Conversation

@navnit-elastic
Copy link
Copy Markdown
Contributor

@navnit-elastic navnit-elastic commented Aug 29, 2025

Proposed commit message

rapid7_insightvm: use terminate function to stop pagination in HTTPJSON templates

Previously, it relied on a template error to stop pagination, which resulted in
the fleet's health status changing to degraded.

The "terminate" processor is available starting from stack version 8.19.3 and 9.1.3.
Reference: https://github.com/elastic/beats/pull/45810.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@navnit-elastic navnit-elastic self-assigned this Aug 29, 2025
@navnit-elastic navnit-elastic added enhancement New feature or request Integration:rapid7_insightvm Rapid7 InsightVM Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Aug 29, 2025
@navnit-elastic navnit-elastic force-pushed the rapid7_insightvm-2.3.0 branch from 5fea525 to 4bab472 Compare August 29, 2025 06:39
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Aug 29, 2025

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Aug 29, 2025

💚 Build Succeeded

cc @navnit-elastic

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Aug 29, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@navnit-elastic navnit-elastic marked this pull request as ready for review August 29, 2025 07:16
@navnit-elastic navnit-elastic requested a review from a team as a code owner August 29, 2025 07:16
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Aug 29, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6
efd6 reviewed Aug 29, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@efd6 efd6 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs assert.hit_count in the system tests (looking at this, I'm not sure that the tests would actually pass in the old state; AFAICS there should be 3 events in the assets test and 7 for the vulnerability test; but they give only 2 each).

With this change, in the assets data stream, I see

2025/08/29 18:01:48 DEBUG found 0 hits in logs-rapid7_insightvm.asset-38517 data stream
2025/08/29 18:01:49 DEBUG found 1300 hits in logs-rapid7_insightvm.asset-38517 data stream
2025/08/29 18:01:53 DEBUG found 6913 hits in logs-rapid7_insightvm.asset-38517 data stream

(in the vulnerability stream, it stays at 2)

@navnit-elastic
Copy link
Copy Markdown
Contributor Author

navnit-elastic commented Aug 29, 2025

Hi @efd6, by looking at the agent logs, It appears that the template execution is not getting terminated when it should be. The message template execution terminated appears multiple time in logs. Did I make the correct use of the terminate function?

@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Aug 29, 2025
@efd6
Copy link
Copy Markdown
Contributor

efd6 commented Aug 31, 2025

@navnit-elastic The terminate helper prevents any error from being emitted, so the pagination will not stop. It terminates the evaluation of the template, nothing else. The option that is needed here is ignore_empty_value, but this is only on the cursor assignment templates. I do not think we have a way of suppressing the same thing when the destination is not the cursor.

@efd6 efd6 mentioned this pull request Sep 1, 2025
Merged
6 tasks
@navnit-elastic
Copy link
Copy Markdown
Contributor Author

navnit-elastic commented Sep 1, 2025

Thanks @efd6 for the clarification, The confusion is arises from the template error and fleet health status being degraded (which is the root cause of flaky tests failures) representing different aspects.
We need a way to stop pagination without updating the fleet health status to degraded, which I believe be addressed after PR elastic/beats#46332 is merged. This PR should be closed.

@efd6
Copy link
Copy Markdown
Contributor

efd6 commented Sep 1, 2025

Yeah. When the fix in beats is merged, the fix here becomes a matter of just bumping the kibana.version.

This was referenced Sep 5, 2025
Closed
Merged
Merged
Merged
@kcreddy
Copy link
Copy Markdown
Contributor

kcreddy commented Sep 5, 2025

@navnit-elastic, the backport PRs are also merged.
You can reopen this PR to use do_not_log_failure option as per the PR.

@navnit-elastic
Copy link
Copy Markdown
Contributor Author

navnit-elastic commented Sep 5, 2025

@kcreddy, the changes will be available starting from the next release. I'll raise a separate PR to bump the Kibana version after 8.19.4 and 9.1.4 are released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 left review comments

Assignees

@navnit-elastic navnit-elastic

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:rapid7_insightvm Rapid7 InsightVM Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@navnit-elastic @elasticmachine @efd6 @kcreddy @andrewkroh