Updated "output" to test package field.

Transform's uniqueness is defined similar to Qualys, instead of following the guide.

cc: @maxcold @alexreal1314

I think you can can avoid these by building the token strings from chars.

In Painless Lab, put this:

String lf = (String)(char)10;
String s = "line1" + lf + "line2";

"s:"+lf
+"---"+lf
+s+lf
+"---"+lf
+lf
+"s.splitOnToken(lf)[1]:"+lf
+s.splitOnToken(lf)[1]

And you'll see the output:

s:
---
line1
line2
---

s.splitOnToken(lf)[1]:
line2

For the following Painless script, I think it would be nice to define a function that does most of the work without being dependent on a specific context, then have minimal code to apply the function where necessary. That makes it easier to test or modify in the future, because you can copy the function out and work on it in other contexts.

Thanks for the suggestion @chrisberkhout.
For this script, we are trying to replace a pattern and not a single character. I will use the regexp inside the script processor itself and add functions inside the script itself as you suggested so that it will be easy to maintain.

Done in f9f1dc2

Sorry, for some reason I thought it was newlines and tabs, and that the problem was making a string literal with those characters in it.

A pattern is different. Moving it into the script is nice.

Splitting on the pattern rather than doing the substitutions is an option: https://discuss.elastic.co/t/can-we-use-split-processor-with-painless-scripting/98536/2

My original idea was to wrap everything in a function. Something like this:

List packagesFromOutput(String output) {
  // ...
}

List results = packagesFromOutput(ctx.tenable_io.vulnerability.output);

ctx.package = ctx.package ?: [];
ctx.vulnerability = ctx.vulnerability ?: [:];
ctx.vulnerability.package = ctx.vulnerability.package ?: [];

ctx.package.addAll(results);
ctx.vulnerability.package.addAll(results);

Anyway, these are all optional suggestions.

@kcreddy currently we support in the UI where each package.name, package.version and package.fixed_version are separate fields. Introducing this field a nested type in Tenable is breaking the client as i have tested it.

Qualys + wiz + CNVM

Tenable

IMO nested is the correct type here. but we should decide on the correct data model and type before continuing.

cc @maxcold

Thanks @alexreal1314 for the testing. Does it mean we need to have a separate package_nested field for storing nested package data?
We did this in Qualys where package is an object type and package_nested is nested type.

@kcreddy creating a separate package_nested field won't fix the issue because we rely in the UI on the package field which is of type object. Also package_nested is not part of ECS.
Need to think how to overcome this issue, meanwhile I would think that we should store ctx.tenable_io?.vulnerability?.output fields in the format we have today package.name: array of packages, package.version: array of versions and package.fixed_version array of fixed versions.
WDYT @maxcold ?

Related discussion previously: https://github.com/elastic/security-team/issues/9987#issuecomment-2548605862. We wanted to have package.* fields for the UI, but also package_nested.* nested fields to preserve the relationships.

@alexreal1314 @maxcold, this seems to be the only pending comment to merge this PR.
Let me know if there's anything needed here from my side.

just getting back to this problem, discussed on Slack. My suggestion to be consistent with what we have in Qualys and deal with the relationship between fields in case of multiple packages later. Also I think it makes sense to have package_nested as a way to keep the information about the relation

Updated in 21b2ac8.
Now the package and vulnerability.package fields are mapped as objects and the custom field tenable_io.vulnerability.package_nested is mapped as nested.

@kcreddy Is it possible to share some docs from Tenable? I would like to understand which fields they have. Because I see for example that out of 1844 documents, 52 are missing host.name which is quite important because alerts calculations are based on it. In comparison in Qualys out of 45k documents only 39 are missing this field.

@alexreal1314, here's the relevant discussion which lead to host.name copied from asset.fqdn: https://docs.google.com/spreadsheets/d/1ovxuXOtSXyNGZ_db7TaD5D0pt5m_60yUPe1oxJ1AQ8I/edit?gid=0#gid=0&range=G21
Here's the Tenable API doc: https://developer.tenable.com/reference/exports-vulns-download-chunk.
In case asset.fqdn field is missing, we could populate with some other field asset.hostname or asset.netbios_name. Would that be accepted solution?

In case asset.fqdn field is missing, we could populate with some other field asset.hostname or asset.netbios_name. Would that be accepted solution?

Yes, that’s a great suggestion. If the asset.fqdn field isn’t available, falling back to asset.hostname makes sense and if that’s also missing, using asset.netbios_name as a final fallback will be a solid approach, especially since this is a crucial ECS field that powers alert and entity(host) flyouts with vulnerability contextualization.

Cc @tinnytintin10 @maxcold

@alexreal1314, as per Nick's suggestion, I updated in 0fab27e

@kcreddy We've been given guidance by the leadership team that Security Solution treats 8.19 as a bugfix release and that we shouldn't add new features to this release, only 9.1.0. Does Security Integrations follow the same approach? If yes, we need to limit this version to only ^9.1.0 stack version.
For Qyalys we already have the integration published with ^8.19.0 || ^9.1.0 stack condition, we will need to figure out what we need to do on Kibana side to support that, most likely backport all the changes we made to support multiple CVEs, packages etc.

We've been given guidance by the leadership team that Security Solution treats 8.19 as a bugfix release

@maxcold I wasn't aware of this. Also, Qualys VMDR is the only integration that seems to define this version.
Do you know if this specifically with 8.19.0 and we can still target ^8.19.1?

Let me know which versions are you targeting for backport, I can update this accordingly.
BTW 8.17.7 and 8.18.2 FF is today.

@jamiehynds, it seems that we shouldn't be releasing new features targeting ^8.19.0. Does it also mean new integrations shouldn't be targeting this version?
Its not GA yet, but would like to know in case any future PRs comes up targeting this version.

I'm waiting for the confirmation from @nick-alayil on the approach to stack versions and availability of CDR related changes in integrations. If we follow the guidance for Security Solution and treat 8.19.0 as bugfix release (it applies to all 8.x versions going forward as far as I understand) , then for Tenable we should have version: "^9.1.0"
For Qualys as the integration has already been published, I think we need to backport Kibana changes to 8.19.0, but that's solely on us. Let me know if I can give more context on this topic

The main problem going only with ^9.1.0 and skip entire 8.x versions is that any future changes such as bugfixes and enhancements for this integration need to be backported to 8.x versions following our backporting guide which needs 3 PRs per every change.
I am also unsure if customers in 8.x would likely upgrade to 9.1 any sooner because its a major version upgrade.

After discussing with @maxcold, we are going ahead with version constraint: version: "^8.19.0 || ^9.1.0". Will go ahead and merge the PR.

cc: @nick-alayil

I think it would be helpful to specify what the breaking changes are and include any advice we might have for users who are upgrading (if applicable).

Just missed this comment and merged it 😞.

I will create a quick PR with list of changes and update documentation regarding transform and its requirement.
Essentially there are few important mappings that were updated and addition of a transform adds atleast 2X data, and also requires Transform nodes, all of which the users could be made aware of.

Created #14130