Skip to content

[Cloud Security] [Fleet] Add cloud connector access scope for input or package level credential definitions#250280

Merged
seanrathier merged 9 commits intoelastic:mainfrom
seanrathier:cloud-connector-access-scope
Jan 29, 2026
Merged

[Cloud Security] [Fleet] Add cloud connector access scope for input or package level credential definitions#250280
seanrathier merged 9 commits intoelastic:mainfrom
seanrathier:cloud-connector-access-scope

Conversation

@seanrathier
Copy link
Copy Markdown
Contributor

@seanrathier seanrathier commented Jan 23, 2026

Summary

Implements a scope-aware solution for Cloud Connector credential variables, addressing the inconsistency where credentials could be stored at either package-level (policy.vars) or input/stream-level (policy.inputs[].streams[].vars).

Allows to easily support integration with credential aliases.

Key Changes

  • New Accessor Module (fleet/common/services/cloud_connectors/)
  • types.ts - Core types for storage modes, targets, and credential schemas
  • schemas.ts - Credential schemas for AWS/Azure with variable key mappings and aliases
  • var_accessor.ts - Main accessor with detectStorageMode(), extractRawCredentialVars(), readCredentials(), writeCredentials()
  • Publicly exported from @kbn/fleet-plugin/common for clean API access
  • Fleet Server side backend and Cloud Connector UI using same access scopes

Test Plan

  • Unit tests for accessor module (var_accessor.test.ts, schemas.test.ts)
  • Updated use_cloud_connector_setup.test.ts for new hook signature
  • Updated cloud_connector_setup.test.tsx assertions
  • Updated integration_helpers.test.ts with packageInfo parameter
  • Updated package_policy.test.ts for createCloudConnectorForPackagePolicy
  • E2E: Verify AWS Cloud Connector creation/existing flow
  • E2E: Verify Azure Cloud Connector creation/existing flow

Checklist

  • Unit or functional tests were updated or added to match the most common scenarios
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Identify risks

None

@seanrathier seanrathier self-assigned this Jan 23, 2026
@seanrathier seanrathier added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Fleet Team label for Observability Data Collection Fleet team Team:Cloud Security Cloud Security team related ci:cloud-deploy Create or update a Cloud deployment ci:project-deploy-security Create a Security Serverless Project labels Jan 23, 2026
@seanrathier seanrathier marked this pull request as ready for review January 26, 2026 14:46
@seanrathier seanrathier requested review from a team as code owners January 26, 2026 14:46
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/contextual-security-apps (Team:Cloud Security)

@nchaulet nchaulet self-requested a review January 26, 2026 15:57
@nchaulet
Copy link
Copy Markdown
Member

Hi @seanrathier it would be great if you can give us more information to maybe test this locally what integrations are expected to be at the input/package level? thanks a lot


// Input mode: update the correct stream
const { inputIndex, streamIndex } = target;
if (inputIndex === -1 || streamIndex === -1) {
Copy link
Copy Markdown
Contributor

@Omolola-Akinleye Omolola-Akinleye Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm why would index be -1 here? Maybe store -1 as store as

const INVALID_INDEX = -1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updatePolicyWithNewCredentials,
updatePolicyWithExistingCredentials,
} = useCloudConnectorSetup(input, newPolicy, updatePolicy);
} = useCloudConnectorSetup(input, newPolicy, updatePolicy, packageInfo);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Can we derive input from newPolicy? I wonder if remove input and create a util to extract from newPolicy?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, did not notice the input was still in the params and unused. It has been removed

packageInfo: PackageInfo
): NewPackagePolicy => {
const mode = detectStorageMode(packageInfo);
const { target } = resolveVarTarget(policy, mode);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this function is really about updating vars based on their scope, not just a “target.”
Some ideas to make this clearer:

  • Rename target → scope throughout.
  • Rename updatePolicyVarsAtTarget → updatePolicyVarsByScope or updatePackagePolicyVarsScope.

Add a brief comment explaining the two var scopes:

  • Package scope: policy.vars (global to the package)
  • Input/stream scope: policy.inputs[].streams[].vars

Copy link
Copy Markdown
Contributor Author

@seanrathier seanrathier Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for bringing this also to my attention, I found a way deduplicate updating vars, server side and client side.

https://github.com/elastic/kibana/pull/250280/changes#diff-7ea88fb880abbd507bb63081807f1dd4017aa68aabbb8f3f11e803999afc1eb2R110

https://github.com/elastic/kibana/pull/250280/changes#diff-bd2dda36254f5eb14f4c4fb82802df30bb7d7c0448afce5504ae078c3361b325R113

Changed updatePolicyVarsAtTarget -> updatePolicyVarsByScope

Rename target → scope throughout.

I changed mode to scope, but target makes sense to me. I dedicated most of the code review changes on this comment 😅

The target is there mostly for the inputs to return the index of the enabled input. For consistency I would like to keep this as target

const scope = getCredentialStorageScope(packageInfo);
const { target } = resolveVarTarget(policy, scope);

const packageVars: RegistryVarsEntry[] = packageInfo.vars || [];
const supportedVarNames = getAllSupportedVarNames();

const hasPackageLevelCredentials = packageVars.some((varDef) =>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name detectStorageMode is a bit unclear about what’s being checked. It seems this function is really determining where connector credentials are stored—package/global vs input/stream level.
Maybe consider a name like getConnectorVarScope or getConnectorCredentialScope to make the intent clearer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to updatePolicyVarsByScope

const updatePolicyVarsByScope = (
  policy: NewPackagePolicy,
  updatedVars: PackagePolicyConfigRecord,
  packageInfo: PackageInfo
): NewPackagePolicy => {
  const scope = getCredentialStorageScope(packageInfo);
  const { target } = resolveVarTarget(policy, scope);

  return applyVarsAtTarget(policy, updatedVars, target);
};

const enabledInputIndex = packagePolicy.inputs.findIndex((input) => input.enabled);
if (enabledInputIndex === -1) {
return {
target: { mode: 'input', inputIndex: -1, streamIndex: -1 },
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storing -1 as const INVALID_INDEX or INDEX_NOT_FOUND

Copy link
Copy Markdown
Contributor Author

@seanrathier seanrathier Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// Look for tenant_id using schema-defined keys
let tenantIdVar: PackagePolicyConfigRecordEntry | undefined;
for (const key of tenantIdKeys) {
if (vars[key]) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed there are quite a few for/map loops here (O(n) operations) to access or update vars both on the FE and BE. Wondering if there’s a way to optimize some of these lookups to O(1), e.g., by indexing vars by key or using a lookup map. This could improve performance especially in case there is scale in the increase fields for each cloud provider and processing Cloud connector API request.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to come back to this a create a helper, thanks for pointing this out.

I created a new function in the var_accessor called findFirstVarEntry to reduce the repeating for-loop to get the first var that matches any key.

const tenantIdVar = findFirstVarEntry(vars, tenantIdKeys);
const clientIdVar = findFirstVarEntry(vars, clientIdKeys);
const azureCredentials = findFirstVarEntry(vars, connectorIdKeys);

I need to give some thought about replacing the for-loop though.

varKeys is typically 2-4 items (credential field names like ['role_arn', 'aws.credentials.role_arn'])

Building a lookup map would add complexity and the performance difference is small for these arrays

Copy link
Copy Markdown
Contributor

@Omolola-Akinleye Omolola-Akinleye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just have some feedback on naming and mainly the use of for-loops.

@seanrathier seanrathier force-pushed the cloud-connector-access-scope branch from f4e7020 to 0f2dcb3 Compare January 28, 2026 16:22
@seanrathier
Copy link
Copy Markdown
Contributor Author

@nchaulet

Hi @seanrathier it would be great if you can give us more information to maybe test this locally what integrations are expected to be at the input/package level? thanks a lot

You can test this locally using the CSPM or Cloud Asset Discovery integrations.

Credential Scopes

CSPM and Cloud Asset Discovery has credentials in the input level, while the AWS integration has credentials in the package level

This change is needed before we move the Cloud Connector UI to Fleet to have the CC enabled for AWS and other integration.

This change also reduces the current complexity around checking for preferred credentials var names and the aliases with the prefixes.

* @param packageInfo - The package info containing var definitions
* @returns The storage scope ('package' or 'input')
*/
export function getCredentialStorageScope(packageInfo: PackageInfo): CloudConnectorVarStorageMode {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: suggestion on the naming getCredentialVariablesScope

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😅 I am not going to change this since its a :nit, I just changed it to getCredentialStorageScope

Comment on lines +76 to +82
const enabledInputIndex = packagePolicy.inputs.findIndex((input) => input.enabled);
if (enabledInputIndex === INVALID_INDEX) {
return {
target: { mode: 'input', inputIndex: INVALID_INDEX, streamIndex: INVALID_INDEX },
vars: undefined,
};
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With that model It will never be possible to have multiple enabled input with cloud connector variables? I do not think a package exists with that currently

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Cloud Connectors, I don't see how it is possible, ATM, to have multiple enabled inputs with each having their own access credentials. We can only have one cloud connector id per policy.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is a potential future case where a multi-cloud integration, such as Sentinel Cloud, might need to support Cloud Connector. Sentinel Cloud integrations already enable AWS, GCP, and Azure inputs with credentials. While this is not something we need to account for right now, it’s possible that if this use case becomes relevant in the future, we would need to refactor the Cloud Connector model to better support multi-cloud enabled inputs .

@seanrathier seanrathier requested a review from nchaulet January 28, 2026 20:01
Copy link
Copy Markdown
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@seanrathier seanrathier enabled auto-merge (squash) January 28, 2026 20:24
@seanrathier seanrathier disabled auto-merge January 28, 2026 20:24
@seanrathier seanrathier enabled auto-merge (squash) January 29, 2026 16:47
Copy link
Copy Markdown
Contributor

@Omolola-Akinleye Omolola-Akinleye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🥇

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 29, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Scout: [ observability / observability ] plugin / serverless-oblt - Alert Details Page - should show an error when the alert does not exist
  • [job] [logs] Scout: [ observability / observability ] plugin / should show an error when the alert does not exist

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
fleet 1393 1397 +4

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
fleet 1611 1620 +9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
cloudSecurityPosture 659.5KB 659.1KB -428.0B
securitySolution 10.8MB 10.8MB -428.0B
total -856.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
fleet 111 114 +3

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
fleet 182.6KB 187.2KB +4.5KB
Unknown metric groups

API count

id before after diff
fleet 1746 1794 +48

References to deprecated APIs

id before after diff
@kbn/cloud-security-posture 18 16 -2

Unreferenced deprecated APIs

id before after diff
@kbn/cloud-security-posture 18 16 -2

History

cc @seanrathier

@seanrathier seanrathier merged commit f2fa7da into elastic:main Jan 29, 2026
16 checks passed
mbondyra added a commit to mbondyra/kibana that referenced this pull request Jan 30, 2026
…iew_cps

* commit '5f7fec57cb01883038810bd735a0666683b49904': (116 commits)
  [Security Solution][Attacks/Alerts][Setup and miscellaneous] Advanced setting to control feature visibility (elastic#250157) (elastic#250830)
  Fix synthtrace `fetch` usage (elastic#250950)
  [APM] Add Nodes and Edges components and selection logic (elastic#250937)
  [Docs] Update alerting-settings.md and add serverless value for one parameter (elastic#250842)
  [Agent Builder] filestore: initial implementation (elastic#250043)
  [CPS] Support CPS in Vega ESQL (elastic#250693)
  Adjustments to cascade document esql helpers (elastic#250560)
  [Security Solutions] Trial Companion - adds ai chat and elastic agent detectors (elastic#250908)
  [Obs Presentation] Code Scanning Alert Fixes (elastic#250858)
  [performance] add return and refresh render scenarios to dashboard journeys (elastic#250939)
  skip failing test suite (elastic#245458)
  Add Cloud Forwarder onboarding tile to O11y Solution (elastic#250325)
  [Traces] Remove APM unified trace waterall embeddable registration (elastic#250808)
  [Discover] [Metrics] Fix: metrics grid titles do not update on order change (elastic#250963)
  [a11y] Fix Eui modal title annoucment (elastic#250459)
  [Cloud Security] [Fleet] Add cloud connector access scope for input or package level credential definitions (elastic#250280)
  [WorkplaceAI] SharePoint Online stack connector (elastic#248737)
  [Response Ops][Task Manager] Update functions do not handle API key invalidation (elastic#249109)
  [Osquery] Remove @kbn/timelines-plugin dependency from osquery plugin (elastic#250055)
  [One Discover][Logs UX] Update OpenTelemetry Semantic Conventions (elastic#250346)
  ...
hannahbrooks pushed a commit to hannahbrooks/kibana that referenced this pull request Jan 30, 2026
@seanrathier seanrathier deleted the cloud-connector-access-scope branch March 6, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting ci:cloud-deploy Create or update a Cloud deployment ci:project-deploy-security Create a Security Serverless Project release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related Team:Fleet Team label for Observability Data Collection Fleet team v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants