Skip to content

core: CrossRegionExportReader: Response object is too long #25114

@antoncohen

Description

@antoncohen

Describe the bug

When using crossRegionReferences: true the cdk deploy fails in the reader region with Response object is too long.

Similar to #23958, but on the reader side, likely with a different root cause.

Expected Behavior

I expect crossRegionReferences: true to work, and it be possible to reference variables across stacks in different regions.

Current Behavior

When referencing a variable in a stack in a different region, it fails in the reader region with this error:

testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:01 PM | CREATE_COMPLETE      | AWS::Lambda::Function           | Custom::CrossRegionExportReaderCustomResourceProvider/Handler (CustomCrossRegionExportReaderCustomResourceProviderHandler46647B68)
[12:28:03] Stack testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:06 PM | CREATE_IN_PROGRESS   | Custom::CrossRegionExportReader | ExportsReader/Resource/Default (ExportsReader8B249524)
[12:28:08] Stack testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[12:28:13] Stack testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:15 PM | CREATE_FAILED        | Custom::CrossRegionExportReader | ExportsReader/Resource/Default (ExportsReader8B249524) Response object is too long.
	new CustomResource (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource.ts:135:21)
	\_ new ExportReader (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-reader-provider.ts:62:27)
	\_ Function.getOrCreate (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-reader-provider.ts:29:9)
	\_ ExportWriter.addToExportReader (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-writer-provider.ts:122:39)
	\_ ExportWriter.exportValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-writer-provider.ts:114:17)
	\_ createCrossRegionImportValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:240:33)
	\_ resolveValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:121:12)
	\_ resolveValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:105:12)
	\_ resolveReferences (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:34:24)
	\_ prepareApp (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/prepare-app.ts:30:20)
	\_ synthesize (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/synthesis.ts:45:13)
	\_ App.synth (/path/to/code/node_modules/aws-cdk-lib/core/lib/stage.ts:217:33)
	\_ process.<anonymous> (/path/to/code/node_modules/aws-cdk-lib/core/lib/app.ts:195:45)
	\_ Object.onceWrapper (node:events:628:26)
	\_ process.emit (node:events:513:28)
	\_ process.emit (node:domain:489:12)
	\_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/path/to/code/node_modules/@cspotcode/source-map-support/source-map-support.js:745:40)
testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:16 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack      | testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 The following resource(s) failed to create: [ExportsReader8B249524]. Rollback requested by user.

This is using Nested Stacks, and long stack names, with multiple cross-region references.

Reproduction Steps

Node.js: 18.15.0
CDK bin and lib: 2.73.0
TypeScript: 5.0.4
Command: npx cdk deploy --all --require-approval=never --verbose --debug --app 'npx ts-node app.ts'

app.ts:

#!/usr/bin/env npx ts-node

import * as awscdk from 'aws-cdk-lib'

const ACCOUNT_ID = '111222333444'
const STACK_PREFIX = 'testing-cdk-cross-region-references-as-example-for-github-issue'
// 8 or less works, 9 or more fails with "Response object is too long"
const NUMBER_OF_KEYS_TO_CREATE = 9

function createRootStack (app: awscdk.App, region: string): awscdk.Stack {
  const stackName = `${STACK_PREFIX}-rootstack-${region}`

  const props = {
    env: {
      account: ACCOUNT_ID,
      region: region
    },
    crossRegionReferences: true,
    synthesizer: new awscdk.LegacyStackSynthesizer()
  }

  const stack = new awscdk.Stack(app, stackName, props)

  return stack
}

function createPrimaryKeyStack (rootStack: awscdk.Stack, keyName: string): string {
  const nestedStack = new awscdk.NestedStack(rootStack, `${STACK_PREFIX}-nestedstack-${keyName}`)

  const keyProps = getKeyProps(keyName)
  const kmsCfnKey = new awscdk.aws_kms.CfnKey(nestedStack, `${STACK_PREFIX}-key-${keyName}`, keyProps)

  const aliasName = `alias/${STACK_PREFIX}-${keyName}`
  const aliasProps = getAliasProps(aliasName, kmsCfnKey.attrKeyId)
  new awscdk.aws_kms.CfnAlias(nestedStack, `${STACK_PREFIX}-alias-${keyName}`, aliasProps)

  return kmsCfnKey.attrArn
}

function createReplicaKeyStack (rootStack: awscdk.Stack, keyName: string, primaryKeyArn: string) {
  const nestedStack = new awscdk.NestedStack(rootStack, `${STACK_PREFIX}-nestedstack-${keyName}`)

  const keyProps = getReplicaKeyProps(keyName, primaryKeyArn)
  const kmsCfnKey = new awscdk.aws_kms.CfnReplicaKey(nestedStack, `${STACK_PREFIX}-key-${keyName}`, keyProps)

  const aliasName = `alias/${STACK_PREFIX}-${keyName}`
  const aliasProps = getAliasProps(aliasName, kmsCfnKey.attrKeyId)
  new awscdk.aws_kms.CfnAlias(nestedStack, `${STACK_PREFIX}-alias-${keyName}`, aliasProps)
}

function getKeyPolicy (): awscdk.aws_iam.PolicyDocument {
  const thisPrincipal = new awscdk.aws_iam.AccountPrincipal(ACCOUNT_ID)

  const policyStatement = new awscdk.aws_iam.PolicyStatement({
    sid: 'Allow all key access',
    effect: awscdk.aws_iam.Effect.ALLOW,
    actions: [
      'kms:*'
    ],
    resources: ['*'],
    principals: [thisPrincipal]
  })

  const policyDocument = new awscdk.aws_iam.PolicyDocument({ statements: [policyStatement] })

  return policyDocument
}

function getKeyProps (keyName: string): awscdk.aws_kms.CfnKeyProps {
  const policyDocument = getKeyPolicy()

  const keyProps: awscdk.aws_kms.CfnKeyProps = {
    description: `Testing cross-region references in CDK - ${keyName}`,
    keyPolicy: policyDocument,
    multiRegion: true,
    enableKeyRotation: false,
    enabled: true,
    keyUsage: 'ENCRYPT_DECRYPT',
    pendingWindowInDays: 7
  }

  return keyProps
}

function getReplicaKeyProps (keyName: string, primaryKeyArn: string): awscdk.aws_kms.CfnReplicaKeyProps {
  const policyDocument = getKeyPolicy()

  const keyProps: awscdk.aws_kms.CfnReplicaKeyProps = {
    description: `Testing cross-region references in CDK - ${keyName}`,
    primaryKeyArn: primaryKeyArn,
    keyPolicy: policyDocument,
    enabled: true,
    pendingWindowInDays: 7
  }

  return keyProps
}

function getAliasProps (aliasName:string, targetKeyId: string): awscdk.aws_kms.CfnAliasProps {
  const aliasProps: awscdk.aws_kms.CfnAliasProps = {
    aliasName: aliasName,
    targetKeyId: targetKeyId
  }

  return aliasProps
}

function main (args: Array<string> | undefined = undefined) {
  const app = new awscdk.App()

  const rootStackEast = createRootStack(app, 'us-east-1')
  const rootStackWest = createRootStack(app, 'us-west-2')

  for (let i = 1; i <= NUMBER_OF_KEYS_TO_CREATE; i++) {
    const primaryKeyArn = createPrimaryKeyStack(rootStackEast, i.toString())
    createReplicaKeyStack(rootStackWest, i.toString(), primaryKeyArn)
  }
}

if (require.main === module) {
  main()
}

At the top of app.ts the constant ACCOUNT_ID will need to be changed. The constant NUMBER_OF_KEYS_TO_CREATE can be lowered below 9 to not trigger the error.

I don't see any errors in the Lambda logs. Parameter Store values are written. And in the reader region the Parameter Store tags are set and removed, according to the Lambda logs.

Possible Solution

My guess is that it is triggered long stack names, including nested stacks creating even longer reference names.

Additional Information/Context

No response

CDK CLI Version

2.73.0 (build 43e681e)

Framework Version

No response

Node.js Version

18.15.0

OS

macOS

Language

Typescript

Language Version

TypeScript (5.0.4)

Other information

No response

Metadata

Metadata

Assignees

Labels

@aws-cdk/coreRelated to core CDK functionalitybugThis issue is a bug.effort/mediumMedium work item – several days of effortp2

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions