Skip to content

(cli): Cross-account asset publishing broken in v1.135 #17988

@msessa

Description

@msessa

What is the problem?

CDK asset publishing doesn't correctly assume the target asset publishing role in certain cases and fails to deploy S3 artifacts.

The issue seems to have surfaced with the release of cdk 1.135.0 but was possibly introduced in earlier versions.

Initial investigation suggests this line isn't passing the stack's asset publishing role and instead making API calls as the base role.

When v1.135 is used to deploy against an account bootstrapped with stack < v9, this code branch will trigger the above account() call which causes this.host.aborted to become true and this line to never be reached.

Reproduction Steps

  • Have role in AccountA
  • Bootstrap AccountB with stack v8 ( cdk <= 1.134.0 ), and --trust role in AccountA
  • Create CDK application with some S3 asset (lambda functions or similar), point environment to AccountB
  • Synth application with @aws-cdk/core:newStyleStackSynthesis=true
  • npx aws-cdk@1.135.0 deploy --debug -vvvvv

What did you expect to happen?

CDK deploy assumes the asset publishing role in AccountB and correctly uploads artifacts

What actually happened?

fail: Need to perform AWS calls for account <AccountB>, but the current credentials are for <AccountA>

CDK CLI Version

1.135.0

Framework Version

No response

Node.js Version

v14.16.0

OS

MacOS

Language

Typescript

Language Version

No response

Other information

Complete deploy log:

CDK toolkit version: 1.135.0 (build 3825f59)
Command line arguments: {
  _: [ 'deploy' ],
  debug: true,
  v: 5,
  verbose: 5,
  'outputs-file': 'cdk-outputs.json',
  O: 'cdk-outputs.json',
  outputsFile: 'cdk-outputs.json',
  app: 'cdk.out/',
  a: 'cdk.out/',
  'require-approval': 'any-change',
  requireApproval: 'any-change',
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  all: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  '$0': '../.npm/_npx/69094/bin/cdk',
  STACKS: [ 'cdk-issue-debug' ],
  'S-t-a-c-k-s': [ 'cdk-issue-debug' ]
}
cdk.json: {
  "app": "npx ts-node -P tsconfig.json --prefer-ts-exts src/main.ts",
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": "true",
    "aws-cdk:enableDiffNoFail": true,
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
    "@aws-cdk/core:enableStackNameDuplicates": true,
    "@aws-cdk/core:stackRelativeExports": true,
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
    "@aws-cdk/aws-kms:defaultKeyPolicies": true,
    "@aws-cdk/aws-s3:grantWriteWithoutAcl": true,
    "@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
    "@aws-cdk/aws-efs:defaultEncryptionAtRest": true,
    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true
  },
  "output": "cdk.out",
  "build": "npx projen bundle",
  "watch": {
    "include": [
      "src/**/*.ts",
      "test/**/*.ts"
    ],
    "exclude": [
      "README.md",
      "cdk*.json",
      "**/*.d.ts",
      "**/*.js",
      "tsconfig.json",
      "package*.json",
      "yarn.lock",
      "node_modules"
    ]
  },
  "//": "~~ Generated by projen. To modify, edit .projenrc.js and run \"npx projen\"."
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  context: {
    '@aws-cdk/core:newStyleStackSynthesis': 'true',
    'aws-cdk:enableDiffNoFail': true,
    '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
    '@aws-cdk/core:enableStackNameDuplicates': true,
    '@aws-cdk/core:stackRelativeExports': true,
    '@aws-cdk/aws-ecr-assets:dockerIgnoreSupport': true,
    '@aws-cdk/aws-secretsmanager:parseOwnedSecretName': true,
    '@aws-cdk/aws-kms:defaultKeyPolicies': true,
    '@aws-cdk/aws-s3:grantWriteWithoutAcl': true,
    '@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount': true,
    '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
    '@aws-cdk/aws-efs:defaultEncryptionAtRest': true,
    '@aws-cdk/aws-lambda:recognizeVersionProps': true,
    '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true
  },
  app: 'cdk.out/',
  build: 'npx projen bundle',
  watch: {
    include: [ 'src/**/*.ts', 'test/**/*.ts' ],
    exclude: [
      'README.md',
      'cdk*.json',
      '**/*.d.ts',
      '**/*.js',
      'tsconfig.json',
      'package*.json',
      'yarn.lock',
      'node_modules'
    ]
  },
  '//': '~~ Generated by projen. To modify, edit .projenrc.js and run "npx projen".',
  debug: true,
  assetMetadata: true,
  requireApproval: 'any-change',
  toolkitBucket: {},
  staging: true,
  outputsFile: 'cdk-outputs.json',
  bundlingStacks: [ '*' ],
  lookups: true
}
Determining if we're on an EC2 instance.
Does not look like an EC2 instance.
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to ap-southeast-2
Resolving default credentials
Looking up default account ID from STS
[AWS sts 200 0.181s 0 retries] getCallerIdentity({})
Default account ID: AccountA
Setting "CDK_DEFAULT_ACCOUNT" environment variable to AccountA
context: {
  '@aws-cdk/core:newStyleStackSynthesis': 'true',
  'aws-cdk:enableDiffNoFail': true,
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
  '@aws-cdk/core:enableStackNameDuplicates': true,
  '@aws-cdk/core:stackRelativeExports': true,
  '@aws-cdk/aws-ecr-assets:dockerIgnoreSupport': true,
  '@aws-cdk/aws-secretsmanager:parseOwnedSecretName': true,
  '@aws-cdk/aws-kms:defaultKeyPolicies': true,
  '@aws-cdk/aws-s3:grantWriteWithoutAcl': true,
  '@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount': true,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
  '@aws-cdk/aws-efs:defaultEncryptionAtRest': true,
  '@aws-cdk/aws-lambda:recognizeVersionProps': true,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true,
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
--app points to a cloud assembly, so we bypass synth
Reading existing template for stack cdk-issue-debug.
Retrieved account ID AccountA from disk cache
Assuming role 'arn:aws:iam::AccountB:role/cdk-hnb659fds-deploy-role-AccountB-ap-southeast-2'.
[AWS cloudformation 200 0.291s 0 retries] describeStacks({ StackName: 'cdk-issue-debug' })
[AWS cloudformation 200 0.175s 0 retries] getTemplate({ StackName: 'cdk-issue-debug', TemplateStage: 'Original' })
cdk-issue-debug: deploying...
Retrieved account ID AccountA from disk cache
Assuming role 'arn:aws:iam::AccountB:role/cdk-hnb659fds-deploy-role-AccountB-ap-southeast-2'.
Waiting for stack CDKToolkit to finish creating or updating...
[AWS cloudformation 200 0.212s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
[AWS ssm 200 0.236s 0 retries] getParameter({ Name: '/cdk-bootstrap/hnb659fds/version' })
[0%] start: Publishing ea955e1dd13eac7a289683a4c42a8f95466ebb1e7002c2b524369eab874ebd6c:AccountB-ap-southeast-2
Retrieved account ID AccountA from disk cache
Assuming role 'arn:aws:iam::AccountB:role/cdk-hnb659fds-file-publishing-role-AccountB-ap-southeast-2'.
[0%] check: Check s3://cdk-hnb659fds-assets-AccountB-ap-southeast-2/ea955e1dd13eac7a289683a4c42a8f95466ebb1e7002c2b524369eab874ebd6c.zip
[AWS s3 200 0.183s 0 retries] getBucketLocation({ Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2' })
[AWS s3 200 0.169s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2',
  Prefix: 'ea955e1dd13eac7a289683a4c42a8f95466ebb1e7002c2b524369eab874ebd6c.zip',
  MaxKeys: 1
})
[AWS s3 403 0.147s 0 retries] getBucketEncryption({ Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2' })
Call failed: getBucketEncryption({"Bucket":"cdk-hnb659fds-assets-AccountB-ap-southeast-2"}) => Access Denied (code=AccessDenied)
[50%] fail: Need to perform AWS calls for account AccountB, but the current credentials are for AccountA
[50%] start: Publishing 4ed6eb06863732b08308385e96b4da247bb35b3e8b6aca611ee88573da8a870a:AccountB-ap-southeast-2
Retrieved account ID AccountA from disk cache
[50%] check: Check s3://cdk-hnb659fds-assets-AccountB-ap-southeast-2/4ed6eb06863732b08308385e96b4da247bb35b3e8b6aca611ee88573da8a870a.json
[AWS s3 200 0.165s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2',
  Prefix: '4ed6eb06863732b08308385e96b4da247bb35b3e8b6aca611ee88573da8a870a.json',
  MaxKeys: 1
})
[100%] fail: Need to perform AWS calls for account AccountB, but the current credentials are for AccountA

 ❌  cdk-issue-debug failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at CloudFormationDeployments.publishStackAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:305:7)
    at CloudFormationDeployments.deployStack (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:202:5)
    at CdkToolkit.deploy (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:194:24)
    at initCommandLine (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)
Failed to publish one or more assets. See the error messages above for more information.
Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at CloudFormationDeployments.publishStackAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:305:7)
    at CloudFormationDeployments.deployStack (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:202:5)
    at CdkToolkit.deploy (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:194:24)
    at initCommandLine (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.needs-triageThis issue or PR still needs to be triaged.package/toolsRelated to AWS CDK Tools or CLI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions