Skip to content

(Triggers): Permissions race condition: not authorized to perform: lambda:InvokeFunction #19272

@deuscapturus

Description

@deuscapturus

What is the problem?

Trigger stack creations succeed, but updates fails with error:

12:37:55 PM | UPDATE_FAILED        | Custom::Trigger       | PublishDevOpsDashboardMetrics
Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: User: arn:aws:sts::000000000000:assumed-role/search-test-trigger-delet-AWSCDKTriggerCustomResou-1WJNIVYDBY6D2/search-test-trigger-delet-AWS
CDKTriggerCustomResou-pOS3k3dmifGe is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:000000000000:function:search-test-trigger-delet-PublishDevOpsDashboardMe-h7bTtHL1pu4B:2 because no identity-bas
ed policy allows the lambda:InvokeFunction action

This Trigger lambda publishes change notifications. The dynamic parts of this notification as passed to the lambda in the Environment. This however forces an update of the Lambda resource and IAM role policy allowing lambda:InvokeFunction. I believe a race condition is occurring where the IAM role policy is not updated in time for the custom resource Lambda invocation.

Reproduction Steps

Create the example construct below

import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as triggers from 'aws-cdk-lib/triggers'
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class CdkTriggerStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const trigger = new triggers.Trigger(this, "PublishDevOpsDashboardMetrics", { 
      handler: new lambda.Function(this, "PublishDevOpsDashboardMetricsLambda", {
        code: new lambda.InlineCode(`#!/usr/bin/python3\nimport os\ndef handler(event, context):\n  print(event, os.getenv('DEPLOYMENT_VERSION'))`),
        handler: 'index.handler',
        runtime: lambda.Runtime.PYTHON_3_9,
        environment: {
            DEPLOYMENT_VERSION: "1",
        }
      })
    })
  }
}

Update environment DEPLOYMENT_VERSION to 2 and update the stack. This fails every time for me, but may not fail every time if this is a race condition.

What did you expect to happen?

no error

What actually happened?

AccessDeniedException error

CDK CLI Version

2.15.0

Framework Version

No response

Node.js Version

v14.17.0

OS

Fedora

Language

Typescript

Language Version

No response

Other information

I think the Triggers implementation is flawed. It would be better to have a one Lambda for the custom resource and allow parameters to be passed via resource parameters. This way the Lambda resources doesn't need to update with each deploy.

I recommend not using Triggers and instead implementing your own custom resource.

Metadata

Metadata

Assignees

No one assigned

    Labels

    @aws-cdk/triggersRelated to the triggers packagebugThis issue is a bug.effort/smallSmall work item – less than a day of effortp1

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions