@nija-at let me know whenever you have a minute to review :).

@nija-at I don't have the specific error since I fixed it in my local node_modules 5 weeks ago before I went on paternity leave. Let me see if it's still in the account itself to get you the exact error message but let me also try and explain below my understanding of the dependencies within the APIG Stack being created. I'll post my example CDK stack for more clarity below before I explain.

class ApigStack extends Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, { domainName, cert, definition, ...props }) {
    super(scope, id, props);
    // The code that defines your stack goes here
    const certificate = Certificate.fromCertificateArn(this, `cert-${domainName}`, cert.CertificateArn);

    this.api = new SpecRestApi(this, 'apig', {
      apiDefinition: ApiDefinition.fromInline(definition),
      endpointConfiguration: {
        types: [ EndpointType.REGIONAL ]
      },
      deployOptions: {
        stageName: 'production',
        metricsEnabled: true,
        loggingLevel: MethodLoggingLevel.INFO
      }
    });

    this.domain = new DomainName(this, 'domain', {
      domainName,
      certificate,
      endpointType: EndpointType.REGIONAL,
      securityPolicy: SecurityPolicy.TLS_1_2
    });

    this.domain.addBasePathMapping(this.api, { basePath: '', stage: this.api.deploymentStage });
  }
}

In this APIG stack example, we have metricsEnabled which will create the CfnAccount as well as the IAM roles in order to write metrics to cloudwatch. The Stage created within the APIG depends upon that CfnAccount and those IAM roles for it to be created due to the deployOptions I defined. The error that would occur is that the stage would fail to create because the IAM roles did not finish creating before the Stage was being created. The Stage within the APIG only depends upon the root APIG resource in the generated CFN template from this stack. The change I added ensures that the IAM role for cloudwatch that live under that CfnAccount, are created before the Stage that depends on them is created. Currently that dependency doesn't exist which causes the race condition I am expressing here.

I debugged this with cloudtrail originally with one of my coworkers and I was able to add the dependency locally and fix the problem. The problem would only occur with creating the IAM role and CfnAccount was slower than creating the APIG -> Deploy -> Stage since the resources were created independently and in parallel by CFN.

Thanks for your response @jcrugzz. I still have a couple of clarifications.

The Stage created within the APIG depends upon that CfnAccount and those IAM roles for it to be created due to the deployOptions I defined.

The change I added ensures that the IAM role for cloudwatch that live under that CfnAccount, are created before the Stage that depends on them is created

Why do you say Stage depends on CfnAccount? As far as I can tell, the only dependency is CfnAccount depending on the RestApi.

Which IAM roles are you referring to here? As far as I can tell, The IAM role used by CfnAccount is not used anywhere else.

@nija-at Sorry language gets a little tricky here. In my first statement you pointed out the following and ill include your response

The Stage created within the APIG depends upon that CfnAccount and those IAM roles for it to be created due to the deployOptions I defined.

Why do you say Stage depends on CfnAccount? As far as I can tell, the only dependency is CfnAccount depending on the RestApi.

My change is what adds the explicit depends where Stage depends upon CfnAccount. What I learned through debugging is that in the current Cloudformation template generated by CDK, there is a race condition due to an implicit dependency that I am trying to make explicit with my changes. So when I said depends upon here, its that the actual sequence of creating the Stage component within APIGateway requires an IAM role in order to have cloudwatch logs enabled before the Stage can successfully be created. Prior to my changes, the IAM role for writing to cloudwatch wasn't guaranteed to exist before the Stage is attempted to be created within the Cloudformation template. I add a dependency of Stage -> CfnAccount since CfnAccount is the parent of the explicit IAM Role needed. Maybe its more correct to have Stage depend upon this role instead when cloudwatch logging is enabled? I just figured since it was consumed by the account that that was the correct construct to depend on.

Which IAM roles are you referring to here? As far as I can tell, The IAM role used by CfnAccount is not used anywhere else.

The IAM role linked above is what I am referring to and it is what is used by the Stage in an indirect way since the Stage has ownership over enabling cloudwatch logging within the various components within an APIGateway.

Please let me know if I didn't improve clarity here. I'm happy to make time to hop on a call to help hash this out if this is inadequate. I created 8 APIGateways before even seeing this issue for the first time and thought it was a red herring until I was able to understand the subtle dependency issue when it was persistent in happening in particular regions.

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
• Commit ID: 721c8c7
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Thanks @jcrugzz for the explanation. That explanation made it super clear.
I've gone ahead and re-written the commit title and message based on that. Hope that's ok.

This bug still seems to be present - is there anything we can help with to get it fixed / reviewed in more detail?

@ChrisSargent - please see the one comment I have above. Fixing it, re-syncing with the latest code and master and re-submitting the PR will move this forward.

Closing this old pull request. Please submit a new one when ready.