feat(stepfunctions-tasks): step functions task for cross-region AWS API call by tmokmss · Pull Request #30061 · aws/aws-cdk

tmokmss · 2024-05-04T14:34:48Z

Issue # (if applicable)

Reason for this change

It would be useful if we could call AWS API across regions from a Step Functions state machine. Currently it is not officially supported even with AWS SDK integration tasks.

Our usecase is to automate a cross-region failover scenario in a multi-region application. This requires you to orchestrate multiple API calls for both active and standby regions (e.g. failover Aurora DB cluster, rewrite AppConfig parameter, etc), and it would be great if we can manage these operations in a single state machine.

Description of changes

This PR adds a new construct CallAwsServiceCrossRegion that deploys 1. a Lambda function to call AWS API in different regions 2. SFn task to call the function.

Because most properties are compatible with the existing CallAwsService construct, you can use the new construct by just adding the region property.

Additionally, it also allows to set endpoint to override AWS API endpoint, because some AWS APIs requires you to override it. (e.g. Route53 ARC)

Here's example code to define this new SFn task:

    const pollTable = new tasks.CallAwsServiceCrossRegion(this, 'DescribeTable', {
      service: 'dynamodb',
      action: 'describeTable',
      parameters: {
        TableName: tableName,
      },
      // You can set any region in the same partition
      region: 'ap-northeast-1',
    });

Description of how you validated changes

Added unit tests and integ tests.

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…WS API call

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts

nmussy

Looks good overall, just some concerns and minor comments. I'm also wondering if it's better for maintainability not to create a new class, and to reuse the existing CallAwsService, and just to add an optional crossRegionTarget prop

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts

...mework-integ/test/aws-stepfunctions-tasks/test/lambda/integ.call-aws-service-cross-region.ts

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts

nmussy · 2024-05-06T14:31:47Z

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts

+
+  try {
+    // esbuild-disable unsupported-require-call -- not esbuildable but that's fine
+    const pkg = require(`@aws-sdk/client-${event.service}`);


Correct me if I'm wrong, but normalizeServiceName is not being ran for your handler? It would allow us to keep the same pattern as other services props, see AwsSdkCall:

aws-cdk/packages/@aws-cdk/aws-custom-resource-sdk-adapter/lib/api-call.ts

Line 65 in fff9cf6

this.service = normalizeServiceName(service);

aws-cdk/packages/@aws-cdk/aws-custom-resource-sdk-adapter/lib/sdk-info.ts

Lines 10 to 15 in fff9cf6

export function normalizeServiceName(service: string) {

service = service.toLowerCase(); // Lowercase

service = service.replace(/^@aws-sdk\/client-/, ''); // Strip the start of a V3 package name

service = v2ToV3Mapping()?.[service] ?? service; // Optionally map v2 name -> v3 name

return service;

}

I'd like to keep this feature and API as simple as possible to lower the maintenance cost. AwsSdkCall code has become such complicated mostly for backward compatibility reason. Since this feature is new, we don't have to introduce the complexity at all.

Also the API is aimed to be compatible with the existing CallAwsService construct (and it should), which also does not support normalizing service names.

I'll let the maintainer weigh in, I think it's a nice piece of QoL, but I understand your maintainability concern

To help me understand, can you elaborate on your concern? What maintenance cost are you concerned with using AwsSdkCall code.

AwsSdkCall's code is mainly to solve the v2-v3 compatibility problem. Since this construct does not require such compatibility, I think there is little advantage to using the existing code. After all, the new Lambda is only code under 100 lines. Also, if we want to reuse the existing code, we need to refactor it first (for example, currently it's only for CFn custom resource handler, but this one is for invocation from SFn). Given the ROI, I'd rather keep them independent.

nmussy · 2024-05-06T14:32:06Z

packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

+ */
+export interface CallAwsServiceCrossRegionProps extends sfn.TaskStateBaseProps {
+  /**
+   * The AWS service to call in AWS SDK for JavaScript v3 style.


See comment about normalizeServiceName

nmussy · 2024-05-06T14:33:53Z

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts

+    // esbuild-disable unsupported-require-call -- not esbuildable but that's fine
+    const pkg = require(`@aws-sdk/client-${event.service}`);
+    const Client = findV3ClientConstructor(pkg);
+    const Command = findCommandClass(pkg, event.action);


Similarly, you should be able to use normalizeActionName, see AwsSdkCall:

aws-cdk/packages/@aws-cdk/aws-custom-resource-sdk-adapter/lib/sdk-info.ts

Lines 10 to 16 in fff9cf6

export function normalizeServiceName(service: string) {

service = service.toLowerCase(); // Lowercase

service = service.replace(/^@aws-sdk\/client-/, ''); // Strip the start of a V3 package name

service = v2ToV3Mapping()?.[service] ?? service; // Optionally map v2 name -> v3 name

return service;

}

Same as service name, I'd like to intentionally only support action names in camelCase.

nmussy · 2024-05-06T14:34:08Z

packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

+  /**
+   * The API action to call.
+   *
+   * Use camelCase.


See comment about normalizeActionName

packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

tmokmss · 2024-05-07T04:15:09Z

Hi @nmussy, thanks for the comments.

reuse the existing CallAwsService, and just to add an optional crossRegionTarget prop

I considered this approach as well but chose the current approach, because with the current approach we can easily avoid from future breaking changes when SFn will officially start to support cross-region API call; we can just deprecate CallAwsServiceCrossRegion construct and add the official functionality to CallAwsService construct.

nmussy

LGTM, just a couple of pending comments

GavinZZ

Left some minor feedbacks. Also have a high-level questions I want to clarify on. There's already a stepfunctions task for CallAwsService. Is the reason of building this cross-region-aws-sdk-handler using custom resource lambda function because the AWS SDK service integrations do not support cross region invocation?

packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

GavinZZ · 2024-06-18T20:51:05Z

packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

+   *
+   * @default - service:action
+   */
+  readonly iamAction?: string;


Question: does it only support one single action? What would users do if I want to support multiple actions?

In that case, you can use additionalIamStatements. I agree that this should ideally be iamActions?: string[], but then the API becomes less compatible with CallAwsService construct. I think it's okay to leave this as-is because most API only requires a single IAM action anyway. Wdyt?

Can you share an example how to use additionalIamStatements with another action? I also think it would be nice to add this example to the readme file.

Sure, but that is same as CallAwsService, which is described here:

aws-cdk/packages/aws-cdk-lib/aws-stepfunctions-tasks/README.md

Lines 233 to 245 in c826d8f

```ts

const detectLabels = new tasks.CallAwsService(this, 'DetectLabels', {

service: 'rekognition',

action: 'detectLabels',

iamResources: ['*'],

additionalIamStatements: [

new iam.PolicyStatement({

actions: ['s3:getObject'],

resources: ['arn:aws:s3:::my-bucket/*'],

}),

],

});

```

so I added a line as below:

Other properties such as additionalIamStatements can be used in the same way as the CallAwsService task.

Sounds good, thank you!

packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

GavinZZ · 2024-06-18T21:14:16Z

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts

+
+  try {
+    // esbuild-disable unsupported-require-call -- not esbuildable but that's fine
+    const pkg = require(`@aws-sdk/client-${event.service}`);


To help me understand, can you elaborate on your concern? What maintenance cost are you concerned with using AwsSdkCall code.

tmokmss · 2024-06-19T12:05:15Z

Hi @GavinZZ, thank you so much for the review!

Is the reason of building this cross-region-aws-sdk-handler using custom resource lambda function because the AWS SDK service integrations do not support cross region invocation

Yes. According to the doc:

Currently, cross-Region AWS SDK integration and cross-Region AWS resource access aren't available in Step Functions.
https://docs.aws.amazon.com/step-functions/latest/dg/concepts-access-cross-acct-resources.html

mergify · 2024-06-21T01:04:39Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2024-06-21T01:08:53Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Pull request has been modified.

tmokmss · 2024-06-21T12:58:22Z

@GavinZZ Integ test failed due to a recent mainline change #30108. Looking forward to your re-approval 🙏 Thanks!

mergify · 2024-06-24T17:18:34Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2024-06-24T17:48:45Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 6e6693e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2024-06-24T17:49:09Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2024-07-25T21:11:23Z

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

feat(aws-stepfunctions-tasks): Step Functions task for cross-region A…

214f00f

…WS API call

github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 star-contributor [Pilot] contributed between 25-49 PRs to the CDK labels May 4, 2024

aws-cdk-automation requested a review from a team May 4, 2024 14:35

aws-cdk-automation previously requested changes May 4, 2024

View reviewed changes

tmokmss changed the title ~~feat(aws-stepfunctions-tasks): Step Functions task for cross-region AWS API call~~ feat(stepfunctions-tasks): step functions task for cross-region AWS API call May 4, 2024

Merge branch 'main' into cross-region-api-sfn

19fbe96

nmussy reviewed May 4, 2024

View reviewed changes

...k/custom-resource-handlers/lib/aws-stepfunctions-tasks/cross-region-aws-sdk-handler/index.ts Show resolved Hide resolved

Update call-aws-service-cross-region.ts

87eb546

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label May 4, 2024

update comments

0769697

github-actions bot mentioned this pull request May 6, 2024

Weekly PR metrics report #30073

Closed

nmussy suggested changes May 6, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label May 6, 2024

address review comments

9ec0fa6

nmussy approved these changes May 7, 2024

View reviewed changes

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label May 7, 2024

update comments

e387716

github-actions bot mentioned this pull request Jun 1, 2024

Monthly PR metrics report #30414

Closed

GavinZZ reviewed Jun 18, 2024

View reviewed changes

tmokmss added 2 commits June 19, 2024 21:10

Update call-aws-service-cross-region.ts

ebc831d

Update README.md

e4f99fd

GavinZZ previously approved these changes Jun 21, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jun 21, 2024

mergify bot and others added 2 commits June 21, 2024 01:39

Merge branch 'main' into cross-region-api-sfn

210a707

update integ

98d836b

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jun 21, 2024

GavinZZ approved these changes Jun 24, 2024

View reviewed changes

Merge branch 'main' into cross-region-api-sfn

6e6693e

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jun 24, 2024

mergify bot merged commit 1397737 into aws:main Jun 24, 2024

tmokmss deleted the cross-region-api-sfn branch June 24, 2024 23:35

ADringer mentioned this pull request Jul 9, 2024

stepfunctions-tasks: CallAwsServiceCrossRegion uses wrong casing for ECS #30799

Closed

aws locked as resolved and limited conversation to collaborators Jul 25, 2024

	export function normalizeServiceName(service: string) {
	service = service.toLowerCase(); // Lowercase
	service = service.replace(/^@aws-sdk\/client-/, ''); // Strip the start of a V3 package name
	service = v2ToV3Mapping()?.[service] ?? service; // Optionally map v2 name -> v3 name
	return service;
	}

	```ts
	const detectLabels = new tasks.CallAwsService(this, 'DetectLabels', {
	service: 'rekognition',
	action: 'detectLabels',
	iamResources: ['*'],
	additionalIamStatements: [
	new iam.PolicyStatement({
	actions: ['s3:getObject'],
	resources: ['arn:aws:s3:::my-bucket/*'],
	}),
	],
	});
	```

Conversation

tmokmss commented May 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue # (if applicable)

Reason for this change

Description of changes

Description of how you validated changes

Checklist

Uh oh!

aws-cdk-automation left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nmussy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmokmss May 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tmokmss commented May 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nmussy left a comment

Choose a reason for hiding this comment

Uh oh!

GavinZZ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmokmss Jun 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmokmss Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmokmss commented Jun 19, 2024

Uh oh!

mergify bot commented Jun 21, 2024

Uh oh!

mergify bot commented Jun 21, 2024

Uh oh!

tmokmss commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Jun 24, 2024

Uh oh!

aws-cdk-automation commented Jun 24, 2024

tmokmss commented May 4, 2024 •

edited

Loading

tmokmss May 7, 2024 •

edited

Loading

tmokmss commented May 7, 2024 •

edited

Loading

tmokmss Jun 19, 2024 •

edited

Loading

tmokmss Jun 20, 2024 •

edited

Loading

tmokmss commented Jun 21, 2024 •

edited

Loading