feat(stepfunctions-tasks): add support for SageMakerCreateTransformJob to take instance type as a parameter#14064
feat(stepfunctions-tasks): add support for SageMakerCreateTransformJob to take instance type as a parameter#14064PPPW wants to merge 6 commits intoaws:masterfrom
Conversation
…b to take instance type as a parameter
| * ML compute instance type for the transform job. | ||
| */ | ||
| readonly instanceType: ec2.InstanceType; | ||
| readonly instanceType: ec2.InstanceType | string; |
There was a problem hiding this comment.
Typescript union types will not be compatible with other languages.
This will need to be represented by a single type. The way we do it in the cdk is to create a union like class with APIs like fromInstanceType and fromString that return that type.
There was a problem hiding this comment.
I see, I can define a instanceType class here to use it as the type of this field, something like this:
export class InstanceType {
public static fromEc2InstanceType(instanceType: ec2.InstanceType) {
return new InstanceType(`ml.${instanceType}`);
}
public static fromString(instanceType: string) {
return new InstanceType(instanceType);
}
constructor(private readonly instanceType: string) {
}
public toString(): string {
return this.instanceType;
}
}
But it won't be backward compatible, e.g. current users need to change their code: instanceType: ec2.InstanceType.of(ec2.InstanceClass.P3, ec2.InstanceSize.XLARGE2). Should we make it backward compatible somehow, or it's fine to do this?
There was a problem hiding this comment.
Hi @shivlaks, could you take a quick look at the above comments when you get a chance? Thanks!
There was a problem hiding this comment.
Hi @shivlaks, looks like yarn compat is not happy with this change, because it changed the API. So should I make the InstanceType class defined in above to extend ec2. InstanceType?
packages/@aws-cdk/aws-stepfunctions-tasks/lib/sagemaker/create-transform-job.ts
Outdated
Show resolved
Hide resolved
|
Dear @shivlaks, when you get a chance, could let me know if my fix looks fine? The instance type part is not backward compatible, I appreciate if you could provide some suggestions. Thank you for your time! |
packages/@aws-cdk/aws-stepfunctions-tasks/test/sagemaker/create-transform-job.test.ts
Outdated
Show resolved
Hide resolved
packages/@aws-cdk/aws-stepfunctions-tasks/lib/sagemaker/base-types.ts
Outdated
Show resolved
Hide resolved
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
shivlaks
left a comment
There was a problem hiding this comment.
pending resolution re: breaking change from modifying type of instanceType from ec2.InstanceType to one that is defined locally within the module.
| * ML compute instance type for the transform job. | ||
| */ | ||
| readonly instanceType: ec2.InstanceType; | ||
| readonly instanceType: InstanceType; |
There was a problem hiding this comment.
I know we removed the @experimental annotations after you had initially published this PR, but I believe this is now a breaking change.
Let me check with a team member or two to see whether we would need to:
- support the old parameter but mark it as
@deprecated - make this change as it is, but call it out as a
BREAKINGchange.
It is a breaking change if we ship it as it is as clients who upgrade will see breakage.
There was a problem hiding this comment.
Right, in earlier comments I was checking whether we should make it backward compatible. Let me know if we have to extend ec2. InstanceType here, thanks
There was a problem hiding this comment.
As Shiv mentions above, we will need to create a new property (maybe mlInstanceType) and mark the old property as deprecated. Unfortunately we cannot extend this type with a type union due to jsii restrictions
|
This will probably be superseded by #15726 as a simpler solution |
Thanks @BenChaimberg, that looks good. Feel free to borrow my README and unit tests (or I can contribute if we'd like to). |
Currently to create a SageMaker Batch Transform job, we can't specify the instance type as a parameter (e.g.,
'InstanceType.$': '$.InstanceType'in CFN template), because theTransformResources.instanceTypeis of typeec2.InstanceType, and we will add a "ml." prefix to this EC2 instance type as the final instance type. We'd like to build step functions with SageMaker Batch Transform state and be able to support different instance types from user input.With this change, user can pass instance type as a parameter, with values such as "ml.p2.xlarge".
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license