feat(stepfunctions-tasks): add step concurrency level to EmrCreateCluster by mousabaker · Pull Request #15242 · aws/aws-cdk

mousabaker · 2021-06-22T07:33:34Z

Added support for step concurrency when creating EMR clusters through Step Functions. This feature allows users to run multiple steps in parallel on a cluster created through SFN.

closes #15223.

As a byproduct, adds validation for releaseLabel to ensure that it follows the correct format laid out here.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…EMR cluster

gitpod-io · 2021-06-22T07:33:36Z

mergify · 2021-06-22T07:34:13Z

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

shivlaks

@MousaZeidBaker thanks for opening a contribution.

I think we also need to add some tests to go with this added feature.

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts

Pull request has been modified.

mousabaker · 2021-06-22T21:08:07Z

Thanks for your comments @shivlaks. They should now all be resolved.

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts

…luster.ts Co-authored-by: Ben Chaimberg <youppi3@gmail.com>

Pull request has been modified.

…luster.ts Co-authored-by: Ben Chaimberg <youppi3@gmail.com>

mousabaker · 2021-07-12T19:17:28Z

Thanks for the suggestions @BenChaimberg, they should now all be committed. Do you know why the AWS CodeBuild is failing?

BenChaimberg · 2021-07-12T19:52:26Z

Click "Build Logs" on the aws-cdk-automation comment. You have a compilation error

…y_level

BenChaimberg

please add a small blurb to the README

packages/@aws-cdk/aws-stepfunctions-tasks/test/emr/emr-create-cluster.test.ts

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts

HsiehShuJeng · 2021-09-06T08:28:13Z

For those who have been waiting for this feature by accident, they could consider implementing the following class in advance:

import * as sfn from '@aws-cdk/aws-stepfunctions';
import * as tasks from '@aws-cdk/aws-stepfunctions-tasks';

interface ExtendedEmrCreateClusterProps extends tasks.EmrCreateClusterProps {
  /**
   * Specifies the step concurrency level to allow multiple steps to run in parallel
   *
   * Requires EMR release label 5.28.0 or above.
   * Must be in range [1, 256].
   *
   * @default 1 - no step concurrency allowed
   */
  readonly stepConcurrencyLevel?: number;
}

/**
 * A Step Functions Task to create an EMR Cluster.
 * 
 * The ClusterConfiguration is defined as Parameters in the state machine definition.
 * 
 * OUTPUT: the ClusterId.
 * 
 * @see https://github.com/aws/aws-cdk/issues/14408
 * @see https://github.com/aws/aws-cdk/issues/15223
 * @see https://github.com/aws/aws-cdk/pull/15242/
 */
class ExtendedEmrCreateCluster extends tasks.EmrCreateCluster {
  protected readonly stepConcurrencyLevel: number;
  constructor(scope: cdk.Construct, id: string, props: ExtendedEmrCreateClusterProps) {
    super(scope, id, props);
    this.stepConcurrencyLevel = props.stepConcurrencyLevel ?? 1;
  }
  protected _renderTask(): any {
    const originalObject = super._renderTask();
    const extensionObject = {};
    Object.assign(extensionObject, originalObject, { Parameters: { StepConcurrencyLevel: cdk.numberToCloudFormation(this.stepConcurrencyLevel), ...originalObject.Parameters } })
    return extensionObject
  }
}

kaizencc · 2021-09-30T17:28:38Z

Cleaning up a stale PR inherited from @BenChaimberg. I can't request your review Ben, but I'm happy to take comments if you have them.

BenChaimberg

One small comment, rest looks great!

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts

madeline-k

Looks great! Nice work, @kaizen3031593 and @MousaZeidBaker!

mergify · 2021-10-04T23:54:26Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2021-10-05T01:14:35Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: 4e75b74
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2021-10-05T01:15:14Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

* '15588' of https://github.com/xykkong/aws-cdk: (47 commits) chore: rollback `GenericSSMParameterImage` deprecation (backport aws#16798) (aws#16800) chore(deps): bump actions/setup-node from 2.4.0 to 2.4.1 (aws#16778) Update CHANGELOG.md chore(release): 1.126.0 feat(assertions): matcher support for `templateMatches()` API (aws#16789) feat(stepfunctions-tasks): add step concurrency level to EmrCreateCluster (aws#15242) docs(s3): correct heading levels Object Ownership / Bucket deletion (aws#16790) chore(individual-pkg-gen): fix bug in setting alpha package visibility (aws#16787) fix(s3): setting `autoDeleteObjects` to `false` empties the bucket (aws#16756) fix(iam): `User.fromUserArn` does not work for ARNs that include a path (aws#16269) fix(cli): progress bar overshoots count by 1 for stack updates (aws#16168) fix(config): add SourceAccount condition to Lambda permission (aws#16617) docs(events): add a note about not using `EventPattern` with `CfnRule` (aws#16715) docs(core): fix reference to nonexistant enum value (aws#16716) chore(s3-deployments): update python version on BucketDeployment handler (aws#16771) chore: set response-requested length to 2 and closing-soon to 5 (aws#16763) fix(revert): "fix: CDK does not honor NO_PROXY settings (aws#16751)" (aws#16761) docs(GitHub issue templates): Upgrade to GitHub Issues v2 (aws#16592) chore: reset jsii-rosetta worker count to default (aws#16755) fix: CDK does not honor NO_PROXY settings (aws#16751) ...

…ster (#15242) Added support for step concurrency when creating EMR clusters through Step Functions. This feature allows users to run multiple steps in parallel on a cluster created through SFN. closes #15223. As a byproduct, adds validation for `releaseLabel` to ensure that it follows the correct format laid out [here](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html). ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…ster (aws#15242) Added support for step concurrency when creating EMR clusters through Step Functions. This feature allows users to run multiple steps in parallel on a cluster created through SFN. closes aws#15223. As a byproduct, adds validation for `releaseLabel` to ensure that it follows the correct format laid out [here](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html). ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

feat(aws-stepfunctions-tasks) set StepConcurrencyLevel when creating …

c5eb4fc

…EMR cluster

mousabaker changed the title ~~feat(aws-stepfunctions-tasks) set StepConcurrencyLevel when creating EMR cluster~~ feat(aws-stepfunctions-tasks) add StepConcurrencyLevel property Jun 22, 2021

Merge branch 'master' into emr_step_concurrency_level

ba6a817

shivlaks previously requested changes Jun 22, 2021

View reviewed changes

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts Outdated Show resolved Hide resolved

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts Outdated Show resolved Hide resolved

Mousa Zeid Baker added 2 commits June 22, 2021 21:38

add test for StepConcurrencyLevel

70626ff

update documentation

7f396b2

validate release label when step Concurrency Level is specified

8e86cf5

MousaZeidBaker added 3 commits June 22, 2021 23:08

Merge branch 'master' into emr_step_concurrency_level

521d356

Merge branch 'master' into emr_step_concurrency_level

c76f5d1

Merge branch 'master' into emr_step_concurrency_level

98fe675

peterwoodworth assigned BenChaimberg Jul 7, 2021

peterwoodworth added the @aws-cdk/aws-stepfunctions-tasks label Jul 7, 2021

BenChaimberg previously requested changes Jul 12, 2021

View reviewed changes

MousaZeidBaker and others added 3 commits July 12, 2021 20:47

Update packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-c…

a2a05ad

…luster.ts Co-authored-by: Ben Chaimberg <youppi3@gmail.com>

Update packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-c…

2e1b333

…luster.ts Co-authored-by: Ben Chaimberg <youppi3@gmail.com>

Update packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-c…

fec6016

…luster.ts Co-authored-by: Ben Chaimberg <youppi3@gmail.com>

Update packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-c…

c1a951c

…luster.ts Co-authored-by: Ben Chaimberg <youppi3@gmail.com>

Mousa Zeid Baker and others added 3 commits July 12, 2021 22:48

update dependencies

20f1a8e

complete unterminated string literal

69e07df

Merge remote-tracking branch 'origin/master' into emr_step_concurrenc…

b745bb3

…y_level

BenChaimberg changed the title ~~feat(aws-stepfunctions-tasks) add StepConcurrencyLevel property~~ feat(stepfunctions-tasks): add step concurrency level to EmrCreateCluster Jul 21, 2021

BenChaimberg suggested changes Jul 21, 2021

View reviewed changes

packages/@aws-cdk/aws-stepfunctions-tasks/test/emr/emr-create-cluster.test.ts Outdated Show resolved Hide resolved

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts Outdated Show resolved Hide resolved

kaizencc mentioned this pull request Sep 23, 2021

aws-stepfunctions-tasks: Add support for emr-containers #16621

Closed

2 tasks

kaizencc assigned kaizencc and unassigned BenChaimberg Sep 24, 2021

RomainMuller force-pushed the master branch from 924c117 to ebfd5f2 Compare September 29, 2021 13:07

kaizencc and others added 5 commits September 30, 2021 11:55

Merge branch 'master' into emr_step_concurrency_level

0f7a584

incoporate PCR comments on stale PR.

0c9b7b9

fix edge case where StepConcurrencyLevel set to 1

3b5151e

add README blurb

d48052f

1 line change

88d9fd1

kaizencc requested a review from a team September 30, 2021 17:26

BenChaimberg suggested changes Sep 30, 2021

View reviewed changes

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts Show resolved Hide resolved

add validateReleaseLabel to EmrCreateCluster

581ea13

BenChaimberg approved these changes Sep 30, 2021

View reviewed changes

packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts Show resolved Hide resolved

kaizencc and others added 4 commits September 30, 2021 16:02

add documentation for validateReleaseLabel

020f1c4

add documentation for validateReleaseLabel

3300891

Merge branch 'master' into emr_step_concurrency_level

8034932

Merge branch 'master' into emr_step_concurrency_level

0e9382d

madeline-k approved these changes Oct 4, 2021

View reviewed changes

Merge branch 'master' into emr_step_concurrency_level

4e75b74

mergify bot merged commit 1deea90 into aws:master Oct 5, 2021

Conversation

mousabaker commented Jun 22, 2021 • edited by kaizencc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitpod-io bot commented Jun 22, 2021

Uh oh!

mergify bot commented Jun 22, 2021

Uh oh!

shivlaks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mousabaker commented Jun 22, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mousabaker commented Jul 12, 2021

Uh oh!

BenChaimberg commented Jul 12, 2021

Uh oh!

BenChaimberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HsiehShuJeng commented Sep 6, 2021

Uh oh!

kaizencc commented Sep 30, 2021

Uh oh!

BenChaimberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

madeline-k left a comment

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Oct 4, 2021

Uh oh!

aws-cdk-automation commented Oct 5, 2021

AWS CodeBuild CI Report

Uh oh!

mergify bot commented Oct 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mousabaker commented Jun 22, 2021 •

edited by kaizencc

Loading