feat(stepfunctions): distributed map construct #28821

abdelnn · 2024-01-23T01:48:40Z

Adds support for Step Functions Map state in Distributed mode. Currently, in order to create a Distributed Map in CDK, CDK users have to define a Custom State containing their Amazon States Language definition.

This solution consists of the creation of a new L2 construct, DistributedMap. This design decision was made due to the fact that some fields are exclusive to Distributed Maps, such as ItemReader. Adding support for it through the existing Map L2 construct would lead to some fields being conditionally available.

Some design decisions that were made:

I created an abstract class MapBase that encapsulates all fields currently supported by both inline and distributed maps. This includes all currently supported fields in the CDK except for iterator and parameters (deprecated fields). Those are now part of the Map subclass which extends MapBase. All new Distributed Maps fields are part of the new DistributedMap construct (also a subclass of MapBase)
Permissions specific to Distributed Maps are added as part of this new construct

Thanks to @beck3905 and their PR #24331 for inspiration. A lot of the ideas here are re-used from the PR cited.

Closes #23216

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map.ts

dominhhai · 2024-01-24T06:58:45Z

IMO, the ItemBatcher should be configurable

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map-base.ts

Co-authored-by: Roman <491247+moltar@users.noreply.github.com>

abdelnn · 2024-01-24T19:59:42Z

IMO, the ItemBatcher should be configurable

I agree, I've changed it so that it is. Thanks for the feedback!

kaizencc · 2024-02-01T21:57:14Z

I'm going to continue looking at this tomorrow. I haven't finished my review but there were some style changes that I wanted to get in first.

kaizencc

Ok. I started on a preliminary review. This is going to take a while though, since it is quite involved.

packages/aws-cdk-lib/aws-stepfunctions/lib/states/distributed-map.ts

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map-base.ts

kaizencc · 2024-02-02T18:54:08Z

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map-base.ts

+ *
+ * @see https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-map-state.html
+ */
+export abstract class MapBase extends State implements INextable {


Whats the motivation behind moving MapBase out of Map?

With this PR, I wanted to do two things:

Implement a new Distributed Maps construct

Ensure that this new construct does not support deprecated ASL fields.

The way I decided to go about this was to abstract the common properties of both distributed and inline maps, and only support deprecated ASL fields (parameters and iterator) in the existing Map construct for backwards compatibility concerns.

Two other alternatives that I briefly considered but ultimately rejected:

Adding to the existing Map construct the required Distributed Map fields. This was rejected as having fields be conditionally available based on what kind of map it is seemed unnecessarily convoluted

Extending the existing Map construct to create the Distributed Map one. This was rejected as the new construct would contain deprecated ASL fields

packages/aws-cdk-lib/aws-stepfunctions/lib/states/state.ts

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map.ts

Co-authored-by: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com>

Pull request has been modified.

abdelnn · 2024-02-02T19:25:45Z

Yes doing this will have some backwards compatibility concerns we have to skirt

That shouldn't be the case anymore though, right? We're now correctly returning the Map type just like we were before

kaizencc

Nice! Thanks for the hefty PR @abdelnn. lets give it a try...

mergify · 2024-02-09T00:54:14Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2024-02-09T00:57:58Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2024-02-09T01:25:15Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 169c6dd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2024-02-09T01:25:36Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@beck3905

Adds support for Step Functions Map state in Distributed mode. Currently, in order to create a Distributed Map in CDK, CDK users have to define a Custom State containing their Amazon States Language definition. This solution consists of the creation of a new L2 construct, `DistributedMap`. This design decision was made due to the fact that some fields are exclusive to Distributed Maps, such as `ItemReader`. Adding support for it through the existing `Map` L2 construct would lead to some fields being conditionally available. Some design decisions that were made: - I created an abstract class `MapBase` that encapsulates all fields currently supported by both `inline` and `distributed` maps. This includes all currently supported fields in the CDK except for `iterator` and `parameters` (deprecated fields). Those are now part of the Map subclass which extends `MapBase`. All new Distributed Maps fields are part of the new `DistributedMap` construct (also a subclass of `MapBase`) - Permissions specific to Distributed Maps are added as part of this new construct Thanks to @beck3905 and their PR #24331 for inspiration. A lot of the ideas here are re-used from the PR cited. Closes #23216 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

orekav · 2024-02-22T02:06:53Z

packages/aws-cdk-lib/aws-stepfunctions/lib/state-machine.ts

+      graph = new StateGraph(definitionBody.chainable.startState, 'State Machine definition');
+      graph.timeout = props.timeout;
+      for (const statement of graph.policyStatements) {
+        this.addToRolePolicy(statement);


@abdelnn is it possible that this is adding the role policies on the wrong place?

#29203

As soon as I remove this for and put it back where it was before (Only the for).
The bug disappears.

I believe you're right, this should be

this.role.addToPrincipalPolicy(statement)

… executionType in the ProcessorConfig (#30301) ### Issue # (if applicable) Closes #30194 ### Reason for this change In #27913, the ItemProcessor was introduced for use with the Map Class. With the executionType in the ProcessorConfig, it was possible to specify the executionType for the Map. On the other hand, in #28821, the DistributedMap Class was introduced. The mapExecutionType of the DistributedMap class always overwrites the executionType of the ProcessorConfig. Therefore, when using the DistributedMap class, the implementation ignores the executionType of the ProcessorConfig. However, this behavior cannot be inferred from the documentation. ### Description of changes * Added to the docs that when using the DistributedMap Class, the executionType in the ProcessorConfig is ignored. * Also added a warning. ### Description of how you validated changes Add unit tests and integ tests ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

… executionType in the ProcessorConfig (aws#30301) ### Issue # (if applicable) Closes aws#30194 ### Reason for this change In aws#27913, the ItemProcessor was introduced for use with the Map Class. With the executionType in the ProcessorConfig, it was possible to specify the executionType for the Map. On the other hand, in aws#28821, the DistributedMap Class was introduced. The mapExecutionType of the DistributedMap class always overwrites the executionType of the ProcessorConfig. Therefore, when using the DistributedMap class, the implementation ignores the executionType of the ProcessorConfig. However, this behavior cannot be inferred from the documentation. ### Description of changes * Added to the docs that when using the DistributedMap Class, the executionType in the ProcessorConfig is ignored. * Also added a warning. ### Description of how you validated changes Add unit tests and integ tests ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

feat(stepfunctions): add distributed map construct

9592e28

github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 labels Jan 23, 2024

aws-cdk-automation requested a review from a team January 23, 2024 01:48

github-actions bot added the beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK label Jan 23, 2024

fix backwards incompatible api changes

701160d

abdelnn commented Jan 23, 2024

View reviewed changes

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map.ts Show resolved Hide resolved

Fix README code snippets

7aab8a6

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jan 23, 2024

moltar reviewed Jan 24, 2024

View reviewed changes

packages/aws-cdk-lib/aws-stepfunctions/lib/states/map-base.ts Outdated Show resolved Hide resolved

abdelnn and others added 3 commits January 24, 2024 09:31

Merge branch 'main' into distributedmap

5780a74

Typo fix

f48f39e

Co-authored-by: Roman <491247+moltar@users.noreply.github.com>

make itemBatcher configurable

3331868

Merge branch 'main' into distributedmap

ec0c1f7

github-actions bot mentioned this pull request Feb 1, 2024

Monthly PR metrics report - Jan 2024 #28952

Closed

kaizencc changed the title ~~feat(stepfunctions): add distributed map construct~~ feat(stepfunctions): distributed map construct Feb 1, 2024

fix style

17f53fc

kaizencc previously requested changes Feb 2, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 2, 2024

Apply suggestions from code review

c848a83

Co-authored-by: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com>

Merge branch 'aws:main' into distributedmap

422ed73

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 2, 2024

abdelnn and others added 2 commits February 2, 2024 23:02

File refactor, addressing feedback from PR

171dd4d

Merge branch 'main' into distributedmap

7ec4ff6

Merge branch 'main' into distributedmap

b62c7a3

kaizencc approved these changes Feb 9, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 9, 2024

Merge branch 'main' into distributedmap

169c6dd

mergify bot merged commit 97e3827 into aws:main Feb 9, 2024
9 checks passed

abdelnn mentioned this pull request Feb 15, 2024

stepfunctions: using itemProcessor with mode = DISTRIBUTED doesn't work out of the box due to permission error #28820

Open

orekav reviewed Feb 22, 2024

View reviewed changes

abdelnn deleted the distributedmap branch March 7, 2024 18:04

khushail mentioned this pull request May 16, 2024

aws-stepfunctions: Distributed Map State Item Processor executionType doesn't recognize ProcessorType.EXPRESS #30194

Closed

mazyu36 mentioned this pull request May 22, 2024

fix(stepfunctions): add docs and warning DistributedMap Class ignores executionType in the ProcessorConfig #30301

Merged

1 task

This was referenced Jul 12, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#108

Open

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#109

Open

This was referenced Jul 19, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#110

Open

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#111

Open

NOUIY mentioned this pull request Aug 1, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#112

Open

gitafolabi mentioned this pull request Aug 16, 2024

[Snyk] Upgrade aws-cdk from 2.44.0 to 2.150.0 gitafolabi/kreuzlaker#2

Open

NOUIY mentioned this pull request Aug 20, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#113

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(stepfunctions): distributed map construct #28821

feat(stepfunctions): distributed map construct #28821

abdelnn commented Jan 23, 2024

dominhhai commented Jan 24, 2024

abdelnn commented Jan 24, 2024

kaizencc commented Feb 1, 2024

kaizencc left a comment

kaizencc Feb 2, 2024

abdelnn Feb 2, 2024

abdelnn commented Feb 2, 2024

kaizencc left a comment

mergify bot commented Feb 9, 2024

mergify bot commented Feb 9, 2024

aws-cdk-automation commented Feb 9, 2024

mergify bot commented Feb 9, 2024

orekav Feb 22, 2024 •

edited

Loading

orekav Feb 22, 2024

abdelnn Feb 23, 2024

feat(stepfunctions): distributed map construct #28821

feat(stepfunctions): distributed map construct #28821

Conversation

abdelnn commented Jan 23, 2024

dominhhai commented Jan 24, 2024

abdelnn commented Jan 24, 2024

kaizencc commented Feb 1, 2024

kaizencc left a comment

Choose a reason for hiding this comment

kaizencc Feb 2, 2024

Choose a reason for hiding this comment

abdelnn Feb 2, 2024

Choose a reason for hiding this comment

abdelnn commented Feb 2, 2024

kaizencc left a comment

Choose a reason for hiding this comment

mergify bot commented Feb 9, 2024

mergify bot commented Feb 9, 2024

aws-cdk-automation commented Feb 9, 2024

AWS CodeBuild CI Report

mergify bot commented Feb 9, 2024

orekav Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

orekav Feb 22, 2024

Choose a reason for hiding this comment

abdelnn Feb 23, 2024

Choose a reason for hiding this comment

orekav Feb 22, 2024 •

edited

Loading