aws-cdk-lib(step_functions): Creating an S3ObjectsItemReader with a dynamic path but fixed bucket name does not show up correctly in the StepFunction console.

[urkle]

Describe the bug

I am creating a DistributedMap step like so

map_for_s3 = sfn.DistributedMap(
    self, 'List Files',
    max_concurrency=1,
    item_reader=sfn.S3ObjectsItemReader(
        bucket=self.incoming_mail_bucket,
        prefix=sfn.JsonPath.string_at('$.pending_path'),
    ),
    item_batcher=sfn.ItemBatcher(
        max_items_per_batch=500,
    )
)

This generates a working step where the item payload looks like this.

{
    "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {
            "Bucket": "my-bucket-name",
            "Prefix.$": "$.pending_path"
        }
   },
}

However, the Step Function console page shows the step as incomplete, as the Step function console only recognizes either both json path values or fixed values

I tried to change it to use bucket_name_path instead, however that failed to add the required permissions into the Step function role for the S3 Bucket.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The step function console UI to allow a mixed of fixed & dynamic definition

Current Behavior

the UI does not show the configuration correctly, and doing any reworking to allow it to show correctly (e.g. adding a Pass above the DistributedMap that sets a bucket var to the fixed value and the path to the input param) does not auto-add the needed permissions to the generated step function execution role

Reproduction Steps

Documented above.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.166.0

Framework Version

No response

Node.js Version

18.16.1

OS

macOS 12.7.6

Language

Python

Language Version

Python 3.11

Other information

No response

[markmansur]

If you open the S3 item source drop down you'll notice two options

1. Get bucket and prefix at runtime from state input
2. Enter bucket and prefix

It's not possible to mix the two. Either both come from the state input or both are static values.

Regarding the permissions not being added. I believe this is because CDK does not know the bucket/prefix to which it should add permissions at synth time, since you're getting these values at runtime from the state input. In this case you'll need to explicitly add the S3 read permissions to the state machine execution role yourself.

Hope this helps!

[urkle]

@markmansur that is what I noticed as well. IMHO Kind of limiting. And would require an extra step just to assign state input to the fixed bucket value just to have it show on the UI correctly. Right now it "works" with the mixed configuration. It is just limiting on the UI to restrict things that way when the backend will work correctly regardless.

[markmansur]

@urkle Yes, it is a bit annoying to have to add a pass state when you want a static bucket name and a dynamic prefix (or vica-versa). From the CDK pov I think there's nothing that can be done. This is something that the StepFunctions service first needs to support.