aws-lambda-nodejs: How to setup AWS CDK to use ESM with layers / top level await

[pfried]

Describe the issue

Lambda is one of the most important services of the AWS portfolio. Significant improvements have been made allowing the NodeJS runtime (14+) to use ESM and top level await:

1. ESM Announcement Top Level Await
2. NodeJS 18 Annoucement
3. NodeJS 18 Blog Post (aws-cdk_v3, ESM via NODE_PATH, global fetch, ...)

Yet I cannot really use that in my application due to several reasons:

1. Lambda Layers are a de facto requirement for most bigger applications. Before the NodeJS 18 release there was no support for ESM Lambda Layers. I have not checked if the new

Support for ES module resolution using NODE_PATH

solves this issue since I did not test it ever since. Has anyone?

2. When having CDK project files with the lambda function handler code side by side in a project you will have issues with typechecking since the .tsconfig file shipped with aws-cdk does not play nice with handler code which needs other settings in .tsconfig. On the other hand if you change your .tsconfig file to other target/module/library settings the project will fail (no module errors, if you change package.json to "type"="module" it will not like the *.ts` extension, etc. etc. its a rabbit hole to which bottom I never got)

3. The state of Lambda Layer bundling is a wild landscape of self-baked solutions (of course I have my own). The issues of this are widespread and maybe better in its own thread)

I would heavily welcome if someone at AWS / AWS-CDK takes a deep dive on how to setup an aws-cdk TypeScript project with ESM / top level await / ESM layers and shows it to the rest of the world. There is too much potential improvement on the line to keep this lingering in its current state (is it only the docs?)

The whole JavaScript ecosystem with CJS / ESM etc. is a mess only few developers will fully understand (excluding me). AWS-CDK is a powerful but complex software (which I love). Implementing the above requires a lot of knowledge (at least when the solution must be compatible with future plans / releases)

Related Issues:

1. Appsync with NodejsFunction lambda resolver doesn't work with ESM format aws-appsync-community#213
2. Lambda: Can't use typescript module files #21635
3. (lambdanodejs): using top level await fails at runtime #21329
4. Cannot find package when using ES Module and Lambda Layer aws-sdk-js-v3#3386
5. Using ES Modules breaks when a directory with the same name as the module exists aws-lambda-nodejs-runtime-interface-client#93 (comment)

Partial solutions / approaches to some problems and additional content

1. https://github.com/coderbyheart/aws-lambda-esm-with-layer
2. https://github.com/vibe/aws-esm-modules-layer-support
3. https://twitter.com/adamdotdev/status/1523282408417677312 + https://github.com/adamelmore/cdk-top-level-await
4. https://github.com/huntharo/lambda-docker-typescript-esm

Links

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs-readme.html

[revmischa]

I have a CDK Severless-Stack project template that uses ESM and top-level await here: https://github.com/jetbridge/sst-prisma
It has layers (prisma and sharp for nextjs) that are CJS but are loaded fine by ESM lambda functions with the use of a goofy compat shim

[pfried]

I created a repository here: https://github.com/Nantis-GmbH/cdk-lambda-with-layers to show my approach to lambda and layer bundling.

Short summary:

• I do use package.json files which will be bundled into a Layer using Docker and the SAM Docker images for the runtime. Resolution of this works in the latest v18 runtime
• The Lambda functions then use that layer and declare all layer dependcies as external (+aws-sdk).
• The function will then be bundled (+everything not in the layer) normally as an ESM. This supports top level await.

Downsides:

• You will have to put the dependencies of your lambda functions in your package.json and you will need to make sure the versions of the layer match those.
• When using a monorepo setup bundling local packages with Docker might not be ideal and you will have to adjust the working directory etc. to make it work. In my setup where I have a yarn monorepo I do prefer to have the dependencies locally installed (not using docker) (resolves my own packages nicely by following the symlinks). This uses that fact that workspaces run npm install also in the packages (issue: also dev-dependencies).

[otterley]

Just FYI, our use of the / separator in our Construct Library packages causes issues with some linters, namely, es-lint-plugin-n (see eslint-community/eslint-plugin-n#21)

[daveharig]

Confirmed w/ AWS support

1. CDK and Lambda Node 16+ w/ Top level Await are both supported features of AWS.

2. They can't be used together at the same time.

3. The service team has known about the issue for almost a year.

4. They have yet to provide a workaround.

I'll keep requesting escalations w/ AWS Support until I get a solution worth reporting back with.

[mrgrain]

@daveharig Did you mean to mention layers somewhere in your list?

[daveharig]

The issue is bigger than Layers. It's any TS CDK project with any Lambda wanting to use ESM at the same time. This Layers case was just the first one to discover it and publicly post about it. This guy sums up the broader issue really well. https://twitter.com/adamdotdev/status/1523282408417677312?s=20

[mrgrain]

Thanks for the link @daveharig. Going by the video, this appears possible already (not the layers stuff maybe). I also agree with the OP that ESM vs CJS is a mess and hard to understand. Would you mind clarifying what your ask is specifically?

[daveharig]

The Twitter workaround is hack-y at best. Depending on the SDK's used in your Lambdas you end up chasing your tail even more.

My ask is that the CDK service team provide official documentation on how to support Lambda w/ ESM in a TS CDK project.

[ThePlenkov]

In my case I just changed to "app": "npx tsx src/bin/app.ts" and it just works.
No need even to compile. Check what tsx project is.

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[coderbyheart]

@pfried can you add aws/aws-lambda-nodejs-runtime-interface-client#93 (comment) to your related issues?

[daveharig]

@ThePlenkov this allowed you to change to "type": "module" in your package.json and "target": "ESNext" "module": "ESNext" in your tsconfig.json?

[ThePlenkov]

Of course. Tsx doesn't care

[daveharig]

@ThePlenkov thank you! I'll give it a try.

[pedroGenio]

could anyone please clarify how to use TSX to get top level await working with Typescript and CDK ?
I'm quite new to both and my setup is not working properly.

[domengabrovsek]

In my case I just changed to "app": "npx tsx src/bin/app.ts" and it just works. No need even to compile. Check what tsx project is.

This worked for me as well to deploy the cdk project. I haven't tried the top level await yet. I'll report how that goes.

[daveharig]

@domengabrovsek that is fantastic! Would you be willing to share how you have your compiler options set in tsconfig.json?

[Phil1216]

I too have been down the ts-node rabbit hole. Apparently using "module": "ESNext" with ts-node requires esm, and the issue here TypeStrong/ts-node#1997 suggests that using esm with ts-node is broken for newer versions of node.

When I switched to tsx, things started working.

[cjnoname]

swc-node will save your life