Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement code generated handler framework #28251

Merged
merged 77 commits into from
Dec 16, 2023
Merged
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
b983212
initial commit - framework, classes, constructors, modules
colifran Dec 4, 2023
7145c7a
refactor
colifran Dec 4, 2023
dd2ff42
refactor and bug fixes
colifran Dec 5, 2023
f7b4593
initial setup for config file
colifran Dec 5, 2023
2038bcf
updated outfile location and config format
colifran Dec 5, 2023
d888a5d
Merge branch 'main' into colifran/codegen-handler-framework
colifran Dec 5, 2023
c87613b
initial work on build script
colifran Dec 5, 2023
6de2be2
refactor
colifran Dec 5, 2023
5873fe4
migrated handler framework into custom resource handlers and updated …
colifran Dec 5, 2023
dccc985
migrated handler-framework into custom-resource-handlers
colifran Dec 5, 2023
688a69d
imports for external modules
colifran Dec 6, 2023
c920b1e
refactored airlift custom resource handlers
colifran Dec 6, 2023
c78566a
formatting
colifran Dec 6, 2023
0f8c8e5
exclude dist from compilation
colifran Dec 6, 2023
00fc75a
code to generate custom super props
colifran Dec 6, 2023
ba9c570
path import and path literal in config
colifran Dec 6, 2023
3f450da
remove cdk handler and runtime determiner
colifran Dec 6, 2023
ba044b6
compatible framework runtimes
colifran Dec 6, 2023
43e60d7
singleton interface creation
colifran Dec 6, 2023
8ef2e15
refactor for generate script
colifran Dec 6, 2023
7ef5139
removed unneeded console log
colifran Dec 6, 2023
0c45784
added to config
colifran Dec 7, 2023
1647a82
no op
colifran Dec 7, 2023
7da55a3
config
colifran Dec 7, 2023
2210613
refactor for cdk handler changes and added immutable to property spec…
colifran Dec 7, 2023
9d950d9
cdk-handler and runtime determiner
colifran Dec 7, 2023
e19f090
get or create cdk handler to force singleton cdk handler and migration
colifran Dec 7, 2023
6712a4e
merge conflicts
colifran Dec 7, 2023
fd0b93f
refactor for cdk handler change and updated config
colifran Dec 7, 2023
bacd7b6
move runtime determiner and create runtime class to break circular de…
colifran Dec 8, 2023
ca3be84
renamed to custom resource framework
colifran Dec 8, 2023
721c59f
migrate
colifran Dec 8, 2023
f2504ae
refactor
colifran Dec 8, 2023
98cbe11
migrate
colifran Dec 8, 2023
aa3c176
migrate to codegen providers
colifran Dec 8, 2023
c89f470
airlift core to core
colifran Dec 8, 2023
b0e9ba4
core internal
colifran Dec 8, 2023
d6d5cbf
finished migrating core providers
colifran Dec 8, 2023
ca0ad0a
reverted drain hook change
colifran Dec 8, 2023
2ca1f33
use lambda runtime and auto generate component name
colifran Dec 8, 2023
e036649
lambda runtime isdeprecated flag
colifran Dec 8, 2023
b8da22d
generate names better
colifran Dec 8, 2023
13a0b48
formatting
colifran Dec 8, 2023
1496b55
update custom resources to use auto generated names
colifran Dec 9, 2023
972836b
remove console log
colifran Dec 9, 2023
7b87231
drain hook revert
colifran Dec 9, 2023
c4a20ba
runtimes determiner testing
colifran Dec 10, 2023
510bcaa
testing and empty readme
colifran Dec 11, 2023
a7b74e1
alpha
colifran Dec 11, 2023
228aafd
runtime deprecated
colifran Dec 11, 2023
c3fdb61
naming
colifran Dec 11, 2023
187c7df
refactor and naming update
colifran Dec 11, 2023
f857264
naming
colifran Dec 11, 2023
2db1c20
docstring
colifran Dec 11, 2023
3c81f54
fix framework unit tests
colifran Dec 11, 2023
cc69345
fix failures
colifran Dec 11, 2023
034c7ce
unit tests
colifran Dec 11, 2023
7ba9f86
selective import from core internal stack module
colifran Dec 11, 2023
1db5e8c
formatting
colifran Dec 12, 2023
c11b42a
update importing modules
colifran Dec 12, 2023
3744d65
documentation
colifran Dec 13, 2023
a64afe4
readme and naming
colifran Dec 13, 2023
3a8757a
readme
colifran Dec 13, 2023
7a326ca
fix unit tests
colifran Dec 13, 2023
8582ad6
Merge branch 'main' into colifran/codegen-handler-framework
colifran Dec 13, 2023
cb75044
Merge branch 'main' into colifran/codegen-handler-framework
colifran Dec 14, 2023
812ac0a
naming
colifran Dec 14, 2023
7b3e7c1
name generator
colifran Dec 14, 2023
cf435a0
updated generated component names
colifran Dec 14, 2023
4d25941
fixed unit tests with new naming
colifran Dec 14, 2023
e8f5a3a
reverted lambda runtime deprecated flag
colifran Dec 14, 2023
46aa221
fix constructor props
colifran Dec 14, 2023
ab259ed
node18 as esbuild target
colifran Dec 14, 2023
11f9476
readme
colifran Dec 15, 2023
f77cacf
readme
colifran Dec 15, 2023
dedc574
utils tests
colifran Dec 15, 2023
2dffd0a
Merge branch 'main' into colifran/codegen-handler-framework
mergify[bot] Dec 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# AWS CDK Custom Resource Framework

The CDK custom resource framework is an internal framework developed to establish best practices, runtime enforcement, and consistency for custom resource providers and their associated source code.

## Custom Resource Framework Concepts

Custom resource providers can be created in one of three forms - `CdkFunction`, `CdkSingletonFunction`, or `CdkCustomResourceProvider`. These three custom resource provider formats are a code generated wrapper around the lambda `Function`, lambda `SingletonFunction`, and `CustomResourceProvider` constructs, respectively. These new CDK prefixed constructs will offer the same behavior and initialization options as their parent minus the ability to configure the following properties during construction:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This framing around CdkFunction, CdkSingletonFunction etc is confusing me a little.

As far as I can tell, those exact classes are gone (which is great!) so now these are just concepts. That's fine, but as a concept a CdkFunction doesn't really mean anything to me.

Can we not give these names after what you would use them for:

  • CdkFunction -> BuiltinLambdaFunction (because I think these are never used as custom resource providers?)
  • SingletonFunction -> actually confusing to me. Are these usually/always used for custom resource providers? Or no? Are they also built-in Lambda functions that are not actually used as custom resource providers? If true, the naming of this section is confusing, because you're heavily stating it's all about custom resources everywhere, but it might not actually be? Maybe it's just non-Custom Resource Lambdas we provision for you?
  • CdkCustomResourceProvider -> this should make a distinction between the core.CustomResourceProvider and the custom-resources.CustomResourceProvider. Maybe name them something like CoreCustomResource and LongStabilizationCustomResource?

Maybe the flavors of Lambdas we generate here are:

  • RuntimeLambda({ singleton: boolean })
  • CoreCustomResource
  • LongStabilizationCustomResource({ onEvent, isComplete })

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I'm a little worried we're only adding more layers of indirection, and it looks like we're only adding complexity instead of taking it away...)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(If CdkFunction, CdkSingletonFunction etc are still in from the previous commit, then I think we can take them out again, and we can just codegen their implementations)

- `handler` - the name of the method within your code that the provider calls to execute your function
- `code` - the source code of your provider
- `codeDirectory` - a local file system directory with the provider's code
- `runtime` - the runtime environment for the provider that you are uploading

Instead, these properties will be automatically generated using the `ProviderProps` configured in the [config](./config.ts) file:
- `type`: the custom resource provider type to generate
- `sourceCode`: the source code that will be executed by the provider
- `compatibleRuntimes`: runtimes that are compatible with the provider's source code
- `handler`: the name of the method within your code that the provider calls to execute your function

The [config](./config.ts) file is structured with the top most level mapping to a `aws-cdk-lib` module, i.e., `aws-s3`, `aws-dynamodb`, etc. Each service can contain one or more provider modules. Provider modules are containers for custom resource providers and will be rendered as a code generated file. Each provider module can contain one or more `ProviderProps` objects. Each `ProviderProps` object contains all the necessary information required to generate a single custom resource provider. The following example shows a more structural breakdown of how the [config](./config.ts) file is configured:

```ts
const config = {
'aws-s3': { // the aws-cdk-lib module
'replica-provider': [ // the custom resource provider module
// custom resource provider defined as ProviderProps object
{
// the custom resource provider type - `CdkFunction`
type: ComponentType.CDK_FUNCTION,
// the source code that the provider will execute
sourceCode: path.resolve(__dirname, '..', 'aws-dynamodb', 'replica-handler', 'index.ts'),
// runtimes that are compatible with the source code
compatibleRuntimes: [Runtime.NODEJS_18_X],
// the handler in the source code that the provider will execute
handler: 'index.onEventHandler',
},
],
},
'aws-stepfunctions-tasks': {
// contains multiple custom resource provider modules
'eval-nodejs-provider': [
{
// the custom resource provider type - `CdkSingletonFunction`
type: ProviderType.CDK_SINGLETON_FUNCTION,
sourceCode: path.resolve(__dirname, '..', 'aws-stepfunctions-tasks', 'eval-nodejs-handler', 'index.ts'),
compatibleRuntimes: [Runtime.NODEJS_18_X],
},
],
'role-policy-provider': [
{
type: ProviderType.CDK_SINGLETON_FUNCTION,
sourceCode: path.resolve(__dirname, '..', 'aws-stepfunctions-tasks', 'role-policy-handler', 'index.py'),
compatibleRuntimes: [Runtime.PYTHON_3_9],
// prevent minify and bundle flag is set since the source code is a python file
preventMinifyAndBundle: true,
},
],
},
};
```

Code generation for the provider modules is triggered when this package - `@aws-cdk/custom-resource-handlers` - is built. Importantly, this framework is also responsible for minifying and bundling the custom resource providers' source code and dependencies. A flag named `preventMinifyAndBundle` can be configured as part of the `ProviderProps` to prevent minifying and bundling the source code for a specific provider. This flag is `false` by default and is only needed for Python files or for JavaScript/TypeScript files containing require imports.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Swap this around to be the positive: minifyAndBundle: true is the default and it can be set to false.


Once built, all generated code and bundled source code will be written to `@aws-cdk/custom-resource-handlers/dist`. The top level field in the [config](./config.ts) file defining individual `aws-cdk-lib` modules will be used to create specific directories within `@aws-cdk/custom-resource-handlers/dist` and each provider module will be a separate code generated file within these directories named as `<provider-module>.generated.ts`. As an example, the sample [config](./config.ts) file above would create the following file structure:

|--- @aws-cdk
| |--- custom-resource-handlers
| | |--- dist
| | | |--- aws-s3
| | | | |--- replica-handler
| | | | | |--- index.js
| | | | |--- replica-provider.generated.ts
| | | |--- aws-stepfunctions-tasks
| | | | |--- eval-nodejs-handler
| | | | | |--- index.js
| | | | |--- role-policy-handler
| | | | | |--- index.py
| | | | |--- eval-nodejs-provider.generated.ts
| | | | |--- role-policy-provider.generated.ts

The code generated custom resource providers are usable in `aws-cdk-lib` once `aws-cdk-lib` is built. The custom resource providers will be consumable from `aws-cdk-lib/custom-resource-handlers/dist` and the file structure therein will match what was generated in `@aws-cdk/custom-resource-handlers/dist` except for providers defined in `core`. To prevent circular dependencies, all custom resource providers defined in `core`and any associated source code will be consumable from `aws-cdk-lib/core/dist/core`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aws-cdk-lib/core/dist/core as a path is a bit confusing. Can it be something like aws-cdk-lib/core/dist/custom-resource-handlers


## Creating a Custom Resource Provider

Creating a new custom resource provider involves three steps:
1. Add the custom resource provider's source code to `@aws-cdk/custom-resource-handlers/lib/<aws-cdk-lib-module>`
2. Update the [config](./config.ts) file with the custom resource provider to generate by specifying all required `ProviderProps`.
3. At this point you can directly build `@aws-cdk/custom-resource-handlers` with `yarn build` to view the generated provider in `@aws-cdk/custom-resource-handlers/dist`. Alternatively, you can build `aws-cdk-lib` with `npx lerna run build --scope=aws-cdk-lib --skip-nx-cache` to make the generated provider available for use within `aws-cdk-lib`
Loading
Loading