Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions): Distributed Map State #1

Closed
wants to merge 43 commits into from
Closed

Conversation

beck3905
Copy link
Owner

Adds support for Step Functions Map state in Distributed mode. I decided to implement this as a new state type rather than as properties on the existing Map state. I thought that would make for a simpler user experience rather than having a bunch of complicated combinations of configuration properties.

Closes aws#23265.
Closes aws#23216.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

TheRealAmazonKendra and others added 30 commits February 21, 2023 02:03
Lock `@types/conventional-commits-parser`.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…ws#24252)

According to the Lambda docs, there is no space for FunctionUpdate. This commit uses the correct value, as specified here:

https://docs.aws.amazon.com/lambda/latest/dg/API_PutRuntimeManagementConfig.html#API_PutRuntimeManagementConfig_RequestSyntax

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Add support for Apache Kafka versions 3.1.1, 3.2.0 and 3.3.1 in Amazon MSK.

Announcements:
- [3.1.1 and 3.2.0](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-msk-adds-support-apache-kafka-version-3-1-1-3-2-0/) (Posted On: Jun 22, 2022)
- [3.3.1](https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-msk-support-apache-kafka-version-3-3-1) (Posted On: Oct 26, 2022)

Closes aws#23899 
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
When using `Vpc.fromLookup()`, a call may be made to determine whether a VPN Gatway is attached. This call can be undesirable, for example when the lookup role does not have permissions to make this call. This adds a flag to allow opting out of VPN Gateways being queried to alleviate this requirement.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
This is to help vet that the new release line correctly interoperates with `jsii@1.x` before issuing a stable release.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
A recent change in a dependency we use to enforce the order of imports included a small update  to sort import from `./` before `../`.
We previously warned about violations of this rule, but did not enforce it.
This frequently causes files unrelated to a change being updated.

With this change I've updated all violations of this rule and the lint step is now free of warnings.
Most of these updates where automated, by passing `--fix` to the lint command.
For a few I had to manual resolve the problem.
In a small number of cases the import order could not be changed without causing a test failure.
For these I have just disabled the rule per line/block and did not investigate further.

This changes also updates the `import/order` rule to `error` to avoid any future issues.

Existing PRs will be fine, we enforce update from main anyway.
In some circumstances where new imports are added this will cause conflicts and require manual intervention.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
This PR fixes the integ tests errors in aws-eks and addresses the following issues:

`integ.alb-controller.ts` has `runAsNonRoot` error and should define a non-root user in securityContext of the pod definition to ensure successful k8s deployment.

`integ.eks-service-account-sdk-call.ts` should define a non-root user as well and increase the timeout of `BucketPinger` from the default 1 min to 3 min as the deployment could take up to 2 minutes or more. The base docker image from docker hub for this test has been replaced with the same image in ECR public to avoid potential throttling.

I have manually run the two integ tests in my account and both successfully deploy with no error now.

Closes aws#24266

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
MagagedRule up-to-date whenever possible.

fixes aws#21896



----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Construct Runtime Dependencies:

* [ ] This PR adds new construct runtime dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-construct-runtime-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Closes aws#24243.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
aws#24269)

The [Filter policy constraints docs](https://docs.aws.amazon.com/sns/latest/dg/subscription-filter-policy-constraints.html) says:

> For the complexity of the filter policy, the total combination of values must not exceed 150. Calculate the total combination by multiplying the number of values in each array.

This PR sets the complexity limit to the correct value.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…ror (aws#24163)

When the subnet list is passed to the EKS Cluster construct in a different order, an update is triggered to the EKS cluster. 
The update process fails as it falsely identifies a change for an unsupported update, although the list has the same items.

The solution is to change the analyzeUpdate function to return `updateVpc: false` if only the securityGroups/subnetsId order has been changed. 


Fixes: [aws#24162](aws#24162)

---
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
feat(aws-cdk):Adding Support for DefaultCapacityProviderStrategy for L2_cluster
A capacity provider strategy determines whether ECS tasks are launched on EC2 instances or Fargate/Fargate Spot. It can be specified at the cluster, service, or task level, and consists of one or more capacity providers. You can specify an optional base and weight value for finer control of how tasks are launched. The `base` specifies a minimum number of tasks on one capacity provider, and the `weight`s of each capacity provider determine how tasks are distributed after `base` is satisfied.

You can associate a default capacity provider strategy with an Amazon ECS cluster. After you do this, a default capacity provider strategy is used when creating a service or running a standalone task in the cluster and whenever a custom capacity provider strategy or a launch type isn't specified. We recommend that you define a default capacity provider strategy for each cluster.

For more information visit https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html

When the service does not have a capacity provider strategy, the cluster's default capacity provider strategy will be used. Default Capacity Provider Strategy can be added by using the method `addDefaultCapacityProviderStrategy`. A capacity provider strategy cannot contain a mix of EC2 Autoscaling Group capacity providers and Fargate providers.

```ts
declare const capacityProvider: ecs.CapacityProvider;

const cluster = new ecs.Cluster(stack, 'EcsCluster', {
  enableFargateCapacityProviders: true,
});
cluster.addAsgCapacityProvider(capacityProvider);

cluster.addDefaultCapacityProviderStrategy([
  { capacityProvider: 'FARGATE', base: 10, weight: 50 },
  { capacityProvider: 'FARGATE_SPOT', weight: 50 },
]);
```

```ts
declare const capacityProvider: ecs.CapacityProvider;

const cluster = new ecs.Cluster(stack, 'EcsCluster', {
  enableFargateCapacityProviders: true,
});
cluster.addAsgCapacityProvider(capacityProvider);

cluster.addDefaultCapacityProviderStrategy([
  { capacityProvider: capacityProvider.capacityProviderName },
]);
```

Related aws#15230
yarn build && yarn test results
<img width="680" alt="image" src="https://user-images.githubusercontent.com/115483524/216092468-82864aea-b809-48de-9cec-f4b54ec1e541.png">

> Describe the reason for this change, what the solution is, and any
> important design decisions you made. 
>
> Remember to follow the [CONTRIBUTING GUIDE] and [DESIGN GUIDELINES] for any
> code you submit.
>
> [CONTRIBUTING GUIDE]: https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md
> [DESIGN GUIDELINES]: https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Updates cloudwatch tests to use `IntegTest` construct.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
All this tells us is that we've updated this dependency and it causes our automatic workflows to fail on dependency upgrades.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Adds `fromAsset` to the `BuildSpec` class to allow referencing local files as a project's buildspec. Uploads the file to s3 and references the object arn in the `buildSpec` property of `Codebuild.Project`.

`isImmediate` is true for AssetBuildSpec because it's actual meaning is not can be defined at synth time, but it exsists somewhere other  than the project's source code, which in this case is true.

Requires referencing of the project so adds `scope` as an optional parameter to `toBuildSpec` method.

fixes: aws#1138

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Closes aws#24243.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
fix helm deploy login for public ECR repositories

I have tested this issue fixed in `us-east-1` and `us-west-2` integ testing for
```
yarn integ-runner integ.eks-helm-asset.js --force --parallel-regions us-east-1
yarn integ-runner integ.eks-helm-asset.js --force --parallel-regions us-west-2
```

Closes aws#23977.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
just a little cleanup.



----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Keeping the snapshot folders clean

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TheRealAmazonKendra and others added 9 commits February 23, 2023 23:45
…4295)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
aws-dynamodb-global was deprecated before v2 so we can safely delete the package.



----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…etrics (aws#24313)

Closes [aws#20136](aws#20136).

It is intended that all metric identifiers referenced in a MathExpression are included in the usingMetrics map and we will raise warnings if the customer does not follow this contract. 
However for SEARCH and METRICS queries, we can refer directly to metrics attribute values inside the query. Therefore we should not raise warnings.

Change made based on work done in aws@55108b9 with regex extended to a few other expressions. Looks like integ requests will not be required based on that commit.

I had some firsthand experience getting thousands of this warning message after upgrading to CDK 2 and decided it would be easier to fix than suppress the excessive warnings.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
**Description**

This PR Updates for the fact table to include Layer ARNs published during the February release of the ADOT Lambda Layers. 

> [CONTRIBUTING GUIDE]: https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md
> [DESIGN GUIDELINES]: https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@beck3905 beck3905 self-assigned this Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet