diff --git a/CHANGELOG.md b/CHANGELOG.md index e8bd27c2..daa365d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## UNRELEASED +### **Added** +- added `mlflow-ai-gw-image` module + +### **Changed** + + +## v1.7.0 + ### **Added** - added GitHub as code repository option along with AWS CodeCommit for sagemaker templates batch_inference, finetune_llm_evaluation, hf_import_models and xgboost_abalone - added `ray-orchestrator` module - added GitHub as alternate option for code repository support along with AWS CodeCommit for sagemaker-templates-service-catalog module +- added SageMaker ground truth labeling module ### **Changed** - updated manifests to idf release 1.12.0 diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 354dbc1b..c69fb0b8 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -5,7 +5,7 @@ 1. Clone the repository and checkout a release branch using the below command: ``` -git clone --origin upstream --branch release/1.6.0 https://github.com/awslabs/aiops-modules +git clone --origin upstream --branch release/1.7.0 https://github.com/awslabs/aiops-modules ``` The release version can be replaced with the version of interest. diff --git a/README.md b/README.md index 04dba393..c1597272 100644 --- a/README.md +++ b/README.md @@ -50,13 +50,15 @@ End-to-end example use-cases built using modules in this repository. | [SageMaker Model Package Promote Pipeline Module](modules/sagemaker/sagemaker-model-package-promote-pipeline/README.md) | Deploy a Pipeline to promote SageMaker Model Packages in a multi-account setup. The pipeline can be triggered through an EventBridge rule in reaction of a SageMaker Model Package Group state event change (Approved/Rejected). Once the pipeline is triggered, it will promote the latest approved model package, if one is found. | | [SageMaker Model Monitoring Module](modules/sagemaker/sagemaker-model-monitoring/README.md) | Deploy data quality, model quality, model bias, and model explainability monitoring jobs which run against a SageMaker Endpoint. | | [SageMaker Model CICD Module](modules/sagemaker/sagemaker-model-cicd/README.md) | Creates a comprehensive CICD pipeline using AWS CodePipelines to build and deploy a ML model on SageMaker. | +| [SageMaker Ground Truth Labeling Module](modules/sagemaker/sagemaker-ground-truth-labeling/README.md) | Creates a state machine to allow labeling of images and text file, uploaded to the upload bucket, using various built-in task types in SageMaker Ground Truth. | ### Mlflow Modules -| Type | Description | -|-------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Mlflow Image Module](modules/mlflow/mlflow-image/README.md) | Creates Mlflow Docker container image and pushes the image to Elastic Container Registry | -| [Mlflow on AWS Fargate Module](modules/mlflow/mlflow-fargate/README.md) | Runs Mlflow container on AWS Fargate in a load-balanced Elastic Container Service. Supports Elastic File System and Relational Database Store for metadata persistence, and S3 for artifact store | +| Type | Description | +|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Mlflow Image Module](modules/mlflow/mlflow-image/README.md) | Creates Mlflow Tracing Server Docker image and pushes the image to Elastic Container Registry | +| [Mlflow on AWS Fargate Module](modules/mlflow/mlflow-fargate/README.md) | Runs Mlflow container on AWS Fargate in a load-balanced Elastic Container Service. Supports Elastic File System and Relational Database Store for metadata persistence, and S3 for artifact store | +| [Mlflow AI Gateway Image Module](modules/mlflow/mlflow-ai-gw-image/README.md) | Creates Mlflow AI Gateway Docker image and pushes the image to Elastic Container Registry | ### FMOps/LLMOps Modules diff --git a/VERSION b/VERSION index ce6a70b9..9dbb0c00 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.6.0 \ No newline at end of file +1.7.0 \ No newline at end of file diff --git a/examples/manifests/event-bus-modules.yaml b/examples/manifests/event-bus-modules.yaml index 1d43f846..8b53ec28 100644 --- a/examples/manifests/event-bus-modules.yaml +++ b/examples/manifests/event-bus-modules.yaml @@ -1,5 +1,5 @@ name: event-bus -path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/event-bus?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/event-bus?ref=release/1.7.0&depth=1 targetAccount: tooling parameters: - name: event_bus_name diff --git a/examples/manifests/fmops-modules.yaml b/examples/manifests/fmops-modules.yaml index d8e712ee..4ec5456d 100644 --- a/examples/manifests/fmops-modules.yaml +++ b/examples/manifests/fmops-modules.yaml @@ -1,5 +1,5 @@ name: jumpstart-hf-asrwhisper-endpoint -path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-jumpstart-fm-endpoint?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-jumpstart-fm-endpoint?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: jump-start-model-name diff --git a/examples/manifests/personas-modules.yaml b/examples/manifests/personas-modules.yaml index b0b4657d..7d083722 100644 --- a/examples/manifests/personas-modules.yaml +++ b/examples/manifests/personas-modules.yaml @@ -1,5 +1,5 @@ name: personas -path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/personas?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/personas?ref=release/1.7.0&depth=1 parameters: - name: bucket-name value: my-bucket diff --git a/examples/manifests/sagemaker-endpoints-modules.yaml b/examples/manifests/sagemaker-endpoints-modules.yaml index 36947a21..ea2dbdcd 100644 --- a/examples/manifests/sagemaker-endpoints-modules.yaml +++ b/examples/manifests/sagemaker-endpoints-modules.yaml @@ -1,7 +1,7 @@ # This is an example manifest group. # Replace the parameters with the parameters for your model below prior the deployment. name: endpoint -path: git::https://github.com/awslabs/aiops-modules.git//sagemaker/fmops/sagemaker-endpoint?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//sagemaker/fmops/sagemaker-endpoint?ref=release/1.7.0&depth=1 parameters: - name: sagemaker_project_id value: project-1 diff --git a/examples/manifests/sagemaker-hugging-face.yml b/examples/manifests/sagemaker-hugging-face.yml index 4dd0486e..1abf2d50 100644 --- a/examples/manifests/sagemaker-hugging-face.yml +++ b/examples/manifests/sagemaker-hugging-face.yml @@ -1,5 +1,5 @@ name: hugging-face-mistral-endpoint -path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-hugging-face-endpoint?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-hugging-face-endpoint?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: hugging-face-model-id diff --git a/examples/manifests/sagemaker-model-monitoring-modules.yaml b/examples/manifests/sagemaker-model-monitoring-modules.yaml index db30551f..83d09c8f 100644 --- a/examples/manifests/sagemaker-model-monitoring-modules.yaml +++ b/examples/manifests/sagemaker-model-monitoring-modules.yaml @@ -1,7 +1,7 @@ # This is an example manifest group. # Replace the parameters with the parameters for your model below prior the deployment. name: monitor -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-monitoring?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-monitoring?ref=release/1.7.0&depth=1 parameters: - name: sagemaker_project_id value: project-1 diff --git a/examples/manifests/sagemaker-model-package-group-modules.yaml b/examples/manifests/sagemaker-model-package-group-modules.yaml index b04de8f1..ee9fbd59 100644 --- a/examples/manifests/sagemaker-model-package-group-modules.yaml +++ b/examples/manifests/sagemaker-model-package-group-modules.yaml @@ -1,5 +1,5 @@ name: source-model-package-group -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-group?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-group?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: model_package_group_name diff --git a/examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml b/examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml index 382040f5..1fa3ea52 100644 --- a/examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml +++ b/examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml @@ -1,5 +1,5 @@ name: model-pipeline -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-promote-pipeline?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-promote-pipeline?ref=release/1.7.0&depth=1 targetAccount: tooling parameters: - name: source_model_package_group_arn diff --git a/examples/manifests/sagemaker-notebook-modules.yaml b/examples/manifests/sagemaker-notebook-modules.yaml index 94b6fb36..af841d96 100644 --- a/examples/manifests/sagemaker-notebook-modules.yaml +++ b/examples/manifests/sagemaker-notebook-modules.yaml @@ -1,5 +1,5 @@ name: notebook -path: git::https://github.com/awslabs/aiops-modules.git//modules/modules/sagemaker/sagemaker-notebook?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/modules/sagemaker/sagemaker-notebook?ref=release/1.7.0&depth=1 parameters: - name: notebook_name value: dummy diff --git a/examples/manifests/sagemaker-templates-modules-codecommit.yaml b/examples/manifests/sagemaker-templates-modules-codecommit.yaml index 94f14c31..6d320952 100644 --- a/examples/manifests/sagemaker-templates-modules-codecommit.yaml +++ b/examples/manifests/sagemaker-templates-modules-codecommit.yaml @@ -1,5 +1,5 @@ name: service-catalog -path: modules/sagemaker/sagemaker-templates-service-catalog +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: repository-type diff --git a/examples/manifests/sagemaker-templates-modules-github.yaml b/examples/manifests/sagemaker-templates-modules-github.yaml index f53199d5..3e41a5fe 100644 --- a/examples/manifests/sagemaker-templates-modules-github.yaml +++ b/examples/manifests/sagemaker-templates-modules-github.yaml @@ -1,5 +1,5 @@ name: service-catalog -path: modules/sagemaker/sagemaker-templates-service-catalog +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: repository-type diff --git a/examples/sagemaker-ground-truth-labeling/README.md b/examples/sagemaker-ground-truth-labeling/README.md new file mode 100644 index 00000000..6d6371d4 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/README.md @@ -0,0 +1,40 @@ +# SageMaker Ground truth labeling examples + +### Description + +This folder contains examples for each of the built-in task types for the sagemaker ground truth module. Each folder contains an example manifest as well as any necessary templates. Please upload the templates to an S3 bucket and update the manifest with the correct location. + +### Additional workers + +For tasks without a verification step (all except `image_bounding_box` and `image_semantic_segmentation`) we recommend increasing the number of human reviewers per object to increase accuracy. This will only work if you have at least that many reviewers in your workteam, as the same reviewer cannot review the same item twice. To adjust the number of workers add the additional parameters below to your manifest: + +```yaml + - name: labeling-human-task-config + value: + NumberOfHumanWorkersPerDataObject: 5 + TaskAvailabilityLifetimeInSeconds: 21600 + TaskTimeLimitInSeconds: 300 +``` + +### Using public workforce + +As mentioned in the README you can use a public workforce for your task if you wish (at an additional cost). More information on using a public workforce like Amazon Mechanical Turk is available [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-public.html). Labeling and verification task prices is specified in USD, see [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_PublicWorkforceTaskPrice.html) for allowed values. [This page](https://aws.amazon.com/sagemaker/groundtruth/pricing/) provides suggested pricing based on task type. To use a public workforce add / adjust the following parameters to your manifest: + +```yaml + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker::394669845002:workteam/public-crowd/default' + - name: labeling-task-price + value: + AmountInUsd: + Dollars: 0 + Cents: 3 + TenthFractionsOfACent: 6 + - name: verification-workteam-arn + value: 'arn:aws:sagemaker::394669845002:workteam/public-crowd/default' + - name: verification-task-price + value: + AmountInUsd: + Dollars: 0 + Cents: 3 + TenthFractionsOfACent: 6 +``` diff --git a/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_categories.json b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_categories.json new file mode 100644 index 00000000..87ea07f8 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Plane"}, {"label": "Boat"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_template.html b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_template.html new file mode 100644 index 00000000..2a3e890a --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_template.html @@ -0,0 +1,28 @@ + + + + +
    +
  1. Inspect the image
  2. +
  3. Determine if the specified label is/are visible in the picture.
  4. +
  5. Outline each instance of the specified label in the image using the provided “Box” tool.
  6. +
+
    +
  • Boxes should fit tight around each object
  • +
  • Do not include parts of the object are overlapping or that cannot be seen, even though you think you can interpolate the whole shape.
  • +
  • Avoid including shadows.
  • +
  • If the target is off screen, draw the box up to the edge of the image.
  • +
+
+ + + Outline each instance of the specified label in the image using the provided “Box” tool. + + +
+
diff --git a/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_categories.json b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_categories.json new file mode 100644 index 00000000..dd879e0d --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_categories.json @@ -0,0 +1 @@ +{"labels":[{"label":"Label(s) correct"},{"label":"Incorrect label - missed object"},{"label":"Incorrect label - bounding box not accurate enough"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_template.liquid b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_template.liquid new file mode 100644 index 00000000..3d529af9 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_template.liquid @@ -0,0 +1,39 @@ + + + + +
    +
  1. Read the task carefully and inspect the image.
  2. +
  3. Read the options and review the examples provided to understand more about the labels.
  4. +
  5. Choose the appropriate label that best suits the image.
  6. +
+
+ + Choose the appropriate label that best suits the image. + + +
+
diff --git a/examples/sagemaker-ground-truth-labeling/image_bounding_box/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/image_bounding_box/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..d0b274d0 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_bounding_box/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,34 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'plane-and-boat-bounding-box' + - name: task_type + value: 'image_bounding_box' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///image_bounding_box_labeling_template.html' + - name: labeling-categories-s3-uri + value: 's3:///image_bounding_box_labeling_categories.json' + - name: labeling-task-title + value: 'Labeling - Bounding boxes: Draw bounding boxes around all planes and boats in the image' + - name: labeling-task-description + value: 'Draw bounding boxes around all planes and boats in the image' + - name: labeling-task-keywords + value: ['image', 'object', 'detection'] + + - name: verification-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: verification-instructions-template-s3-uri + value: 's3:///image_bounding_box_verification_template.liquid' + - name: verification-categories-s3-uri + value: 's3:///image_bounding_box_verification_categories.json' + - name: verification-task-title + value: 'Label verification - Bounding boxes: Review the existing labels on the objects and choose the appropriate option.' + - name: verification-task-description + value: 'Verify that all of the planes and boats in the image are correctly labeled' + - name: verification-task-keywords + value: ['image', 'object', 'detection', 'label verification', 'bounding boxes'] diff --git a/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_categories.json b/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_categories.json new file mode 100644 index 00000000..87ea07f8 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Plane"}, {"label": "Boat"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_template.html b/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_template.html new file mode 100644 index 00000000..32c4cfa5 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_template.html @@ -0,0 +1,21 @@ + + + + +

If more than one label applies to the image, select multiple labels.

+

If no labels apply, select None of the above

+
+ + +

Read the task carefully and inspect the image.

+

Choose the appropriate label(s) that best suit the image.

+ +
+
+
\ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..27f2b442 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_multi_label_classification/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,21 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'vehicle-classification' + - name: task_type + value: 'image_multi_label_classification' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///image_multi_label_labeling_template.html' + - name: labeling-categories-s3-uri + value: 's3:///image_multi_label_labeling_categories.json' + - name: labeling-task-title + value: 'Labeling - Multi-Classification: Classify all images as containing a plane and/or a boat' + - name: labeling-task-description + value: 'Classify all images as containing a plane and/or a boat, selecting all of the appropriate labels' + - name: labeling-task-keywords + value: ['image', 'object', 'multi classification'] diff --git a/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_labeling_categories.json b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_labeling_categories.json new file mode 100644 index 00000000..87ea07f8 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_labeling_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Plane"}, {"label": "Boat"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_labeling_template.html b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_labeling_template.html new file mode 100644 index 00000000..fb8ed208 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_labeling_template.html @@ -0,0 +1,22 @@ + + + + +
    +
  1. Read the task carefully and inspect the image.
  2. +
  3. Read the options and review the examples provided to understand more about the labels.
  4. +
  5. Choose the appropriate label that best suits the image.
  6. +
+
+ + + Use the tools to label the requested items in the image + + +
+
diff --git a/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_verification_categories.json b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_verification_categories.json new file mode 100644 index 00000000..eb09eb86 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_verification_categories.json @@ -0,0 +1 @@ +{"labels":[{"label":"Label(s) correct"},{"label":"Incorrect label - missed object"},{"label":"Incorrect label - segmentation not accurate enough"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_verification_template.liquid b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_verification_template.liquid new file mode 100644 index 00000000..8dff9576 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/image_semantic_segmentation_verification_template.liquid @@ -0,0 +1,44 @@ + + + + +
    +
  1. Read the task carefully and inspect the image.
  2. +
  3. Read the options and review the examples provided to understand more about the labels.
  4. +
  5. Choose the appropriate label that best suits the image.
  6. +
+
+ + Choose the appropriate label that best suits the image. + + +
+
\ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..6db32a69 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_semantic_segmentation/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,34 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'plane-and-boat-sem-seg' + - name: task_type + value: 'image_semantic_segmentation' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///image_semantic_segmentation_labeling_template.html' + - name: labeling-categories-s3-uri + value: 's3:///image_semantic_segmentation_labeling_categories.json' + - name: labeling-task-title + value: 'Labeling - Semantic segmentation: Fill all planes and boats in the image' + - name: labeling-task-description + value: 'Fill all planes and boats in the image using the appropriate label' + - name: labeling-task-keywords + value: ['image', 'object', 'detection'] + + - name: verification-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: verification-instructions-template-s3-uri + value: 's3:///image_semantic_segmentation_verification_template.liquid' + - name: verification-categories-s3-uri + value: 's3:///image_semantic_segmentation_verification_categories.json' + - name: verification-task-title + value: 'Label verification - Semantic segmentation: Review the existing labels on the objects and choose the appropriate option.' + - name: verification-task-description + value: 'Verify that all of the planes and boats in the image are correctly labeled' + - name: verification-task-keywords + value: ['image', 'object', 'detection', 'label verification', 'semantic segmentation'] diff --git a/examples/sagemaker-ground-truth-labeling/image_single_label_classification/image_single_label_labeling_categories.json b/examples/sagemaker-ground-truth-labeling/image_single_label_classification/image_single_label_labeling_categories.json new file mode 100644 index 00000000..6f17cf3e --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_single_label_classification/image_single_label_labeling_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Plane"}, {"label": "Boat"}, {"label": "Neither"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_single_label_classification/image_single_label_labeling_template.html b/examples/sagemaker-ground-truth-labeling/image_single_label_classification/image_single_label_labeling_template.html new file mode 100644 index 00000000..fd8e707c --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_single_label_classification/image_single_label_labeling_template.html @@ -0,0 +1,19 @@ + + + + +

Read the task carefully and inspect the image.

+

Choose the appropriate label that best suits the image.

+
+ + + Choose the appropriate label that best suits the image. + + +
+
\ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/image_single_label_classification/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/image_single_label_classification/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..039a5d65 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/image_single_label_classification/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,21 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'vehicle-classification' + - name: task_type + value: 'image_single_label_classification' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///image_single_label_labeling_template.html' + - name: labeling-categories-s3-uri + value: 's3:///image_single_label_labeling_categories.json' + - name: labeling-task-title + value: 'Labeling - Classification: Classify all images as containing a plane or a boat' + - name: labeling-task-description + value: 'Classify all images as containing a plane or a boat using the appropriate label' + - name: labeling-task-keywords + value: ['image', 'object', 'classification'] diff --git a/examples/sagemaker-ground-truth-labeling/named_entity_recognition/named_entity_recognition_categories.json b/examples/sagemaker-ground-truth-labeling/named_entity_recognition/named_entity_recognition_categories.json new file mode 100644 index 00000000..b03d4dc5 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/named_entity_recognition/named_entity_recognition_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Person"}, {"label": "Organisation"}, {"label": "Time"}, {"label": "Location"}, {"label": "Capital City"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/named_entity_recognition/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/named_entity_recognition/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..eeca491d --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/named_entity_recognition/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,19 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'named-entity-recognition' + - name: task_type + value: 'named_entity_recognition' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-categories-s3-uri + value: 's3:///named_entity_recognition_categories.json' + - name: labeling-task-title + value: 'Labeling - Named entity recognition: Indentify all entities in the text' + - name: labeling-task-description + value: 'Evaluate all texts and select the identify the entities from the provided values' + - name: labeling-task-keywords + value: ['text', 'named entity recognition', 'detection'] diff --git a/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..96833e99 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,21 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'text-multi-classification' + - name: task_type + value: 'text_multi_label_classification' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///text_multi_label_labeling_template.html' + - name: labeling-categories-s3-uri + value: 's3:///text_multi_label_labeling_categories.json' + - name: labeling-task-title + value: 'Labeling - Multi-Classification: Classify all texts using the labels' + - name: labeling-task-description + value: 'Classify all texts selecting all of the appropriate labels' + - name: labeling-task-keywords + value: ['text', 'multi classification', 'sentiment'] diff --git a/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/text_multi_label_labeling_categories.json b/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/text_multi_label_labeling_categories.json new file mode 100644 index 00000000..413500f9 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/text_multi_label_labeling_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Positive"}, {"label": "Negative"}, {"label": "Neutral"}, {"label": "Review"}, {"label": "Message"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/text_multi_label_labeling_template.html b/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/text_multi_label_labeling_template.html new file mode 100644 index 00000000..fcd71e7c --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/text_multi_label_classification/text_multi_label_labeling_template.html @@ -0,0 +1,25 @@ + + + + + {{ task.input.taskObject }} + + + +

Positive sentiment include: joy, excitement, delight

+

Negative sentiment include: anger, sarcasm, anxiety

+

Neutral: neither positive or negative, such as stating a fact

+

When the sentiment is mixed, such as both joy and sadness, use your judgment to choose the stronger emotion.

+
+ + + Choose all categories that are expressed by the text. + + +
+
\ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/text_single_label_classification/sagemaker-ground-truth-labeling.yaml b/examples/sagemaker-ground-truth-labeling/text_single_label_classification/sagemaker-ground-truth-labeling.yaml new file mode 100644 index 00000000..3e6a45c8 --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/text_single_label_classification/sagemaker-ground-truth-labeling.yaml @@ -0,0 +1,21 @@ +name: ground-truth-labeling +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1 +targetAccount: primary +parameters: + - name: job_name + value: 'sentiment-classification' + - name: task_type + value: 'text_single_label_classification' + + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///text_single_label_labeling_template.html' + - name: labeling-categories-s3-uri + value: 's3:///text_single_label_labeling_categories.json' + - name: labeling-task-title + value: 'Labeling - Classification: Classify all texts as either positive, neutral or negative' + - name: labeling-task-description + value: 'Classify all texts as either positive, neutral or negative' + - name: labeling-task-keywords + value: ['text', 'classification', 'sentiment'] diff --git a/examples/sagemaker-ground-truth-labeling/text_single_label_classification/text_single_label_labeling_categories.json b/examples/sagemaker-ground-truth-labeling/text_single_label_classification/text_single_label_labeling_categories.json new file mode 100644 index 00000000..4b91087f --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/text_single_label_classification/text_single_label_labeling_categories.json @@ -0,0 +1 @@ +{"labels": [{"label": "Positive"}, {"label": "Negative"}, {"label": "Neutral"}]} \ No newline at end of file diff --git a/examples/sagemaker-ground-truth-labeling/text_single_label_classification/text_single_label_labeling_template.html b/examples/sagemaker-ground-truth-labeling/text_single_label_classification/text_single_label_labeling_template.html new file mode 100644 index 00000000..dbc5c22c --- /dev/null +++ b/examples/sagemaker-ground-truth-labeling/text_single_label_classification/text_single_label_labeling_template.html @@ -0,0 +1,24 @@ + + + + + {{ task.input.taskObject }} + + + +

Positive sentiment include: joy, excitement, delight

+

Negative sentiment include: anger, sarcasm, anxiety

+

Neutral: neither positive or negative, such as stating a fact

+

When the sentiment is mixed, such as both joy and sadness, use your judgment to choose the stronger emotion.

+
+ + + Choose the most relevant category that is expressed by the text. + + +
+
\ No newline at end of file diff --git a/manifests/bedrock-finetuning-sfn/bedrock-finetuning-modules.yaml b/manifests/bedrock-finetuning-sfn/bedrock-finetuning-modules.yaml index c266a41e..8b220553 100644 --- a/manifests/bedrock-finetuning-sfn/bedrock-finetuning-modules.yaml +++ b/manifests/bedrock-finetuning-sfn/bedrock-finetuning-modules.yaml @@ -1,5 +1,5 @@ name: bedrock-finetuning -path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/bedrock-finetuning?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/bedrock-finetuning?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: bedrock-base-model-ID diff --git a/manifests/fine-tuning-6b/core-modules.yaml b/manifests/fine-tuning-6b/core-modules.yaml index 6791838f..9bd91c3e 100644 --- a/manifests/fine-tuning-6b/core-modules.yaml +++ b/manifests/fine-tuning-6b/core-modules.yaml @@ -45,7 +45,7 @@ parameters: usage: core - eks_ng_name: ng-gpu eks_node_quantity: 6 - eks_node_max_quantity: 15 + eks_node_max_quantity: 10 eks_node_min_quantity: 6 eks_node_disk_size: 400 eks_node_instance_type: "g4dn.4xlarge" diff --git a/manifests/fine-tuning-6b/images-modules.yaml b/manifests/fine-tuning-6b/images-modules.yaml index 4edd51cb..77b559a1 100644 --- a/manifests/fine-tuning-6b/images-modules.yaml +++ b/manifests/fine-tuning-6b/images-modules.yaml @@ -1,5 +1,5 @@ name: ray -path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-image?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-image?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: EcrRepoName diff --git a/manifests/fine-tuning-6b/ray-cluster-modules.yaml b/manifests/fine-tuning-6b/ray-cluster-modules.yaml index 45757324..bbc7e363 100644 --- a/manifests/fine-tuning-6b/ray-cluster-modules.yaml +++ b/manifests/fine-tuning-6b/ray-cluster-modules.yaml @@ -1,5 +1,5 @@ name: ray-cluster -path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-cluster?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-cluster?ref=release/1.7.0&depth=1 parameters: - name: EksClusterAdminRoleArn valueFrom: diff --git a/manifests/fine-tuning-6b/ray-operator-modules.yaml b/manifests/fine-tuning-6b/ray-operator-modules.yaml index 38ee34f0..6f033043 100644 --- a/manifests/fine-tuning-6b/ray-operator-modules.yaml +++ b/manifests/fine-tuning-6b/ray-operator-modules.yaml @@ -1,5 +1,5 @@ name: ray-operator -path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-operator?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-operator?ref=release/1.7.0&depth=1 parameters: - name: EksClusterAdminRoleArn valueFrom: diff --git a/manifests/fine-tuning-6b/ray-orchestrator-modules.yaml b/manifests/fine-tuning-6b/ray-orchestrator-modules.yaml index aa75b5f2..a55a8d42 100644 --- a/manifests/fine-tuning-6b/ray-orchestrator-modules.yaml +++ b/manifests/fine-tuning-6b/ray-orchestrator-modules.yaml @@ -1,5 +1,5 @@ name: ray-orchestrator -path: modules/eks/ray-orchestrator +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-orchestrator?ref=release/1.7.0&depth=1 parameters: - name: Namespace valueFrom: diff --git a/manifests/fmops-qna-rag/qna-rag-modules.yaml b/manifests/fmops-qna-rag/qna-rag-modules.yaml index 0b98db47..ce0c6dc6 100644 --- a/manifests/fmops-qna-rag/qna-rag-modules.yaml +++ b/manifests/fmops-qna-rag/qna-rag-modules.yaml @@ -1,5 +1,5 @@ name: qna-rag -path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/qna-rag?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/qna-rag?ref=release/1.7.0&depth=1 parameters: - name: cognito-pool-id # Replace below value with valid congnito pool id diff --git a/manifests/mlflow-tracking/deployment.yaml b/manifests/mlflow-tracking/deployment.yaml index af3310e3..ed5929f7 100644 --- a/manifests/mlflow-tracking/deployment.yaml +++ b/manifests/mlflow-tracking/deployment.yaml @@ -12,6 +12,8 @@ groups: path: manifests/mlflow-tracking/images-modules.yaml - name: mlflow path: manifests/mlflow-tracking/mlflow-modules.yaml + - name: mlflow-ai-gw + path: manifests/mlflow-tracking/images-ai-gw-modules.yaml targetAccountMappings: - alias: primary accountId: diff --git a/manifests/mlflow-tracking/images-ai-gw-modules.yaml b/manifests/mlflow-tracking/images-ai-gw-modules.yaml new file mode 100644 index 00000000..1dfd0491 --- /dev/null +++ b/manifests/mlflow-tracking/images-ai-gw-modules.yaml @@ -0,0 +1,10 @@ +name: mlflow-ai-gw-image +path: modules/mlflow/mlflow-ai-gw-image/ +targetAccount: primary +parameters: + - name: ecr-repository-name + valueFrom: + moduleMetadata: + group: storage + name: ecr-mlflow-ai-gw + key: EcrRepositoryName diff --git a/manifests/mlflow-tracking/images-modules.yaml b/manifests/mlflow-tracking/images-modules.yaml index d062505f..d463d74d 100644 --- a/manifests/mlflow-tracking/images-modules.yaml +++ b/manifests/mlflow-tracking/images-modules.yaml @@ -1,5 +1,5 @@ name: mlflow-image -path: git::https://github.com/awslabs/aiops-modules.git//modules/mlflow/mlflow-image?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/mlflow/mlflow-image?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: ecr-repository-name diff --git a/manifests/mlflow-tracking/mlflow-modules.yaml b/manifests/mlflow-tracking/mlflow-modules.yaml index 3b67218a..8d0979e1 100644 --- a/manifests/mlflow-tracking/mlflow-modules.yaml +++ b/manifests/mlflow-tracking/mlflow-modules.yaml @@ -1,5 +1,5 @@ name: mlflow-fargate -path: git::https://github.com/awslabs/aiops-modules.git//modules/mlflow/mlflow-fargate?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/mlflow/mlflow-fargate?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: vpc-id diff --git a/manifests/mlflow-tracking/sagemaker-studio-modules.yaml b/manifests/mlflow-tracking/sagemaker-studio-modules.yaml index 49243063..93be1e4d 100644 --- a/manifests/mlflow-tracking/sagemaker-studio-modules.yaml +++ b/manifests/mlflow-tracking/sagemaker-studio-modules.yaml @@ -1,5 +1,5 @@ name: studio -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: vpc_id diff --git a/manifests/mlflow-tracking/storage-modules.yaml b/manifests/mlflow-tracking/storage-modules.yaml index 921e71d2..d94ec95b 100644 --- a/manifests/mlflow-tracking/storage-modules.yaml +++ b/manifests/mlflow-tracking/storage-modules.yaml @@ -11,6 +11,19 @@ parameters: - name: removal-policy value: DESTROY --- +name: ecr-mlflow-ai-gw +path: git::https://github.com/awslabs/idf-modules.git//modules/storage/ecr?ref=release/1.12.0&depth=1 +targetAccount: primary +parameters: + - name: image-tag-mutability + value: MUTABLE + - name: image-scan-on-push + value: True + - name: encryption + value: KMS_MANAGED + - name: removal-policy + value: DESTROY +--- name: buckets path: git::https://github.com/awslabs/idf-modules.git//modules/storage/buckets?ref=release/1.12.0&depth=1 targetAccount: primary diff --git a/manifests/mlops-sagemaker-multiacc/kernels-modules.yaml b/manifests/mlops-sagemaker-multiacc/kernels-modules.yaml index b0a47092..f7f6707e 100644 --- a/manifests/mlops-sagemaker-multiacc/kernels-modules.yaml +++ b/manifests/mlops-sagemaker-multiacc/kernels-modules.yaml @@ -1,5 +1,5 @@ name: sagemaker-custom-kernel -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-custom-kernel?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-custom-kernel?ref=release/1.7.0&depth=1 targetAccount: dev parameters: - name: ecr-repo-name diff --git a/manifests/mlops-sagemaker-multiacc/sagemaker-model-cicd-modules.yaml b/manifests/mlops-sagemaker-multiacc/sagemaker-model-cicd-modules.yaml index 0c453a04..ad9e6426 100644 --- a/manifests/mlops-sagemaker-multiacc/sagemaker-model-cicd-modules.yaml +++ b/manifests/mlops-sagemaker-multiacc/sagemaker-model-cicd-modules.yaml @@ -1,5 +1,5 @@ name: model # replace with name of the ML model you prefer -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-cicd?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-cicd?ref=release/1.7.0&depth=1 targetAccount: tooling parameters: - name: infra-repo diff --git a/manifests/mlops-sagemaker-multiacc/sagemaker-studio-modules.yaml b/manifests/mlops-sagemaker-multiacc/sagemaker-studio-modules.yaml index 45b9823c..b37b494b 100644 --- a/manifests/mlops-sagemaker-multiacc/sagemaker-studio-modules.yaml +++ b/manifests/mlops-sagemaker-multiacc/sagemaker-studio-modules.yaml @@ -1,5 +1,5 @@ name: studio -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.7.0&depth=1 targetAccount: dev parameters: - name: vpc_id diff --git a/manifests/mlops-sagemaker-multiacc/sagemaker-templates-modules.yaml b/manifests/mlops-sagemaker-multiacc/sagemaker-templates-modules.yaml index f29fce55..64b85e3e 100644 --- a/manifests/mlops-sagemaker-multiacc/sagemaker-templates-modules.yaml +++ b/manifests/mlops-sagemaker-multiacc/sagemaker-templates-modules.yaml @@ -1,5 +1,5 @@ name: service-catalog -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.7.0&depth=1 targetAccount: dev parameters: - name: portfolio-access-role-arn diff --git a/manifests/mlops-sagemaker/kernels-modules.yaml b/manifests/mlops-sagemaker/kernels-modules.yaml index 6b2ff5b6..79e27409 100644 --- a/manifests/mlops-sagemaker/kernels-modules.yaml +++ b/manifests/mlops-sagemaker/kernels-modules.yaml @@ -1,5 +1,5 @@ name: sagemaker-custom-kernel -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-custom-kernel?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-custom-kernel?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: ecr-repo-name diff --git a/manifests/mlops-sagemaker/sagemaker-studio-modules.yaml b/manifests/mlops-sagemaker/sagemaker-studio-modules.yaml index 49243063..93be1e4d 100644 --- a/manifests/mlops-sagemaker/sagemaker-studio-modules.yaml +++ b/manifests/mlops-sagemaker/sagemaker-studio-modules.yaml @@ -1,5 +1,5 @@ name: studio -path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: vpc_id diff --git a/manifests/mlops-sagemaker/sagemaker-templates-modules.yaml b/manifests/mlops-sagemaker/sagemaker-templates-modules.yaml index bf78d64b..bdd46914 100644 --- a/manifests/mlops-sagemaker/sagemaker-templates-modules.yaml +++ b/manifests/mlops-sagemaker/sagemaker-templates-modules.yaml @@ -1,6 +1,5 @@ name: service-catalog - -path: modules/sagemaker/sagemaker-templates-service-catalog +path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: portfolio-access-role-arn diff --git a/manifests/mlops-stepfunctions/mlops-stepfunctions.yaml b/manifests/mlops-stepfunctions/mlops-stepfunctions.yaml index a4a4630a..b7b467f5 100644 --- a/manifests/mlops-stepfunctions/mlops-stepfunctions.yaml +++ b/manifests/mlops-stepfunctions/mlops-stepfunctions.yaml @@ -1,5 +1,5 @@ name: stepfunctions -path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/mlops-stepfunctions?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/mlops-stepfunctions?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: model-name diff --git a/manifests/mwaa-ml-training/mwaa-dag-modules.yaml b/manifests/mwaa-ml-training/mwaa-dag-modules.yaml index 89806698..14f00272 100644 --- a/manifests/mwaa-ml-training/mwaa-dag-modules.yaml +++ b/manifests/mwaa-ml-training/mwaa-dag-modules.yaml @@ -1,5 +1,5 @@ name: dags -path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/airflow-dags?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/airflow-dags?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: dag-bucket-name diff --git a/manifests/ray-on-eks/core-modules.yaml b/manifests/ray-on-eks/core-modules.yaml index c1354283..270a0897 100644 --- a/manifests/ray-on-eks/core-modules.yaml +++ b/manifests/ray-on-eks/core-modules.yaml @@ -36,10 +36,10 @@ parameters: value: eks_nodegroup_config: - eks_ng_name: ng1 - eks_node_quantity: 2 - eks_node_max_quantity: 5 + eks_node_quantity: 1 + eks_node_max_quantity: 1 eks_node_min_quantity: 1 - eks_node_disk_size: 200 + eks_node_disk_size: 400 eks_node_instance_type: "m5.12xlarge" eks_node_labels: usage: core @@ -47,7 +47,7 @@ parameters: eks_node_quantity: 1 eks_node_max_quantity: 10 eks_node_min_quantity: 1 - eks_node_disk_size: 200 + eks_node_disk_size: 400 eks_node_instance_type: "g4dn.4xlarge" eks_node_labels: usage: gpu diff --git a/manifests/ray-on-eks/images-modules.yaml b/manifests/ray-on-eks/images-modules.yaml index 4edd51cb..77b559a1 100644 --- a/manifests/ray-on-eks/images-modules.yaml +++ b/manifests/ray-on-eks/images-modules.yaml @@ -1,5 +1,5 @@ name: ray -path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-image?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-image?ref=release/1.7.0&depth=1 targetAccount: primary parameters: - name: EcrRepoName diff --git a/manifests/ray-on-eks/ray-cluster-modules.yaml b/manifests/ray-on-eks/ray-cluster-modules.yaml index 03a69d16..bbc7e363 100644 --- a/manifests/ray-on-eks/ray-cluster-modules.yaml +++ b/manifests/ray-on-eks/ray-cluster-modules.yaml @@ -1,5 +1,5 @@ name: ray-cluster -path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-cluster?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-cluster?ref=release/1.7.0&depth=1 parameters: - name: EksClusterAdminRoleArn valueFrom: @@ -34,8 +34,8 @@ parameters: cpu: "1" memory: "8G" limits: - cpu: "1" - memory: "8G" + cpu: "4" + memory: "16G" - name: WorkerReplicas value: 1 - name: WorkerMinReplicas @@ -45,11 +45,11 @@ parameters: - name: WorkerResources value: requests: - cpu: "1" + cpu: "4" memory: "8G" limits: - cpu: "1" - memory: "8G" + cpu: "14" + memory: "60G" - name: DataBucketName valueFrom: moduleMetadata: diff --git a/manifests/ray-on-eks/ray-operator-modules.yaml b/manifests/ray-on-eks/ray-operator-modules.yaml index 38ee34f0..6f033043 100644 --- a/manifests/ray-on-eks/ray-operator-modules.yaml +++ b/manifests/ray-on-eks/ray-operator-modules.yaml @@ -1,5 +1,5 @@ name: ray-operator -path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-operator?ref=release/1.6.0&depth=1 +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-operator?ref=release/1.7.0&depth=1 parameters: - name: EksClusterAdminRoleArn valueFrom: diff --git a/manifests/ray-on-eks/ray-orchestrator-modules.yaml b/manifests/ray-on-eks/ray-orchestrator-modules.yaml index aa75b5f2..a55a8d42 100644 --- a/manifests/ray-on-eks/ray-orchestrator-modules.yaml +++ b/manifests/ray-on-eks/ray-orchestrator-modules.yaml @@ -1,5 +1,5 @@ name: ray-orchestrator -path: modules/eks/ray-orchestrator +path: git::https://github.com/awslabs/aiops-modules.git//modules/eks/ray-orchestrator?ref=release/1.7.0&depth=1 parameters: - name: Namespace valueFrom: diff --git a/modules/mlflow/mlflow-ai-gw-image/README.md b/modules/mlflow/mlflow-ai-gw-image/README.md new file mode 100644 index 00000000..87344bd5 --- /dev/null +++ b/modules/mlflow/mlflow-ai-gw-image/README.md @@ -0,0 +1,39 @@ +# Mlflow AI Gateway Image + +## Description + +This module creates a [MLFlow AI Gateway](https://mlflow.org/docs/latest/llms/index.html#id2) server container image and pushes to the specified Elastic Container Repository. + +## Inputs/Outputs + +### Input Parameters + +#### Required + +- `ecr-repository-name`: The name of the ECR repository to push the image to. + +### Sample manifest declaration + +```yaml +name: mlflow-image +path: modules/mlflow/mlflow-image +parameters: + - name: ecr-repository-name + valueFrom: + moduleMetadata: + group: storage + name: ecr-mlflow-ai-gw + key: EcrRepositoryName +``` + +### Module Metadata Outputs + +- `MlflowAIGWImageUri`: Mlflow AI Gateway image URI + +#### Output Example + +```json +{ + "MlflowAIGWImageUri": "xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/ecr-mlflow-ai-gw:latest" +} +``` diff --git a/modules/mlflow/mlflow-ai-gw-image/deployspec.yaml b/modules/mlflow/mlflow-ai-gw-image/deployspec.yaml new file mode 100644 index 00000000..8eb9f191 --- /dev/null +++ b/modules/mlflow/mlflow-ai-gw-image/deployspec.yaml @@ -0,0 +1,21 @@ +publishGenericEnvVariables: true +deploy: + phases: + build: + commands: + - aws ecr describe-repositories --repository-names ${SEEDFARMER_PARAMETER_ECR_REPOSITORY_NAME} || aws ecr create-repository --repository-name ${SEEDFARMER_PARAMETER_ECR_REPOSITORY_NAME} --image-scanning-configuration scanOnPush=true + - export COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7) + - export IMAGE_TAG=${COMMIT_HASH:=latest} + - export REPOSITORY_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${SEEDFARMER_PARAMETER_ECR_REPOSITORY_NAME} + - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com + - echo Building the Docker image... + - cd src/ && docker build -t $REPOSITORY_URI:latest . + - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG + - docker push $REPOSITORY_URI:$IMAGE_TAG + - seedfarmer metadata add -k ImageUri -v $REPOSITORY_URI:$IMAGE_TAG +destroy: + phases: + build: + commands: + - aws ecr delete-repository --repository-name ${SEEDFARMER_PARAMETER_ECR_REPOSITORY_NAME} --force +# build_type: BUILD_GENERAL1_LARGE diff --git a/modules/mlflow/mlflow-ai-gw-image/modulestack.yaml b/modules/mlflow/mlflow-ai-gw-image/modulestack.yaml new file mode 100644 index 00000000..9ea9c80b --- /dev/null +++ b/modules/mlflow/mlflow-ai-gw-image/modulestack.yaml @@ -0,0 +1,42 @@ +AWSTemplateFormatVersion: 2010-09-09 +Description: This stack deploys a Module specific IAM permissions + +Parameters: + # DeploymentName: + # Type: String + # Description: The name of the deployment + # ModuleName: + # Type: String + # Description: The name of the Module + RoleName: + Type: String + Description: The name of the IAM Role + ECRRepositoryName: + Type: String + Description: The name of the ECR repository + +Resources: + Policy: + Type: "AWS::IAM::Policy" + Properties: + PolicyDocument: + Statement: + - Effect: Allow + Action: + - "ecr:Describe*" + - "ecr:Get*" + - "ecr:List*" + Resource: "*" + - Action: + - "ecr:Create*" + - "ecr:Delete*" + - "ecr:*LayerUpload" + - "ecr:UploadLayerPart" + - "ecr:Batch*" + - "ecr:Put*" + Effect: Allow + Resource: + - !Sub "arn:${AWS::Partition}:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepositoryName}" + Version: 2012-10-17 + PolicyName: "modulespecific-policy" + Roles: [!Ref RoleName] diff --git a/modules/mlflow/mlflow-ai-gw-image/src/Dockerfile b/modules/mlflow/mlflow-ai-gw-image/src/Dockerfile new file mode 100644 index 00000000..aa3b7565 --- /dev/null +++ b/modules/mlflow/mlflow-ai-gw-image/src/Dockerfile @@ -0,0 +1,15 @@ +FROM public.ecr.aws/docker/library/python:3.10.15 + +RUN pip install \ + 'mlflow[gateway]' \ + mlflow==2.17.0 && \ + mkdir /mlflow/ + +EXPOSE 7000 + +COPY config.yaml ./mlflow/config.yaml + +CMD mlflow gateway start \ + --host 0.0.0.0 \ + --port 7000 \ + --config-path ./mlflow/config.yaml; diff --git a/modules/mlflow/mlflow-ai-gw-image/src/config.yaml b/modules/mlflow/mlflow-ai-gw-image/src/config.yaml new file mode 100644 index 00000000..fca27555 --- /dev/null +++ b/modules/mlflow/mlflow-ai-gw-image/src/config.yaml @@ -0,0 +1,13 @@ +endpoints: + - name: chat + endpoint_type: llm/v1/chat + model: + provider: bedrock + name: anthropic.claude-3-5-sonnet-20240620-v1:0 + config: + aws_config: + aws_region: 'AWS_REGION' + aws_role_arn: 'AWS_ROLE_ARN' + limit: + renewal_period: minute + calls: 10 diff --git a/modules/mlflow/mlflow-image/src/Dockerfile b/modules/mlflow/mlflow-image/src/Dockerfile index bd6bad93..ee2f83cd 100644 --- a/modules/mlflow/mlflow-image/src/Dockerfile +++ b/modules/mlflow/mlflow-image/src/Dockerfile @@ -1,7 +1,7 @@ FROM python:3.10.12 RUN pip install \ - mlflow==2.16.0 \ + mlflow==2.17.0 \ pymysql==1.1.1 \ boto3==1.34.45 && \ mkdir /mlflow/ diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/README.md b/modules/sagemaker/sagemaker-ground-truth-labeling/README.md new file mode 100644 index 00000000..1d9aef67 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/README.md @@ -0,0 +1,146 @@ +# SageMaker Ground truth labeling + +## Description + +This module creates a workflow for labeling data using SageMaker ground truth. + +A bucket is created to store the raw data. Data uploaded to the S3 bucket is then sent to a created SQS queue. If a text job type is selected the contents of `.txt` files uploaded to the bucket is sent to the SQS queue, instead of the file location. A step function is created that runs on a schedule, pulling the unlabeled data from the SQS queue. The function then runs a labeling job, followed by a verification job (only on supported job types, see below) to increase the accuracy of the labeling. Labeled items that fail validation are returned to the SQS queue for relabelling. New labels are then saved to a created Sagemaker feature group. + +This module assumes that uploaded content will be free of `Personally Identifiable Information (PII)` and `Adult content`. If this is not the case please remove the appropiate content classifiers from the `create_labeling_job` method. + +### Architecture + +![SageMaker Ground Truth Labeling Module Architecture](docs/_static/sagemaker-ground-truth-labeling-module-architecture.png "SageMaker Ground Truth Labeling Module Architecture") + +### Step function example + +![Step function graph](docs/_static/stepfunctions_graph.png "Step function graph") + +#### With verification step + +![Step function graph with verification step](docs/_static/stepfunctions_graph_with_verification_step.png "Step function graph with verification step") + +## Inputs/Outputs + +### Input Parameters + +#### Required + +- `job-name`: Used as prefix for created resources and executions of workflow +- `task-type`: The labeling task type to be carried out (read more [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-task-types.html)). Currently this module supports all built in task types for images and text. +Allowed values are: + - [`image_bounding_box`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-bounding-box.html) + - [`image_semantic_segmentation`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-semantic-segmentation.html) + - [`image_single_label_classification`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-image-classification.html) + - [`image_multi_label_classification`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-image-classification-multilabel.html) + - [`text_single_label_classification`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-text-classification.html) + - [`text_multi_label_classification`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-text-classification-multilabel.html) + - [`named_entity_recognition`](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-named-entity-recg.html) +- `labeling-workteam-arn` - ARN of the workteam to carry out the labeling task, can be public or private + - `labeling-task-price` - Required if public team is to be used +- `labeling-instructions-template-s3-uri` - S3 URI of the labeling template `.html` or `.liquid` file + - Required for all labeling types _except_ `named_entity_recognition` +- `labeling-categories-s3-uri` - S3 URI of the labeling categories `.json` file +- `labeling-task-title` +- `labeling-task-description` +- `labeling-task-keywords` + +For job types supporting verification, currently `image_bounding_box` and `image_semantic_segmentation` further additional fields are required + +- `verification-workteam-arn` - ARN of the workteam to carry out the verification task, can be public or private + - `verification-task-price` - Required if public team is to be used +- `verification-instructions-template-s3-uri` - S3 URI of the verification template `.html` or `.liquid` file +- `verification-categories-s3-uri` - S3 URI of the verification categories `.json` file. The first label must be the label to pass validation, all other labels are validation failures. +- `verification-task-title` +- `verification-task-description` +- `verification-task-keywords` + +For more information and examples of the templates please look at the examples. There are also multiple templates available [here](https://github.com/aws-samples/amazon-sagemaker-ground-truth-task-uis/tree/master). + +Labeling and verification task title, description and keywords are used to create the task config which will be sent to the human carrying out the labeling or verification job. + +More information on using a public workforce like Amazon Mechanical Turk is available [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-public.html). Labeling and verification task prices is specified in USD, see [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_PublicWorkforceTaskPrice.html) for allowed values. [This page](https://aws.amazon.com/sagemaker/groundtruth/pricing/) provides suggested pricing based on task type. + +#### Optional + +- `labeling-human-task-config`: Additional configuration parameters for labeling job. For tasks without a verification step we recommend increasing the number of human workers per data object, to increase accuracy. Depending on task complexity you might want to increase the task time limit. Default is: + - `NumberOfHumanWorkersPerDataObject`: 1 + - `TaskAvailabilityLifetimeInSeconds`: 21600 (6 hours) + - `TaskTimeLimitInSeconds`: 300 (5 minutes) +- `verification-human-task-config`: Additional configuration parameters for verification job. Default is: + - `NumberOfHumanWorkersPerDataObject`: 1 + - `TaskAvailabilityLifetimeInSeconds`: 21600 (6 hours) + - `TaskTimeLimitInSeconds`: 300 (5 minutes) +- `labeling-workflow-schedule`: CRON schedule for how often the workflow should run. Default is `cron(0 12 * * ? *)` (midday UTC daily), empty string ('') to disable +- `sqs-queue-retention-period`: Upload queue retention period in minutes. Default is 20160 (14 days) +- `sqs-queue-visibility-timeout`: Upload queue visibility timeout in minutes. Default is 720 (12 hours) +- `sqs-queue-max-receive-count`: Default is 3 +- `sqs-dlq-retention-period`: DLQ retention period in minutes, suggest setting to a high value to ensure they are caught and re-driven before deletion. Default is 20160 (14 days) +- `sqs-dlq-visibility-timeout`: DLQ visibility timeout in minutes. Default is 720 (12 hours) +- `sqs-dlq-alarm-threshold` - Number of messages in the DLQ on which to alarm. Default is 1, 0 to disable + +`labeling-workflow-schedule` and the SQS queue parameters should be set to values that ensure the workflow can run at least as many times as the `maxRecieveCount` before the `retentionPeriod` is reached, to avoid messages being deleted upon reaching the `retentionPeriod`, instead of being sent to the DLQ + +### Sample manifest declaration + +```yaml +name: ground-truth-labeling +path: modules/sagemaker/sagemaker-ground-truth-labeling +parameters: + - name: job_name + value: 'plane-identification' + - name: task_type + value: 'image_bounding_box' + - name: labeling-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: labeling-instructions-template-s3-uri + value: 's3:///' + - name: labeling-categories-s3-uri + value: 's3:///' + - name: labeling-task-title + value: 'Labeling - Bounding boxes: Draw bounding boxes around all planes in the image' + - name: labeling-task-description + value: 'Draw bounding boxes around all planes in the image' + - name: labeling-task-keywords + value: [ 'image', 'object', 'detection' ] + - name: verification-workteam-arn + value: 'arn:aws:sagemaker:::workteam/private-crowd/' + - name: verification-instructions-template-s3-uri + value: 's3:///' + - name: verification-categories-s3-uri + value: 's3:///' + - name: verification-task-title + value: 'Label verification - Bounding boxes: Review the existing labels on the objects and choose the appropriate option.' + - name: verification-task-description + value: 'Verify that the planes are correctly labeled' + - name: verification-task-keywords + value: ['image', 'object', 'detection', 'label verification', 'bounding boxes'] +``` + +### Module Metadata Outputs + +- `DataStoreBucketName`: Name of the created S3 bucket where the user will upload the raw data +- `DataStoreBucketArn`: ARN of the created S3 bucket where the user will upload the raw data +- `SqsQueueName`: Name of the created SQS queue +- `SqsQueueArn`: ARN of the created SQS queue +- `SqsDlqName`: Name of the created SQS DLQ +- `SqsDlqArn`: ARN of the created SQS DLQ +- `LabelingStateMachineName`: Name of the labeling state machine +- `LabelingStateMachineArn`: ARN of the labeling state machine +- `FeatureGroupName`: Name of the feature group + +#### Output Example + +```json +{ + "DataStoreBucketName": "aiops-mlops-sagemaker-sagemaker--upload-bucket", + "DataStoreBucketArn": "arn:aws:s3:::aiops-mlops-sagemaker-sagemaker--upload-bucket", + "SqsQueueName": "aiops-mlops-sagemaker-sagemaker-ground-truth-ground-truth--upload-queue", + "SqsQueueArn": "arn:aws:sqs:::aiops-mlops-sagemaker-sagemaker-ground-truth-ground-truth--upload-queue", + "SqsDlqName": "aiops-mlops-sagemaker-sagemaker-ground-truth-ground-truth--upload-dlq", + "SqsDlqArn": "arn:aws:sqs:::aiops-mlops-sagemaker-sagemaker-ground-truth-ground-truth--upload-dlq", + "LabelingStateMachineName": "aiops-mlops-sagemaker-sagemaker-ground-truth-ground-truth--state-machine", + "LabelingStateMachineArn": "arn:aws:states:::stateMachine:aiops-mlops-sagemaker-sagemaker-ground-truth-ground-truth--state-machine", + "FeatureGroupName": "aiops-mlops-sagemaker-sagemaker--sagemaker-feature-group" +} +``` diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/app.py b/modules/sagemaker/sagemaker-ground-truth-labeling/app.py new file mode 100644 index 00000000..c8f59049 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/app.py @@ -0,0 +1,71 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +import aws_cdk +import cdk_nag + +from settings import ApplicationSettings +from stack import DeployGroundTruthLabelingStack + +app = aws_cdk.App() +app_settings = ApplicationSettings() + +stack = DeployGroundTruthLabelingStack( + scope=app, + id=app_settings.seedfarmer_settings.app_prefix, + job_name=app_settings.module_settings.job_name, + task_type=app_settings.module_settings.task_type, + sqs_queue_retention_period=app_settings.module_settings.sqs_queue_retention_period, + sqs_queue_visibility_timeout=app_settings.module_settings.sqs_queue_visibility_timeout, + sqs_queue_max_receive_count=app_settings.module_settings.sqs_queue_max_receive_count, + sqs_dlq_retention_period=app_settings.module_settings.sqs_dlq_retention_period, + sqs_dlq_visibility_timeout=app_settings.module_settings.sqs_dlq_visibility_timeout, + sqs_dlq_alarm_threshold=app_settings.module_settings.sqs_dlq_alarm_threshold, + labeling_workteam_arn=app_settings.module_settings.labeling_workteam_arn, + labeling_instructions_template_s3_uri=app_settings.module_settings.labeling_instructions_template_s3_uri, + labeling_categories_s3_uri=app_settings.module_settings.labeling_categories_s3_uri, + labeling_task_title=app_settings.module_settings.labeling_task_title, + labeling_task_description=app_settings.module_settings.labeling_task_description, + labeling_task_keywords=app_settings.module_settings.labeling_task_keywords, + labeling_human_task_config=app_settings.module_settings.labeling_human_task_config, + labeling_task_price=app_settings.module_settings.labeling_task_price, + verification_workteam_arn=app_settings.module_settings.verification_workteam_arn, + verification_instructions_template_s3_uri=app_settings.module_settings.verification_instructions_template_s3_uri, + verification_categories_s3_uri=app_settings.module_settings.verification_categories_s3_uri, + verification_task_title=app_settings.module_settings.verification_task_title, + verification_task_description=app_settings.module_settings.verification_task_description, + verification_task_keywords=app_settings.module_settings.verification_task_keywords, + verification_human_task_config=app_settings.module_settings.verification_human_task_config, + verification_task_price=app_settings.module_settings.verification_task_price, + labeling_workflow_schedule=app_settings.module_settings.labeling_workflow_schedule, + env=aws_cdk.Environment( + account=app_settings.cdk_settings.account, + region=app_settings.cdk_settings.region, + ), +) + +aws_cdk.CfnOutput( + scope=stack, + id="metadata", + value=stack.to_json_string( + { + "DataStoreBucketName": stack.upload_bucket.bucket_name, + "DataStoreBucketArn": stack.upload_bucket.bucket_arn, + "SqsQueueName": stack.upload_queue.queue_name, + "SqsQueueArn": stack.upload_queue.queue_arn, + "SqsDlqName": stack.upload_dlq.queue_name, + "SqsDlqArn": stack.upload_dlq.queue_arn, + "LabelingStateMachineName": stack.labeling_state_machine.state_machine_name, + "LabelingStateMachineArn": stack.labeling_state_machine.state_machine_arn, + "FeatureGroupName": stack.feature_group.feature_group_name, + } + ), +) + +aws_cdk.Aspects.of(app).add(cdk_nag.AwsSolutionsChecks(log_ignores=True)) + +aws_cdk.Tags.of(app).add("SeedFarmerDeploymentName", app_settings.seedfarmer_settings.deployment_name) +aws_cdk.Tags.of(app).add("SeedFarmerModuleName", app_settings.seedfarmer_settings.module_name) +aws_cdk.Tags.of(app).add("SeedFarmerProjectName", app_settings.seedfarmer_settings.project_name) + +app.synth() diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/constants.py b/modules/sagemaker/sagemaker-ground-truth-labeling/constants.py new file mode 100644 index 00000000..6b30982c --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/constants.py @@ -0,0 +1,27 @@ +import aws_cdk.aws_lambda as lambda_ + +MAX_BUCKET_NAME_LENGTH = 63 +MAX_SQS_QUEUE_NAME_LENGTH = 80 +MAX_FEATURE_GROUP_NAME_LENGTH = 64 +MAX_ROLE_NAME_LENGTH = 64 +MAX_STATE_MACHINE_NAME_LENGTH = 80 +MAX_LAMBDA_FUNCTION_NAME_LENGTH = 64 + +# map of lambda function Id, you can see the current list here: +# https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AnnotationConsolidationConfig.html#SageMaker-Type-AnnotationConsolidationConfig-AnnotationConsolidationLambdaArn +AC_ARN_MAP = { + "us-east-1": "432418664414", + "us-east-2": "266458841044", + "us-west-2": "081040173940", + "eu-west-1": "568282634449", + "ap-northeast-1": "477331159723", + "ap-southeast-2": "454466003867", + "ap-south-1": "565803892007", + "eu-central-1": "203001061592", + "ap-northeast-2": "845288260483", + "eu-west-2": "487402164563", + "ap-southeast-1": "377565633583", + "ca-central-1": "918755190332", +} + +LAMBDA_RUNTIME = lambda_.Runtime.PYTHON_3_12 diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/deployspec.yaml b/modules/sagemaker/sagemaker-ground-truth-labeling/deployspec.yaml new file mode 100644 index 00000000..fea2cc1d --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/deployspec.yaml @@ -0,0 +1,23 @@ +publishGenericEnvVariables: true +deploy: + phases: + install: + commands: + - env + # Install whatever additional build libraries + - npm install -g aws-cdk@2.158.0 + - pip install -r requirements.txt + build: + commands: + - cdk deploy --require-approval never --progress events --app "python app.py" --outputs-file ./cdk-exports.json +destroy: + phases: + install: + commands: + # Install whatever additional build libraries + - npm install -g aws-cdk@2.158.0 + - pip install -r requirements.txt + build: + commands: + # execute the CDK + - cdk destroy --force --app "python app.py" diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/sagemaker-ground-truth-labeling-module-architecture.png b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/sagemaker-ground-truth-labeling-module-architecture.png new file mode 100644 index 00000000..f8b6b655 Binary files /dev/null and b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/sagemaker-ground-truth-labeling-module-architecture.png differ diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/sagemaker-ground-truth-labeling-module-architecture.xml b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/sagemaker-ground-truth-labeling-module-architecture.xml new file mode 100644 index 00000000..307327da --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/sagemaker-ground-truth-labeling-module-architecture.xml @@ -0,0 +1 @@ +7V1rV6M4GP41njP7IR4C4fbRVp0dV0dXnbHuF0+A0DLTQgeobefXb8Kl5Vbajq0Fm113NSGEXN7keXjekJxI3dHss4/HgxvPIsMTUbBmJ9L5iShCVdToLxYzj2N0SYoj+r5jJYmWEQ/Ob5JECknsxLFIkEsYet4wdMb5SNNzXWKGuTjs+940n8z2hvmnjnGflCIeTDwsxz45VjiIYzVRXcb/TZz+IH0yVPT4yginiZOaBANsedNMlHRxInV9zwvjv0azLhmyxkvb5bn/im5vv+r/GEB+MX6P8eMAOSDJzSWzkKX/Yn3Hw0lSWElb5HtPAm/im+ScBKbvjEPPpzf5SWSceuK7J9LZDND2dfoucNxgTFuQJpTOXiH9X5+4jgUEeKUKz8rT94te5/npCV393bt71v651/4FtQUMiO/gofMbh47nglfiB/R3/ODXJAlO+savKGtSjQcywm7omOc4xF3PDbHjEn+T3OO7Q99x+9dOSHw8jI0kJG64QfOOfW9M/DAx2kEYMnM7OxEv6Q/NxBt6/flpQMyJ74TzUzzCvz331CKv9LLtTVwrKhcNWA7u+3gEXp1gsigvjceiKMu6rABTsmyATNEEBtJ0GhRUW9GwYeqxhV3GNfly/6XUf1uVqtTJ4qXpjcaeS9sjoAENYU0wZBvIChIBwlAFuqzJgBi2ZSiyjQxi7rZpgnkQkhEYsSmD9hKNEZAs6bKuAiRqEkC2IgNN0AWAiaLauiURHenZRqF/VNtHerViEKSXktG2zchbTAX+JjZ04FEIWz0KIR+FRzcKPeMHA3BRGGKDkoioXokRU3MxneHjfJy0OR0T1PJMYA4wRf1hnCodvOal+PX2qzS6f/rpXNm66549vVwCuITGxXAPwnkK88SiqJ8EPT8ceH3PxcOLZWzHZ61G2DMEGhqEI1ZEOkA7tOv8eY/Fn8pp8DlJFgXOZ7nQPAn9IGE4TwgPnoQejVo++drzxkn+Fg4G0XNZIK4qK22Bnqybk1LuhP0+CesaCsUJfcv+NqaDJq5yhlMFAQlZT3zPjnZpyYa+JOl/XQffER5g9TO6dAcXvjuefLvN9MFn4o0IbY9oQA2pOb6SwoPSvk7TLW698xyXGUpCMjWYPD4ZDFApzGyx+SV35Qw3U4x3m6/r7LMFE3Z98Zs2Y9dWZaWJ7r8egmyasgRNYIkIAqSbEp1kLVoZS1JV1ZSRqNgcedYgz5sbJVeePnuBBUHoT8xw4hMao8hQVhVRACoyaKFUgdD+0S1g6IYtK4ZiGshY1SaLIkyn09OpdOr5rIZQ13VWUVYHUQR0kgXBnA7PGXBpJ0ghQ7g1FdzT/IN227kVjQlVg1DkAqZNbQthAQKsyhbQILEkjFRRlsR90ApW8JhZrGQZX8mUKRT0KfTXJ2dEUSz4a3fUI0bHfVCPHDXYlE/YtCWTZFDcAaVoEVNAqaqU2LdYZAoxPSoxhVJGUFyTURsoR9oXLaUcC1NqE+WomWo2xRBREASGIdT6LyMAMQe06Sl6xDPasePj0bOij0J5D0YAQjIL/wj+H37/GPvX8p2C/StlguV7Ec1ugaTsC/43RfylQpHTJ5ZyxQqFIgrd0fpSuGLtHkXuTY5Y0Xpy61hGkRxIQiPIwVn3+aF3g8Srp97dVe9SvflPQk+PHVBntS0gB/XFbxU5aNHkzJH3GPWILahtBkTfh9juax6Uj1gYORGVYcgw36H1UPphhPhxlJFG3ONMtwf0YSRNQh9gFG+LpqBSZpmoTw/SX6sSVzKyQNpIg6nlYGMGxRGKyR36Q62wG/8n06tdFnMqyhWRVXFqORKWk9FfsOoJxciqOLUcCcvJWCgtdT6yKk6VyyUu3g0r7oaFu+kPJaGTkM6gpLtYkiMkslOXDhM/anKJ/nvJ+qPDOJ1DctcU4UyS1My1c8enGcXUwGUUl+XnDIfZ/FQVQoXG06Hl/SSZK3b0T5a5suJQrkEHEKXJzNrvvMBJsje8MPRGmQRnQzqd0Asho9MdnIRMWio25LJ0vCisReHEyNgjcTQj0YDtzFg5OsEAR1Y8mkWzwimeBqJwmk5rX0xWoA4Nxn8VklG7T8cEKypJyef+1bsyYU7ltYIohpK5c5pZOJUkGWTWTGkFHpll2Omkd0CVrCU8uLb0TaPBe6ePhCBkSDoGJjIMgARKbgxdgUC0kUZMqEsktbrjoY+CgQjBpgYERGSACGXTmqIzji8rtqEj1VAPSB/LPCKmAvthkB/lPexdyWC1IuZ7k/EmFAzWc7A4nxygrkS3ZLEvNtKb07F8CNSDgi7ncU8SSrinV+AehI0GPthu5EuLf3TQd5TKybpGQbZMwQrawJZMhTYKoo1iCAIdFZKsCNjQKTHYK/YdN7hzRN0CUc05HUUW8TcBVb0WU9MXzEWOUmc6oD30MMaRDU6pGeUh14h9TdfGIgKbP/uRB+o2fqtfh81ZDK6fmXcLwiXIVVv3qqm3GnB1jrccbznecrxtMN4WfBeXBMfdud5xYfjFmGCM3aXXAvfJDf5J/KzzIptiBdYzN9lGL8/LqbkK6KNsckBOLZ/2ibEQn/MuDqbSsxUjQSJXVynb1QI4rZNJ7e4xWlwCUIOYwDwP8WtevqVGM4GUzrSUCsAda6ut4QIc9vYNe5zObbvkswByOaji4npjqEklPQhoX41YX23CEeq/cOVrHBq1xgGddUT9bLs1DhAq57TzjmiNw8L8m0Mzk4eLyQLTdI2vWGKdklbBOrUms852f4Lc0C+QOWHh+tNHJOKc5L3/Cor6D4lbu4JCQfkNPBalyaIpqhBxYMNVnHZ/YJsWnyMqR1SOqBxRG4mo6z9Qid5+o0ElTD3/pz30pqudOicZ188n+hJKLYSMgT1xo1fzAKQ5bPeNSk1Gm+A+2l7YKYk6RUGnJObkhZySSFIUSEriSF6mKSk5RbmnpAnlZaOSclOUd0oaUK0wUxRgXI8tZMkypap1MevEjmptI6Vhfb9S04guv9C/X5hFvCws4mVhERX6TvecsaSCIpRUol7EGRI7XLrtrqPQuVQhVi0ekZOQ3qK6SDtkii1ngqjVRHDHe0NxHsh5IOeBf8gDK9jMGp7CfXzNJaufFh6Ol3iR7Qs1xHCw5RfQlXlsQizrd6bhHkPuMWytxzA/FppDZJOHp0vSknkwFUCzNFcus9w0rpkkty07BtWWnpNcTnI5yT0wyV1NiziZbTCZHeKRYeHtyGt8zyZkVeVktT1k9VLVLgS0HVk9F+QuVI+IrCa23zhyutjNOmWn4ml5Jx9UwU9Ro/mp2mp+mh44d2z8lCgihPRFFlhYEhlT0QA2RAgEwYKKIhiiQvhOPs1ibwsewNlaQ9jaA35lH4C4bItOBY8YDLpGwH5FiYIV5OxPztk6+G7XuQ2qd33YRXZEf5Tzs3Qtv3/RpvtVn/k+nmeSJRQ8k6K8NTYUCnslaQXKsOIEjpMmHZwB2610wYZKXR9lXueSHd8bu7hJ/QJ/s3jbsI2xG3ZgmAVtU1AUAiRNYVZOBKBJAgKKSWwk2kjAstAc8au07LDrs3oJPzwaI1Db6PfZrL7RukPySg0HfAqdETViHBDrr+00tfL9GzG3eoHtTYpQslStIPh0ukxOLwk+SeLGaT0RvyH+xSuJlUa4Uv+JOuCFtf9L1P6bSjzv9sWFquW/uEgFn4y+k0Zl9Z3iKWfN0ndguwUeeKwKz1HSGe6BbLSGVYXe1bjMNa6GaFzfXFp5kwQMb9kHhEWqNaKX2HGLf6J1oXOvP5iq41+BF3yZms+fCeo/Aal+U6731rreqm21SLJCYp6/QFQA9k0Pci1mJBYzOtBBrh0Iny+0x8eOoPf+1h80Fepq7w7U2WELiE998ZtGfGqrkp9t8nMLP8mVb+vFT3I9CAmID6HbA9TXf4P5BqgnMyfsZf6Oj2ZV5SS4PJqVBeaZQO3BrJsSiLdyhux4rW45UWsduyjuR1FyiB3ISbUdKWjLN4f1xW8VKaiafTgZ4GSAk4Fduai2mgLFHW8J9+F8VLkje39NyGTDLdA/Bb+CLb9R/BVsQrNEvodpmxZ5Swhp4naLvFf4/D7uIm9m+U1z/yG4QvVq2kmtfzTdt5Tx7hqtWuP/kzURqaJpAVVSVFYgAnSsIoBlCwlIkVVN3fHLQAP8f632jpV5Q8IIuDOsITrYPbF855W47+IGg/vSxg4iabVIqELaGjfYCu/V2oxKbrBWKF6pJtlO/E+L3zT8/yizb/3p48l8ybU7rt3tni02WfY5v/730KJP/SG2XPThog8XffYg+nwIzactZybXF79pnI9rPlzzqdN8KGngik+zFJ87LyJOdjT3uvtaBlV/As1blkG5FFN62UC8EEqU0/ByJVQUmmdDjVgLFWskm0j8LZKYSmfzFL8Ba8UC6bYcxFNf/KaxhI8yS9fWY+28ykUiLhLti162dmGUAlWDUCwFpk3bHmEBAqzKFtAgsSRMqb+cnJjanFP46tjQw+8fY/9avlOwf6VMsHwvotktkOq/mN/5ovDt14QvWJWQZ1Vqe1hVdjis6Ab5Q/CvfYgpZ93nh94NEq+eendXvUv15j8JPT12QJ09t4Am1Re/aTTpTRP7lj0o73ti33bF61GjGidyx0jkPspr0btysvVey9Nwxu6ynSGJGzVyrDBEqfBVeu5wvnhrC0l0Z7KZAzXhpR9zzU5Mu9rRvG625o7Pljg++Zbm6x2f221p/m6+z4Wzc7HtaDOdn3/G9trK13dMVlvj/ORbmrfN+bmae6xlGnw39IOyTdZ+nhdmkn5mtnfDRgKN/B8= \ No newline at end of file diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/stepfunctions_graph.png b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/stepfunctions_graph.png new file mode 100644 index 00000000..c1ab409e Binary files /dev/null and b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/stepfunctions_graph.png differ diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/stepfunctions_graph_with_verification_step.png b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/stepfunctions_graph_with_verification_step.png new file mode 100644 index 00000000..8760ca9a Binary files /dev/null and b/modules/sagemaker/sagemaker-ground-truth-labeling/docs/_static/stepfunctions_graph_with_verification_step.png differ diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/__init__.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/labeling_state_machine.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/labeling_state_machine.py new file mode 100644 index 00000000..7128359d --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/labeling_state_machine.py @@ -0,0 +1,1076 @@ +import json +from typing import Any, Dict, List, Optional, Tuple + +import aws_cdk.aws_iam as iam +import aws_cdk.aws_kms as kms +import aws_cdk.aws_lambda as lambda_ +import aws_cdk.aws_logs as logs +import aws_cdk.aws_s3 as s3 +import aws_cdk.aws_scheduler as scheduler +import aws_cdk.aws_sqs as sqs +import aws_cdk.aws_stepfunctions as sfn +import aws_cdk.aws_stepfunctions_tasks as tasks +from aws_cdk import Duration, RemovalPolicy, Stack +from cdk_nag import NagSuppressions + +from constants import ( + AC_ARN_MAP, + LAMBDA_RUNTIME, + MAX_BUCKET_NAME_LENGTH, + MAX_LAMBDA_FUNCTION_NAME_LENGTH, + MAX_STATE_MACHINE_NAME_LENGTH, +) +from task_type_config import TaskTypeConfig, get_task_type_config + +LAMBDA_ASSET_FOLDER = "labeling_step_function/lambda" + + +def setup_and_create_state_machine( + scope: Stack, + id: str, + job_name: str, + queue: sqs.Queue, + queue_kms_key_arn: str, + upload_bucket_arn: str, + upload_bucket_kms_key_arn: str, + log_bucket: s3.Bucket, + task_type: str, + feature_group_name: str, + labeling_workteam_arn: str, + labeling_instructions_template_s3_uri: str, + labeling_categories_s3_uri: str, + labeling_task_title: str, + labeling_task_description: str, + labeling_task_keywords: List[str], + labeling_human_task_config: Dict[str, Any], + labeling_task_price: Dict[str, Dict[str, int]], + verification_workteam_arn: str, + verification_instructions_template_s3_uri: str, + verification_categories_s3_uri: str, + verification_task_title: str, + verification_task_description: str, + verification_task_keywords: List[str], + verification_human_task_config: Dict[str, Any], + verification_task_price: Dict[str, Dict[str, int]], + labeling_workflow_schedule: str, +) -> sfn.StateMachine: + ground_truth_output_bucket, ground_truth_output_bucket_kms_key = create_ground_truth_output_bucket( + scope=scope, job_name=job_name, id=id, log_bucket=log_bucket + ) + + task_type_config = get_task_type_config(task_type) + + poll_sqs_queue_lambda = create_poll_sqs_queue_lambda( + scope=scope, + queue=queue, + queue_kms_key_arn=queue_kms_key_arn, + ground_truth_output_bucket=ground_truth_output_bucket, + ground_truth_output_bucket_kms_key_arn=ground_truth_output_bucket_kms_key.key_arn, + job_name=job_name, + id=id, + task_type=task_type, + task_media_type=task_type_config.media_type, + ) + + ground_truth_role = create_ground_truth_role( + scope=scope, + upload_bucket_arn=upload_bucket_arn, + ground_truth_output_bucket_arn=ground_truth_output_bucket.bucket_arn, + labeling_instructions_template_s3_uri=labeling_instructions_template_s3_uri, + labeling_categories_s3_uri=labeling_categories_s3_uri, + task_type_config_verification_attribute_name=task_type_config.verification_attribute_name, + verification_instructions_template_s3_uri=verification_instructions_template_s3_uri, + verification_categories_s3_uri=verification_categories_s3_uri, + ground_truth_output_bucket_kms_key_arn=ground_truth_output_bucket_kms_key.key_arn, + upload_bucket_kms_key_arn=upload_bucket_kms_key_arn, + ) + + labeling_job_name = "labeling-job" + verification_job_name = "verification-job" + + run_ground_truth_job_lambda_execution_role = create_run_ground_truth_job_lambda_execution_role( + scope=scope, + ground_truth_output_bucket_kms_key_arn=ground_truth_output_bucket_kms_key.key_arn, + ground_truth_output_bucket_arn=ground_truth_output_bucket.bucket_arn, + labeling_job_name=labeling_job_name, + verification_job_name=verification_job_name, + ground_truth_role_arn=ground_truth_role.role_arn, + ) + + run_labeling_job_lambda = create_run_labeling_job_lambda( + scope=scope, + job_name=job_name, + id=id, + run_ground_truth_job_lambda_execution_role=run_ground_truth_job_lambda_execution_role, + labeling_job_name=labeling_job_name, + ground_truth_role_arn=ground_truth_role.role_arn, + ground_truth_output_bucket_name=ground_truth_output_bucket.bucket_name, + task_type_config=task_type_config, + labeling_workteam_arn=labeling_workteam_arn, + labeling_instructions_template_s3_uri=labeling_instructions_template_s3_uri, + labeling_categories_s3_uri=labeling_categories_s3_uri, + labeling_task_title=labeling_task_title, + labeling_task_description=labeling_task_description, + labeling_task_keywords=labeling_task_keywords, + labeling_human_task_config=labeling_human_task_config, + labeling_task_price=labeling_task_price, + ) + + run_verification_job_lambda = create_run_verification_job_lambda( + scope=scope, + task_type_config=task_type_config, + job_name=job_name, + id=id, + run_ground_truth_job_lambda_execution_role=run_ground_truth_job_lambda_execution_role, + verification_job_name=verification_job_name, + ground_truth_role_arn=ground_truth_role.role_arn, + ground_truth_output_bucket_name=ground_truth_output_bucket.bucket_name, + verification_workteam_arn=verification_workteam_arn, + verification_instructions_template_s3_uri=verification_instructions_template_s3_uri, + verification_categories_s3_uri=verification_categories_s3_uri, + verification_task_title=verification_task_title, + verification_task_description=verification_task_description, + verification_task_keywords=verification_task_keywords, + verification_human_task_config=verification_human_task_config, + verification_task_price=verification_task_price, + ) + + update_feature_store_lambda = create_update_feature_store_lambda( + scope=scope, + ground_truth_output_bucket=ground_truth_output_bucket, + ground_truth_output_bucket_kms_key_arn=ground_truth_output_bucket_kms_key.key_arn, + feature_group_name=feature_group_name, + queue=queue, + task_type_config=task_type_config, + job_name=job_name, + id=id, + ) + + return_messages_to_sqs_queue_lambda = create_return_messages_to_sqs_queue_lambda( + scope=scope, + queue=queue, + ground_truth_output_bucket=ground_truth_output_bucket, + ground_truth_output_bucket_kms_key_arn=ground_truth_output_bucket_kms_key.key_arn, + job_name=job_name, + task_media_type=task_type_config.media_type, + id=id, + ) + + state_machine = create_state_machine( + scope=scope, + id=id, + job_name=job_name, + poll_sqs_queue_lambda=poll_sqs_queue_lambda, + run_labeling_job_lambda=run_labeling_job_lambda, + run_verification_job_lambda=run_verification_job_lambda, + update_feature_store_lambda=update_feature_store_lambda, + return_messages_to_sqs_queue_lambda=return_messages_to_sqs_queue_lambda, + labeling_job_name=labeling_job_name, + verification_job_name=verification_job_name, + ) + + create_labeling_workflow_schedule( + scope=scope, + labeling_workflow_schedule=labeling_workflow_schedule, + state_machine_arn=state_machine.state_machine_arn, + ) + + return state_machine + + +def create_ground_truth_output_bucket( + scope: Stack, job_name: str, id: str, log_bucket: s3.Bucket +) -> Tuple[s3.Bucket, kms.Key]: + ground_truth_output_bucket_kms_key = kms.Key( + scope, + "GroundTruthOutputBucketKMSKey", + description="Key for encrypting the Ground Truth output bucket", + enable_key_rotation=True, + ) + ground_truth_output_bucket_name_suffix = f"{job_name}-ground-truth-output-bucket" + ground_truth_output_bucket_name = ( + f"{id[: MAX_BUCKET_NAME_LENGTH - len(ground_truth_output_bucket_name_suffix)]}" + f"{ground_truth_output_bucket_name_suffix}" + ) + ground_truth_output_bucket = s3.Bucket( + scope, + "GroundTruthOutputBucket", + bucket_name=ground_truth_output_bucket_name, + enforce_ssl=True, + encryption_key=ground_truth_output_bucket_kms_key, + server_access_logs_prefix="ground_truth_output_logs/", + server_access_logs_bucket=log_bucket, + removal_policy=RemovalPolicy.DESTROY, + auto_delete_objects=True, + cors=[ + s3.CorsRule( + allowed_headers=[], + allowed_methods=[s3.HttpMethods.GET], + allowed_origins=["*"], + exposed_headers=["Access-Control-Allow-Origin"], + ) + ], + ) + + return ground_truth_output_bucket, ground_truth_output_bucket_kms_key + + +def create_poll_sqs_queue_lambda( + scope: Stack, + queue: sqs.Queue, + queue_kms_key_arn: str, + ground_truth_output_bucket: s3.Bucket, + ground_truth_output_bucket_kms_key_arn: str, + job_name: str, + id: str, + task_type: str, + task_media_type: str, +) -> lambda_.Function: + poll_sqs_queue_lambda_execution_role = iam.Role( + scope, + "PollSqsQueueLambdaExecutionRole", + assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"), + ) + poll_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + actions=[ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents", + ], + resources=["*"], + ) + ) + poll_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + queue.queue_arn, + ], + actions=[ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + ], + ) + ) + poll_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + queue_kms_key_arn, + ], + actions=[ + "kms:Decrypt", + ], + ) + ) + poll_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket.bucket_arn, + f"{ground_truth_output_bucket.bucket_arn}/*", + ], + actions=[ + "s3:PutObject", + ], + ) + ) + poll_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_kms_key_arn, + ], + actions=[ + "kms:GenerateDataKey", + ], + ) + ) + + poll_sqs_queue_lambda_name_suffix = f"{job_name}-poll-sqs-queue-lambda" + poll_sqs_queue_lambda_name = ( + f"{id[: MAX_LAMBDA_FUNCTION_NAME_LENGTH - len(poll_sqs_queue_lambda_name_suffix)]}" + f"{poll_sqs_queue_lambda_name_suffix}" + ) + poll_sqs_queue_lambda = lambda_.Function( + scope, + "PollSqsQueueFunction", + runtime=LAMBDA_RUNTIME, + code=lambda_.Code.from_asset(LAMBDA_ASSET_FOLDER), + handler="poll_sqs_queue.handler", + function_name=poll_sqs_queue_lambda_name, + memory_size=512, + role=poll_sqs_queue_lambda_execution_role, + timeout=Duration.seconds(600), + environment={ + "SQS_QUEUE_URL": queue.queue_url, + "OUTPUT_BUCKET": ground_truth_output_bucket.bucket_name, + "TASK_TYPE": task_type, + "TASK_MEDIA_TYPE": task_media_type, + }, + ) + + return poll_sqs_queue_lambda + + +def create_ground_truth_role( + scope: Stack, + upload_bucket_arn: str, + ground_truth_output_bucket_arn: str, + labeling_instructions_template_s3_uri: str, + labeling_categories_s3_uri: str, + task_type_config_verification_attribute_name: Optional[str], + verification_instructions_template_s3_uri: str, + verification_categories_s3_uri: str, + ground_truth_output_bucket_kms_key_arn: str, + upload_bucket_kms_key_arn: str, +) -> iam.Role: + ground_truth_role = iam.Role( + scope, + "GroundTruthRole", + assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com"), + managed_policies=[ + iam.ManagedPolicy.from_aws_managed_policy_name( + "AmazonSageMakerGroundTruthExecution", + ), + ], + ) + ground_truth_role_s3_get_object_resources = [ + upload_bucket_arn, + f"{upload_bucket_arn}/*", + ground_truth_output_bucket_arn, + f"{ground_truth_output_bucket_arn}/*", + get_s3_arn_from_uri(labeling_categories_s3_uri, partition=scope.partition), + ] + if labeling_instructions_template_s3_uri: + ground_truth_role_s3_get_object_resources.append( + get_s3_arn_from_uri(labeling_instructions_template_s3_uri, partition=scope.partition) + ) + if task_type_config_verification_attribute_name: + ground_truth_role_s3_get_object_resources.extend( + [ + get_s3_arn_from_uri(verification_instructions_template_s3_uri, partition=scope.partition), + get_s3_arn_from_uri(verification_categories_s3_uri, partition=scope.partition), + ] + ) + ground_truth_role.add_to_policy( + iam.PolicyStatement( + resources=ground_truth_role_s3_get_object_resources, + actions=[ + "s3:GetObject", + ], + ) + ) + ground_truth_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_arn, + f"{ground_truth_output_bucket_arn}/*", + ], + actions=[ + "s3:PutObject", + ], + ) + ) + ground_truth_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_kms_key_arn, + ], + actions=[ + "kms:GenerateDataKey", + "kms:Decrypt", + ], + ) + ) + ground_truth_role.add_to_policy( + iam.PolicyStatement( + resources=[ + upload_bucket_kms_key_arn, + ], + actions=[ + "kms:Decrypt", + ], + ) + ) + NagSuppressions.add_resource_suppressions( + ground_truth_role, + [ + { + "id": "AwsSolutions-IAM4", + "reason": "AWS manged policy used for Ground Truth", + } + ], + ) + + return ground_truth_role + + +def create_run_ground_truth_job_lambda_execution_role( + scope: Stack, + ground_truth_output_bucket_kms_key_arn: str, + ground_truth_output_bucket_arn: str, + labeling_job_name: str, + verification_job_name: str, + ground_truth_role_arn: str, +) -> iam.Role: + run_ground_truth_job_lambda_execution_role = iam.Role( + scope, + "GroundTruthJobLambdaExecutionRole", + assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"), + ) + run_ground_truth_job_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + actions=[ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents", + ], + resources=["*"], + ) + ) + run_ground_truth_job_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_kms_key_arn, + ], + actions=["kms:GenerateDataKey", "kms:Decrypt"], + ) + ) + run_ground_truth_job_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_arn, + f"{ground_truth_output_bucket_arn}/*", + ], + actions=[ + "s3:GetObject", + "s3:PutObject", + ], + ) + ) + run_ground_truth_job_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + f"arn:{scope.partition}:sagemaker:{scope.region}:{scope.account}:labeling-job/{labeling_job_name}-*", + f"arn:{scope.partition}:sagemaker:{scope.region}:{scope.account}:labeling-job/{verification_job_name}-*", + ], + actions=[ + "sagemaker:CreateLabelingJob", + ], + ) + ) + run_ground_truth_job_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_role_arn, + ], + actions=[ + "iam:PassRole", + ], + ) + ) + + return run_ground_truth_job_lambda_execution_role + + +def create_run_labeling_job_lambda( + scope: Stack, + job_name: str, + id: str, + run_ground_truth_job_lambda_execution_role: iam.Role, + labeling_job_name: str, + ground_truth_role_arn: str, + ground_truth_output_bucket_name: str, + task_type_config: TaskTypeConfig, + labeling_workteam_arn: str, + labeling_instructions_template_s3_uri: str, + labeling_categories_s3_uri: str, + labeling_task_title: str, + labeling_task_description: str, + labeling_task_keywords: List[str], + labeling_human_task_config: Dict[str, Any], + labeling_task_price: Dict[str, Dict[str, int]], +) -> lambda_.Function: + run_labeling_job_lambda_name_suffix = f"{job_name}-labeling-job-lambda" + run_labeling_job_lambda_name = ( + f"{id[: MAX_LAMBDA_FUNCTION_NAME_LENGTH - len(run_labeling_job_lambda_name_suffix)]}" + f"{run_labeling_job_lambda_name_suffix}" + ) + run_labeling_job_lambda = lambda_.Function( + scope, + "LabelingJobFunction", + runtime=LAMBDA_RUNTIME, + code=lambda_.Code.from_asset(LAMBDA_ASSET_FOLDER), + handler="run_labeling_job.handler", + function_name=run_labeling_job_lambda_name, + memory_size=512, + role=run_ground_truth_job_lambda_execution_role, + timeout=Duration.seconds(15), + environment={ + "AC_ARN_MAP": json.dumps(AC_ARN_MAP), + "TASK_TYPE": task_type_config.task_type, + "SOURCE_KEY": task_type_config.source_key, + "LABELING_JOB_NAME": labeling_job_name, + "GROUND_TRUTH_ROLE_ARN": ground_truth_role_arn, + "OUTPUT_BUCKET": ground_truth_output_bucket_name, + "FUNCTION_NAME": task_type_config.function_name, + "WORKTEAM_ARN": labeling_workteam_arn, + "INSTRUCTIONS_TEMPLATE_S3_URI": labeling_instructions_template_s3_uri, + "LABEL_CATEGORIES_S3_URI": labeling_categories_s3_uri, + "TASK_TITLE": labeling_task_title, + "TASK_DESCRIPTION": labeling_task_description, + "TASK_KEYWORDS": json.dumps(labeling_task_keywords), + "HUMAN_TASK_CONFIG": json.dumps(labeling_human_task_config), + "TASK_PRICE": json.dumps(labeling_task_price), + "LABELING_ATTRIBUTE_NAME": task_type_config.labeling_attribute_name, + }, + ) + + return run_labeling_job_lambda + + +def create_run_verification_job_lambda( + scope: Stack, + task_type_config: TaskTypeConfig, + job_name: str, + id: str, + run_ground_truth_job_lambda_execution_role: iam.Role, + verification_job_name: str, + ground_truth_role_arn: str, + ground_truth_output_bucket_name: str, + verification_workteam_arn: str, + verification_instructions_template_s3_uri: str, + verification_categories_s3_uri: str, + verification_task_title: str, + verification_task_description: str, + verification_task_keywords: List[str], + verification_human_task_config: Dict[str, Any], + verification_task_price: Dict[str, Dict[str, int]], +) -> Optional[lambda_.Function]: + if task_type_config.verification_attribute_name: + run_verification_job_lambda_name_suffix = f"{job_name}-verification-job-lambda" + run_verification_job_lambda_name = ( + f"{id[: MAX_LAMBDA_FUNCTION_NAME_LENGTH - len(run_verification_job_lambda_name_suffix)]}" + f"{run_verification_job_lambda_name_suffix}" + ) + run_verification_job_lambda = lambda_.Function( + scope, + "VerificationJobFunction", + runtime=LAMBDA_RUNTIME, + code=lambda_.Code.from_asset(LAMBDA_ASSET_FOLDER), + handler="run_verification_job.handler", + function_name=run_verification_job_lambda_name, + memory_size=512, + role=run_ground_truth_job_lambda_execution_role, + timeout=Duration.seconds(30), + environment={ + "AC_ARN_MAP": json.dumps(AC_ARN_MAP), + "SOURCE_KEY": task_type_config.source_key, + "VERIFICATION_JOB_NAME": verification_job_name, + "GROUND_TRUTH_ROLE_ARN": ground_truth_role_arn, + "OUTPUT_BUCKET": ground_truth_output_bucket_name, + "FUNCTION_NAME": task_type_config.function_name, + "WORKTEAM_ARN": verification_workteam_arn, + "INSTRUCTIONS_TEMPLATE_S3_URI": verification_instructions_template_s3_uri, + "LABEL_CATEGORIES_S3_URI": verification_categories_s3_uri, + "TASK_TITLE": verification_task_title, + "TASK_DESCRIPTION": verification_task_description, + "TASK_KEYWORDS": json.dumps(verification_task_keywords), + "HUMAN_TASK_CONFIG": json.dumps(verification_human_task_config), + "TASK_PRICE": json.dumps(verification_task_price), + "VERIFICATION_ATTRIBUTE_NAME": task_type_config.verification_attribute_name, + "LABELING_ATTRIBUTE_NAME": task_type_config.labeling_attribute_name, + }, + ) + return run_verification_job_lambda + else: + return None + + +def create_update_feature_store_lambda( + scope: Stack, + ground_truth_output_bucket: s3.Bucket, + ground_truth_output_bucket_kms_key_arn: str, + feature_group_name: str, + queue: sqs.Queue, + task_type_config: TaskTypeConfig, + job_name: str, + id: str, +) -> lambda_.Function: + update_feature_store_lambda_execution_role = iam.Role( + scope, + "UpdateFeatureStoreLambdaExecutionRole", + assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"), + ) + update_feature_store_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + actions=[ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents", + ], + resources=["*"], + ) + ) + update_feature_store_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket.bucket_arn, + f"{ground_truth_output_bucket.bucket_arn}/*", + ], + actions=[ + "s3:GetObject", + "s3:PutObject", + ], + ) + ) + update_feature_store_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_kms_key_arn, + ], + actions=[ + "kms:GenerateDataKey", + "kms:Decrypt", + ], + ) + ) + update_feature_store_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + f"arn:{scope.partition}:sagemaker:{scope.region}:{scope.account}:feature-group/{feature_group_name}", + ], + actions=[ + "sagemaker:PutRecord", + ], + ) + ) + update_feature_store_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + queue.queue_arn, + ], + actions=[ + "sqs:DeleteMessage", + ], + ) + ) + + feature_group_definitions = { + definition.feature_name: definition.output_key for definition in task_type_config.feature_group_config + } + + update_feature_store_lambda_name_suffix = f"{job_name}-update-feature-store-lambda" + update_feature_store_lambda_name = ( + f"{id[: MAX_LAMBDA_FUNCTION_NAME_LENGTH - len(update_feature_store_lambda_name_suffix)]}" + f"{update_feature_store_lambda_name_suffix}" + ) + update_feature_store_lambda_environment = { + "FEATURE_GROUP_NAME": feature_group_name, + "FEATURE_GROUP_DEFINITIONS": json.dumps(feature_group_definitions), + "SQS_QUEUE_URL": queue.queue_url, + "OUTPUT_BUCKET": ground_truth_output_bucket.bucket_name, + "LABELING_ATTRIBUTE_NAME": task_type_config.labeling_attribute_name, + "TASK_MEDIA_TYPE": task_type_config.media_type, + "SOURCE_KEY": task_type_config.source_key, + } + if task_type_config.verification_attribute_name: + update_feature_store_lambda_environment["VERIFICATION_ATTRIBUTE_NAME"] = ( + task_type_config.verification_attribute_name + ) + update_feature_store_lambda = lambda_.Function( + scope, + "UpdateFeatureStoreFunction", + runtime=LAMBDA_RUNTIME, + code=lambda_.Code.from_asset(LAMBDA_ASSET_FOLDER), + handler="update_feature_store.handler", + function_name=update_feature_store_lambda_name, + memory_size=512, + role=update_feature_store_lambda_execution_role, + timeout=Duration.seconds(900), + environment=update_feature_store_lambda_environment, + ) + + return update_feature_store_lambda + + +def create_return_messages_to_sqs_queue_lambda( + scope: Stack, + queue: sqs.Queue, + ground_truth_output_bucket: s3.Bucket, + ground_truth_output_bucket_kms_key_arn: str, + job_name: str, + task_media_type: str, + id: str, +) -> lambda_.Function: + return_messages_to_sqs_queue_lambda_execution_role = iam.Role( + scope, + "ReturnMessagesToSqsQueueLambdaExecutionRole", + assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"), + ) + return_messages_to_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + actions=[ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents", + ], + resources=["*"], + ) + ) + return_messages_to_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + queue.queue_arn, + ], + actions=[ + "sqs:ChangeMessageVisibility", + ], + ) + ) + return_messages_to_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket.bucket_arn, + f"{ground_truth_output_bucket.bucket_arn}/*", + ], + actions=[ + "s3:GetObject", + ], + ) + ) + return_messages_to_sqs_queue_lambda_execution_role.add_to_policy( + iam.PolicyStatement( + resources=[ + ground_truth_output_bucket_kms_key_arn, + ], + actions=[ + "kms:Decrypt", + ], + ) + ) + + return_messages_to_sqs_queue_lambda_name_suffix = f"{job_name}-return-messages-to-sqs-queue-lambda" + return_messages_to_sqs_queue_lambda_name = ( + f"{id[: MAX_LAMBDA_FUNCTION_NAME_LENGTH - len(return_messages_to_sqs_queue_lambda_name_suffix)]}" + f"{return_messages_to_sqs_queue_lambda_name_suffix}" + ) + return_messages_to_sqs_queue_lambda = lambda_.Function( + scope, + "ReturnMessagesToSqsQueueFunction", + runtime=LAMBDA_RUNTIME, + code=lambda_.Code.from_asset(LAMBDA_ASSET_FOLDER), + handler="return_messages_to_sqs_queue.handler", + function_name=return_messages_to_sqs_queue_lambda_name, + memory_size=512, + role=return_messages_to_sqs_queue_lambda_execution_role, + timeout=Duration.seconds(600), + environment={ + "SQS_QUEUE_URL": queue.queue_url, + "OUTPUT_BUCKET": ground_truth_output_bucket.bucket_name, + "TASK_MEDIA_TYPE": task_media_type, + }, + ) + + return return_messages_to_sqs_queue_lambda + + +def create_state_machine( + scope: Stack, + id: str, + job_name: str, + poll_sqs_queue_lambda: lambda_.Function, + run_labeling_job_lambda: lambda_.Function, + run_verification_job_lambda: Optional[lambda_.Function], + update_feature_store_lambda: lambda_.Function, + return_messages_to_sqs_queue_lambda: lambda_.Function, + labeling_job_name: str, + verification_job_name: str, +) -> sfn.StateMachine: + state_machine_log_group = logs.LogGroup( + scope, + "StateMachineLogGroup", + log_group_name=f"{id}-state-machine", + retention=logs.RetentionDays.ONE_MONTH, + removal_policy=RemovalPolicy.DESTROY, + ) + + state_machine_name_suffix = f"-{job_name}-state-machine" + state_machine_name = ( + f"{id[: MAX_STATE_MACHINE_NAME_LENGTH - len(state_machine_name_suffix)]}{state_machine_name_suffix}" + ) + state_machine = sfn.StateMachine( + scope, + "LabelingStateMachine", + state_machine_name=state_machine_name, + definition=get_state_machine_definition( + scope, + poll_sqs_queue_lambda=poll_sqs_queue_lambda, + run_labeling_job_lambda=run_labeling_job_lambda, + run_verification_job_lambda=run_verification_job_lambda, + update_feature_store_lambda=update_feature_store_lambda, + return_messages_to_sqs_queue_lambda=return_messages_to_sqs_queue_lambda, + labeling_job_name=labeling_job_name, + verification_job_name=verification_job_name, + ), + logs=sfn.LogOptions(destination=state_machine_log_group, level=sfn.LogLevel.ALL), + tracing_enabled=True, + ) + + return state_machine + + +def get_state_machine_definition( + scope: Stack, + poll_sqs_queue_lambda: lambda_.Function, + run_labeling_job_lambda: lambda_.Function, + run_verification_job_lambda: Optional[lambda_.Function], + update_feature_store_lambda: lambda_.Function, + return_messages_to_sqs_queue_lambda: lambda_.Function, + labeling_job_name: str, + verification_job_name: str, +) -> sfn.Chain: + success = sfn.Succeed(scope, "Labeling Pipeline execution succeeded") + fail = sfn.Fail(scope, "Labeling Pipeline execution failed") + + poll_sqs_queue = tasks.LambdaInvoke( + scope, + "PollSqsQueue", + lambda_function=poll_sqs_queue_lambda, + payload=sfn.TaskInput.from_object( + { + "ExecutionId": sfn.JsonPath.string_at("$$.Execution.Id"), + } + ), + result_path="$.Output", + result_selector={ + "MessagesCount.$": "$.Payload.MessagesCount", + "RecordSourceToReceiptHandleS3Key.$": "$.Payload.RecordSourceToReceiptHandleS3Key", + }, + ) + + run_labeling_job = tasks.LambdaInvoke( + scope, + "RunLabelingJob", + lambda_function=run_labeling_job_lambda, + payload=sfn.TaskInput.from_object( + { + "ExecutionId": sfn.JsonPath.string_at("$$.Execution.Id"), + "RecordSourceToReceiptHandleS3Key": sfn.JsonPath.string_at("$.Output.RecordSourceToReceiptHandleS3Key"), + } + ), + result_path="$.Output.RunLabelingJobOutput", + result_selector={ + "LabelingJobName.$": "$.Payload.LabelingJobName", + }, + ) + + run_verification_job = None + return_unlabeled_messages_to_sqs_queue = None + if run_verification_job_lambda: + run_verification_job = tasks.LambdaInvoke( + scope, + "RunVerificationJob", + lambda_function=run_verification_job_lambda, + payload=sfn.TaskInput.from_object( + { + "ExecutionId": sfn.JsonPath.string_at("$$.Execution.Id"), + "LabelingJobOutputUri": sfn.JsonPath.string_at( + "$.DescribeLabelingJobOutput.LabelingJobOutput.OutputDatasetS3Uri" + ), + "RecordSourceToReceiptHandleS3Key": sfn.JsonPath.string_at( + "$.Output.RecordSourceToReceiptHandleS3Key" + ), + } + ), + result_path="$.Output", + result_selector={ + "RunLabelingJobOutput": { + "LabelingJobName.$": "$.Payload.LabelingJobName", + }, + "UnlabeledRecordSourceToReceiptHandleS3Key.$": "$.Payload.UnlabeledRecordSourceToReceiptHandleS3Key", + "RecordSourceToReceiptHandleS3Key.$": "$.Payload.RecordSourceToReceiptHandleS3Key", + }, + ) + + return_unlabeled_messages_to_sqs_queue = tasks.LambdaInvoke( + scope, + "ReturnUnlabeledMessagesToSqsQueue", + lambda_function=return_messages_to_sqs_queue_lambda, + payload=sfn.TaskInput.from_object( + { + "RecordSourceToReceiptHandleS3Key": sfn.JsonPath.string_at( + "$.Output.UnlabeledRecordSourceToReceiptHandleS3Key" + ), + } + ), + result_path=sfn.JsonPath.DISCARD, + ) + + update_feature_store = tasks.LambdaInvoke( + scope, + "UpdateFeatureStore", + lambda_function=update_feature_store_lambda, + payload=sfn.TaskInput.from_object( + { + "ExecutionId": sfn.JsonPath.string_at("$$.Execution.Id"), + "LabelingJobOutputUri": sfn.JsonPath.string_at( + "$.DescribeLabelingJobOutput.LabelingJobOutput.OutputDatasetS3Uri" + ), + "RecordSourceToReceiptHandleS3Key": sfn.JsonPath.string_at("$.Output.RecordSourceToReceiptHandleS3Key"), + } + ), + result_path="$.Output", + result_selector={ + "RejectedLabelsRecordSourceToReceiptHandleS3Key.$": ( + "$.Payload.RejectedLabelsRecordSourceToReceiptHandleS3Key" + ), + }, + ) + + # invoke multiple times as we can't have multiple next steps + return_messages_to_sqs_queue_on_failure = tasks.LambdaInvoke( + scope, + "ReturnMessagesToSqsQueueOnFailure", + lambda_function=return_messages_to_sqs_queue_lambda, + payload=sfn.TaskInput.from_object( + { + "RecordSourceToReceiptHandleS3Key": sfn.JsonPath.string_at("$.Output.RecordSourceToReceiptHandleS3Key"), + } + ), + ) + return_messages_to_sqs_queue_on_failure.next(fail) + + return_remaining_label_messages_to_sqs_queue = tasks.LambdaInvoke( + scope, + "ReturnRemainingLabelMessagesToSqsQueue", + lambda_function=return_messages_to_sqs_queue_lambda, + payload=sfn.TaskInput.from_object( + { + "RecordSourceToReceiptHandleS3Key": sfn.JsonPath.string_at( + "$.Output.RejectedLabelsRecordSourceToReceiptHandleS3Key" + ), + } + ), + ) + + post_labeling_job_step = update_feature_store.next(return_remaining_label_messages_to_sqs_queue.next(success)) + + if run_verification_job and return_unlabeled_messages_to_sqs_queue: + post_labeling_job_step = run_verification_job.next( + return_unlabeled_messages_to_sqs_queue.next( + create_labeling_job_waiter( + scope, + verification_job_name, + return_messages_to_sqs_queue_on_failure, + post_labeling_job_step, + ) + ) + ) + + definition = poll_sqs_queue.next( + sfn.Choice(scope, "New messages?") + .when( + sfn.Condition.number_equals("$.Output.MessagesCount", 0), + success, + ) + .otherwise( + run_labeling_job.next( + create_labeling_job_waiter( + scope, + labeling_job_name, + return_messages_to_sqs_queue_on_failure, + post_labeling_job_step, + ) + ) + ) + ) + + return definition + + +def create_labeling_job_waiter( + scope: Stack, + labeling_job_name: str, + fail: sfn.IChainable, + next: sfn.IChainable, +) -> sfn.Chain: + get_labeling_job_status = tasks.CallAwsService( + scope, + f"Get {labeling_job_name} status", + service="sagemaker", + action="describeLabelingJob", + parameters={"LabelingJobName": sfn.JsonPath.string_at("$.Output.RunLabelingJobOutput.LabelingJobName")}, + iam_resources=[ + f"arn:{scope.partition}:sagemaker:{scope.region}:{scope.account}:labeling-job/{labeling_job_name}-*", + ], + result_path="$.DescribeLabelingJobOutput", + ) + + wait_x = sfn.Wait( + scope, + f"Waiting for - {labeling_job_name} - completion", + time=sfn.WaitTime.duration(Duration.seconds(30)), + ) + + return wait_x.next(get_labeling_job_status).next( + sfn.Choice(scope, f"{labeling_job_name} Complete?") + .when( + sfn.Condition.string_equals("$.DescribeLabelingJobOutput.LabelingJobStatus", "Failed"), + fail, + ) + .when( + sfn.Condition.string_equals("$.DescribeLabelingJobOutput.LabelingJobStatus", "Stopped"), + fail, + ) + .when( + sfn.Condition.and_( + sfn.Condition.string_equals("$.DescribeLabelingJobOutput.LabelingJobStatus", "Completed"), + sfn.Condition.number_equals("$.DescribeLabelingJobOutput.LabelCounters.TotalLabeled", 0), + ), + fail, + ) + .when( + sfn.Condition.and_( + sfn.Condition.string_equals("$.DescribeLabelingJobOutput.LabelingJobStatus", "Completed"), + sfn.Condition.number_greater_than("$.DescribeLabelingJobOutput.LabelCounters.TotalLabeled", 0), + ), + next, + ) + .otherwise(wait_x) + ) + + +def create_labeling_workflow_schedule(scope: Stack, labeling_workflow_schedule: str, state_machine_arn: str) -> None: + if labeling_workflow_schedule: + scheduler_role = iam.Role( + scope, + "SchedulerRole", + assumed_by=iam.ServicePrincipal("scheduler.amazonaws.com"), + ) + scheduler_role.add_to_policy( + iam.PolicyStatement( + resources=[ + state_machine_arn, + ], + actions=[ + "states:StartExecution", + ], + ) + ) + scheduler.CfnSchedule( + scope, + "LabelingStateMachineSchedule", + flexible_time_window=scheduler.CfnSchedule.FlexibleTimeWindowProperty(mode="OFF"), + schedule_expression=labeling_workflow_schedule, + target=scheduler.CfnSchedule.TargetProperty( + arn=state_machine_arn, + role_arn=scheduler_role.role_arn, + ), + ) + + +def get_s3_arn_from_uri(uri: str, partition: str) -> str: + if not uri: + return "" + bucket_name = uri.split("/")[2] + object_key = "/".join(uri.split("/")[3:]) + return f"arn:{partition}:s3:::{bucket_name}/{object_key}" diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/_utils.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/_utils.py new file mode 100644 index 00000000..70c2cfe4 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/_utils.py @@ -0,0 +1,100 @@ +import json +from typing import Any, Dict, List, Optional +from urllib.parse import urlparse + +import boto3 + +s3 = boto3.client("s3") +sagemaker = boto3.client("sagemaker") + +IMAGE = "image" +TEXT = "text" + + +def upload_json_to_s3(json_data: Dict[str, str], bucket: str, prefix: str, filename: str) -> str: + key = f"{prefix}/{filename}" + s3.put_object(Body=json.dumps(json_data), Bucket=bucket, Key=key) + return key + + +def download_json_dict_from_s3(s3_key: str, bucket: str) -> Dict[str, str]: + response = s3.get_object(Bucket=bucket, Key=s3_key) + json_data = json.loads(response["Body"].read().decode("utf-8")) + + if not isinstance(json_data, dict): + raise ValueError("The JSON data is not a dictionary") + + str_dict: Dict[str, str] = {key: str(value) for key, value in json_data.items()} + + return str_dict + + +def create_labeling_job( + human_task_config: Dict[str, Any], + prehuman_arn: str, + acs_arn: str, + task_title: str, + task_description: str, + task_keywords: List[str], + workteam_arn: str, + task_price: Dict[str, Dict[str, int]], + manifest_uri: str, + output_uri: str, + job_name: str, + ground_truth_role_arn: str, + label_attribute_name: str, + label_categories_s3_uri: str, + instructions_template_s3_uri: Optional[str] = None, + human_task_ui_arn: Optional[str] = None, +) -> None: + human_task_config = human_task_config.copy() + human_task_config.update( + { + "PreHumanTaskLambdaArn": prehuman_arn, + "AnnotationConsolidationConfig": { + "AnnotationConsolidationLambdaArn": acs_arn, + }, + "TaskTitle": task_title, + "TaskDescription": task_description, + "TaskKeywords": task_keywords, + "WorkteamArn": workteam_arn, + } + ) + + if human_task_ui_arn: + ui_config = {"HumanTaskUiArn": human_task_ui_arn} + elif instructions_template_s3_uri: + ui_config = {"UiTemplateS3Uri": instructions_template_s3_uri} + else: + raise ValueError("Either human_task_ui_arn or instructions_template_s3_uri must be provided") + human_task_config["UiConfig"] = ui_config + + if task_price: + human_task_config["PublicWorkforceTaskPrice"] = task_price + + sagemaker.create_labeling_job( + InputConfig={ + "DataSource": {"S3DataSource": {"ManifestS3Uri": manifest_uri}}, + "DataAttributes": { + "ContentClassifiers": [ + "FreeOfPersonallyIdentifiableInformation", + "FreeOfAdultContent", + ] + }, + }, + OutputConfig={"S3OutputPath": output_uri}, + HumanTaskConfig=human_task_config, + LabelingJobName=job_name, + RoleArn=ground_truth_role_arn, + LabelAttributeName=label_attribute_name, + LabelCategoryConfigS3Uri=label_categories_s3_uri, + ) + + +def get_s3_string_object_from_uri(s3_uri: str) -> str: + parsed_url = urlparse(s3_uri, allow_fragments=False) + bucket = parsed_url.netloc + key = parsed_url.path.lstrip("/") + response = s3.get_object(Bucket=bucket, Key=key) + object = response["Body"].read().decode("utf-8") + return str(object) diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/poll_sqs_queue.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/poll_sqs_queue.py new file mode 100644 index 00000000..e51789bd --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/poll_sqs_queue.py @@ -0,0 +1,132 @@ +import json +import logging +import os +from typing import Any, Dict, List + +import boto3 +from _utils import IMAGE, TEXT, upload_json_to_s3 + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +sqs = boto3.client("sqs") + +SQS_QUEUE_URL = os.environ["SQS_QUEUE_URL"] +OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"] +TASK_TYPE = os.environ["TASK_TYPE"] +TASK_MEDIA_TYPE = os.environ["TASK_MEDIA_TYPE"] + +MAX_BATCH_SIZE = 10 +MAX_LONG_POLLING_DURATION = 20 + +MAX_NUMBER_OF_LABELING_ITEMS = 100000 +MAX_NUMBER_OF_LABELING_ITEMS_SEMANTIC_SEGMENTATION = 20000 + + +def handler(event: Dict[str, Any], context: object) -> Dict[str, str]: + item_limit = ( + MAX_NUMBER_OF_LABELING_ITEMS_SEMANTIC_SEGMENTATION + if TASK_TYPE == "semantic_segmentation" + else MAX_NUMBER_OF_LABELING_ITEMS + ) + logger.info(f"Item limit set: {item_limit}") + + record_source_to_receipt_handle = poll_sqs_queue( + sqs_queue_url=SQS_QUEUE_URL, + item_limit=item_limit, + messages_per_batch=MAX_BATCH_SIZE, + wait_time_seconds=MAX_LONG_POLLING_DURATION, + task_media_type=TASK_MEDIA_TYPE, + ) + + execution_id = event["ExecutionId"].rsplit(":", 1)[-1] + record_source_to_receipt_handle_s3_key = upload_json_to_s3( + json_data=record_source_to_receipt_handle, + bucket=OUTPUT_BUCKET, + prefix=f"runs/{execution_id}", + filename="record_source_to_receipt_handle.json", + ) + logger.info("Uploaded record source to receipt handles to S3") + + return { + "MessagesCount": str(len(record_source_to_receipt_handle)), + "RecordSourceToReceiptHandleS3Key": record_source_to_receipt_handle_s3_key, + } + + +def poll_sqs_queue( + sqs_queue_url: str, + item_limit: int, + messages_per_batch: int, + wait_time_seconds: int, + task_media_type: str, +) -> Dict[str, str]: + receipt_handles_of_duplicates: List[str] = [] + record_source_to_receipt_handle: Dict[str, str] = {} + + while True: + if len(record_source_to_receipt_handle) >= item_limit: + break + if len(record_source_to_receipt_handle) + messages_per_batch > item_limit: + messages_per_batch = item_limit - len(record_source_to_receipt_handle) + response = sqs.receive_message( + QueueUrl=sqs_queue_url, + MaxNumberOfMessages=messages_per_batch, + WaitTimeSeconds=wait_time_seconds, + ) + + if not response.get("Messages"): + break + + for message in response["Messages"]: + try: + if task_media_type == IMAGE: + body = json.loads(message["Body"]) + s3_event = body["Records"][0]["s3"] + bucket_name = s3_event["bucket"]["name"] + object_key = s3_event["object"]["key"] + record_source = f"s3://{bucket_name}/{object_key}" + elif task_media_type == TEXT: + record_source = message["Body"] + else: + raise ValueError(f"Unsupported task media type: {task_media_type}") + + if record_source in record_source_to_receipt_handle: + receipt_handles_of_duplicates.append(message["ReceiptHandle"]) + else: + record_source_to_receipt_handle[record_source] = message["ReceiptHandle"] + + except (KeyError, json.JSONDecodeError, IndexError) as e: + logger.error(f"Error processing message: {str(e)}") + + if receipt_handles_of_duplicates: + delete_duplicate_record_source_messages( + sqs_queue_url=sqs_queue_url, + messages_per_batch=messages_per_batch, + receipt_handles_of_duplicates=receipt_handles_of_duplicates, + ) + + logger.info(f"Total messages received: {len(record_source_to_receipt_handle)}") + + return record_source_to_receipt_handle + + +def delete_duplicate_record_source_messages( + sqs_queue_url: str, + messages_per_batch: int, + receipt_handles_of_duplicates: List[str], +) -> None: + logger.info(f"Deleting {len(receipt_handles_of_duplicates)} duplicate messages from the queue.") + + receipt_handles_batches = [ + receipt_handles_of_duplicates[i : i + messages_per_batch] + for i in range(0, len(receipt_handles_of_duplicates), messages_per_batch) + ] + for batch in receipt_handles_batches: + try: + sqs.delete_message_batch( + QueueUrl=sqs_queue_url, + Entries=[{"Id": str(i), "ReceiptHandle": receipt_handle} for i, receipt_handle in enumerate(batch)], + ) + except Exception as e: + logger.error(f"Error deleting messages: {str(e)}") diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/return_messages_to_sqs_queue.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/return_messages_to_sqs_queue.py new file mode 100644 index 00000000..a2ea83a1 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/return_messages_to_sqs_queue.py @@ -0,0 +1,69 @@ +import logging +import os +from typing import Any, Dict, List + +import boto3 +from _utils import IMAGE, TEXT, download_json_dict_from_s3 + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +sqs = boto3.client("sqs") + +QUEUE_URL = os.environ["SQS_QUEUE_URL"] +OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"] +TASK_MEDIA_TYPE = os.environ["TASK_MEDIA_TYPE"] + +BATCH_SIZE_LIMIT = 10 + + +def handler(event: Dict[str, Any], context: object) -> None: + record_source_to_receipt_handle_s3_key = event["RecordSourceToReceiptHandleS3Key"] + record_source_to_receipt_handle = download_json_dict_from_s3( + s3_key=record_source_to_receipt_handle_s3_key, bucket=OUTPUT_BUCKET + ) + logger.info( + f"Downloaded record source to receipt handles from S3, " + f"{len(record_source_to_receipt_handle)} messages to return to SQS queue" + ) + + batches = split_dict_into_batches(data=record_source_to_receipt_handle, batch_size=BATCH_SIZE_LIMIT) + + logger.info("Starting to return messages to SQS queue") + + for batch in batches: + change_visibility_batch(queue_url=QUEUE_URL, batch=batch, task_media_type=TASK_MEDIA_TYPE) + + logger.info("Finished returning messages to SQS queue") + + +def split_dict_into_batches(data: Dict[str, str], batch_size: int) -> List[Dict[str, str]]: + items = list(data.items()) + return [dict(items[i : i + batch_size]) for i in range(0, len(items), batch_size)] + + +def change_visibility_batch(queue_url: str, batch: Dict[str, str], task_media_type: str) -> None: + entries = [] + for i, (record_source, receipt_handle) in enumerate(batch.items()): + if task_media_type == IMAGE: + filename = record_source.split("/")[-1] + # id request can only contain alphanumeric, hyphen and underscores + id = filename.replace(".", "_") + elif task_media_type == TEXT: + id = str(i) + else: + raise ValueError(f"Unsupported task media type: {task_media_type}") + + entries.append( + { + "Id": id, + "ReceiptHandle": receipt_handle, + "VisibilityTimeout": 0, + } + ) + + response = sqs.change_message_visibility_batch(QueueUrl=queue_url, Entries=entries) + + if "Failed" in response: + for failed in response["Failed"]: + logger.error(f"Failed to return message: {failed['Id']}") diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/run_labeling_job.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/run_labeling_job.py new file mode 100644 index 00000000..05597852 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/run_labeling_job.py @@ -0,0 +1,97 @@ +import json +import logging +import os +from typing import Any, Dict + +import boto3 +from _utils import create_labeling_job, download_json_dict_from_s3 + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +s3 = boto3.client("s3") + +OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"] +TASK_TYPE = os.environ["TASK_TYPE"] +SOURCE_KEY = os.environ["SOURCE_KEY"] +LABELING_JOB_NAME = os.environ["LABELING_JOB_NAME"] +HUMAN_TASK_CONFIG = json.loads(os.environ["HUMAN_TASK_CONFIG"]) +AWS_REGION = os.environ["AWS_REGION"] +AC_ARN_MAP = json.loads(os.environ["AC_ARN_MAP"]) +FUNCTION_NAME = os.environ["FUNCTION_NAME"] +TASK_TITLE = os.environ["TASK_TITLE"] +TASK_DESCRIPTION = os.environ["TASK_DESCRIPTION"] +TASK_KEYWORDS = json.loads(os.environ["TASK_KEYWORDS"]) +WORKTEAM_ARN = os.environ["WORKTEAM_ARN"] +TASK_PRICE = json.loads(os.environ["TASK_PRICE"]) +INSTRUCTIONS_TEMPLATE_S3_URI = os.getenv("INSTRUCTIONS_TEMPLATE_S3_URI", None) +GROUND_TRUTH_ROLE_ARN = os.environ["GROUND_TRUTH_ROLE_ARN"] +LABEL_CATEGORIES_S3_URI = os.environ["LABEL_CATEGORIES_S3_URI"] +LABELING_ATTRIBUTE_NAME = os.environ["LABELING_ATTRIBUTE_NAME"] + + +def handler(event: Dict[str, Any], context: object) -> Dict[str, str]: + record_source_to_receipt_handle_s3_key = event["RecordSourceToReceiptHandleS3Key"] + record_source_to_receipt_handle = download_json_dict_from_s3( + s3_key=record_source_to_receipt_handle_s3_key, + bucket=OUTPUT_BUCKET, + ) + logger.info( + f"Downloaded record source to receipt handles from S3, {len(record_source_to_receipt_handle)} items to label" + ) + + execution_id = event["ExecutionId"].rsplit(":", 1)[-1] + manifest_uri = create_and_upload_manifest( + record_source_to_receipt_handle=record_source_to_receipt_handle, + bucket=OUTPUT_BUCKET, + prefix=f"runs/{execution_id}", + ) + + output_uri = f"s3://{OUTPUT_BUCKET}/runs/{execution_id}/" + job_name = f"{LABELING_JOB_NAME}-{execution_id}" + prehuman_arn = f"arn:aws:lambda:{AWS_REGION}:{AC_ARN_MAP[AWS_REGION]}:function:PRE-{FUNCTION_NAME}" + acs_arn = f"arn:aws:lambda:{AWS_REGION}:{AC_ARN_MAP[AWS_REGION]}:function:ACS-{FUNCTION_NAME}" + args = { + "human_task_config": HUMAN_TASK_CONFIG, + "prehuman_arn": prehuman_arn, + "acs_arn": acs_arn, + "task_title": TASK_TITLE, + "task_description": TASK_DESCRIPTION, + "task_keywords": TASK_KEYWORDS, + "workteam_arn": WORKTEAM_ARN, + "task_price": TASK_PRICE, + "manifest_uri": manifest_uri, + "output_uri": output_uri, + "job_name": job_name, + "ground_truth_role_arn": GROUND_TRUTH_ROLE_ARN, + "label_attribute_name": LABELING_ATTRIBUTE_NAME, + "label_categories_s3_uri": LABEL_CATEGORIES_S3_URI, + } + if TASK_TYPE == "named_entity_recognition": + args["human_task_ui_arn"] = f"arn:aws:sagemaker:{AWS_REGION}:394669845002:human-task-ui/NamedEntityRecognition" + else: + args["instructions_template_s3_uri"] = INSTRUCTIONS_TEMPLATE_S3_URI + + create_labeling_job(**args) + logger.info("Created labeling job") + + return {"LabelingJobName": job_name} + + +def create_and_upload_manifest( + record_source_to_receipt_handle: Dict[str, str], + bucket: str, + prefix: str, +) -> str: + logger.info("Creating manifest") + + manifest_name = "labeling.manifest" + tmp_file = f"/tmp/{manifest_name}" + with open(tmp_file, "w") as f: + for record_source in record_source_to_receipt_handle: + line = f'{{"{SOURCE_KEY}": "{record_source}"}}\n' + f.write(line) + s3.upload_file(Filename=tmp_file, Bucket=bucket, Key=f"{prefix}/{manifest_name}") + logger.info("Uploaded manifest to S3") + + return f"s3://{bucket}/{prefix}/{manifest_name}" diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/run_verification_job.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/run_verification_job.py new file mode 100644 index 00000000..108869ae --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/run_verification_job.py @@ -0,0 +1,155 @@ +import json +import logging +import os +from typing import Any, Dict, List, Tuple + +import boto3 +from _utils import ( + create_labeling_job, + download_json_dict_from_s3, + get_s3_string_object_from_uri, + upload_json_to_s3, +) + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +s3 = boto3.client("s3") + +OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"] +SOURCE_KEY = os.environ["SOURCE_KEY"] +VERIFICATION_JOB_NAME = os.environ["VERIFICATION_JOB_NAME"] +HUMAN_TASK_CONFIG = json.loads(os.environ["HUMAN_TASK_CONFIG"]) +AWS_REGION = os.environ["AWS_REGION"] +AC_ARN_MAP = json.loads(os.environ["AC_ARN_MAP"]) +FUNCTION_NAME = os.environ["FUNCTION_NAME"] +TASK_TITLE = os.environ["TASK_TITLE"] +TASK_DESCRIPTION = os.environ["TASK_DESCRIPTION"] +TASK_KEYWORDS = json.loads(os.environ["TASK_KEYWORDS"]) +WORKTEAM_ARN = os.environ["WORKTEAM_ARN"] +TASK_PRICE = json.loads(os.environ["TASK_PRICE"]) +INSTRUCTIONS_TEMPLATE_S3_URI = os.environ["INSTRUCTIONS_TEMPLATE_S3_URI"] +GROUND_TRUTH_ROLE_ARN = os.environ["GROUND_TRUTH_ROLE_ARN"] +LABEL_CATEGORIES_S3_URI = os.environ["LABEL_CATEGORIES_S3_URI"] +VERIFICATION_ATTRIBUTE_NAME = os.environ["VERIFICATION_ATTRIBUTE_NAME"] +LABELING_ATTRIBUTE_NAME = os.environ["LABELING_ATTRIBUTE_NAME"] + + +def handler(event: Dict[str, Any], context: object) -> Dict[str, str]: + record_source_to_receipt_handle_s3_key = event["RecordSourceToReceiptHandleS3Key"] + record_source_to_receipt_handle = download_json_dict_from_s3( + s3_key=record_source_to_receipt_handle_s3_key, + bucket=OUTPUT_BUCKET, + ) + logger.info("Downloaded record source to receipt handles from S3") + + labeling_job_output_uri = event["LabelingJobOutputUri"] + labeling_job_output = get_s3_string_object_from_uri(s3_uri=labeling_job_output_uri) + logger.info("Downloaded labeling job output from S3") + + ( + labeled_records, + unlabeled_record_source_to_receipt_handle, + record_source_to_receipt_handle, + ) = parse_labeling_job_output( + labeling_job_output=labeling_job_output, + record_source_to_receipt_handle=record_source_to_receipt_handle, + ) + + execution_id = event["ExecutionId"].rsplit(":", 1)[-1] + manifest_uri = create_and_upload_manifest( + labeled_records=labeled_records, + bucket=OUTPUT_BUCKET, + prefix=f"runs/{execution_id}", + ) + + output_uri = f"s3://{OUTPUT_BUCKET}/runs/{execution_id}/" + job_name = f"{VERIFICATION_JOB_NAME}-{execution_id}" + prehuman_arn = f"arn:aws:lambda:{AWS_REGION}:{AC_ARN_MAP[AWS_REGION]}:function:PRE-Verification{FUNCTION_NAME}" + acs_arn = f"arn:aws:lambda:{AWS_REGION}:{AC_ARN_MAP[AWS_REGION]}:function:ACS-Verification{FUNCTION_NAME}" + create_labeling_job( + human_task_config=HUMAN_TASK_CONFIG, + prehuman_arn=prehuman_arn, + acs_arn=acs_arn, + task_title=TASK_TITLE, + task_description=TASK_DESCRIPTION, + task_keywords=TASK_KEYWORDS, + workteam_arn=WORKTEAM_ARN, + task_price=TASK_PRICE, + instructions_template_s3_uri=INSTRUCTIONS_TEMPLATE_S3_URI, + manifest_uri=manifest_uri, + output_uri=output_uri, + job_name=job_name, + ground_truth_role_arn=GROUND_TRUTH_ROLE_ARN, + label_attribute_name=VERIFICATION_ATTRIBUTE_NAME, + label_categories_s3_uri=LABEL_CATEGORIES_S3_URI, + ) + logger.info("Created verification labeling job") + + unlabeled_record_source_to_receipt_handle_s3_key = upload_json_to_s3( + json_data=unlabeled_record_source_to_receipt_handle, + bucket=OUTPUT_BUCKET, + prefix=f"runs/{execution_id}", + filename="unlabeled_record_source_to_receipt_handle.json", + ) + logger.info("Uploaded unlabeled record source to receipt handles to S3") + + record_source_to_receipt_handle_s3_key = upload_json_to_s3( + json_data=record_source_to_receipt_handle, + bucket=OUTPUT_BUCKET, + prefix=f"runs/{execution_id}", + filename="record_source_to_receipt_handle.json", + ) + logger.info("Uploaded record source to receipt handles to S3") + + return { + "LabelingJobName": job_name, + "UnlabeledRecordSourceToReceiptHandleS3Key": unlabeled_record_source_to_receipt_handle_s3_key, + "RecordSourceToReceiptHandleS3Key": record_source_to_receipt_handle_s3_key, + } + + +def parse_labeling_job_output( + labeling_job_output: str, + record_source_to_receipt_handle: Dict[str, str], +) -> Tuple[List[str], Dict[str, str], Dict[str, str]]: + logger.info("Parsing labeling job output") + + unlabeled_record_source_to_receipt_handle = {} + labeled_records = [] + for line in labeling_job_output.splitlines(): + item = json.loads(line) + if "failure-reason" in item[f"{LABELING_ATTRIBUTE_NAME}-metadata"]: + record_source = item[SOURCE_KEY] + receipt_handle = record_source_to_receipt_handle.pop(record_source) + unlabeled_record_source_to_receipt_handle[record_source] = receipt_handle + else: + labeled_records.append(line) + + logger.info( + f"Parsed labeling job output. {len(labeled_records)} labeled records, " + f"{len(unlabeled_record_source_to_receipt_handle)} unlabeled records." + ) + return ( + labeled_records, + unlabeled_record_source_to_receipt_handle, + record_source_to_receipt_handle, + ) + + +def create_and_upload_manifest( + labeled_records: List[str], + bucket: str, + prefix: str, +) -> str: + logger.info("Creating manifest") + + manifest_name = "verification.manifest" + tmp_file = f"/tmp/{manifest_name}" + with open(tmp_file, "w") as f: + for line in labeled_records: + f.write(f"{line}\n") + s3.upload_file(Filename=tmp_file, Bucket=bucket, Key=f"{prefix}/{manifest_name}") + logger.info("Uploaded manifest to S3") + + return f"s3://{bucket}/{prefix}/{manifest_name}" diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/update_feature_store.py b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/update_feature_store.py new file mode 100644 index 00000000..baa74de2 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/labeling_step_function/lambda/update_feature_store.py @@ -0,0 +1,242 @@ +import concurrent.futures +import datetime +import json +import logging +import operator +import os +from functools import reduce +from typing import Any, Dict, List, Union + +import boto3 +from _utils import ( + IMAGE, + TEXT, + download_json_dict_from_s3, + get_s3_string_object_from_uri, + upload_json_to_s3, +) +from botocore.config import Config + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"] +TASK_MEDIA_TYPE = os.environ["TASK_MEDIA_TYPE"] +SOURCE_KEY = os.environ["SOURCE_KEY"] +FEATURE_GROUP_NAME = os.environ["FEATURE_GROUP_NAME"] +FEATURE_GROUP_DEFINITIONS = json.loads(os.environ["FEATURE_GROUP_DEFINITIONS"]) +QUEUE_URL = os.environ["SQS_QUEUE_URL"] +LABELING_ATTRIBUTE_NAME = os.environ["LABELING_ATTRIBUTE_NAME"] +VERIFICATION_ATTRIBUTE_NAME = os.getenv("VERIFICATION_ATTRIBUTE_NAME", None) +VERIFICATION_STEP = VERIFICATION_ATTRIBUTE_NAME is not None + +VERIFICATION_APPROVED_CLASS = 0 +APPROVED_STATUS = "APPROVED" +REJECTED_STATUS = "REJECTED" + +MAX_SQS_BATCH_SIZE = 10 +THREAD_POOL_SIZE = 5 +CONCURRENT_PROCESSING_BATCH_SIZE = 1000 + +boto3_config = Config(max_pool_connections=THREAD_POOL_SIZE) +sqs = boto3.client("sqs", config=boto3_config) +sagemaker_featurestore = boto3.client("sagemaker-featurestore-runtime", config=boto3_config) + + +def handler(event: Dict[str, Any], context: object) -> Dict[str, str]: + labeling_job_output_uri = event["LabelingJobOutputUri"] + labeling_job_output = get_s3_string_object_from_uri(s3_uri=labeling_job_output_uri) + logger.info("Downloaded labeling job output from S3") + + record_source_to_receipt_handle_s3_key = event["RecordSourceToReceiptHandleS3Key"] + record_source_to_receipt_handle = download_json_dict_from_s3( + s3_key=record_source_to_receipt_handle_s3_key, bucket=OUTPUT_BUCKET + ) + logger.info("Downloaded record source to receipt handles from S3") + + rejected_labels_record_source_to_receipt_handle = process_labeling_job_output( + feature_group_name=FEATURE_GROUP_NAME, + queue_url=QUEUE_URL, + labeling_job_output=labeling_job_output, + record_source_to_receipt_handle=record_source_to_receipt_handle, + feature_group_definitions=FEATURE_GROUP_DEFINITIONS, + task_media_type=TASK_MEDIA_TYPE, + ) + + execution_id = event["ExecutionId"].rsplit(":", 1)[-1] + rejected_labels_record_source_to_receipt_handle_s3_key = upload_json_to_s3( + json_data=rejected_labels_record_source_to_receipt_handle, + bucket=OUTPUT_BUCKET, + prefix=f"runs/{execution_id}", + filename="rejected_labels_record_source_to_receipt_handle.json", + ) + logger.info("Uploaded rejected labels record source to receipt handles to S3") + + return {"RejectedLabelsRecordSourceToReceiptHandleS3Key": rejected_labels_record_source_to_receipt_handle_s3_key} + + +def process_labeling_job_output( + feature_group_name: str, + queue_url: str, + labeling_job_output: str, + record_source_to_receipt_handle: Dict[str, str], + feature_group_definitions: Dict[str, List[Union[str, int]]], + task_media_type: str, +) -> Dict[str, str]: + logger.info("Processing labeling job output") + + delete_entries = [] + feature_store_records = {} + for i, line in enumerate(labeling_job_output.splitlines()): + labeling_result = json.loads(line) + record = transform_labeling_result_to_record( + labeling_result=labeling_result, + feature_group_definitions=feature_group_definitions, + ) + + if record: + record_source = labeling_result[SOURCE_KEY] + if record_source in record_source_to_receipt_handle: + receipt_handle = record_source_to_receipt_handle.pop(record_source) + if task_media_type == IMAGE: + filename = record_source.split("/")[-1] + # id request can only contain alphanumeric, hyphen and underscores + id = filename.replace(".", "_") + elif task_media_type == TEXT: + id = str(i) + else: + raise ValueError(f"Unsupported task media type: {task_media_type}") + delete_entries.append({"Id": id, "ReceiptHandle": receipt_handle}) + feature_store_records[id] = record + + # multi thread deletes in batches to prevent all records being held in memory at the same time + if len(delete_entries) == CONCURRENT_PROCESSING_BATCH_SIZE: + multi_threaded_batch_delete_message_from_sqs_and_save_to_feature_store( + queue_url=queue_url, + feature_group_name=feature_group_name, + delete_entries=delete_entries, + feature_store_records=feature_store_records, + ) + delete_entries = [] + feature_store_records = {} + + # Delete any remaining messages after for loop + if delete_entries: + multi_threaded_batch_delete_message_from_sqs_and_save_to_feature_store( + queue_url=queue_url, + feature_group_name=feature_group_name, + delete_entries=delete_entries, + feature_store_records=feature_store_records, + ) + + logger.info(f"Processed labeling job output, {len(record_source_to_receipt_handle)} remaining messages") + + return record_source_to_receipt_handle + + +def transform_labeling_result_to_record( + labeling_result: Dict[str, Any], + feature_group_definitions: Dict[str, List[Union[str, int]]], +) -> List[Dict[str, str]]: + labeling_result_location = VERIFICATION_ATTRIBUTE_NAME if VERIFICATION_STEP else LABELING_ATTRIBUTE_NAME + if "failure-reason" in labeling_result[f"{labeling_result_location}-metadata"]: + return [] + + creation_date = datetime.datetime.strptime( + labeling_result[f"{LABELING_ATTRIBUTE_NAME}-metadata"]["creation-date"], + "%Y-%m-%dT%H:%M:%S.%f", + ).timestamp() + + record = [ + create_record_feature(SOURCE_KEY.replace("-", "_"), str(labeling_result[SOURCE_KEY])), + create_record_feature("event_time", str(int(round(creation_date)))), + create_record_feature( + "labeling_job", + str(labeling_result[f"{LABELING_ATTRIBUTE_NAME}-metadata"]["job-name"]), + ), + ] + + if VERIFICATION_STEP: + status = ( + APPROVED_STATUS + if labeling_result[f"{VERIFICATION_ATTRIBUTE_NAME}"] == VERIFICATION_APPROVED_CLASS + else REJECTED_STATUS + ) + + if status == REJECTED_STATUS: + return [] + else: + record.append(create_record_feature("status", status)) + + for feature_name, output_key in feature_group_definitions.items(): + # access nested value using key list + feature_value = reduce(operator.getitem, output_key, labeling_result) # type: ignore + record.append(create_record_feature(feature_name, str(feature_value))) + return record + + +def create_record_feature(feature_name: str, feature_value: str) -> Dict[str, str]: + return {"FeatureName": feature_name, "ValueAsString": feature_value} + + +def multi_threaded_batch_delete_message_from_sqs_and_save_to_feature_store( + queue_url: str, + feature_group_name: str, + delete_entries: List[Dict[str, str]], + feature_store_records: Dict[str, List[Dict[str, str]]], +) -> None: + logger.info(f"Batch deleting {len(delete_entries)} messages from SQS and saving to feature store") + + delete_entries_split = [ + delete_entries[i : i + MAX_SQS_BATCH_SIZE] for i in range(0, len(delete_entries), MAX_SQS_BATCH_SIZE) + ] + with concurrent.futures.ThreadPoolExecutor(max_workers=THREAD_POOL_SIZE) as executor: + futures = [] + for delete_entries_for_batch in delete_entries_split: + feature_store_records_for_batch = { + entry["Id"]: feature_store_records[entry["Id"]] for entry in delete_entries_for_batch + } + future = executor.submit( + batch_delete_message_from_sqs_and_save_to_feature_store, + queue_url=queue_url, + feature_group_name=feature_group_name, + delete_entries=delete_entries_for_batch, + feature_store_records=feature_store_records_for_batch, + executor=executor, + ) + futures.append(future) + + concurrent.futures.wait(futures) + + +def batch_delete_message_from_sqs_and_save_to_feature_store( + queue_url: str, + feature_group_name: str, + delete_entries: List[Dict[str, str]], + feature_store_records: Dict[str, List[Dict[str, str]]], + executor: concurrent.futures.ThreadPoolExecutor, +) -> None: + try: + response = sqs.delete_message_batch(QueueUrl=queue_url, Entries=delete_entries) + + if "Successful" in response: + for success in response["Successful"]: + source_key = success["Id"] + executor.submit( + put_record_to_feature_store, + feature_group_name=feature_group_name, + record=feature_store_records[source_key], + ) + if "Failed" in response: + for failed in response["Failed"]: + logger.error(f"Failed to delete message {failed['Id']} from SQS") + + except Exception as e: + logger.error(f"Error occurred while deleting messages from SQS: {e}") + + +def put_record_to_feature_store(feature_group_name: str, record: List[Dict[str, str]]) -> None: + try: + sagemaker_featurestore.put_record(FeatureGroupName=feature_group_name, Record=record) + except Exception as e: + logger.error(f"Error occurred while putting record to feature store: {e}") diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/lambda/txt_file_s3_to_sqs_relay.py b/modules/sagemaker/sagemaker-ground-truth-labeling/lambda/txt_file_s3_to_sqs_relay.py new file mode 100644 index 00000000..94fb3422 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/lambda/txt_file_s3_to_sqs_relay.py @@ -0,0 +1,32 @@ +import logging +import os +from typing import Any, Dict + +import boto3 + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + +s3 = boto3.client("s3") +sqs = boto3.client("sqs") + +SQS_QUEUE_URL = os.environ["SQS_QUEUE_URL"] + + +def handler(event: Dict[str, Any], context: object) -> None: + for record in event["Records"]: + bucket = record["s3"]["bucket"]["name"] + key = record["s3"]["object"]["key"] + + if not key.endswith(".txt"): + logger.info(f"Skipping non-txt file {key}") + continue + + try: + response = s3.get_object(Bucket=bucket, Key=key) + file_content = response["Body"].read().decode("utf-8") + + sqs.send_message(QueueUrl=SQS_QUEUE_URL, MessageBody=file_content) + logger.info("Sent file contents to SQS queue") + except Exception as e: + logger.error(f"An error occurred: {str(e)}") diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/pyproject.toml b/modules/sagemaker/sagemaker-ground-truth-labeling/pyproject.toml new file mode 100644 index 00000000..b0e4e337 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/pyproject.toml @@ -0,0 +1,45 @@ +[tool.ruff] +exclude = [ + ".eggs", + ".git", + ".hg", + ".mypy_cache", + ".ruff_cache", + ".tox", + ".venv", + "_build", + "buck-out", + "build", + "dist", + "codeseeder", +] +line-length = 120 +target-version = "py38" + +[tool.ruff.lint] +select = ["F", "I", "E", "W"] +fixable = ["ALL"] + +[tool.mypy] +python_version = "3.8" +strict = true +ignore_missing_imports = true +disallow_untyped_decorators = false +exclude = "codeseeder.out/|example/|tests/" +warn_unused_ignores = false + +plugins = [ + "pydantic.mypy" +] + +[tool.pytest.ini_options] +addopts = "-v --cov=. --cov-report term" +pythonpath = [ + "." +] + +[tool.coverage.run] +omit = ["tests/*"] + +[tool.coverage.report] +fail_under = 80 diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/requirements.txt b/modules/sagemaker/sagemaker-ground-truth-labeling/requirements.txt new file mode 100644 index 00000000..4635d113 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/requirements.txt @@ -0,0 +1,3 @@ +aws-cdk-lib==2.158.0 +cdk-nag==2.28.195 +pydantic-settings==2.5.2 \ No newline at end of file diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/settings.py b/modules/sagemaker/sagemaker-ground-truth-labeling/settings.py new file mode 100644 index 00000000..08fee657 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/settings.py @@ -0,0 +1,101 @@ +"""Defines the stack settings.""" + +from abc import ABC +from typing import Any, Dict, List + +from pydantic import Field, computed_field +from pydantic_settings import BaseSettings, SettingsConfigDict + +DEFAULT_TASK_CONFIG = { + "NumberOfHumanWorkersPerDataObject": 1, + "TaskAvailabilityLifetimeInSeconds": 21600, + "TaskTimeLimitInSeconds": 300, +} + + +class CdkBaseSettings(BaseSettings, ABC): + """Defines common configuration for settings.""" + + model_config = SettingsConfigDict( + case_sensitive=False, + env_nested_delimiter="__", + protected_namespaces=(), + extra="ignore", + populate_by_name=True, + ) + + +class ModuleSettings(CdkBaseSettings): + """Seedfarmer Parameters. + + These parameters are required for the module stack. + """ + + model_config = SettingsConfigDict(env_prefix="SEEDFARMER_PARAMETER_") + + job_name: str + task_type: str + labeling_workteam_arn: str + labeling_instructions_template_s3_uri: str = Field(default="") + labeling_categories_s3_uri: str + labeling_task_title: str + labeling_task_description: str + labeling_task_keywords: List[str] + verification_workteam_arn: str = Field(default="") + verification_instructions_template_s3_uri: str = Field(default="") + verification_categories_s3_uri: str = Field(default="") + verification_task_title: str = Field(default="") + verification_task_description: str = Field(default="") + verification_task_keywords: List[str] = Field(default=[]) + + sqs_queue_retention_period: int = Field(default=60 * 24 * 14) + sqs_queue_visibility_timeout: int = Field(default=60 * 12) + sqs_queue_max_receive_count: int = Field(default=3) + sqs_dlq_retention_period: int = Field(default=60 * 24 * 14) + sqs_dlq_visibility_timeout: int = Field(default=60 * 12) + sqs_dlq_alarm_threshold: int = Field(default=1) + labeling_human_task_config: Dict[str, Any] = Field(default=DEFAULT_TASK_CONFIG) + labeling_task_price: Dict[str, Dict[str, int]] = Field(default={}) + verification_human_task_config: Dict[str, Any] = Field(default=DEFAULT_TASK_CONFIG) + verification_task_price: Dict[str, Dict[str, int]] = Field(default={}) + labeling_workflow_schedule: str = Field(default="cron(0 12 * * ? *)") + + +class SeedFarmerSettings(CdkBaseSettings): + """Seedfarmer Settings. + + These parameters comes from seedfarmer by default. + """ + + model_config = SettingsConfigDict(env_prefix="SEEDFARMER_") + + project_name: str = Field(default="") + deployment_name: str = Field(default="") + module_name: str = Field(default="") + + @computed_field # type: ignore + @property + def app_prefix(self) -> str: + """Application prefix.""" + prefix = "-".join([self.project_name, self.deployment_name, self.module_name]) + return prefix + + +class CDKSettings(CdkBaseSettings): + """CDK Default Settings. + + These parameters come from AWS CDK by default. + """ + + model_config = SettingsConfigDict(env_prefix="CDK_DEFAULT_") + + account: str + region: str + + +class ApplicationSettings(CdkBaseSettings): + """Application settings.""" + + seedfarmer_settings: SeedFarmerSettings = Field(default_factory=SeedFarmerSettings) + module_settings: ModuleSettings = Field(default_factory=ModuleSettings) + cdk_settings: CDKSettings = Field(default_factory=CDKSettings) diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/stack.py b/modules/sagemaker/sagemaker-ground-truth-labeling/stack.py new file mode 100644 index 00000000..6055dae0 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/stack.py @@ -0,0 +1,504 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 +from typing import Any, Dict, List, Tuple, Union + +import aws_cdk.aws_lambda as lambda_ +import constructs +from aws_cdk import Duration, RemovalPolicy, Stack +from aws_cdk import aws_cloudwatch as cloudwatch +from aws_cdk import aws_iam as iam +from aws_cdk import aws_kms as kms +from aws_cdk import aws_s3 as s3 +from aws_cdk import aws_s3_notifications as s3n +from aws_cdk import aws_sqs as sqs +from aws_cdk.aws_sagemaker import CfnFeatureGroup +from cdk_nag import NagPackSuppression, NagSuppressions + +from constants import ( + LAMBDA_RUNTIME, + MAX_BUCKET_NAME_LENGTH, + MAX_FEATURE_GROUP_NAME_LENGTH, + MAX_SQS_QUEUE_NAME_LENGTH, +) +from labeling_step_function.labeling_state_machine import setup_and_create_state_machine +from task_type_config import IMAGE, TEXT, TaskTypeConfig, get_task_type_config + + +class DeployGroundTruthLabelingStack(Stack): + def __init__( + self, + scope: constructs.Construct, + id: str, + job_name: str, + task_type: str, + sqs_queue_retention_period: int, + sqs_queue_visibility_timeout: int, + sqs_queue_max_receive_count: int, + sqs_dlq_retention_period: int, + sqs_dlq_visibility_timeout: int, + sqs_dlq_alarm_threshold: int, + labeling_workteam_arn: str, + labeling_instructions_template_s3_uri: str, + labeling_categories_s3_uri: str, + labeling_task_title: str, + labeling_task_description: str, + labeling_task_keywords: List[str], + labeling_human_task_config: Dict[str, Union[str, int]], + labeling_task_price: Dict[str, Any], + verification_workteam_arn: str, + verification_instructions_template_s3_uri: str, + verification_categories_s3_uri: str, + verification_task_title: str, + verification_task_description: str, + verification_task_keywords: List[str], + verification_human_task_config: Dict[str, Any], + verification_task_price: Dict[str, Dict[str, int]], + labeling_workflow_schedule: str, + **kwargs: Any, + ) -> None: + super().__init__(scope, id, **kwargs) + + log_bucket = self.create_log_bucket(job_name=job_name, id=id) + + self.upload_bucket, upload_bucket_kms_key = self.create_upload_bucket( + job_name=job_name, + id=id, + log_bucket=log_bucket, + ) + + self.upload_dlq = self.create_upload_dlq( + job_name=job_name, + id=id, + sqs_dlq_retention_period=sqs_dlq_retention_period, + sqs_dlq_visibility_timeout=sqs_dlq_visibility_timeout, + sqs_dlq_alarm_threshold=sqs_dlq_alarm_threshold, + ) + + self.upload_queue, upload_queue_kms_key = self.create_upload_queue( + job_name=job_name, + id=id, + sqs_queue_retention_period=sqs_queue_retention_period, + sqs_queue_visibility_timeout=sqs_queue_visibility_timeout, + sqs_queue_max_receive_count=sqs_queue_max_receive_count, + upload_dlq=self.upload_dlq, + ) + + task_type_config = get_task_type_config(task_type) + + self.create_s3_upload_notification( + task_type_config_media_type=task_type_config.media_type, + upload_bucket=self.upload_bucket, + upload_bucket_kms_key_arn=upload_bucket_kms_key.key_arn, + upload_queue=self.upload_queue, + upload_queue_kms_key_arn=upload_queue_kms_key.key_arn, + ) + + feature_group_bucket, feature_group_bucket_kms_key = self.create_feature_group_bucket( + job_name=job_name, + id=id, + log_bucket=log_bucket, + ) + + feature_group_name_suffix = f"-{job_name}-sagemaker-feature-group" + feature_group_name = ( + f"{id[: MAX_FEATURE_GROUP_NAME_LENGTH - len(feature_group_name_suffix)]}{feature_group_name_suffix}" + ) + + feature_group_role = self.create_feature_group_role( + feature_group_bucket_arn=feature_group_bucket.bucket_arn, + feature_group_bucket_kms_key_arn=feature_group_bucket_kms_key.key_arn, + feature_group_name=feature_group_name, + ) + + feature_group_bucket_kms_key.add_to_resource_policy( + iam.PolicyStatement( + actions=["kms:Decrypt", "kms:GenerateDataKey"], + principals=[feature_group_role], + resources=["*"], + ) + ) + + self.feature_group = self.create_feature_group( + task_type_config=task_type_config, + feature_group_name=feature_group_name, + feature_group_bucket_name=feature_group_bucket.bucket_name, + feature_group_bucket_kms_key_arn=feature_group_bucket_kms_key.key_arn, + feature_group_role=feature_group_role, + ) + + self.labeling_state_machine = setup_and_create_state_machine( + scope=self, + id=id, + job_name=job_name, + queue=self.upload_queue, + queue_kms_key_arn=upload_queue_kms_key.key_arn, + upload_bucket_arn=self.upload_bucket.bucket_arn, + upload_bucket_kms_key_arn=upload_bucket_kms_key.key_arn, + log_bucket=log_bucket, + task_type=task_type, + feature_group_name=self.feature_group.feature_group_name, + labeling_workteam_arn=labeling_workteam_arn, + labeling_instructions_template_s3_uri=labeling_instructions_template_s3_uri, + labeling_categories_s3_uri=labeling_categories_s3_uri, + labeling_task_title=labeling_task_title, + labeling_task_description=labeling_task_description, + labeling_task_keywords=labeling_task_keywords, + labeling_human_task_config=labeling_human_task_config, + labeling_task_price=labeling_task_price, + verification_workteam_arn=verification_workteam_arn, + verification_instructions_template_s3_uri=verification_instructions_template_s3_uri, + verification_categories_s3_uri=verification_categories_s3_uri, + verification_task_title=verification_task_title, + verification_task_description=verification_task_description, + verification_task_keywords=verification_task_keywords, + verification_human_task_config=verification_human_task_config, + verification_task_price=verification_task_price, + labeling_workflow_schedule=labeling_workflow_schedule, + ) + + NagSuppressions.add_stack_suppressions( + self, + suppressions=[ + NagPackSuppression( + id="AwsSolutions-IAM4", + reason="AWS managed policy is used by CDK-managed Lambda for S3 notifications", + applies_to=[ + "Policy::arn::iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", + ], + ), + NagPackSuppression( + id="AwsSolutions-IAM5", + reason="CDK-managed Lambda for S3 notifications requires these permissions", + ), + ], + ) + + def create_log_bucket( + self, + job_name: str, + id: str, + ) -> s3.Bucket: + log_bucket_kms_key = kms.Key( + self, + "Log Bucket KMS Key", + description="key used for encryption of data in Amazon S3", + enable_key_rotation=True, + ) + + # limit bucket name length to 63 chars as that is the limit in S3 + log_bucket_name_suffix = f"-{job_name}-bucket-logs" + log_bucket_name = f"{id[: MAX_BUCKET_NAME_LENGTH - len(log_bucket_name_suffix)]}{log_bucket_name_suffix}" + log_bucket = s3.Bucket( + self, + "LogBucket", + bucket_name=log_bucket_name, + enforce_ssl=True, + removal_policy=RemovalPolicy.DESTROY, + auto_delete_objects=True, + encryption_key=log_bucket_kms_key, + ) + + return log_bucket + + def create_upload_bucket(self, job_name: str, id: str, log_bucket: s3.Bucket) -> Tuple[s3.Bucket, kms.Key]: + upload_bucket_kms_key = kms.Key( + self, + "Upload Bucket KMS Key", + description="key used for encryption of data in Amazon S3", + enable_key_rotation=True, + ) + + upload_bucket_name_suffix = f"-{job_name}-upload-bucket" + upload_bucket_name = ( + f"{id[: MAX_BUCKET_NAME_LENGTH - len(upload_bucket_name_suffix)]}{upload_bucket_name_suffix}" + ) + upload_bucket = s3.Bucket( + self, + "UploadBucket", + bucket_name=upload_bucket_name, + enforce_ssl=True, + removal_policy=RemovalPolicy.DESTROY, + auto_delete_objects=True, + encryption_key=upload_bucket_kms_key, + server_access_logs_prefix="upload_bucket_logs/", + server_access_logs_bucket=log_bucket, + cors=[ + s3.CorsRule( + allowed_headers=[], + allowed_methods=[s3.HttpMethods.GET], + allowed_origins=["*"], + exposed_headers=["Access-Control-Allow-Origin"], + ) + ], + ) + + return upload_bucket, upload_bucket_kms_key + + def create_upload_dlq( + self, + job_name: str, + id: str, + sqs_dlq_retention_period: int, + sqs_dlq_visibility_timeout: int, + sqs_dlq_alarm_threshold: int, + ) -> sqs.Queue: + upload_dlq_kms_key = kms.Key( + self, + "Upload DLQ KMS Key", + description="key used for encryption of data in Amazon SQS", + enable_key_rotation=True, + ) + + upload_dlq_name_suffix = f"-{job_name}-upload-dlq" + upload_dlq_name = f"{id[: MAX_SQS_QUEUE_NAME_LENGTH - len(upload_dlq_name_suffix)]}{upload_dlq_name_suffix}" + upload_dlq = sqs.Queue( + self, + "UploadDLQ", + queue_name=upload_dlq_name, + retention_period=Duration.minutes(sqs_dlq_retention_period), + visibility_timeout=Duration.minutes(sqs_dlq_visibility_timeout), + enforce_ssl=True, + encryption=sqs.QueueEncryption.KMS, + encryption_master_key=upload_dlq_kms_key, + ) + + if sqs_dlq_alarm_threshold != 0: + cloudwatch.Alarm( + self, + "DlqMessageCountAlarm", + alarm_name=f"{id}-{job_name}-dlq-message-count-alarm", + metric=upload_dlq.metric_approximate_number_of_messages_visible(), + threshold=sqs_dlq_alarm_threshold, + evaluation_periods=1, + alarm_description="Alarm when more than 1 message is in the DLQ", + ) + + return upload_dlq + + def create_upload_queue( + self, + job_name: str, + id: str, + sqs_queue_max_receive_count: int, + sqs_queue_retention_period: int, + sqs_queue_visibility_timeout: int, + upload_dlq: sqs.Queue, + ) -> Tuple[sqs.Queue, kms.Key]: + upload_queue_kms_key = kms.Key( + self, + "Upload Queue KMS Key", + description="key used for encryption of data in Amazon SQS", + enable_key_rotation=True, + ) + + upload_queue_name_suffix = f"-{job_name}-upload-queue" + upload_queue_name = ( + f"{id[: MAX_SQS_QUEUE_NAME_LENGTH - len(upload_queue_name_suffix)]}{upload_queue_name_suffix}" + ) + upload_queue = sqs.Queue( + self, + "UploadQueue", + queue_name=upload_queue_name, + dead_letter_queue=sqs.DeadLetterQueue(max_receive_count=sqs_queue_max_receive_count, queue=upload_dlq), + retention_period=Duration.minutes(sqs_queue_retention_period), + visibility_timeout=Duration.minutes(sqs_queue_visibility_timeout), + enforce_ssl=True, + encryption=sqs.QueueEncryption.KMS, + encryption_master_key=upload_queue_kms_key, + ) + + return upload_queue, upload_queue_kms_key + + def create_s3_upload_notification( + self, + task_type_config_media_type: str, + upload_bucket: s3.Bucket, + upload_bucket_kms_key_arn: str, + upload_queue: sqs.Queue, + upload_queue_kms_key_arn: str, + ) -> None: + if task_type_config_media_type == IMAGE: + upload_bucket.add_event_notification(s3.EventType.OBJECT_CREATED, s3n.SqsDestination(upload_queue)) + elif task_type_config_media_type == TEXT: + txt_file_s3_to_sqs_relay_lambda_role = iam.Role( + self, + "TxtFileS3ToSqsRelayLambdaRole", + assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"), + ) + txt_file_s3_to_sqs_relay_lambda_role.add_to_policy( + iam.PolicyStatement( + actions=[ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents", + ], + resources=["*"], + ) + ) + txt_file_s3_to_sqs_relay_lambda_role.add_to_policy( + iam.PolicyStatement( + actions=["s3:GetObject"], + resources=[ + upload_bucket.bucket_arn, + f"{upload_bucket.bucket_arn}/*", + ], + ) + ) + txt_file_s3_to_sqs_relay_lambda_role.add_to_policy( + iam.PolicyStatement( + actions=["kms:Decrypt"], + resources=[upload_bucket_kms_key_arn], + ) + ) + txt_file_s3_to_sqs_relay_lambda_role.add_to_policy( + iam.PolicyStatement( + actions=["sqs:SendMessage"], + resources=[upload_queue.queue_arn], + ) + ) + txt_file_s3_to_sqs_relay_lambda_role.add_to_policy( + iam.PolicyStatement( + actions=["kms:GenerateDataKey"], + resources=[upload_queue_kms_key_arn], + ) + ) + txt_file_s3_to_sqs_relay_lambda = lambda_.Function( + self, + "TxtFileS3ToSqsRelayLambda", + runtime=LAMBDA_RUNTIME, + code=lambda_.Code.from_asset("lambda"), + handler="txt_file_s3_to_sqs_relay.handler", + memory_size=128, + role=txt_file_s3_to_sqs_relay_lambda_role, + timeout=Duration.seconds(5), + environment={ + "SQS_QUEUE_URL": upload_queue.queue_url, + }, + ) + upload_bucket.add_event_notification( + s3.EventType.OBJECT_CREATED, + s3n.LambdaDestination(txt_file_s3_to_sqs_relay_lambda), + ) + else: + raise ValueError(f"Unsupported media type: {task_type_config_media_type}") + + def create_feature_group_bucket(self, job_name: str, id: str, log_bucket: s3.Bucket) -> Tuple[s3.Bucket, kms.Key]: + feature_group_bucket_kms_key = kms.Key( + self, + "FeatureStoreBucket KMS Key", + description="key used for encryption of data in Amazon S3", + enable_key_rotation=True, + ) + + feature_group_bucket_name_suffix = f"-{job_name}-feature-store-bucket" + feature_group_bucket_name = ( + f"{id[: MAX_BUCKET_NAME_LENGTH - len(feature_group_bucket_name_suffix)]}{feature_group_bucket_name_suffix}" + ) + feature_group_bucket = s3.Bucket( + self, + "FeatureStoreBucket", + bucket_name=feature_group_bucket_name, + enforce_ssl=True, + encryption_key=feature_group_bucket_kms_key, + server_access_logs_prefix="feature_store_bucket_logs/", + server_access_logs_bucket=log_bucket, + removal_policy=RemovalPolicy.DESTROY, + auto_delete_objects=True, + ) + + return feature_group_bucket, feature_group_bucket_kms_key + + def create_feature_group_role( + self, + feature_group_bucket_arn: str, + feature_group_bucket_kms_key_arn: str, + feature_group_name: str, + ) -> iam.Role: + feature_group_role = iam.Role( + self, + "FeatureGroupRole", + assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com"), + ) + feature_group_role.add_to_policy( + iam.PolicyStatement( + actions=[ + "s3:PutObject", + "s3:GetBucketAcl", + "s3:PutObjectAcl", + ], + resources=[ + feature_group_bucket_arn, + f"{feature_group_bucket_arn}/*", + ], + ) + ) + feature_group_role.add_to_policy( + iam.PolicyStatement( + actions=["kms:Decrypt", "kms:GenerateDataKey"], + resources=[feature_group_bucket_kms_key_arn], + ) + ) + feature_group_role.add_to_policy( + iam.PolicyStatement( + actions=["sagemaker:PutRecord"], + resources=[ + f"arn:{self.partition}:sagemaker:{self.region}:{self.account}:feature-group/{feature_group_name}", + ], + ), + ) + + return feature_group_role + + def create_feature_group( + self, + task_type_config: TaskTypeConfig, + feature_group_name: str, + feature_group_bucket_name: str, + feature_group_bucket_kms_key_arn: str, + feature_group_role: iam.Role, + ) -> CfnFeatureGroup: + source_key = task_type_config.source_key.replace("-", "_") + feature_definitions = [ + CfnFeatureGroup.FeatureDefinitionProperty( + feature_name=source_key, + feature_type="String", + ), + CfnFeatureGroup.FeatureDefinitionProperty( + feature_name="event_time", + feature_type="Fractional", + ), + CfnFeatureGroup.FeatureDefinitionProperty(feature_name="labeling_job", feature_type="String"), + ] + if task_type_config.verification_attribute_name: + feature_definitions.append( + CfnFeatureGroup.FeatureDefinitionProperty( + feature_name="status", + feature_type="String", + ) + ) + for feature_definition in task_type_config.feature_group_config: + feature_definitions.append( + CfnFeatureGroup.FeatureDefinitionProperty( + feature_name=feature_definition.feature_name, + feature_type=feature_definition.feature_type, + ) + ) + + sagemaker_feature_group = CfnFeatureGroup( + self, + "SagemakerFeatureGroup", + feature_group_name=feature_group_name, + event_time_feature_name="event_time", + record_identifier_feature_name=source_key, + feature_definitions=feature_definitions, + offline_store_config={ + "S3StorageConfig": { + "S3Uri": f"s3://{feature_group_bucket_name}/feature-store/", + "KmsKeyId": feature_group_bucket_kms_key_arn, + } + }, + role_arn=feature_group_role.role_arn, + ) + sagemaker_feature_group.node.add_dependency(feature_group_role) + + return sagemaker_feature_group diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/task_type_config.py b/modules/sagemaker/sagemaker-ground-truth-labeling/task_type_config.py new file mode 100644 index 00000000..f5c08729 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/task_type_config.py @@ -0,0 +1,221 @@ +from dataclasses import dataclass +from typing import Dict, List, Optional, Union + +DEFAULT_LABELING_ATTRIBUTE_NAME = "label" +DEFAULT_VERIFICATION_ATTRIBUTE_NAME = "verification" + +# Certain jobs require the attribute end with -ref +ALTERNATIVE_LABELING_ATTRIBUTE_NAME = DEFAULT_LABELING_ATTRIBUTE_NAME + "-ref" + +METADATA = "-metadata" + +IMAGE = "image" +TEXT = "text" + +IMAGE_SOURCE_KEY = "source-ref" +TEXT_SOURCE_KEY = "source" + + +class UndefinedTaskTypeError(Exception): + pass + + +@dataclass +class FeatureDefinitionConfig: + feature_name: str + feature_type: str + output_key: List[Union[str, int]] + + +@dataclass +class TaskTypeConfig: + task_type: str + media_type: str + source_key: str + function_name: str + labeling_attribute_name: str + feature_group_config: List[FeatureDefinitionConfig] + verification_attribute_name: Optional[str] = None + + +class TaskTypeConfigManager: + def __init__(self) -> None: + self.configs: Dict[str, TaskTypeConfig] = {} + + def add_config(self, config: TaskTypeConfig) -> None: + self.configs[config.task_type] = config + + def get_config(self, task_type: str) -> TaskTypeConfig: + if task_type not in self.configs: + raise UndefinedTaskTypeError(f"Task type '{task_type}' is not defined") + return self.configs[task_type] + + +task_type_manager = TaskTypeConfigManager() + +task_type_manager.add_config( + TaskTypeConfig( + task_type="image_bounding_box", + media_type=IMAGE, + source_key=IMAGE_SOURCE_KEY, + function_name="BoundingBox", + labeling_attribute_name=DEFAULT_LABELING_ATTRIBUTE_NAME, + feature_group_config=[ + FeatureDefinitionConfig( + "image_width", + "Integral", + [DEFAULT_LABELING_ATTRIBUTE_NAME, "image_size", 0, "width"], + ), + FeatureDefinitionConfig( + "image_height", + "Integral", + [DEFAULT_LABELING_ATTRIBUTE_NAME, "image_size", 0, "height"], + ), + FeatureDefinitionConfig( + "image_depth", + "Integral", + [DEFAULT_LABELING_ATTRIBUTE_NAME, "image_size", 0, "depth"], + ), + FeatureDefinitionConfig( + "annotations", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME, "annotations"], + ), + FeatureDefinitionConfig( + "class_map", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "class-map"], + ), + FeatureDefinitionConfig( + "objects", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "objects"], + ), + ], + verification_attribute_name=DEFAULT_VERIFICATION_ATTRIBUTE_NAME, + ) +) + +task_type_manager.add_config( + TaskTypeConfig( + task_type="image_semantic_segmentation", + media_type=IMAGE, + source_key=IMAGE_SOURCE_KEY, + function_name="SemanticSegmentation", + labeling_attribute_name=ALTERNATIVE_LABELING_ATTRIBUTE_NAME, + feature_group_config=[ + FeatureDefinitionConfig( + "image_location", + "String", + [ALTERNATIVE_LABELING_ATTRIBUTE_NAME], + ), + FeatureDefinitionConfig( + "internal_color_map", + "String", + [ALTERNATIVE_LABELING_ATTRIBUTE_NAME + METADATA, "internal-color-map"], + ), + ], + verification_attribute_name=DEFAULT_VERIFICATION_ATTRIBUTE_NAME, + ) +) + +single_label_classification_feature_group_config = [ + FeatureDefinitionConfig( + "class_name", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "class-name"], + ), + FeatureDefinitionConfig( + "confidence", + "Fractional", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "confidence"], + ), +] + +task_type_manager.add_config( + TaskTypeConfig( + task_type="image_single_label_classification", + media_type=IMAGE, + source_key=IMAGE_SOURCE_KEY, + function_name="ImageMultiClass", + labeling_attribute_name=DEFAULT_LABELING_ATTRIBUTE_NAME, + feature_group_config=single_label_classification_feature_group_config, + ) +) + +task_type_manager.add_config( + TaskTypeConfig( + task_type="text_single_label_classification", + media_type=TEXT, + source_key=TEXT_SOURCE_KEY, + function_name="TextMultiClass", + labeling_attribute_name=DEFAULT_LABELING_ATTRIBUTE_NAME, + feature_group_config=single_label_classification_feature_group_config, + ) +) + +multi_label_classification_feature_group_config = [ + FeatureDefinitionConfig( + "classes", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME], + ), + FeatureDefinitionConfig( + "class_map", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "class-map"], + ), + FeatureDefinitionConfig( + "confidence_map", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "confidence-map"], + ), +] + +task_type_manager.add_config( + TaskTypeConfig( + task_type="image_multi_label_classification", + media_type=IMAGE, + source_key=IMAGE_SOURCE_KEY, + function_name="ImageMultiClassMultiLabel", + labeling_attribute_name=DEFAULT_LABELING_ATTRIBUTE_NAME, + feature_group_config=multi_label_classification_feature_group_config, + ) +) + +task_type_manager.add_config( + TaskTypeConfig( + task_type="text_multi_label_classification", + media_type=TEXT, + source_key=TEXT_SOURCE_KEY, + function_name="TextMultiClassMultiLabel", + labeling_attribute_name=DEFAULT_LABELING_ATTRIBUTE_NAME, + feature_group_config=multi_label_classification_feature_group_config, + ) +) + +task_type_manager.add_config( + TaskTypeConfig( + task_type="named_entity_recognition", + media_type=TEXT, + source_key=TEXT_SOURCE_KEY, + function_name="NamedEntityRecognition", + labeling_attribute_name=DEFAULT_LABELING_ATTRIBUTE_NAME, + feature_group_config=[ + FeatureDefinitionConfig( + "annotations", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME, "annotations"], + ), + FeatureDefinitionConfig( + "entities_confidence", + "String", + [DEFAULT_LABELING_ATTRIBUTE_NAME + METADATA, "entities"], + ), + ], + ) +) + + +def get_task_type_config(task_type: str) -> TaskTypeConfig: + return task_type_manager.get_config(task_type) diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/tests/__init__.py b/modules/sagemaker/sagemaker-ground-truth-labeling/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/tests/test_app.py b/modules/sagemaker/sagemaker-ground-truth-labeling/tests/test_app.py new file mode 100644 index 00000000..258819f3 --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/tests/test_app.py @@ -0,0 +1,94 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +import os +import sys +from unittest import mock + +import pytest +from pydantic import ValidationError + + +@pytest.fixture(scope="function", autouse=True) +def stack_defaults(): + with mock.patch.dict(os.environ, {}, clear=True): + os.environ["SEEDFARMER_PROJECT_NAME"] = "test-project" + os.environ["SEEDFARMER_DEPLOYMENT_NAME"] = "test-deployment" + os.environ["SEEDFARMER_MODULE_NAME"] = "test-module" + + os.environ["CDK_DEFAULT_ACCOUNT"] = "111111111111" + os.environ["CDK_DEFAULT_REGION"] = "us-east-1" + + os.environ["SEEDFARMER_PARAMETER_JOB_NAME"] = "job-name" + os.environ["SEEDFARMER_PARAMETER_TASK_TYPE"] = "image_bounding_box" + os.environ["SEEDFARMER_PARAMETER_LABELING_WORKTEAM_ARN"] = "labeling-workteam" + os.environ["SEEDFARMER_PARAMETER_LABELING_CATEGORIES_S3_URI"] = "s3://bucket/labeling-categories" + os.environ["SEEDFARMER_PARAMETER_LABELING_TASK_TITLE"] = "labeling-title" + os.environ["SEEDFARMER_PARAMETER_LABELING_TASK_DESCRIPTION"] = "labeling-description" + os.environ["SEEDFARMER_PARAMETER_LABELING_TASK_KEYWORDS"] = '["labeling-keywords"]' + + # Unload the app import so that subsequent tests don't reuse + if "app" in sys.modules: + del sys.modules["app"] + + yield + + +def test_app() -> None: + import app # noqa: F401 + + +def test_job_name() -> None: + del os.environ["SEEDFARMER_PARAMETER_JOB_NAME"] + + with pytest.raises(ValidationError): + import app # noqa: F401 + + +def test_task_type() -> None: + del os.environ["SEEDFARMER_PARAMETER_TASK_TYPE"] + + with pytest.raises(ValidationError): + import app # noqa: F401 + + +def test_task_type_invalid_value() -> None: + os.environ["SEEDFARMER_PARAMETER_TASK_TYPE"] = "task_type" + + with pytest.raises(Exception): + import app # noqa: F401 + + +def test_labeling_workteam_arn() -> None: + del os.environ["SEEDFARMER_PARAMETER_LABELING_WORKTEAM_ARN"] + + with pytest.raises(ValidationError): + import app # noqa: F401 + + +def test_labeling_categories_s3_uri() -> None: + del os.environ["SEEDFARMER_PARAMETER_LABELING_CATEGORIES_S3_URI"] + + with pytest.raises(ValidationError): + import app # noqa: F401 + + +def test_labeling_task_title() -> None: + del os.environ["SEEDFARMER_PARAMETER_LABELING_TASK_TITLE"] + + with pytest.raises(ValidationError): + import app # noqa: F401 + + +def test_labeling_task_description() -> None: + del os.environ["SEEDFARMER_PARAMETER_LABELING_TASK_DESCRIPTION"] + + with pytest.raises(ValidationError): + import app # noqa: F401 + + +def test_labeling_task_keywords() -> None: + del os.environ["SEEDFARMER_PARAMETER_LABELING_TASK_KEYWORDS"] + + with pytest.raises(ValidationError): + import app # noqa: F401 diff --git a/modules/sagemaker/sagemaker-ground-truth-labeling/tests/test_stack.py b/modules/sagemaker/sagemaker-ground-truth-labeling/tests/test_stack.py new file mode 100644 index 00000000..aeab6fce --- /dev/null +++ b/modules/sagemaker/sagemaker-ground-truth-labeling/tests/test_stack.py @@ -0,0 +1,113 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + + +import aws_cdk as cdk +import cdk_nag +import pytest +from aws_cdk.assertions import Annotations, Match, Template + + +@pytest.fixture(scope="function") +def stack(task_type: str, labeling_workflow_schedule: str, sqs_dlq_alarm_threshold: int) -> cdk.Stack: + import stack + + app = cdk.App() + project_name = "test-project" + dep_name = "test-deployment" + mod_name = "test-module" + job_name = "job-name" + sqs_queue_retention_period = 1 + sqs_queue_visibility_timeout = 1 + sqs_queue_max_receive_count = 1 + sqs_dlq_retention_period = 1 + sqs_dlq_visibility_timeout = 1 + labeling_workteam_arn = "labeling_workteam" + labeling_instructions_template_s3_uri = "s3://bucket/labeling_instructions" + labeling_categories_s3_uri = "s3://bucket/labeling_categories" + labeling_task_title = "labeling_title" + labeling_task_description = "labeling_description" + labeling_task_keywords = ["labeling_keywords"] + labeling_human_task_config = {"key": "value"} + labeling_task_price = {"key": {"nested_key": "value"}} + verification_workteam_arn = "verification_workteam" + verification_instructions_template_s3_uri = "s3://bucket/verification_instructions" + verification_categories_s3_uri = "s3://bucket/verification_categories" + verification_task_title = "verification_title" + verification_task_description = "verification_description" + verification_task_keywords = ["verification_keywords"] + verification_human_task_config = {"key": "value"} + verification_task_price = {"key": {"nested_key": "value"}} + + return stack.DeployGroundTruthLabelingStack( + app, + f"{project_name}-{dep_name}-{mod_name}", + job_name=job_name, + task_type=task_type, + sqs_queue_retention_period=sqs_queue_retention_period, + sqs_queue_visibility_timeout=sqs_queue_visibility_timeout, + sqs_queue_max_receive_count=sqs_queue_max_receive_count, + sqs_dlq_retention_period=sqs_dlq_retention_period, + sqs_dlq_visibility_timeout=sqs_dlq_visibility_timeout, + sqs_dlq_alarm_threshold=sqs_dlq_alarm_threshold, + labeling_workteam_arn=labeling_workteam_arn, + labeling_instructions_template_s3_uri=labeling_instructions_template_s3_uri, + labeling_categories_s3_uri=labeling_categories_s3_uri, + labeling_task_title=labeling_task_title, + labeling_task_description=labeling_task_description, + labeling_task_keywords=labeling_task_keywords, + labeling_human_task_config=labeling_human_task_config, + labeling_task_price=labeling_task_price, + verification_workteam_arn=verification_workteam_arn, + verification_instructions_template_s3_uri=verification_instructions_template_s3_uri, + verification_categories_s3_uri=verification_categories_s3_uri, + verification_task_title=verification_task_title, + verification_task_description=verification_task_description, + verification_task_keywords=verification_task_keywords, + verification_human_task_config=verification_human_task_config, + verification_task_price=verification_task_price, + labeling_workflow_schedule=labeling_workflow_schedule, + ) + + +@pytest.mark.parametrize( + "task_type", ["text_single_label_classification", "image_single_label_classification", "image_bounding_box"] +) +@pytest.mark.parametrize("labeling_workflow_schedule", ["cron(0 0 * * ? *)", ""]) +@pytest.mark.parametrize("sqs_dlq_alarm_threshold", [0, 1]) +def test_synthesize_stack( + stack: cdk.Stack, task_type: str, labeling_workflow_schedule: str, sqs_dlq_alarm_threshold: int +) -> None: + template = Template.from_stack(stack) + + template.resource_count_is("AWS::S3::Bucket", 4) + template.resource_count_is("AWS::SQS::Queue", 2) + template.resource_count_is("AWS::SageMaker::FeatureGroup", 1) + + lambda_function_count = { + "text_single_label_classification": 7, + "image_single_label_classification": 6, # one less as no lambda to convert text files to SQS message + "image_bounding_box": 7, # one additional to run verification job + }.get(task_type) + template.resource_count_is("AWS::Lambda::Function", lambda_function_count) + + template.resource_count_is("AWS::StepFunctions::StateMachine", 1) + template.resource_count_is("AWS::Scheduler::Schedule", 0 if labeling_workflow_schedule == "" else 1) + template.resource_count_is("AWS::CloudWatch::Alarm", 0 if sqs_dlq_alarm_threshold == 0 else 1) + + +@pytest.mark.parametrize( + "task_type", ["text_single_label_classification", "image_single_label_classification", "image_bounding_box"] +) +@pytest.mark.parametrize("labeling_workflow_schedule", ["cron(0 0 * * ? *)", ""]) +@pytest.mark.parametrize("sqs_dlq_alarm_threshold", [0, 1]) +def test_no_cdk_nag_errors( + stack: cdk.Stack, task_type: str, labeling_workflow_schedule: str, sqs_dlq_alarm_threshold: int +) -> None: + cdk.Aspects.of(stack).add(cdk_nag.AwsSolutionsChecks()) + + nag_errors = Annotations.from_stack(stack).find_error( + "*", + Match.string_like_regexp(r"AwsSolutions-.*"), + ) + assert not nag_errors, f"Found {len(nag_errors)} CDK nag errors" diff --git a/one-click-launch.yaml b/one-click-launch.yaml index 139f33c6..dc0322fe 100644 --- a/one-click-launch.yaml +++ b/one-click-launch.yaml @@ -3,8 +3,8 @@ Description: Creates AIOps setup for Developers Parameters: VersionTag: Type: String - Default: v1.6.0 - Description: Version. Should reference version tag in the repository e.g v1.6.0. + Default: v1.7.0 + Description: Version. Should reference version tag in the repository e.g v1.7.0. ManifestPath: Type: String Default: manifests/mlops-sagemaker/deployment.yaml