Merge branch 'main' into dependabot/pip/data/mwaa/requirements/mysql-…

…connector-python-9.1.0

CHANGELOG.md

@@ -7,10 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## UNRELEASED
 
+### **Added**
+- added `mlflow-ai-gw-image` module
+
+### **Changed**
+
+
+## v1.7.0
+
 ### **Added**
 - added GitHub as code repository option along with AWS CodeCommit for sagemaker templates batch_inference, finetune_llm_evaluation, hf_import_models and xgboost_abalone
 - added `ray-orchestrator` module
 - added GitHub as alternate option for code repository support along with AWS CodeCommit for sagemaker-templates-service-catalog module
+- added SageMaker ground truth labeling module
 
 ### **Changed**
 - updated manifests to idf release 1.12.0

DEPLOYMENT.md

@@ -5,7 +5,7 @@
 1. Clone the repository and checkout a release branch using the below command:
 
 ```
-git clone --origin upstream --branch release/1.6.0 https://github.com/awslabs/aiops-modules
+git clone --origin upstream --branch release/1.7.0 https://github.com/awslabs/aiops-modules
 ```
 The release version can be replaced with the version of interest.
 

README.md

@@ -50,13 +50,15 @@ End-to-end example use-cases built using modules in this repository.
 | [SageMaker Model Package Promote Pipeline Module](modules/sagemaker/sagemaker-model-package-promote-pipeline/README.md)  | Deploy a Pipeline to promote SageMaker Model Packages in a multi-account setup. The pipeline can be triggered through an EventBridge rule in reaction of a SageMaker Model Package Group state event change (Approved/Rejected). Once the pipeline is triggered, it will promote the latest approved model package, if one is found.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
 | [SageMaker Model Monitoring Module](modules/sagemaker/sagemaker-model-monitoring/README.md)                              | Deploy data quality, model quality, model bias, and model explainability monitoring jobs which run against a SageMaker Endpoint.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
 | [SageMaker Model CICD Module](modules/sagemaker/sagemaker-model-cicd/README.md)                                          | Creates a comprehensive CICD pipeline using AWS CodePipelines to build and deploy a ML model on SageMaker.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| [SageMaker Ground Truth Labeling Module](modules/sagemaker/sagemaker-ground-truth-labeling/README.md)                    | Creates a state machine to allow labeling of images and text file, uploaded to the upload bucket, using various built-in task types in SageMaker Ground Truth.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 
 ### Mlflow Modules
 
-| Type                                                                    | Description                                                                                                                                                                                       |
-|-------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [Mlflow Image Module](modules/mlflow/mlflow-image/README.md)            | Creates Mlflow Docker container image and pushes the image to Elastic Container Registry                                                                                                          |
-| [Mlflow on AWS Fargate Module](modules/mlflow/mlflow-fargate/README.md) | Runs Mlflow container on AWS Fargate in a load-balanced Elastic Container Service. Supports Elastic File System and Relational Database Store for metadata persistence, and S3 for artifact store |
+| Type                                                                          | Description                                                                                                                                                                                       |
+|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Mlflow Image Module](modules/mlflow/mlflow-image/README.md)                  | Creates Mlflow Tracing Server Docker image and pushes the image to Elastic Container Registry                                                                                                     |
+| [Mlflow on AWS Fargate Module](modules/mlflow/mlflow-fargate/README.md)       | Runs Mlflow container on AWS Fargate in a load-balanced Elastic Container Service. Supports Elastic File System and Relational Database Store for metadata persistence, and S3 for artifact store |
+| [Mlflow AI Gateway Image Module](modules/mlflow/mlflow-ai-gw-image/README.md) | Creates Mlflow AI Gateway Docker image and pushes the image to Elastic Container Registry                                                                                                         |
 
 ### FMOps/LLMOps Modules
 

VERSION

@@ -1 +1 @@
-1.6.0
+1.7.0

examples/manifests/event-bus-modules.yaml

@@ -1,5 +1,5 @@
 name: event-bus
-path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/event-bus?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/event-bus?ref=release/1.7.0&depth=1
 targetAccount: tooling
 parameters:
   - name: event_bus_name

examples/manifests/fmops-modules.yaml

@@ -1,5 +1,5 @@
 name: jumpstart-hf-asrwhisper-endpoint
-path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-jumpstart-fm-endpoint?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-jumpstart-fm-endpoint?ref=release/1.7.0&depth=1
 targetAccount: primary
 parameters:
   - name: jump-start-model-name

examples/manifests/personas-modules.yaml

@@ -1,5 +1,5 @@
 name: personas
-path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/personas?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/personas?ref=release/1.7.0&depth=1
 parameters:
   - name: bucket-name
     value: my-bucket

examples/manifests/sagemaker-endpoints-modules.yaml

@@ -1,7 +1,7 @@
 # This is an example manifest group.
 # Replace the parameters with the parameters for your model below prior the deployment.
 name: endpoint
-path: git::https://github.com/awslabs/aiops-modules.git//sagemaker/fmops/sagemaker-endpoint?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//sagemaker/fmops/sagemaker-endpoint?ref=release/1.7.0&depth=1
 parameters:
   - name: sagemaker_project_id
     value: project-1

examples/manifests/sagemaker-hugging-face.yml

@@ -1,5 +1,5 @@
 name: hugging-face-mistral-endpoint
-path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-hugging-face-endpoint?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-hugging-face-endpoint?ref=release/1.7.0&depth=1
 targetAccount: primary
 parameters:
   - name: hugging-face-model-id

examples/manifests/sagemaker-model-monitoring-modules.yaml

@@ -1,7 +1,7 @@
 # This is an example manifest group.
 # Replace the parameters with the parameters for your model below prior the deployment.
 name: monitor
-path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-monitoring?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-monitoring?ref=release/1.7.0&depth=1
 parameters:
   - name: sagemaker_project_id
     value: project-1

examples/manifests/sagemaker-model-package-group-modules.yaml

@@ -1,5 +1,5 @@
 name: source-model-package-group
-path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-group?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-group?ref=release/1.7.0&depth=1
 targetAccount: primary
 parameters:
   - name: model_package_group_name

examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml

@@ -1,5 +1,5 @@
 name: model-pipeline
-path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-promote-pipeline?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-promote-pipeline?ref=release/1.7.0&depth=1
 targetAccount: tooling
 parameters:
   - name: source_model_package_group_arn

examples/manifests/sagemaker-notebook-modules.yaml

@@ -1,5 +1,5 @@
 name: notebook
-path: git::https://github.com/awslabs/aiops-modules.git//modules/modules/sagemaker/sagemaker-notebook?ref=release/1.6.0&depth=1
+path: git::https://github.com/awslabs/aiops-modules.git//modules/modules/sagemaker/sagemaker-notebook?ref=release/1.7.0&depth=1
 parameters:
   - name: notebook_name
     value: dummy

examples/manifests/sagemaker-templates-modules-codecommit.yaml

@@ -1,5 +1,5 @@
 name: service-catalog
-path: modules/sagemaker/sagemaker-templates-service-catalog
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.7.0&depth=1
 targetAccount: primary
 parameters:
   - name: repository-type

examples/manifests/sagemaker-templates-modules-github.yaml

@@ -1,5 +1,5 @@
 name: service-catalog
-path: modules/sagemaker/sagemaker-templates-service-catalog
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-templates-service-catalog?ref=release/1.7.0&depth=1
 targetAccount: primary
 parameters:
   - name: repository-type

examples/sagemaker-ground-truth-labeling/README.md

@@ -0,0 +1,40 @@
+# SageMaker Ground truth labeling examples
+
+### Description
+
+This folder contains examples for each of the built-in task types for the sagemaker ground truth module. Each folder contains an example manifest as well as any necessary templates. Please upload the templates to an S3 bucket and update the manifest with the correct location.
+
+### Additional workers
+
+For tasks without a verification step (all except `image_bounding_box` and `image_semantic_segmentation`) we recommend increasing the number of human reviewers per object to increase accuracy. This will only work if you have at least that many reviewers in your workteam, as the same reviewer cannot review the same item twice. To adjust the number of workers add the additional parameters below to your manifest:
+
+```yaml
+  - name: labeling-human-task-config
+    value:
+      NumberOfHumanWorkersPerDataObject: 5
+      TaskAvailabilityLifetimeInSeconds: 21600
+      TaskTimeLimitInSeconds: 300
+```
+
+### Using public workforce
+
+As mentioned in the README you can use a public workforce for your task if you wish (at an additional cost). More information on using a public workforce like Amazon Mechanical Turk is available [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-public.html). Labeling and verification task prices is specified in USD, see [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_PublicWorkforceTaskPrice.html) for allowed values. [This page](https://aws.amazon.com/sagemaker/groundtruth/pricing/) provides suggested pricing based on task type. To use a public workforce add / adjust the following parameters to your manifest:
+
+```yaml
+  - name: labeling-workteam-arn
+    value: 'arn:aws:sagemaker:<region>:394669845002:workteam/public-crowd/default'
+  - name: labeling-task-price
+    value:
+      AmountInUsd:
+        Dollars: 0
+        Cents: 3
+        TenthFractionsOfACent: 6
+  - name: verification-workteam-arn
+    value: 'arn:aws:sagemaker:<region>:394669845002:workteam/public-crowd/default'
+  - name: verification-task-price
+    value:
+      AmountInUsd:
+        Dollars: 0
+        Cents: 3
+        TenthFractionsOfACent: 6
+```

examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_categories.json

@@ -0,0 +1 @@
+{"labels": [{"label": "Plane"}, {"label": "Boat"}]}

examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_labeling_template.html

@@ -0,0 +1,28 @@
+<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
+<crowd-form>
+  <crowd-bounding-box
+    name="boundingBox"
+    src="{{ task.input.taskObject | grant_read_access }}"
+    header="Please draw a box around all planes and boats in the image."
+    labels="{{ task.input.labels | to_json | escape }}"
+  >
+    <full-instructions header="Bounding box instructions">
+      <ol>
+        <li><strong>Inspect</strong> the image</li>
+        <li><strong>Determine</strong> if the specified label is/are visible in the picture.</li>
+        <li><strong>Outline</strong> each instance of the specified label in the image using the provided “Box” tool.</li>
+      </ol>
+      <ul>
+        <li>Boxes should fit tight around each object</li>
+        <li>Do not include parts of the object are overlapping or that cannot be seen, even though you think you can interpolate the whole shape.</li>
+        <li>Avoid including shadows.</li>
+        <li>If the target is off screen, draw the box up to the edge of the image.</li>
+      </ul>
+    </full-instructions>
+
+    <short-instructions>
+      <Strong>Outline</strong> each instance of the specified label in the image using the provided “Box” tool.
+      <!-- You may wish to include examples of correctly and incorrectly labeled images here -->
+    </short-instructions>
+  </crowd-bounding-box>
+</crowd-form>

examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_categories.json

@@ -0,0 +1 @@
+{"labels":[{"label":"Label(s) correct"},{"label":"Incorrect label - missed object"},{"label":"Incorrect label - bounding box not accurate enough"}]}

examples/sagemaker-ground-truth-labeling/image_bounding_box/image_bounding_box_verification_template.liquid

@@ -0,0 +1,39 @@
+<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
+<crowd-form>
+  <crowd-image-classifier
+    name="annotatedResult"
+    src="{{ task.input.taskObject | grant_read_access }}"
+    header="Review the existing labels on the objects and choose the appropriate option."
+    categories="{{ task.input.labels | to_json | escape }}"
+    overlay="{
+      'boundingBox': {
+        labels: ['Plane','Boat'],
+        value: [
+          {% for box in task.input.manifestLine["label"].annotations %}
+            {% capture class_id %}{{ box.class_id }}{% endcapture %}
+            {% assign label = task.input.manifestLine["label-metadata"].class-map[class_id] %}
+          {
+            label: {{label | to_json}},
+            left: {{box.left}},
+            top: {{box.top}},
+            width: {{box.width}},
+            height: {{box.height}},
+          },
+          {% endfor %}
+        ]
+      }
+    }"
+  >
+    <full-instructions header="Label verification - Bounding box instructions">
+      <ol>
+        <li><strong>Read</strong> the task carefully and inspect the image.</li>
+        <li><strong>Read</strong> the options and review the examples provided to understand more about the labels.</li>
+        <li><strong>Choose</strong> the appropriate label that best suits the image.</li>
+      </ol>
+    </full-instructions>
+    <short-instructions>
+      <strong>Choose</strong> the appropriate label that best suits the image.
+      <!-- You may wish to include examples of correctly and incorrectly labeled images here -->
+    </short-instructions>
+  </crowd-image-classifier>
+</crowd-form>

examples/sagemaker-ground-truth-labeling/image_bounding_box/sagemaker-ground-truth-labeling.yaml

@@ -0,0 +1,34 @@
+name: ground-truth-labeling
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-ground-truth-labeling?ref=release/1.7.0&depth=1
+targetAccount: primary
+parameters:
+  - name: job_name
+    value: 'plane-and-boat-bounding-box'
+  - name: task_type
+    value: 'image_bounding_box'
+
+  - name: labeling-workteam-arn
+    value: 'arn:aws:sagemaker:<region>:<account>:workteam/private-crowd/<workteam_name>'
+  - name: labeling-instructions-template-s3-uri
+    value: 's3://<bucket_name>/image_bounding_box_labeling_template.html'
+  - name: labeling-categories-s3-uri
+    value: 's3://<bucket_name>/image_bounding_box_labeling_categories.json'
+  - name: labeling-task-title
+    value: 'Labeling - Bounding boxes: Draw bounding boxes around all planes and boats in the image'
+  - name: labeling-task-description
+    value: 'Draw bounding boxes around all planes and boats in the image'
+  - name: labeling-task-keywords
+    value: ['image', 'object', 'detection']
+
+  - name: verification-workteam-arn
+    value: 'arn:aws:sagemaker:<region>:<account>:workteam/private-crowd/<workteam_name>'
+  - name: verification-instructions-template-s3-uri
+    value: 's3://<bucket_name>/image_bounding_box_verification_template.liquid'
+  - name: verification-categories-s3-uri
+    value: 's3://<bucket_name>/image_bounding_box_verification_categories.json'
+  - name: verification-task-title
+    value: 'Label verification - Bounding boxes: Review the existing labels on the objects and choose the appropriate option.'
+  - name: verification-task-description
+    value: 'Verify that all of the planes and boats in the image are correctly labeled'
+  - name: verification-task-keywords
+    value: ['image', 'object', 'detection', 'label verification', 'bounding boxes']

examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_categories.json

@@ -0,0 +1 @@
+{"labels": [{"label": "Plane"}, {"label": "Boat"}]}

examples/sagemaker-ground-truth-labeling/image_multi_label_classification/image_multi_label_labeling_template.html

@@ -0,0 +1,21 @@
+<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
+<crowd-form>
+  <crowd-image-classifier-multi-select
+    name="crowd-image-classifier-multi-select"
+    src="{{ task.input.taskObject | grant_read_access }}"
+    header="Please select the correct categories for this image"
+    categories="{{ task.input.labels | to_json | escape }}"
+    exclusion-category="{ text: 'None of the above' }"
+  >
+	<full-instructions header="Classification Instructions">
+      <p>If more than one label applies to the image, select multiple labels.</p>
+      <p>If no labels apply, select <b>None of the above</b></p>
+    </full-instructions>
+
+    <short-instructions>
+      <p>Read the task carefully and inspect the image.</p>
+      <p>Choose the appropriate label(s) that best suit the image.</p>
+      <!-- You may wish to include examples of correctly and incorrectly labeled images here -->
+    </short-instructions>
+  </crowd-image-classifier-multi-select>
+</crowd-form>