Merge pull request #2402 from anuprajpara/anuprajpara-feature-eventbr…

…idge-pipes-sqs-to-cwlog

New serverless pattern - Amazon SQS to CloudWatch Log group via Amazon EventBridge Pipes filter and transformation

eventbridge-pipes-sqs-to-cwlog/README.md

@@ -0,0 +1,78 @@
+# Filter and Transform Amazon SQS messages with Amazon EventBridge Pipes 
+
+This pattern will use Amazon EventBridge Pipe connecting an Amazon SQS queue with an Amazon CloudWatch Log group. The pipe will apply a filter and transformation before sending the message to the CloudWatch Log group.
+This pattern is implemented with AWS Serverless Application Model (AWS SAM).
+
+Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/eventbridge-pipes-sqs-to-cwlog
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Requirements
+
+- [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
+- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+- [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+- [AWS SAM](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) installed and configured
+
+## Deployment Instructions
+
+1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
+   ```
+   git clone https://github.com/aws-samples/serverless-patterns
+   ```
+2. Change directory to the pattern directory:
+   ```
+   cd serverless-patterns/eventbridge-pipes-sqs-to-cwlog/
+   ```
+3. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yaml file:
+    ```
+    sam deploy --guided
+    ```
+4. During the prompts:
+    * Enter a stack name
+    * Enter `us-east-1` or any other AWS Region. 
+    * Allow SAM CLI to create IAM roles with the required permissions. Please keep all other options to default.
+5. Make a note of the output, which will be used during testing.
+
+## How it works
+
+* The template creates an SQS queue `source-queue`, a CloudWatch Log group `target-cw-log-group`, and EventBridge Pipe.
+* The Amazon EventBridge pipe copies messages from `source-queue` to `target-cw-log-group` only if message payload (JSON) contains a `status` attribute with values `REJECTED` or `RETURNED`. The Amazon EventBridge Pipe will filter and transform the messages before sending it to `target-cw-log-group`.
+
+Replace the "SQS_URL" with your SQS URL in the below command to send message to SQS:
+
+```
+ aws sqs send-message \
+ --queue-url=SQS_URL \
+ --message-body '{"id":"123", "status": "COMPLETED"}'
+
+ aws sqs send-message \
+ --queue-url=SQS_URL \
+ --message-body '{"id":"456", "status": "RETURNED"}'
+
+ aws sqs send-message \
+ --queue-url=SQS_URL \
+ --message-body '{"id":"789", "status": "REJECTED"}'
+
+ aws sqs send-message \
+ --queue-url=SQS_URL \
+ --message-body '{"id":"110", "status": "DELIVERED"}'
+
+Validate the result by reviewing the target CloudWatch Log group.
+Amazon EventBridge Pipe filter will allow only messages with a status = "REJECTED" or "RETURNED".
+The CloudWatch log group contains the following transformed messages. 
+    Order ID 456 requires immediate attention. Order status RETURNED
+    Order ID 789 requires immediate attention. Order status REJECTED
+```
+
+## Delete stack
+
+```
+sam delete
+```
+
+---
+
+Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+SPDX-License-Identifier: MIT-0

eventbridge-pipes-sqs-to-cwlog/eventbridge-pipes-sqs-cwlog.json

@@ -0,0 +1,93 @@
+{
+    "title": "Filter and Transform Amazon SQS messages with Amazon EventBridge Pipes",
+    "description": "Amazon SQS to Amazon CloudWatch Log group via Amazon EventBridge Pipes filter and transformation",
+    "language": "",
+    "level": "200",
+    "framework": "SAM",
+    "introBox": {
+        "headline": "How it works",
+        "text": [
+            "This sample project demonstrates how to use Amazon EventBridge Pipes to connect an Amazon SQS queue with an Amazon CloudWatch Log group. The pipe will apply a filter and transformation before sending the message to the CloudWatch Log group.",
+            "This pattern deploys one SQS queue, one EventBridge Pipe, and a CloudWatch Log group"
+        ]
+    },
+    "gitHub": {
+        "template": {
+            "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-sqs-to-cwlog",
+            "templateURL": "serverless-patterns/eventbridge-pipes-sqs-to-cwlog",
+            "projectFolder": "eventbridge-pipes-sqs-to-cwlog",
+            "templateFile": "template.yaml"
+        }
+    },
+    "resources": {
+        "bullets": [
+            {
+                "text": "Configuring a queue to use with EventBridge Pipes",
+                "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-sqs.html#pipes-sqs-configure-queue"
+            },
+            {
+                "text": "Amazon EventBridge Pipes filtering",
+                "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-filtering.html"
+            },
+            {
+                "text": "Amazon EventBridge Pipes input transformation",
+                "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-input-transformation.html"
+            },
+            {
+                "text": "Amazon EventBridge Pipes targets",
+                "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html"
+            }
+        ]
+    },
+    "deploy": {
+        "text": [
+            "sam deploy --guided"
+        ]
+    },
+    "testing": {
+        "text": [
+            "See the GitHub repo for detailed testing instructions."
+        ]
+    },
+    "cleanup": {
+        "text": [
+            "Delete the stack: <code>sam delete</code>."
+        ]
+    },
+    "authors": [
+        {
+            "name": "Anup Rajpara",
+            "image": "https://drive.google.com/file/d/1MqpPNLCqbU4kvvtTspNXZBqD99aVIJI9/view?usp=sharing",
+            "bio": "Anup is passionate about serverless & event-driven architectures.",
+            "linkedin": "anup-rajpara-developer"
+        }
+    ],
+    "patternArch": {
+        "icon1": {
+            "x": 15,
+            "y": 50,
+            "service": "sqs",
+            "label": "Amazon SQS"
+        },
+        "icon2": {
+            "x": 45,
+            "y": 50,
+            "service": "eventbridge-pipes",
+            "label": "Amazon EventBridge Pipes"
+        },
+        "icon3": {
+            "x": 80,
+            "y": 50,
+            "service": "cloudwatch",
+            "label": "Amazon CloudWatch Logs"
+        },
+        "line1": {
+            "from": "icon1",
+            "to": "icon2"
+        },
+        "line2": {
+            "from": "icon2",
+            "to": "icon3"
+        }
+    }
+}

eventbridge-pipes-sqs-to-cwlog/example-pattern.json

@@ -0,0 +1,65 @@
+{
+  "title": "Amazon SQS to CloudWatch Log group via Amazon EventBridge",
+  "description": "Amazon SQS to CloudWatch Log group via Amazon EventBridge Pipes filter and transformation",
+  "language": "",
+  "level": "200",
+  "framework": "SAM",
+  "introBox": {
+    "headline": "How it works",
+    "text": [
+      "This sample project demonstrates how to use Amazon EventBridge Pipes to connect an Amazon SQS queue with an AWS CloudWatch Log group. The pipe will apply a filter and transformation before sending the message to AWS CloudWatch Log group.",
+      "This pattern deploys one SQS queue, EventBridge Pipe, and CloudWatch Log group"
+    ]
+  },
+  "gitHub": {
+    "template": {
+      "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-sqs-to-cwlog",
+      "templateURL": "serverless-patterns/eventbridge-pipes-sqs-to-cwlog",
+      "projectFolder": "eventbridge-pipes-sqs-to-cwlog",
+      "templateFile": "template.yaml"
+    }
+  },
+  "resources": {
+    "bullets": [
+      {
+        "text": "Configuring a queue to use with EventBridge Pipes",
+        "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-sqs.html#pipes-sqs-configure-queue"
+      },
+      {
+        "text": "Amazon EventBridge Pipes filtering",
+        "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-filtering.html"
+      },
+      {
+        "text": "Amazon EventBridge Pipes input transformation",
+        "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-input-transformation.html"
+      },
+      {
+        "text": "Amazon EventBridge Pipes targets",
+        "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html"
+      }
+    ]
+  },
+  "deploy": {
+    "text": [
+      "sam deploy --guided"
+    ]
+  },
+  "testing": {
+    "text": [
+      "See the GitHub repo for detailed testing instructions."
+    ]
+  },
+  "cleanup": {
+    "text": [
+      "Delete the stack: <code>sam delete</code>."
+    ]
+  },
+  "authors": [
+    {
+        "name": "Anup Rajpara",
+        "image": "https://drive.google.com/file/d/1MqpPNLCqbU4kvvtTspNXZBqD99aVIJI9/view?usp=sharing",
+        "bio":"Anup is passionate about serverless & event-driven architectures.",
+        "linkedin": "https://www.linkedin.com/in/anup-rajpara-developer/"
+    }
+  ]
+}

eventbridge-pipes-sqs-to-cwlog/template.yaml

@@ -0,0 +1,79 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+Description: Sample SAM template that shows the filter and transformation capability of Amazon EventBridge Pipes.(uksb-1tthgi812) (tag:eventbridge-pipes-sqs-to-cwlog)
+
+Resources:
+  # source queue
+  SourceQueue:
+    Type: AWS::SQS::Queue
+
+  # CloudWatch Log group as target
+  TargetCWLogGroup:
+    Type: AWS::Logs::LogGroup
+    Properties:
+      RetentionInDays: 7
+
+  # Define the pipe
+  Pipe:
+    Type: AWS::Pipes::Pipe
+    Properties:
+      RoleArn: !GetAtt PipeRole.Arn
+      Source: !GetAtt SourceQueue.Arn
+      Target: !GetAtt TargetCWLogGroup.Arn
+      # filter the incoming SQS messages
+      SourceParameters:
+        SqsQueueParameters:
+          BatchSize: 1
+        FilterCriteria:
+          Filters:
+            - Pattern: '{"body": {"status": ["REJECTED", "RETURNED"]}}'
+      TargetParameters: 
+        InputTemplate: 'Order ID <$.body.id> requires immediate attention. Order status <$.body.status>'
+
+  # Role for Pipe
+  PipeRole:
+    Type: AWS::IAM::Role
+    Properties:
+      AssumeRolePolicyDocument:
+        Version: 2012-10-17
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service:
+                - pipes.amazonaws.com
+            Action:
+              - sts:AssumeRole
+      Policies:
+        - PolicyName: SourceQueuePolicy
+          PolicyDocument:
+            Version: 2012-10-17
+            Statement:
+              - Effect: Allow
+                Action:
+                  - 'sqs:ReceiveMessage'
+                  - 'sqs:DeleteMessage'
+                  - 'sqs:GetQueueAttributes'
+                Resource: !GetAtt SourceQueue.Arn
+        - PolicyName: TargetCWLogGrpPolicy
+          PolicyDocument:
+            Version: 2012-10-17
+            Statement:
+              - Effect: Allow
+                Action:
+                  - 'logs:CreateLogStream'
+                  - 'logs:PutLogEvents'
+                Resource: !GetAtt TargetCWLogGroup.Arn
+
+Outputs:
+  SourceQueueURL:
+    Description: "Source Queue URL"
+    Value: !GetAtt SourceQueue.QueueUrl
+  TargetCWLogGroup:
+    Description: 'Target Log Group Name'
+    Value: !GetAtt TargetCWLogGroup.Arn
+  Pipe:
+    Description: 'Pipe info'
+    Value: !GetAtt Pipe.Arn
+  PipeRole:
+    Description: 'Pipe role'
+    Value: !GetAtt PipeRole.Arn