Changes to serverless pattern

cloudtrail-lambda-dynamo-cdk/README.md

@@ -1,15 +1,14 @@
-# Check Resource Tag Compliance Using CloudTrail, DynamoDB, and Lambda
+# Check S3 Object Tag Compliance Using CloudTrail, DynamoDB, and Lambda
 
-This pattern shows how to leverage CloudTrail resource creation API calls to check for required tags and determine compliance. The resources used in this pattern include CloudTrail, S3, Lambda, and DynamoDB which are all deployed via CDK. From the CloudTrail logs stored in S3, the relevant resource creation events are populated into a DynamoDB table via Lambda. The items written into the table are then checked for the required tags to determine compliance. 
+This pattern shows how to leverage CloudTrail resource creation API calls to check for required S3 object tags and determine compliance. The resources used in this pattern include CloudTrail, S3, Lambda, and DynamoDB which are all deployed via CDK. From the CloudTrail logs stored in S3, PutObject events are populated into a DynamoDB table via Lambda. The items written into the table are then checked for the required tags to determine compliance. 
 
-Learn more about this pattern at Serverless Land Patterns:[Create an AWS account](https://serverlessland.com/patterns/cloudtrail-lambda-dynamodb-cdk)
+Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/cloudtrail-lambda-dynamodb-cdk
 
 Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
 
 ## Requirements
 
-* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS 
-s.
+* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
 * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
 * [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
 * [Node.js installed](https://nodejs.org/en/download)
@@ -29,6 +28,10 @@ s.
     ```
     npm install
     ```
+1. Open the lambda function cloudtrail-lambda-dynamo-cdk/src/lib/lambda/object_tag_checker/index.py, and on line 17 replace the example keys with your required keys:
+    ```
+    required_keys = {"Key1", "Key2", "Key3", "Key4"}
+    ```
 1. Use the following command to generate the AWS CloudFormation template for your CDK application:
     ```
     cdk synth
@@ -40,25 +43,31 @@ s.
 ## Testing
 
 Once the CDK stack has deployed successfully, you can take the following steps to ensure the pattern is working appropriately:
-1. Using the AWS CLI, create a bucket using the following command:
+1. Using the AWS CLI, upload the test_file.txt found in the src folder to the S3 bucket of your choosing using the following command:
     ```
-    aws s3api create-bucket --bucket {bucket-name} --region {your AWS region}
+    aws s3 cp test/test_file.txt s3://<bucket-name>/est/test_file1.txt
     ```
-    If the S3 bucket creation was successful, you should receive the following response:
+    If the file upload was successful, you should receive the following response:
     ```
-    {
-        "Location": "/{bucket-name}"
-    }
+    upload test/test_file.txt to s3://<bucket-name>/test_file1.txt
     ```
-    You can also open the AWS Management Console, navigate to S3, and confirm the created bucket is listed there.
+    You can also open the AWS Management Console, navigate to S3, and confirm the uploaded file is found in the S3 bucket you specified.
+
+1. Using the AWS CLI, upload and tag the test_file.txt found in the src folder to the S3 bucket of your choosing using the following command:
+   ```
+   aws s3 cp test/test_file.txt s3://<bucket-name>/test_file2.txt
+   aws s3api put-object-tagging \
+       --bucket <bucket-name> \
+       --key test_file2.txt \
+       --tagging '{"TagSet": [{ "Key": "Key1", "Value": "key1_value" },{ "Key": "Key2", "Value": "key2_value" },{ "Key": "Key3", "Value": "key3_value" },{ "Key": "Key4", "Value": "key4_value" }]}'
 
-1. Navigate to DynamoDB
+   ```
 
-1. Select the table created for this pattern
+1. In DynamoDB console, Navigate to DynamoDB, "s3-objects-table" table created for this pattern
 
 1. Click 'Explore table items'
 
-1. Within about 5 minutes or less, you should see a new item populated in the DynamoDB table specifying the ARN of the newly created bucket. The 'is_compliant' column should be set to 'false' since the bucket was created with no tags. 
+1. Within a couple of minutes, you should see two new items populated into the DynamoDB table specifying the ARN of the uploaded objects. The ‘is_compliant’ column should be set to ‘false’ for test_file1.txt since the object was uploaded with no tags and set to ‘true’ for test_file2.txt.
 
 ## Cleanup
  

cloudtrail-lambda-dynamo-cdk/src/lib/lambda/object_tag_checker/Event.py

@@ -0,0 +1,12 @@
+class Event:
+    def __init__(self, object_arn, bucket_name, is_compliant, object_key):
+        self.object_arn = object_arn
+        self.bucket_name = bucket_name
+        self.is_compliant = is_compliant
+        self.object_key = object_key
+        self.tags = None
+
+    def validate_compliance(self, required_keys):
+        result = self.tags.difference(required_keys)
+        if result:
+            return result

cloudtrail-lambda-dynamo-cdk/src/lib/lambda/object_tag_checker/index.py

@@ -0,0 +1,93 @@
+import boto3
+import os
+from objects_db import Objects
+from Event import Event
+from botocore.exceptions import ClientError
+from boto3.dynamodb.types import TypeDeserializer
+
+deserializer = TypeDeserializer()
+dynamodb = boto3.resource("dynamodb")
+s3_client = boto3.client("s3")
+table_name = os.environ["TABLE_NAME"]
+
+REQUIRED_KEYS = {
+    "Key1",
+    "Key2",
+    "Key3",
+    "Key4",
+}  # replace with required resource keys
+
+
+def pass_items(event):
+    objects_to_check = []
+
+    for record in event["Records"]:
+        new_image = record["dynamodb"].get("NewImage")
+        python_data = new_image
+        print(python_data)
+        for key, value in python_data.items():
+            if "S" in value:
+                python_data[key] = value["S"]
+            elif "BOOL" in value:
+                python_data[key] = value["BOOL"]
+
+        python_data = Event(**python_data)
+
+        objects_to_check.append(python_data)
+
+    return objects_to_check
+
+
+def add_to_set(tags: list):
+    tags_set = set()
+    for value in tags:
+        tags_set.add(value["Key"].strip())
+    return tags_set
+
+
+def update_keys(item):
+    resource_instance = Objects(dynamodb, table_name)
+    attributes = {"is_compliant": item.is_compliant}
+    resource_instance.add_key(item, attributes)
+
+
+def check_compliance(objects):
+    for i in objects:
+        try:
+            tags_set = set()
+            print(i.bucket_name)
+
+            response = s3_client.get_object_tagging(
+                Bucket=i.bucket_name, Key=i.object_key
+            )
+
+            tags = response["TagSet"]
+
+            if tags:
+                tags_set = add_to_set(tags)
+
+            i.tags = tags_set
+            is_not_compliant = i.validate_compliance(required_keys=REQUIRED_KEYS)
+
+            if is_not_compliant or not i.tags:
+                # further action may be taken here if not compliant (EX: notify admins using SNS)
+
+                i.is_compliant = False
+                print(i.object_key, "is not compliant")
+            else:
+                i.is_compliant = True
+                update_keys(i)
+                print(i.object_key, "is now compliant")
+
+        except ClientError as err:
+            if err.response["Error"]["Code"] == "NoSuchTagSet":
+                print(i.resource_name, "has no tags or does not exist")
+            else:
+                print(err)
+        except Exception as err:
+            print(err)
+
+
+def lambda_handler(event, context):
+    objects = pass_items(event)
+    return check_compliance(objects)

cloudtrail-lambda-dynamo-cdk/src/lib/lambda/object_tag_checker/objects_db.py

@@ -0,0 +1,30 @@
+from botocore.exceptions import ClientError
+
+
+class Objects:
+    def __init__(self, db_client, table_name):
+        self.dyn_client = db_client
+        self.table = self.dyn_client.Table(table_name)
+
+    def add_key(self, item, attributes):
+        update_expression = "SET {}".format(
+            ",".join(f"#{key}=:{key}" for key in attributes)
+        )
+        attribute_values = {f":{key}": value for key, value in attributes.items()}
+        attribute_names = {f"#{key}": key for key in attributes}
+        print(attribute_values)
+
+        try:
+            response = self.table.update_item(
+                Key={
+                    "object_arn": item.object_arn,
+                },
+                UpdateExpression=update_expression,
+                ExpressionAttributeValues=attribute_values,
+                ExpressionAttributeNames=attribute_names,
+            )
+            return response
+
+        except ClientError as err:
+            print(err)
+            raise

cloudtrail-lambda-dynamo-cdk/src/lib/lambda/populate_dynamo/index.py

@@ -1,87 +1,59 @@
 import boto3
-import os
-import json
-import gzip
 from botocore.exceptions import ClientError
+import gzip
+import json
+import os
 
-dynamodb = boto3.resource('dynamodb')
-table_name = os.environ['TABLE_NAME']
+dynamodb = boto3.resource("dynamodb")
+table_name = os.environ["TABLE_NAME"]
 table = dynamodb.Table(table_name)
-s3_client = boto3.client('s3')
+s3_client = boto3.client("s3")
+
 
 def lambda_handler(event, context):
+    print(event)
+
     items_to_add = []
-    bucket_name = os.environ['BUCKET_NAME']
-    
-    for record in event['Records']:
+    cloudtrail_bucket = os.environ["CLOUDTRAIL_BUCKET_NAME"]
+
+    for record in event["Records"]:
         try:
-            if record['s3']['bucket']['name'] != bucket_name:
+            bucket_name = record["s3"]["bucket"]["name"]
+
+            if bucket_name != cloudtrail_bucket:
                 continue
-
-            object_key = record['s3']['object']['key']
-
-            response = s3_client.get_object(Bucket=bucket_name, Key=object_key)
-            log_data = gzip.decompress(response['Body'].read()).decode('utf-8')
-            cloudtrail_logs = json.loads(log_data)['Records']
-
+
+            object_key = record["s3"]["object"]["key"]
+            response = s3_client.get_object(Bucket=cloudtrail_bucket, Key=object_key)
+            log_data = gzip.decompress(response["Body"].read()).decode("utf-8")
+            cloudtrail_logs = json.loads(log_data)["Records"]
+
             for log in cloudtrail_logs:
-                event_name = log['eventName']
-                event_id = log['eventID']
-                event_source = log['eventSource']
-
-                if event_name == 'CreateBucket':
-                    bucket_name = log['requestParameters']['bucketName']
-                    bucket_arn = f'arn:aws:s3:::{bucket_name}'
-
-                    item = {
-                        'resource_arn': bucket_arn,
-                        'event_id': event_id,
-                        'event_source': event_source,
-                        'resource_name': bucket_name,
-                        'is_compliant': False
-                    }
-                    items_to_add.append(item)
-
-                if event_name == 'CreateCluster':
-                    cluster_arn = log['responseElements']['cluster']['clusterArn']
-                    cluster_name = log['requestParameters']['clusterName']
-
+                event_name = log["eventName"]
+                user_identity = log["userIdentity"]
+
+                if (
+                    user_identity.get("type") == "AWSService"
+                    and user_identity.get("invokedBy") == "cloudtrail.amazonaws.com"
+                ):
+                    continue
+
+                if event_name == "PutObject":
+                    bucket_name = log["requestParameters"]["bucketName"]
+                    key = log["requestParameters"]["key"]
+                    object_arn = f"arn:aws:s3:::{bucket_name}/{key}"
                     item = {
-                        'resource_arn': cluster_arn,
-                        'event_id': event_id,
-                        'event_source': event_source,
-                        'resource_name': cluster_name,
-                        'is_compliant': False
+                        "object_arn": object_arn,
+                        "object_key": key,
+                        "bucket_name": bucket_name,
+                        "is_compliant": False,
                     }
                     items_to_add.append(item)
-
-                if event_name.startswith('CreateFunction'):
-                    function_name = log['requestParameters']['functionName']
-                    region = log['awsRegion']
-                    account_id = log['userIdentity']['accountId']
-                    function_arn = f'arn:aws:lambda:{region}:{account_id}:function:{function_name}'
-
-
-                    item = {
-                        'resource_arn': function_arn,
-                        'event_id': event_id,
-                        'event_source': event_source,
-                        'resource_name': function_name,
-                        'is_compliant': False
-                    }
-
-                    if not any(i['resource_arn'] == function_arn for i in items_to_add):
-                        items_to_add.append(item)
-
         except ClientError as err:
             print(err)
-            
+
     if items_to_add:
         with table.batch_writer() as batch:
             for item in items_to_add:
                 batch.put_item(Item=item)
-
-    return {
-        'statusCode': 200,
-        'body': json.dumps('Success') 
-    }
+    return {"statusCode": 200, "body": json.dumps("Success")}