diff --git a/ami-recycle-bin/README.md b/ami-recycle-bin/README.md new file mode 100644 index 000000000..6c3505a71 --- /dev/null +++ b/ami-recycle-bin/README.md @@ -0,0 +1,87 @@ +# AMI de-registration with AWS Lambda and retention in Amazon EC2 Recycle Bin + +## Description + +In this pattern an Amazon EventBridge rule triggers an AWS Lambda function which deregisters an Amazon Machine Image (AMI), deletes the associated snapshot and moves them to the Recycle Bin for retention. + +The template creates all the necessary resources including an Amazon EventBridge Rule that triggers the AWS Lambda function once every day. Additionally, Recycle Bin rules for AMI and EBS Snapshots are created to retain deleted resources matching the resources for a retention period. + +The AWS Lambda function automates the expiration of Amazon Machine Images (AMIs) by moving the AMIs and their associated snapshots to Recycle Bin. Recycle Bin is a feature of Amazon Elastic Compute Cloud (EC2) that allows you to retain AMIs that you have de-registered for a specified retention period, providing an opportunity to recover them if needed. To recover the deleted AMI, its associated snapshot should be recovered first. The Lambda function also adds corresponding tags to both, the AMI and the EBS snapshot, before moving them to Recycle Bin. + + +Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/ami-recycle-bin + +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [Terraform Installed](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) Required Terraform version >= 4.61.0 + +## Pre-requisite for Testing +An AMI to deregister that has the following Tags: + +| Key | Value | +| -------- | ------- | +| Expire-After | Date in Zulu format (e.g. 2024-08-30T17:39:00Z) | +| Same value as resource_tag_key in src/variables.tf | Same value as resource_tag_value in src/variables.tf | + + +## Deployment Instructions +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ``` + git clone https://github.com/aws-samples/serverless-patterns + ``` +2. Change directory to the pattern directory: + ``` + cd ami-recycle-bin + ``` +3. Bootstrap the input variables in the following file that are used in the Terraform configuration + ``` + src/variables.tf + ``` +4. Initialize a new or existing Terraform working directory by downloading required provider plugins and modules + ``` + terraform init + ``` +5. Create an execution plan that shows the changes Terraform will make to your infrastructure based on the current configuration files + ``` + terraform plan + ``` +6. Apply the changes defined in the Terraform configuration to the infrastructure. Provide the prompts as required + ``` + terraform apply + ``` + +## How It Works + +Following is the architectural diagram to demonstrate how the pattern works: + +![alt text](src/ami-recycle-bin.png) + +1. An Amazon Eventbridge rule is configured to run daily (on a schedule) with AWS Lambda function as a target +2. The Lambda function performs the following: +- Verifies that Recycle Bin rules matching the `resource_tag_key` and `resource_tag_value` as bootstraped in the `variables.tf` exists +- Filters AMIs matching the `resource_tag_key` and `resource_tag_value` and contains the `Expire-After` tag +- Determines whether any of the filtered AMI are expired using the `Expire-After` tag +- Tags the expired AMI with its Snapshot Id and the associated snapshot with the AMI Id if required for recovery +- Deprecates the expired AMI and deletes its associated snapshot + +## Testing + +1. Verify the the expired AMI and its snapshot is retained in the Recycle Bin after deletion +2. Verify the AMI and its snapshot can be recovered after deletion before the retention period ends + +## Cleanup + + Delete the resources + +``` +terraform destroy +``` +---- +Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 diff --git a/ami-recycle-bin/ami-recycle-bin.json b/ami-recycle-bin/ami-recycle-bin.json new file mode 100644 index 000000000..b69f89174 --- /dev/null +++ b/ami-recycle-bin/ami-recycle-bin.json @@ -0,0 +1,85 @@ +{ + "title": "AMI de-registration with AWS Lambda and retention in Amazon EC2 Recycle Bin", + "description": "This project demonstrates a pattern to deregister and retain expired AMI and its snapshot with AWS Lambda and Amazon EC2 Recycle Bin", + "language": "", + "level": "200", + "framework": "Terraform", + "introBox": { + "headline": "How it works", + "text": [ + "In this pattern an Amazon EventBridge rule triggers an AWS Lambda function which deregisters an Amazon Machine Image (AMI), deletes the associated snapshot and moves them to the Recycle Bin for retention.", + "The AWS Lambda function automates the expiration of Amazon Machine Images (AMIs) by moving the AMIs and their associated snapshots to Recycle Bin. Recycle Bin is a feature of Amazon Elastic Compute Cloud (EC2) that allows you to retain AMIs that you have de-registered for a specified retention period, providing an opportunity to recover them if needed. To recover the deleted AMI, its associated snapshot should be recovered first. The Lambda function also adds corresponding tags to both, the AMI and the EBS snapshot, before moving them to Recycle Bin." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/ami-recycle-bin", + "templateURL": "serverless-patterns/ami-recycle-bin", + "projectFolder": "ami-recycle-bin", + "templateFile": "main.tf" + } + }, + "resources": { + "bullets": [ + { + "text": "Recover deleted Amazon EBS snapshots and EBS-backed AMIs with Recycle Bin", + "link": "https://docs.aws.amazon.com/ebs/latest/userguide/recycle-bin.html" + } + ] + }, + "deploy": { + "text": [ + "terraform init", + "terraform plan", + "terraform apply" + ] + }, + "testing": { + "text": [ + "See the Github repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "terraform destroy" + ] + }, + "authors": [ + { + "name": "Divya Vijendra Girase", + "image": "https://avatars.githubusercontent.com/u/172667506?v=4", + "bio": "I am a Cloud Infrastructure Architect at AWS and I work with our strategic customers to build, run and maintain their infrastructure on AWS.", + "linkedin": "divya-girase" + } + ], + "patternArch": { + "icon1": { + "x": 20, + "y": 50, + "service": "eventbridge", + "label": "Amazon EventBridge rule" + }, + "icon2": { + "x": 50, + "y": 50, + "service": "lambda", + "label": "AWS Lambda" + }, + "icon3": { + "x": 80, + "y": 50, + "service": "ec2", + "label": "Amazon EC2" + }, + "line1": { + "from": "icon1", + "to": "icon2", + "label": "" + }, + "line2": { + "from": "icon2", + "to": "icon3", + "label": "" + } + } +} diff --git a/ami-recycle-bin/data.tf b/ami-recycle-bin/data.tf new file mode 100644 index 000000000..c059c623a --- /dev/null +++ b/ami-recycle-bin/data.tf @@ -0,0 +1,8 @@ +# data source to lookup information about the current AWS partition in which Terraform is working +data "aws_partition" "current" {} + +# data source to get the access to the effective Account ID, User ID, and ARN in which Terraform is authorized +data "aws_caller_identity" "current" {} + +# data source to obtain the name of the AWS region configured on the provider +data "aws_region" "current" {} \ No newline at end of file diff --git a/ami-recycle-bin/example-pattern.json b/ami-recycle-bin/example-pattern.json new file mode 100644 index 000000000..6e08b07c1 --- /dev/null +++ b/ami-recycle-bin/example-pattern.json @@ -0,0 +1,55 @@ +{ + "title": "AMI de-registration with AWS Lambda and retention in Amazon EC2 Recycle Bin", + "description": "This project demonstrates a pattern to deregister and retain expired AMI and its snapshot with AWS Lambda and Amazon EC2 Recycle Bin", + "language": "", + "level": "200", + "framework": "Terraform", + "introBox": { + "headline": "How it works", + "text": [ + "In this pattern an Amazon EventBridge rule triggers an AWS Lambda function which deregisters an Amazon Machine Image (AMI), deletes the associated snapshot and moves them to the Recycle Bin for retention.", + "The AWS Lambda function automates the expiration of Amazon Machine Images (AMIs) by moving the AMIs and their associated snapshots to Recycle Bin. Recycle Bin is a feature of Amazon Elastic Compute Cloud (EC2) that allows you to retain AMIs that you have de-registered for a specified retention period, providing an opportunity to recover them if needed. To recover the deleted AMI, its associated snapshot should be recovered first. The Lambda function also adds corresponding tags to both, the AMI and the EBS snapshot, before moving them to Recycle Bin." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/ami-recycle-bin", + "templateURL": "serverless-patterns/ami-recycle-bin", + "projectFolder": "ami-recycle-bin", + "templateFile": "main.tf" + } + }, + "resources": { + "bullets": [ + { + "text": "Recover deleted Amazon EBS snapshots and EBS-backed AMIs with Recycle Bin", + "link": "https://docs.aws.amazon.com/ebs/latest/userguide/recycle-bin.html" + } + ] + }, + "deploy": { + "text": [ + "terraform init", + "terraform plan", + "terraform apply" + ] + }, + "testing": { + "text": [ + "See the Github repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "terraform destroy" + ] + }, + "authors": [ + { + "name": "Divya Vijendra Girase", + "image": "https://avatars.githubusercontent.com/u/172667506?v=4", + "bio": "I am a Cloud Infrastructure Architect at AWS and I work with our strategic customers to build, run and maintain their infrastructure on AWS.", + "linkedin": "divya-girase" + } + ] + } \ No newline at end of file diff --git a/ami-recycle-bin/main.tf b/ami-recycle-bin/main.tf new file mode 100644 index 000000000..edfa781d0 --- /dev/null +++ b/ami-recycle-bin/main.tf @@ -0,0 +1,168 @@ +# Default terraform provider configuration to manage resources in a region of the aws cloud provider +# To provision resources in a different region, update the region variable in variables.tf +provider "aws" { + region = var.region +} + +# Terraform block to enable hashicorp/aws provider and use the version constraint +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "5.65.0" + } + } +} + +resource "aws_cloudwatch_event_rule" "event_rule" { + name = "invoke-lambda-daily" + description = "Invoke a Lambda function once per day" + schedule_expression = "rate(1 day)" + tags = { + "Name" = "invoke-lambda-daily" + } +} + +resource "aws_cloudwatch_event_target" "event_target" { + rule = aws_cloudwatch_event_rule.event_rule.name + target_id = "InvokeLambda" + arn = aws_lambda_function.lambda_function.arn +} + +resource "aws_lambda_permission" "lambda_permission" { + statement_id = "AllowExecutionFromEventBridge" + action = "lambda:InvokeFunction" + function_name = aws_lambda_function.lambda_function.function_name + principal = "events.amazonaws.com" + source_arn = aws_cloudwatch_event_rule.event_rule.arn +} + +resource "aws_cloudwatch_log_group" "example" { + name = "/aws/lambda/${aws_lambda_function.lambda_function.function_name}" + retention_in_days = var.cw_retention_period +} + +resource "aws_lambda_function" "lambda_function" { + filename = "src/ami-recycle-lambda.zip" + function_name = var.function_name + description = var.function_description + role = aws_iam_role.lambda_role.arn + handler = "ami-recycle-lambda.lambda_handler" + runtime = var.function_runtime + environment { + variables = { + "RECYCLE_BIN_TAG_KEY" = var.resource_tag_key + "RECYCLE_BIN_TAG_VALUE" = var.resource_tag_value + "RBIN_RETENTION_PERIOD_VALUE" = var.rbin_retention_period_value + "RBIN_RETENTION_PERIOD_UNIT" = var.rbin_retention_period_unit + } + } + timeout = var.function_timeout + memory_size = var.memory_size + source_code_hash = filebase64sha256("src/ami-recycle-lambda.zip") + tags = { + "Name" = var.function_name + } +} + +resource "aws_iam_role" "lambda_role" { + name = "ami-recycle-lambda-role" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Action = "sts:AssumeRole" + Effect = "Allow" + Sid = "" + Principal = { + Service = "lambda.amazonaws.com" + } + }] + }) + + inline_policy { + name = "ami-recycle-lambda-policy" + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Action = [ + "logs:CreateLogGroup" + ] + Resource = "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:*" + }, + { + Effect = "Allow" + Action = [ + "logs:CreateLogStream", + "logs:PutLogEvents" + ] + Resource = "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/lambda/${var.function_name}:*" + }, + { + Effect = "Allow" + Action = [ + "ec2:CreateTags", + "ec2:DeregisterImage", + "ec2:DeleteSnapshot" + ] + Resource = [ + "arn:aws:ec2:${data.aws_region.current.name}::image/*", + "arn:aws:ec2:${data.aws_region.current.name}::snapshot/*" + ] + }, + { + Effect = "Allow" + Action = [ + "ec2:DescribeTags", + "ec2:DescribeImages", + "ec2:DescribeSnapshots", + "rbin:ListRules" + ] + Resource = "*" + } + ] + }) + } + tags = { + "Name" = "ami-recycle-lambda-role" + } +} + +resource "aws_rbin_rule" "snapshot_rbin" { + description = "Recycle bin rule to retain deleted snapshots" + resource_type = "EBS_SNAPSHOT" + + retention_period { + retention_period_value = var.rbin_retention_period_value + retention_period_unit = var.rbin_retention_period_unit + } + + resource_tags { + resource_tag_key = var.resource_tag_key + resource_tag_value = var.resource_tag_value + } + + tags = { + "Name" = "Snapshot-Recycle-Bin" + } +} + +resource "aws_rbin_rule" "image_rbin" { + description = "Recycle bin rule to retain deleted snapshots" + resource_type = "EC2_IMAGE" + + retention_period { + retention_period_value = var.rbin_retention_period_value + retention_period_unit = var.rbin_retention_period_unit + } + + resource_tags { + resource_tag_key = var.resource_tag_key + resource_tag_value = var.resource_tag_value + } + + tags = { + "Name" = "EC2-Image-Recycle-Bin" + } +} diff --git a/ami-recycle-bin/src/ami-recycle-bin.png b/ami-recycle-bin/src/ami-recycle-bin.png new file mode 100644 index 000000000..5a11a60d3 Binary files /dev/null and b/ami-recycle-bin/src/ami-recycle-bin.png differ diff --git a/ami-recycle-bin/src/ami-recycle-lambda.py b/ami-recycle-bin/src/ami-recycle-lambda.py new file mode 100644 index 000000000..762c64f9b --- /dev/null +++ b/ami-recycle-bin/src/ami-recycle-lambda.py @@ -0,0 +1,135 @@ +import logging +import os +import json +import boto3 +from datetime import datetime + +logger = logging.getLogger(__name__) +logger.setLevel(logging.INFO) + +# Create required clients +ec2_client = boto3.client("ec2") +rbin_client = boto3.client("rbin") + +resource_tag_key = os.getenv("RECYCLE_BIN_TAG_KEY", "Project") +resource_tag_value = os.getenv("RECYCLE_BIN_TAG_VALUE", "Test-Retention") + +# Create required tags for the resources +def manage_tags(imageid, snapshotid): + # Add snapshot id to the image + ec2_tag_response = ec2_client.create_tags( + Resources=[imageid], + Tags=[ + {"Key": "SnapshotId", "Value": snapshotid}, + ], + ) + + # Before deleting the snapshot verify that the snapshot has the tags matching the recycle bin rule + # AMIs are already filtered by the required tag name + # Add image id to the snapshot + snapshots = ec2_client.describe_tags( + Filters=[ + {"Name": "resource-id", "Values": [snapshotid]}, + {"Name": "tag:" + resource_tag_key, "Values": [resource_tag_value]}, + {"Name": "resource-type", "Values": ["snapshot"]}, + ] + ) + + snapshot_tags = [{"Key": "ImageId", "Value": imageid}] + # If the response list is empty, add required tags to the snapshot + if not snapshots["Tags"]: + snapshot_name_tag = {"Key": resource_tag_key, "Value": resource_tag_value} + snapshot_tags.append(snapshot_name_tag) + + # Add image id to the snapshot + snapshot_tag_response = ec2_client.create_tags(Resources=[snapshotid], Tags=snapshot_tags) + return ec2_tag_response, snapshot_tag_response + + + +# Deregister the obsolete AMIs and delete the associated snapshots +def deregister_amis(obsolete_amis): + logger.info(f"AMIs to deregister {obsolete_amis}") + for item in obsolete_amis: + ec2_tag_response, snapshot_tag_response = manage_tags(item["ImageId"], item["SnapshotId"]) + # Proceed with deregistration only if the tags are added to the resources + if ( + ec2_tag_response["ResponseMetadata"]["HTTPStatusCode"] + == snapshot_tag_response["ResponseMetadata"]["HTTPStatusCode"] + == 200 + ): + logger.info(f"Successfully added tags to {item['ImageId']} and {item['SnapshotId']}") + logger.info("Deregistering Image and deleting associated snapshot") + ec2_client.deregister_image(ImageId=item["ImageId"]) + ec2_client.delete_snapshot(SnapshotId=item["SnapshotId"]) + else: + logger.info("Failed to add tags to the resources. Cannot proceed with AMI cleanup") + + +# Verify the required recycle bin rules are present +def list_recycle_bin_rules(resource_type): + rbin_rule_present = False + resource_tags = [{"ResourceTagKey": resource_tag_key, "ResourceTagValue": resource_tag_value}] + rbin_rules = rbin_client.list_rules(ResourceType=resource_type, ResourceTags=resource_tags) + if rbin_rules["Rules"]: + rbin_rule_present = True + return rbin_rule_present + + +def lambda_handler(event, context): + try: + logger.info("Verifying the recycle bin rules") + ebs_rule_exists = list_recycle_bin_rules("EBS_SNAPSHOT") + ami_rule_exists = list_recycle_bin_rules("EC2_IMAGE") + + if not ebs_rule_exists or not ami_rule_exists: + logger.info("One or more required recycle bin rules does not exists. Cannot proceed with AMI cleanup") + return + + logger.info("Recycle bin rules present. Starting AMI cleanup") + + # Filter AMIs having the 'Expire-After' tag and 'Name' tag matching the recycle bin rule + # The Expire-After tag can be added to the AMI during the vending process as per the lifecycle rule to retain AMI + response = ec2_client.describe_images( + Owners=["self"], + Filters=[ + {"Name": "tag-key", "Values": ["Expire-After"]}, + {"Name": "tag:" + resource_tag_key, "Values": [resource_tag_value]}, + ], + ) + + # Get today's date in Zulu format + zulu_format = "%Y-%m-%dT%H:%M:%SZ" + today = datetime.strptime((datetime.now()).strftime(zulu_format), (zulu_format)) + + if len(response["Images"]) == 0: + logger.info("No AMIs found for cleanup. Exiting...") + + obsolete_amis = [] + for item in response["Images"]: + ami_expiry = None + if "Tags" in item: + for tag in item["Tags"]: + if tag["Key"] == "Expire-After": + ami_expiry = datetime.strptime((tag["Value"]), zulu_format) + ami_expiration_in_days = (today - ami_expiry).days + break + + if ami_expiry and ami_expiration_in_days >= 0: + deregister_object = { + "ImageId": item["ImageId"], + "SnapshotId": item["BlockDeviceMappings"][0]["Ebs"]["SnapshotId"], + } + obsolete_amis.append(deregister_object) + + if len(obsolete_amis) > 0: + deregister_amis(obsolete_amis) + + except Exception as e: + logger.info(e) + + + + + + diff --git a/ami-recycle-bin/src/ami-recycle-lambda.zip b/ami-recycle-bin/src/ami-recycle-lambda.zip new file mode 100644 index 000000000..0061c7b3d Binary files /dev/null and b/ami-recycle-bin/src/ami-recycle-lambda.zip differ diff --git a/ami-recycle-bin/variables.tf b/ami-recycle-bin/variables.tf new file mode 100644 index 000000000..04d359b89 --- /dev/null +++ b/ami-recycle-bin/variables.tf @@ -0,0 +1,71 @@ +variable "region" { + type = string + default = "us-east-2" + description = "AWS region where Terraform will manage the infrastructure" +} + +variable "rbin_retention_period_value" { + type = number + default = 30 + description = "The period value for which the retention rule is to retain resources" +} + +variable "rbin_retention_period_unit" { + type = string + default = "DAYS" + description = "The unit of time in which the retention period is measured. Currently, only DAYS is supported" +} + +variable "resource_tag_key" { + type = string + default = "Project" + description = "The tag key" +} + +variable "resource_tag_value" { + type = string + default = "Test-Retention" + description = "The tag value" +} + +variable "cw_retention_period" { + type = number + default = 14 + description = "Specifies the number of days you want to retain log events in the specified log group" +} + +variable "function_name" { + type = string + default = "ami-recycle-lambda" + description = "A unique name for your Lambda Function" +} + +variable "function_description" { + type = string + default = "function to automate the expiration of AMI and moving its associated snapshot to Recycle Bin" + description = "Description of what your Lambda Function does" +} + +variable "function_runtime" { + type = string + default = "python3.12" + description = "Identifier of the function's runtime" +} + +variable "function_timeout" { + type = number + default = 15 + description = "The amount of time your Lambda Function has to run in seconds" +} + +variable "memory_size" { + type = number + default = 128 + description = "Amount of memory in MB your Lambda Function can use at runtime" +} + +variable "dry_run" { + type = bool + default = true + description = "If set to true, the script will not delete any resources" +} \ No newline at end of file