Chaos Injection for AWS Lambda - chaos_lambda

chaos_lambda is a small library injecting chaos into AWS Lambda. It offers simple python decorators to do delay, exception and statusCode injection and a Class to add delay to any 3rd party dependencies called from your function. This allows to conduct small chaos engineering experiments for your serverless application in the AWS Cloud.

• Support for Latency injection using delay
• Support for Exception injection using exception_msg
• Support for HTTP Error status code injection using error_code
• Using for SSM Parameter Store to control the experiment using isEnabled
• Support for adding rate of failure using rate. (Default rate = 1)
• Per Lambda function injection control using Environment variable (CHAOS_PARAM)

Install

pip install chaos-lambda

Example

# function.py

import os
from chaos_lambda import inject_delay, inject_exception, inject_statuscode

# this should be set as a Lambda environment variable
os.environ['CHAOS_PARAM'] = 'chaoslambda.config'

@inject_exception
def handler_with_exception(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_exception(exception_type=TypeError, exception_msg='foobar')
def handler_with_exception_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_exception(exception_type=ValueError)
def handler_with_exception_arg_2(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_statuscode
def handler_with_statuscode(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_statuscode(error_code=400)
def handler_with_statuscode_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay
def handler_with_delay(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay(delay=1000)
def handler_with_delay_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_delay(delay=0)
def handler_with_delay_zero(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

When excecuted, the Lambda function, e.g handler_with_exception('foo', 'bar'), will produce the following result:

exception_msg from config I really failed seriously with a rate of 1
corrupting now
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../chaos_lambda.py", line 199, in wrapper
    raise Exception(exception_msg)
Exception: I really failed seriously

Configuration

The configuration for the failure injection is stored in the AWS SSM Parameter Store and accessed at runtime by the get_config() function:

{
    "isEnabled": true,
    "delay": 400,
    "error_code": 404,
    "exception_msg": "I really failed seriously",
    "rate": 1
}

To store the above configuration into SSM using the AWS CLI do the following:

aws ssm put-parameter --region eu-north-1 --name chaoslambda.config --type String --overwrite --value "{ "delay": 400, "isEnabled": true, "error_code": 404, "exception_msg": "I really failed seriously", "rate": 1 }"

AWS Lambda will need to have IAM access to SSM.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeParameters"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:eu-north-1:12345678910:parameter/chaoslambda.config"
        }
    ]
}

Supported Decorators:

chaos_lambda currently supports the following decorators:

• @inject_delay - add delay in the AWS Lambda execution
• @inject_exception - Raise an exception during the AWS Lambda execution
• @inject_statuscode - force AWS Lambda to return a specific HTTP error code

and the following class:

• SessionWithDelay - enabled to sub-classing requests library and call dependencies with delay

More information:

Full Documentation