Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
darkpandarts authored Sep 27, 2023
1 parent 8a59b7d commit 556968f
Showing 1 changed file with 13 additions and 13 deletions.
26 changes: 13 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ A natural solution for this problem is to use AWS lambda because this service pr

![Chain of responsibility](docs/Chained-Microservices-Design-Pattern.png)

Given the limited time to accomplish this task, a benefit of using Serverless is that it is trivial to set up monitoring on a bucket for pushed images because the framework creates the monitoring lambda for us in a few lines of code. When the following code is added to the Serverless.yml file, the monitoring lambda pushes an [event](https://www.Serverless.com/framework/docs/providers/aws/events/s3) to the custom exif-ripper lambda when a file with the suffix of `.jpg` and a s3 key prefix of `incoming/` is created in the bucket called `mysource-bucket`.
Given the limited time to accomplish this task, a benefit of using Serverless is that it is trivial to set up monitoring on a bucket for pushed images because the framework creates the monitoring lambda for us in a few lines of code. When the following code is added to the Serverless.yml file, the monitoring lambda pushes an [event](https://www.Serverless.com/framework/docs/providers/aws/events/s3) to the custom exif-ripper lambda when a file with the suffix of `.jpg` and a S3 key prefix of `incoming/` is created in the bucket called `mysource-bucket`.

```yaml
functions:
Expand All @@ -33,7 +33,7 @@ functions:
```
The lambda Python3 code for exif-ripper is located in `Serverless/exif-ripper/` and it leverages the following libraries to execute the following workflow:
1. [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#s3): Read binary image file from s3 into RAM.
1. [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#s3): Read binary image file from S3 into RAM.
2. [exif](https://pypi.org/project/exif/): Strips any exif data from image
3. Use Boto3 again to write the sanitised file to the destination bucket.

Expand All @@ -60,7 +60,7 @@ Further reading:


### Serverless Function Overview
Exif-Ripper is a Serverless application that creates an event triggering lambda that monitors a source s3 bucket for the upload of jpg files. When this occurs, an AWS event invokes another (Python3) lambda function that strips the exif data from the jpg and writes the "sanitised" jpg to a destination bucket. This lambda function also reads & processes the image directly in memory, and thus does not incur write time-penalties by writing the file to scratch.
Exif-Ripper is a Serverless application that creates an event triggering lambda that monitors a source S3 bucket for the upload of jpg files. When this occurs, an AWS event invokes another (Python3) lambda function that strips the exif data from the jpg and writes the "sanitised" jpg to a destination bucket. This lambda function also reads & processes the image directly in memory, and thus does not incur write time-penalties by writing the file to scratch.

#### The Serverless.yml does the following:
See `Serverless/exif-ripper/Serverless.yml`
Expand Down Expand Up @@ -164,19 +164,19 @@ Some of the pertinent questions with regards to how terraform code is structured
1. `terraform_v1` - [The simplest method](https://github.com/meatware/myCompany_test/blob/master/xxx_pipeline_create.sh#L44-L47)
- Uses a local state file so the terraform.tfstate file is saved to the local disk. In order to facilitate shared team editing, the state file is typically stored in git. This is a potential security concern as sensitive values can be exposed.
- Once the DEV environment is created, it can be copied and pasted to create UAT & DEV environments. Only a few values such as env value (e.g. `dev --> uat`) will have to be changed in the new env. However, the resulting code duplication can result in env-variant configuration drift and uncaught errors.
- Uses publicly available remote modules from the [Terraform registry])(https://registry.terraform.io/) for resources such as s3 to avoid reinventing the wheel.
- Uses publicly available remote modules from the [Terraform registry](https://registry.terraform.io/) for resources such as S3 to avoid reinventing the wheel.
- Uses local modules that are nested in the root of `terraform_v1`. This is a step in the right direction, but any modules defined here cannot be reused for other Terraform consumers. Furthermore, there is no module versioning and changes to these modules will be applicable to DEV, UAT & PROD. We can work around this by checking out specific branches in CI/CD in an env-specific manner, but this is a clunky solution that has suboptimal visibility.
2. `terraform_v2` - [A DRY method](https://github.com/meatware/myCompany_test/blob/master/xxx_tfver2_pipeline_create.sh#L73-L76)
- Uses a remote s3/dynamodb backend with remote state locking. Facilitates multi-user collaboration
- DRY: Leverages passing in tfvar variables (stored in the envs folder) via the `-var-file` CLI argument. e.g. `terraform init -backend-config=../../envs/${myenv}/${myenv}.backend.hcl`, followed by `terraform apply -var-file=../../envs/${myenv}/${myenv}.tfvars` A disadvantage is complexity increase and potential accidental deployment to the wrong environment if deploying from the CLI. Usually not such a big problem because CI/CD is used to deploy. However, something to watch out for.
- Uses a remote S3/dynamodb backend with remote state locking. Facilitates multi-user collaboration
- DRY: Leverages passing in tfvar variables (stored in the envs folder) via the `-var-file` CLI argument. e.g. `terraform init -backend-config=../../envs/${myenv}/${myenv}.backend.hcl`, followed by `terraform apply -var-file=../../envs/${myenv}/${myenv}.tfvars` A disadvantage is complexity increase and potential accidental deployment to the wrong environment if deploying from the CLI. Usually not such a big problem because CI/CD is used to deploy. However, this is something to watch out for.
- Uses custom remote module written by yours truly to provision an IAM role with custom or managed policies. The remote module is versioned with release tags and can be found here: https://github.com/meatware/tfmod-iam-role-with-policies.


#### Terraform_v1 components & workflow
See `xxx_pipeline_create.sh`

1. Creates a global Serverless deployment bucket which can be used by multiple apps. Multiple Serverless projects can be nested in this bucket. This is to avoid the multiple random Serverless buckets being scattered around the root of s3.
2. Creates source & destination s3 buckets for exif image processing
1. Creates a global Serverless deployment bucket which can be used by multiple apps. Multiple Serverless projects can be nested in this bucket. This is to avoid multiple random Serverless buckets being scattered around the root of S3.
2. Creates source & destination S3 buckets for exif image processing
3. Pushes the names of these buckets to SSM
4. Creates a lambda role and policy using a custom remote module pinned to a specific tag
5. Creates two users with RO and RW permissions to the buckets as specified in the brief
Expand All @@ -197,9 +197,9 @@ See `xxx_pipeline_create.sh`
#### Terraform_v2 does the following:
**(Please ensure any infra created by v1 is destroyed before deploying v2!)**
This version is included to illustrate a method that is more DRY than v1. See `xxx_tfver2_pipeline_create.sh`
1. Creates a global s3/dynamodb backend and writes the backend config files to envs folder (`00_setup_remote_s3_backend_{dev,prod}`)
2. Creates Serverless deployment bucket. Multiple Serverless projects can be nested in this bucket. This is to avoid the mess of multiple random Serverless buckets being scattered around the root of s3.
3. Creates source & destination s3 buckets for exif image processing
1. Creates a global S3/dynamodb backend and writes the backend config files to envs folder (`00_setup_remote_s3_backend_{dev,prod}`)
2. Creates Serverless deployment bucket. Multiple Serverless projects can be nested in this bucket. This is to avoid the mess of multiple random Serverless buckets being scattered around the root of S3.
3. Creates source & destination S3 buckets for exif image processing
4. Pushes the names of these buckets to SSM
5. Creates a lambda role and policy using a [remote module](https://github.com/meatware/tfmod-iam-role-with-policies).
- Uses tags so that consumers pin to a specific version of the upstream code
Expand All @@ -225,7 +225,7 @@ This version is included to illustrate a method that is more DRY than v1. See `x


## Deployment notes
As s3 buckets must be unique, a random string is used so that multiple people can run the deployment in their own environments at any given time without error.
As S3 buckets must be unique, a random string is used so that multiple people can run the deployment in their own environments at any given time without error.


## Practical Usage
Expand All @@ -247,7 +247,7 @@ cd -
#### Please ensure you have exported your aws credentials into your shell
This has been test-deployed into an R&D account using Admin credentials. Try to do the same or use an account with the perms to use lambda, s3, iam, dynamodb, and SSM (systems manager) at the least.

An optional method to get a great bash experience via https://github.com/meatware/sys_bashrc
An optional method to get a great BASH experience via https://github.com/meatware/sys_bashrc

```bash
cd
Expand Down

0 comments on commit 556968f

Please sign in to comment.