From 8a59b7dbb441019ee041b72c86aca12c01d20f50 Mon Sep 17 00:00:00 2001 From: darkpandarts <114529342+darkpandarts@users.noreply.github.com> Date: Wed, 27 Sep 2023 22:04:53 +0100 Subject: [PATCH 1/3] Update README.md --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 0851adf..b9ff7a5 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Genomics Test +# A Timed Interview Test (4 days) **Step 1** A company allows their users to upload pictures to an S3 bucket. These pictures are always in the .jpg format. The company wants these files to be stripped from any exif metadata before being shown on their website. Pictures are uploaded to an S3 bucket A. Create a system that retrieves .jpg files when they are uploaded to the S3 bucket A, removes any exif metadata, and save them to another S3 bucket B. The path of the files should be the same in buckets A and B. @@ -13,7 +13,7 @@ To extend this further, we have two users User A and User B. Create IAM users wi ![Exif-ripper architecture](docs/exif_ripper.drawio.png) -A natural solution for this problem is to use AWS lambda because this service provides the ability to monitor an s3 bucket and trigger event based messages that can be sent to any arbitrary downstream image processor. Indeed, a whole pipeline of lambda functions can used in the "chain of responsibility pattern" if desired. +A natural solution for this problem is to use AWS lambda because this service provides the ability to monitor an S3 bucket and trigger event based messages that can be sent to any arbitrary downstream image processor. Indeed, a whole pipeline of lambda functions can used in the "chain of responsibility pattern" if desired. ![Chain of responsibility](docs/Chained-Microservices-Design-Pattern.png) @@ -60,7 +60,7 @@ Further reading: ### Serverless Function Overview -Exif-Ripper is a Serverless application that creates an event triggering lambda that monitors a source s3 bucket for the upload of jpg files. When this occurs, an AWS event invokes another (python3) lambda function that strips the exif data from the jpg and writes the "sanitised" jpg to a destination bucket. This lambda function also reads & processes the image directly in memory, and thus does not incur write time-penalties by writing the file to scratch. +Exif-Ripper is a Serverless application that creates an event triggering lambda that monitors a source s3 bucket for the upload of jpg files. When this occurs, an AWS event invokes another (Python3) lambda function that strips the exif data from the jpg and writes the "sanitised" jpg to a destination bucket. This lambda function also reads & processes the image directly in memory, and thus does not incur write time-penalties by writing the file to scratch. #### The Serverless.yml does the following: See `Serverless/exif-ripper/Serverless.yml` @@ -108,15 +108,15 @@ See `Serverless/exif-ripper/Serverless.yml` The directory structure in this project co-locates the infrastructure code with the dev code. An alternative method is accomplished via separation of the infrastructure code from the dev code into 2 repos: -1. genomics-test (conatins dev Serverless code) -2. genomics-test-infra (contains only terraform code) +1. myCompany-test (conatins dev Serverless code) +2. myCompany-test-infra (contains only terraform code) **Pros and cons of co-location method:** The primary benefit of co-location of the terraform code within a Serverless project is the ostensible ease of deploying the compressed Serverless zip file from a single directory. [ See ./xxx_pipeline_create.sh](./xxx_pipeline_create.sh). This makes sense in the context of this example project because there is a requirement to share an uncomplicated code base. ``` . -└── genomics-test +└── myCompany-test ├── Serverless (code repo) ├── Terraform_v1 (terraform repo) └── Terraform_v2 (terraform repo) @@ -127,7 +127,7 @@ However, if a build server was available, we can escape monorepo-centric notions ```bash . └── build_agent_dir - ├── genomics-test (code repo) + ├── myCompany-test (code repo)    ├── Serverless    └── exif-ripper    ├── config @@ -137,7 +137,7 @@ However, if a build server was available, we can escape monorepo-centric notions ### Note infra repo is accessible at another location on the same build server . └── /opt/all_terraform_consumers - └── genomics-test-infra (terraform repo) + └── myCompany-test-infra (terraform repo) └── terraform_v1 ``` @@ -161,12 +161,12 @@ A few patterns of organising and deploying Terraform code are illustrated in thi Some of the pertinent questions with regards to how terraform code is structured are listed below, but a detailed discussion is beyond the scope of this document. -1. `terraform_v1` - [The simplest method](https://github.com/meatware/genomics_test/blob/master/xxx_pipeline_create.sh#L44-L47) +1. `terraform_v1` - [The simplest method](https://github.com/meatware/myCompany_test/blob/master/xxx_pipeline_create.sh#L44-L47) - Uses a local state file so the terraform.tfstate file is saved to the local disk. In order to facilitate shared team editing, the state file is typically stored in git. This is a potential security concern as sensitive values can be exposed. - Once the DEV environment is created, it can be copied and pasted to create UAT & DEV environments. Only a few values such as env value (e.g. `dev --> uat`) will have to be changed in the new env. However, the resulting code duplication can result in env-variant configuration drift and uncaught errors. - Uses publicly available remote modules from the [Terraform registry])(https://registry.terraform.io/) for resources such as s3 to avoid reinventing the wheel. - Uses local modules that are nested in the root of `terraform_v1`. This is a step in the right direction, but any modules defined here cannot be reused for other Terraform consumers. Furthermore, there is no module versioning and changes to these modules will be applicable to DEV, UAT & PROD. We can work around this by checking out specific branches in CI/CD in an env-specific manner, but this is a clunky solution that has suboptimal visibility. -2. `terraform_v2` - [A DRY method](https://github.com/meatware/genomics_test/blob/master/xxx_tfver2_pipeline_create.sh#L73-L76) +2. `terraform_v2` - [A DRY method](https://github.com/meatware/myCompany_test/blob/master/xxx_tfver2_pipeline_create.sh#L73-L76) - Uses a remote s3/dynamodb backend with remote state locking. Facilitates multi-user collaboration - DRY: Leverages passing in tfvar variables (stored in the envs folder) via the `-var-file` CLI argument. e.g. `terraform init -backend-config=../../envs/${myenv}/${myenv}.backend.hcl`, followed by `terraform apply -var-file=../../envs/${myenv}/${myenv}.tfvars` A disadvantage is complexity increase and potential accidental deployment to the wrong environment if deploying from the CLI. Usually not such a big problem because CI/CD is used to deploy. However, something to watch out for. - Uses custom remote module written by yours truly to provision an IAM role with custom or managed policies. The remote module is versioned with release tags and can be found here: https://github.com/meatware/tfmod-iam-role-with-policies. From 556968f1c286ac0166d098dff5281a8ed5065ffe Mon Sep 17 00:00:00 2001 From: darkpandarts <114529342+darkpandarts@users.noreply.github.com> Date: Wed, 27 Sep 2023 22:12:27 +0100 Subject: [PATCH 2/3] Update README.md --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index b9ff7a5..d0994f1 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ A natural solution for this problem is to use AWS lambda because this service pr ![Chain of responsibility](docs/Chained-Microservices-Design-Pattern.png) -Given the limited time to accomplish this task, a benefit of using Serverless is that it is trivial to set up monitoring on a bucket for pushed images because the framework creates the monitoring lambda for us in a few lines of code. When the following code is added to the Serverless.yml file, the monitoring lambda pushes an [event](https://www.Serverless.com/framework/docs/providers/aws/events/s3) to the custom exif-ripper lambda when a file with the suffix of `.jpg` and a s3 key prefix of `incoming/` is created in the bucket called `mysource-bucket`. +Given the limited time to accomplish this task, a benefit of using Serverless is that it is trivial to set up monitoring on a bucket for pushed images because the framework creates the monitoring lambda for us in a few lines of code. When the following code is added to the Serverless.yml file, the monitoring lambda pushes an [event](https://www.Serverless.com/framework/docs/providers/aws/events/s3) to the custom exif-ripper lambda when a file with the suffix of `.jpg` and a S3 key prefix of `incoming/` is created in the bucket called `mysource-bucket`. ```yaml functions: @@ -33,7 +33,7 @@ functions: ``` The lambda Python3 code for exif-ripper is located in `Serverless/exif-ripper/` and it leverages the following libraries to execute the following workflow: -1. [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#s3): Read binary image file from s3 into RAM. +1. [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#s3): Read binary image file from S3 into RAM. 2. [exif](https://pypi.org/project/exif/): Strips any exif data from image 3. Use Boto3 again to write the sanitised file to the destination bucket. @@ -60,7 +60,7 @@ Further reading: ### Serverless Function Overview -Exif-Ripper is a Serverless application that creates an event triggering lambda that monitors a source s3 bucket for the upload of jpg files. When this occurs, an AWS event invokes another (Python3) lambda function that strips the exif data from the jpg and writes the "sanitised" jpg to a destination bucket. This lambda function also reads & processes the image directly in memory, and thus does not incur write time-penalties by writing the file to scratch. +Exif-Ripper is a Serverless application that creates an event triggering lambda that monitors a source S3 bucket for the upload of jpg files. When this occurs, an AWS event invokes another (Python3) lambda function that strips the exif data from the jpg and writes the "sanitised" jpg to a destination bucket. This lambda function also reads & processes the image directly in memory, and thus does not incur write time-penalties by writing the file to scratch. #### The Serverless.yml does the following: See `Serverless/exif-ripper/Serverless.yml` @@ -164,19 +164,19 @@ Some of the pertinent questions with regards to how terraform code is structured 1. `terraform_v1` - [The simplest method](https://github.com/meatware/myCompany_test/blob/master/xxx_pipeline_create.sh#L44-L47) - Uses a local state file so the terraform.tfstate file is saved to the local disk. In order to facilitate shared team editing, the state file is typically stored in git. This is a potential security concern as sensitive values can be exposed. - Once the DEV environment is created, it can be copied and pasted to create UAT & DEV environments. Only a few values such as env value (e.g. `dev --> uat`) will have to be changed in the new env. However, the resulting code duplication can result in env-variant configuration drift and uncaught errors. - - Uses publicly available remote modules from the [Terraform registry])(https://registry.terraform.io/) for resources such as s3 to avoid reinventing the wheel. + - Uses publicly available remote modules from the [Terraform registry](https://registry.terraform.io/) for resources such as S3 to avoid reinventing the wheel. - Uses local modules that are nested in the root of `terraform_v1`. This is a step in the right direction, but any modules defined here cannot be reused for other Terraform consumers. Furthermore, there is no module versioning and changes to these modules will be applicable to DEV, UAT & PROD. We can work around this by checking out specific branches in CI/CD in an env-specific manner, but this is a clunky solution that has suboptimal visibility. 2. `terraform_v2` - [A DRY method](https://github.com/meatware/myCompany_test/blob/master/xxx_tfver2_pipeline_create.sh#L73-L76) - - Uses a remote s3/dynamodb backend with remote state locking. Facilitates multi-user collaboration - - DRY: Leverages passing in tfvar variables (stored in the envs folder) via the `-var-file` CLI argument. e.g. `terraform init -backend-config=../../envs/${myenv}/${myenv}.backend.hcl`, followed by `terraform apply -var-file=../../envs/${myenv}/${myenv}.tfvars` A disadvantage is complexity increase and potential accidental deployment to the wrong environment if deploying from the CLI. Usually not such a big problem because CI/CD is used to deploy. However, something to watch out for. + - Uses a remote S3/dynamodb backend with remote state locking. Facilitates multi-user collaboration + - DRY: Leverages passing in tfvar variables (stored in the envs folder) via the `-var-file` CLI argument. e.g. `terraform init -backend-config=../../envs/${myenv}/${myenv}.backend.hcl`, followed by `terraform apply -var-file=../../envs/${myenv}/${myenv}.tfvars` A disadvantage is complexity increase and potential accidental deployment to the wrong environment if deploying from the CLI. Usually not such a big problem because CI/CD is used to deploy. However, this is something to watch out for. - Uses custom remote module written by yours truly to provision an IAM role with custom or managed policies. The remote module is versioned with release tags and can be found here: https://github.com/meatware/tfmod-iam-role-with-policies. #### Terraform_v1 components & workflow See `xxx_pipeline_create.sh` -1. Creates a global Serverless deployment bucket which can be used by multiple apps. Multiple Serverless projects can be nested in this bucket. This is to avoid the multiple random Serverless buckets being scattered around the root of s3. -2. Creates source & destination s3 buckets for exif image processing +1. Creates a global Serverless deployment bucket which can be used by multiple apps. Multiple Serverless projects can be nested in this bucket. This is to avoid multiple random Serverless buckets being scattered around the root of S3. +2. Creates source & destination S3 buckets for exif image processing 3. Pushes the names of these buckets to SSM 4. Creates a lambda role and policy using a custom remote module pinned to a specific tag 5. Creates two users with RO and RW permissions to the buckets as specified in the brief @@ -197,9 +197,9 @@ See `xxx_pipeline_create.sh` #### Terraform_v2 does the following: **(Please ensure any infra created by v1 is destroyed before deploying v2!)** This version is included to illustrate a method that is more DRY than v1. See `xxx_tfver2_pipeline_create.sh` -1. Creates a global s3/dynamodb backend and writes the backend config files to envs folder (`00_setup_remote_s3_backend_{dev,prod}`) -2. Creates Serverless deployment bucket. Multiple Serverless projects can be nested in this bucket. This is to avoid the mess of multiple random Serverless buckets being scattered around the root of s3. -3. Creates source & destination s3 buckets for exif image processing +1. Creates a global S3/dynamodb backend and writes the backend config files to envs folder (`00_setup_remote_s3_backend_{dev,prod}`) +2. Creates Serverless deployment bucket. Multiple Serverless projects can be nested in this bucket. This is to avoid the mess of multiple random Serverless buckets being scattered around the root of S3. +3. Creates source & destination S3 buckets for exif image processing 4. Pushes the names of these buckets to SSM 5. Creates a lambda role and policy using a [remote module](https://github.com/meatware/tfmod-iam-role-with-policies). - Uses tags so that consumers pin to a specific version of the upstream code @@ -225,7 +225,7 @@ This version is included to illustrate a method that is more DRY than v1. See `x ## Deployment notes -As s3 buckets must be unique, a random string is used so that multiple people can run the deployment in their own environments at any given time without error. +As S3 buckets must be unique, a random string is used so that multiple people can run the deployment in their own environments at any given time without error. ## Practical Usage @@ -247,7 +247,7 @@ cd - #### Please ensure you have exported your aws credentials into your shell This has been test-deployed into an R&D account using Admin credentials. Try to do the same or use an account with the perms to use lambda, s3, iam, dynamodb, and SSM (systems manager) at the least. -An optional method to get a great bash experience via https://github.com/meatware/sys_bashrc +An optional method to get a great BASH experience via https://github.com/meatware/sys_bashrc ```bash cd From 401deb3bedb7fb7862b3022dea9d5295e240f7e5 Mon Sep 17 00:00:00 2001 From: darkpandarts <114529342+darkpandarts@users.noreply.github.com> Date: Wed, 27 Sep 2023 22:14:30 +0100 Subject: [PATCH 3/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d0994f1..3931acc 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# A Timed Interview Test (4 days) +# A Timed Interview Test (~5 days) **Step 1** A company allows their users to upload pictures to an S3 bucket. These pictures are always in the .jpg format. The company wants these files to be stripped from any exif metadata before being shown on their website. Pictures are uploaded to an S3 bucket A. Create a system that retrieves .jpg files when they are uploaded to the S3 bucket A, removes any exif metadata, and save them to another S3 bucket B. The path of the files should be the same in buckets A and B.