Skip to content

Latest commit

 

History

History
69 lines (52 loc) · 2.5 KB

README.md

File metadata and controls

69 lines (52 loc) · 2.5 KB
Raw

Datalake Buckets

Description

This module creates buckets and policies to support a datalake.

This module:

  • creates buckets to support a datalake
    • Raw Data Bucket
    • Intermediate Data Bucket
    • Curated Data Bucket
    • Logs Data Bucket
    • Artifact Data Bucket
  • creates access policies for the buckets
    • READ-ONLY
    • FULL ACCESS
Please provide deployspec.yaml and modulestack.yaml files which will be consumed by SeedFarmer CLI.

Inputs/Outputs

Input Paramenters

Required

None

Optional

  • encryption-type: the type of encryption on data stored in the buckets
    • SSE or KMS
    • Assumed to be SSE
  • retention-type: type of data retention policy when deleteing the buckets
    • DESTROY or RETAIN
    • Assumed to be DESTROY
  • artifacts-log-retention: the number of days back to keep the logs in artifacts-bucket-logs
    • this is to prevent large number of logs filling from MWAA
    • shoud be an integer - "2"
  • solution-id: a unique identifier for this deployment (must be used with solution-description)
  • solution-name: a unique name for this deployment (must be used with solution-id)
  • solution-version: a unique version for this deployment

The parameters (solution-*) will resolve a custom text that is used as a description of the stack if populated.

Module Metadata Outputs

  • RawBucketName: name of the bucket housing the raw data input
  • CuratedBucketName: name of the bucket housing the data after processing
  • IntermediateBucketName: name of the bucket housing data as it is in process
  • ArtifactsBucketName: name of the bucket housing artifacts used for processing
  • LogsBucketName: name of the bucket housing logs
  • ReadOnlyPolicyArn: ARN of the policy generated giving read-only access to content
  • FullAccessPolicyArn: ARN of the policy generated giving full access to content

Output Example

{
  "ArtifactsBucketName": "addf-dep-artifacts-bucket-us-east-1-12345678901",
  "CuratedBucketName": "addf-dep-curated-bucket-us-east-1-123456789012",
  "FullAccessPolicyArn": "arn:aws:iam::123456789012:policy/addf-dep-optionals-datalake-buckets-us-east-1-123456789012-full-access",
  "IntermediateBucketName": "addf-dep-intermediate-bucket-us-east-1-123456789012",
  "LogsBucketName": "addf-dep-logs-bucket-us-east-1-123456789012",
  "RawBucketName": "addf-dep-raw-bucket-us-east-1-123456789012",
  "ReadOnlyPolicyArn": "arn:aws:iam::123456789012:policy/addf-dep-optionals-datalake-buckets-us-east-1-123456789012-readonly-access"
}