Skip to content

Latest commit

 

History

History
132 lines (112 loc) · 10.5 KB

README.md

File metadata and controls

132 lines (112 loc) · 10.5 KB
Raw

Meltano Infrastructure Module

Terraform module to provision the base infrastructure for a Meltano data platform on AWS. This module deploys foundational resources such an EKS Kubernetes cluster, RDS Databases for Meltano, Airflow and Superset as well as other ancillaries such as ECR, EFS and Prometheus. This module with default values should be enough to host a Meltano Project deployed using the kubernetes/modules/meltano module also in this repository.

Usage

We highly recommend pinning your module source to a specific git Tag, to ensure your infrastructure remains stable over time. The full list of available Tags is here. E.g.

module "infrastructure" {
  # source pinned to Tag v0.1.0
  source = "git::https://gitlab.com/meltano/infra/terraform.git//aws/modules/infrastructure?ref=v0.1.0"
}

The above snippet will deploy an environment marked staging to region us-east-1. This (among other details) can be changed by setting the appropriate variables, listed in full below. E.g.

module "infrastructure" {
  # source pinned to Tag v0.1.0
  source = "git::https://gitlab.com/meltano/infra/terraform.git//aws/modules/infrastructure?ref=v0.1.0"
  region = "eu-west-1"
  environment = "prod"

  # override default VPC details
  vpc = {
    cidr = "10.1.0.0/16"
  }

  # override default RDS details
  rds = {
      meltano_database = {
        instance_class = "db.t4g.small"
      }
      airflow_database = {
        allocated_storage = 20
      }
  }
}

Requirements

Name Version
aws 3.65.0
helm 2.4.1
kubernetes 2.6.1

Providers

Name Version
aws 3.65.0
helm 2.4.1
kubernetes 2.6.1
random 3.1.2

Modules

Name Source Version
airflow_db terraform-aws-modules/rds/aws ~> 3.0
alb_ingress_controller iplabs/alb-ingress-controller/kubernetes 3.4.0
db terraform-aws-modules/rds/aws ~> 3.0
db_security_group terraform-aws-modules/security-group/aws 4.7.0
ecr_airflow cloudposse/ecr/aws 0.32.3
ecr_meltano cloudposse/ecr/aws 0.32.3
efs cloudposse/efs/aws 0.32.2
eks terraform-aws-modules/eks/aws 17.23.0
eks_efs_csi_driver DNXLabs/eks-efs-csi-driver/aws 0.1.4
eks_worker_additional_security_group terraform-aws-modules/security-group/aws 4.7.0
superset_db terraform-aws-modules/rds/aws ~> 3.0
vpc terraform-aws-modules/vpc/aws ~> 3.0

Resources

Name Type
aws_efs_file_system_policy.efs_policy resource
helm_release.aws_efs_pvc resource
helm_release.prometheus resource
kubernetes_namespace.namespace resource
kubernetes_namespace.prometheus resource
random_string.suffix resource
aws_availability_zones.available data source
aws_eks_cluster.eks data source
aws_eks_cluster_auth.eks data source

Inputs

Name Description Type Default Required
aws_region AWS region to deploy to. string "us-east-1" no
eks_cluster_tags EKS cluster tags. map(string) {} no
eks_cluster_version EKS cluster version. string "1.21" no
eks_namespace_name EKS namespace to create. string "meltano" no
eks_worker_group_asg_desired_capacity EKS worker group asg desired capacity. number 6 no
eks_worker_group_asg_max_size EKS worker group asg max size. number 8 no
eks_worker_group_instance_type EKS worker group instance type. string "t3.medium" no
meltano_environment Meltano Environment name. string "staging" no
rds_airflow_database_allocated_storage RDS Airflow database allocated storage. number 10 no
rds_airflow_database_instance_class RDS Airflow database instance class. string "db.t4g.micro" no
rds_airflow_database_port RDS Airflow database port. number 5432 no
rds_backup_window RDS backup window (applied to all databases). string "03:00-06:00" no
rds_deletion_protection RDS deletion protection (applied to all databases). bool true no
rds_maintenance_window RDS maintenance window (applied to all databases). string "Sun:00:00-Sun:03:00" no
rds_meltano_database_allocated_storage RDS Meltano database allocated storage. number 10 no
rds_meltano_database_instance_class RDS Meltano database instance class. string "db.t4g.micro" no
rds_meltano_database_port RDS Meltano database port. number 5432 no
rds_superset_database_allocated_storage RDS Superset database allocated storage. number 10 no
rds_superset_database_instance_class RDS Superset database instance class. string "db.t4g.micro" no
rds_superset_database_port RDS Superset database port. number 5432 no
rds_tags RDS tags (applied to all databases). map(string) {} no
vpc_cidr VPC CIDR block. string "10.0.0.0/16" no
vpc_private_subnets VPC private subnets to create. list(string) 
[
  "10.0.1.0/24",
  "10.0.2.0/24",
  "10.0.3.0/24"
]
 no
vpc_public_subnets VPC public subnets to create. list(string) 
[
  "10.0.4.0/24",
  "10.0.5.0/24",
  "10.0.6.0/24"
]
 no

Outputs

Name Description
airflow_database Map of created RDS database outputs for use by Airflow.
airflow_registry Map of created ECR registry outputs for use by Airflow.
aws_region AWS region deployed to.
kubernetes_cluster Map of created EKS cluster outputs.
meltano_database Map of created RDS database outputs for use by Meltano.
meltano_registry Map of created ECR registry outputs for use by Meltano.
superset_database Map of created RDS database outputs for use by Meltano.
vpc Map of created VPC outputs.