Terraform-Ansible-ELK Stack

Note this stack is a continual work in progress and capabilities will change over time

Table of Contents

• Overview
• Requirements
• Notes about build environment
• Ansible Configuration
• Terraform.py submodule
• Initialization Steps
• Launching the Stack
• Destroying the Stack

Overview

This stack is designed to launch an ELK stack in an AWS environment using Terraform and Ansible for infrastructure creation and provisioning.

The stack consists of the following components:

• Public VPC
• Internet Gateway
• NAT Gateway
• Security Groups
• Instances (one each for now)
  • Elasticsearch
  • Logstash
  • Kibana

Requirements

• Access to an AWS account
• Ubuntu 16.04 AMI. More options to follow later
• AWSCLI installed locally and properly configured
• Python installed locally
• Ansible installed locally.
• Terraform installed locally.

Notes about build environment

The following versions of tools and OS were used to initially develop this stack

• Fedora 24
• Terraform 0.7.9
• Ansible 2.2.1.0
• Python 2.7.13
• awscli 1.11.21

Ansible Configuration

Currently testing these options in /etc/ansible/ansible.cfg for a reliable fact store. See This for details on how this works.

gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/
fact_caching_timeout = 86400

Terraform.py submodule

Terraform.py is a neat submodule that we use to dynamically create an EC2 inventory for ansible to use in place of its inventory file. It can be run at the CLI and will return IP and hostname values for the running environment.

Initialization Steps

Grab the repository

git clone git@github.com:pythianali/TF-Ansible-ELK.git MyProjectFolder

Grab the submodules

cd MyProjectFolder
git submodule update --init --recursive --remote

Create a credentials file in the project root. Update values accordingly for AWS keys. In this case creds.tf was used.

variable "AWS_ACCESS_KEY" {
  default = "MyAccessKEY"
}
variable "AWS_SECRET_KEY" {
  default = "MySecretKEY"
}

variable "KEY_NAME" {
  default = "TF-Ansible-ELK"
}

variable "LOCAL_KEY_NAME" {
  default = "keys/TF-Ansible-ELK.pem"
}

variable "INSTANCE_USERNAME" {
  default = "ubuntu"
}

Create a keys directory and add the EC2 keypair needed to access the instances. The directory structure looks like below when completed

.
├── ansible
│   ├── ansible-elasticsearch
│   └── playbooks
├── creds.tf
├── instance.tf
├── keys
│   └── TF-Ansible-ELK.pem
├── provider.tf
├── README.md
├── sg.tf
├── terraform.py
│   ├── LICENSE
│   ├── README.md
│   ├── requirements.txt
│   ├── terraform.py
│   └── tests
├── vars.tf
└── vpc.tf

Launching the Stack

To launch the stack you can run the following commands

terraform plan  
terraform apply

Plan provides an overview of changes to be applied and Apply will actually apply those changes

Destroying the stack

To destroy the stack simply run:

terraform destroy