Continuous Delivery for Microservices on EC2 with mu

Learn how to deploy a webapp to AWS EC2 using mu. If you want to get into microservices, but aren't ready to take the container plunge, you can deploy applications/services directly to AWS EC2 instances.  Using mu's EC2 mode, you can leverage the benefits of microservices without containerization,  and speed up development feedback loops.

The web app in this article is the smallest possible Flask application. You can adapt this example to deploy other applications that include their own web servers, like Django, NodeJS with Express, Ruby with puma/thin/passenger, Jetty or even Spring Boot.

Stelligent mu is a great open source tool to deploy Microservices to t]he AWS cloud. It has three modes of deployment:

* AWS ECS - (Elastic Container Service, which runs Docker containers on EC2).</li>
* AWS Fargate - (A simpler “serverless” version of ECS)</li>
* AWS EC2 instances without Docker containers.  This is the mode we're covering in this article.</li>

In this blog post, we're going to: * Fork this example repository on Github.com * Change the mu.yml to point to your fork * Create the mu pipeline * Walk through the resources mu creates for you, and * Explain the role each file plays in deploying the app.

Technologies used

• Stelligent mu
• Amazon Web Services
  • AWS Cloudformation
  • AWS CodePipeline
  • AWS CodeBuild
  • AWS CodeDeploy
  • AWS Elastic LoadBalancers, VPC, Subnets
• Upstart, which manages services on Amazon Linux & other CentOS 6.x-based systems.

Prerequisites

• An AWS Account
• a Github Account
• a Github Token - create one here
• Stelligent mu
• AWS Access keys set up in $HOME/.aws

Instructions

Make your own repository

• Fork this repository in github.

• Clone your forked repository to your workstation:

  git clone git@github.com:<your-github-name>/mu-minimal-ec2.git

• Go to the newly cloned directory cd mu-minimal-ec2

• Edit the repo: line 17 of mu.yml, to match your new repository's URL.

• Commit and push your changes

  git commit -a -m "modify the repo: attribute in mu.yml" 
  git push 

• Create the mu pipeline mu pipeline up -t <your-github-token>

Walkthrough of the mu pipeline

The mu pipeline up command creates the CloudFormation stacks for the CodePipeline, CodeBuild and CodeDeploy services.

Navigate to the CloudFormation console, and watch the pipeline stacks form. They'll look something like this:

Once the pipeline stacks are created, open the CodePipeline console. You should have one pipeline listed. Click on it.

The mu pipeline looks like this:

• Source – the CodeBuild service retrieves a copy of your source code from GitHub or AWS CodeCommit.
• Build - your software is built, tested, packaged according to instructions in your repo's buildspec.yml file. This uses AWS CodeBuild.
• Deploy to Acceptance - a new environment called "acceptance" is created (or updated) with your software. This uses AWS CodeDeploy.
• Deploy to Production - after the one-and-only manual step, which is to offer Red and Green buttons to stop the pipeline, or deploy the software that is in Acceptance to the last environment, which is Production. This also uses AWS CodeDeploy.

Once the pipeline stacks are complete, you can watch the CodePipeline execute. The CodePipeline/CodeDeploy/CodeBuild services start building your software within a few moments.

When your pipeline reaches the Deploy stage, the CodeDeploy service creates Infrastructure stacks for each Environment, and will update them with each subsequent build. This is another core concept in DevOps known as infrastructure-as-code, and is fully implemented by mu pipeline.

The additional environment stacks look like this:

The infrastructure created by these additional stacks include:

• networking: VPC, subnets, security groups
• compute: EC2 instances (alternatively, ECS and Fargate are also supported in mu)
• storage: S3 buckets supporting the build artifacts
• resiliency: AutoScaling groups and the Elastic Load Balancer definitions come with mu by default.

This means that if you want to change your infrastructure (AutoScaling parameters, your EC2 or RDS instance sizes, etc), you can implement those changes the same way you do your code changes: Change a file, run git add, git commit, and git push!

Walkthrough of the files

mu.yml

 1    namespace: min-ec2
 2    environments:
 3    - name: acceptance
 4      provider: ec2
 5    - name: production
 6     provider: ec2
 7
 8    service:
 9     name: min-ec2
10    port: 8080
11     healthEndpoint: /
12     pathPatterns:
13     - /*
14     pipeline:
15       source:
16         provider: GitHub
17         repo: stelligent/mu-minimal-ec2

Naturally, this example of mu usage starts with a mu.yml. There are two sections: one for environments, and one for services.

The environments: section from lines 2 to 6 (above), are a simple list of the environment names you wish to have (acceptance, staging, sandbox, demo, production, whatever). The provider value can have one of three different values (ec2, ecs, and fargate).

To learn more about other configuring mu environments, refer to the wiki page for Mu Environments.

The services: section is from lines 8 to 17 (above), and specifies just the bare necessities. We're going to accept incoming connections on port 80, route them to our application on port 8080, checking the / endpoint for 200 OK responses, and use the code in the repo specified on line 17.

To learn more about configuring mu services, refer to the wiki page for Mu Services

buildspec.yml

 1 version: 0.2
 2 
 3 phases:
 4   build:
 5     commands:
 6       - pip install Flask
 7 
 8 artifacts:
 9   files:
10    - "**/*"

mu uses AWS CodePipeline, CodeBuild, and CodeDeploy to turn your github repo into a deployed service. The buildspec.yml contains instructions (lines 3-6), on how to install prerequisites, how to invoke the build, do any postprocessing (minification, compression, archiving, etc).

Finally, you specify what files are to be deployed via the artifacts/files: section from line 8 to 10.

You can (and should) also use buildspec.yml to run unit tests. Make sure all your commands return non-zero if there are any failures.

hello.py

 1  from flask import Flask
 2  app = Flask(__name__)
 3 
 4  @app.route("/")
 5  def hello():
 6     return "Mu minimal ec2 example, v2!\n" 

In this example repo, we have included the smallest possible Flask application possible, predictably named hello.py. It listens on a port number, and responds to incoming GET / requests with a hardcoded string.

So, now we just needs to tell the system how to install it, start it, and stop this service.

appspec.yml

 1    version: 0.0
 2    os: linux
 3    files:
 4     - source: ./hello.py
 5       destination: /home/ec2-home/hello/
 6     - source: ./etc/init/mu-minimal-ec2.conf 
 7       destination: /etc/init/
 8 
 9    hooks:
10     BeforeInstall:
11       - location: codedeploy/remove_previous_install.sh
12         timeout: 30
13     ApplicationStop:
14       - location: codedeploy/stop_server.sh
15         timeout: 30
16     ApplicationStart:
17       - location: codedeploy/start_server.sh
18         timeout: 30

The appspec.yml file contains the commands that install, start and stop the service.

• The files section tells the system which files to move, and where they should reside.
• There are hooks section, with hooks called BeforeInstall (line 8), ApplicationStop (line 11), and ApplicationStart (line 14). Each hook has a script location and a timeout in seconds. These scripts reside in a codedeploy directory, primarily to keep the root directory a little cleaner.

./codedeploy directory

• codedeploy/remove_previous_install.sh removes /home/ec2-user/hello/hello.py and /etc/init/mu-minimal-ec2.conf.

    1	#!/bin/bash -xe
    2	/bin/rm -rf /home/ec2-home/hello* /etc/init/mu-minimal-ec2.conf
    3	exit 0

• codedeploy/start_server.sh tells upstart to start the service.

    1	#!/bin/bash -x
    2	set +e
    3	stop mu-minimal-ec2
    4	start mu-minimal-ec2
    5	exit 0

• codedeploy/stop_server.sh tells upstart to stop the service.

    1	#!/bin/bash -xe
    2	stop mu-minimal-ec2
    3	exit 0

Use your new application

Use mu env show acceptance to display key information about your app. One of the attributes is the ELB endpoint.

Conclusion

This repository demonstrates using mu to deploy software to EC2. You should also know that this is just one of several deployment models that mu supports. mu can also dockerize your application and deploy it to AWS Elastic Container Service (Amazon ECS) and AWS Fargate. For the latest developments in mu, you can watch the mu repository.

Stelligent engineers also staff a live chat/support forum at gitter.im/stelligent/mu. You can also read through the mu wiki or file an issue.