Skip to content
This repository has been archived by the owner on Apr 6, 2024. It is now read-only.
/ ecs-toolkit Public archive

ECS Toolkit is a convenience tool that aims to make it easier to manage an application already running on AWS ECS e.g. deploying within your CI/CI pipeline

License

Apache-2.0 license
5 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

shipatlas/ecs-toolkit

BranchesTags

Repository files navigation

ECS Toolkit

ECS Toolkit is a convenience tool that aims to make it easier to manage an application already running on ECS where an application is a group of different ECS services and tasks.

Introduction

Use Cases

For example, you can use it within your CI/CD pipeline to rollout new changes to your running services or even run new tasks with those new changes. While managing deployments is an obvious use case, the intention is that it will grow to have more functionality where it makes sense to.

If there's a use-case that you would like considered, please file an issue with your request.

Motivation

Simply put, a lot of the existing tooling doesn't do a good job simplifying the update process. You kinda have to jump through hoops to make it work.

For example, when updating application configuration on ECS, at some point you'll need to update the task definition before updating the service or running a new task. Doing so in most cases involves using the AWS CLI to register a new task definition.

Unfortunately using register-task-definition requires the task definition passed in as JSON. You can get that from describe-task-definition but it's not easy use that output as input for register-task-definition without hitting issues such as this, this and this.

And granted, there are other official projects such as ECS CLI and Fargate CLI that try to provide alternative ways of managing ECS deployments but they both felt unsatisfactory. Presenting another take at solving the same problem felt like the natural and best way forward.

Features

Takes inputs as a config file which allows you to:

  • Define all the services and tasks that make up an application in a concise configuration file that's easy to grok at a glance i.e. .ecs-toolkit.yml, you could think of this as an application.yml file.
  • Define all the important configuration options for tasks and services that are most applicable for deployments.

As part of the deployment process, it:

  • Registers a new task definition with a new image cloning all the concurrent configuration from the old one to the new one.
  • Runs new tasks and update services with the new task definition.
  • Runs the tasks and services asynchronously for faster deployments.
  • Watches a service until it's stable or a task until it's stopped.
  • Provides extensive logging and sufficient reporting throughout the process to catch failures and monitor progress.

It can also:

  • Update the image of only select containers in the new task definition.
  • Run pre-deployment tasks before updating services e.g. database migrations, asset syncing.
  • Run post-deployment tasks after updating services e.g. cleanup.
  • Optionally skip pre-deployment and post-deployment tasks during deployment.
  • Perform redeploys using the same image with an option forcing a pull of the image.

If there's a feature that you would like considered, please file an issue with your request.

Installation

Download the binary archive for your platform, and install the binary on your PATH. You can use the provided MD5 or SHA256 hash to verify the integrity of your download:

Configuration

Application Config File

The application config file describes the tasks and services that make up an application. For example, a config file for a typical Rails application that consists of a CDN assets sync, database migration, web-server, background worker and worker scheduler is shown below:

---
version: v1
cluster: example
tasks:
  pre:
    - family: app-asset-sync
      count: 1
      containers:
        - rails
      launch_type: fargate
      network_configuration: 
        vpc_configuration:
          assign_public_ip: true
          security_groups:
            - sg-xxxxxxxxxxxxxxxxx
          subnets:
            - subnet-xxxxxxxxxxxxxxxxx
    - family: app-database-migrate
      count: 1
      containers:
        - rails
      launch_type: ec2
    # add as many tasks as needed
services:
  - name: app-web-server
    containers:
      - rails
  - name: app-worker
    containers:
      - resque
  - name: app-worker-scheduler
    containers:
      - resque-scheduler
  # add as many services as needed

Top-Level Options

# Version of the configuration file i.e. `v1`.
# [Required]
version: <string>

# Name of the ECS cluster which is a logical grouping of tasks or services comprising
# of your application.
# [Required]
cluster: <string>

# List of your application's ECS services to manage. See service options.
# [Required]
tasks: <object>

# List of your application's ECS tasks to manage. See task options.
# [Required]
services: array<object>

Task Options

tasks: <object>

  # List of tasks to run before updating services.
  # [Required]
  pre: array<object>

      # The family for the latest `ACTIVE` revision, family and revision (`family:revision`)
      # for a specific revision in the family, or full Amazon Resource Name (ARN) of the
      # task definition to run a task with.
      # [Required]
    - family: <string>

      # The number of instantiations of the specified task to place on your cluster. You
      # can specify a minimum of one up to ten tasks.
      # [Required]
      count: <integer>

      # List of names of the containers in the task's task definition that should have the
      # image tag updated.
      # [Required]
      containers: array<string>

      # The infrastructure to run your standalone task on i.e. `ec2`, `fargate` or `external`.
      # [Optional]
      launch_type: <string>

      # List of capacity provider strategies to use for the task. If specified, `launch_type`
      # must be omitted. May contain a maximum of 6 capacity providers.
      # [Optional]
      capacity_provider_strategies: array<object>

          # The short name of the capacity provider.
          # [Required]
        - capacity_provider: <string>

          # The base value designates how many tasks, at a minimum, to run on the specified
          # capacity provider. Only one capacity provider in a capacity provider strategy
          # can have a base defined. If no value is specified, the default value of 0 is used.
          # [Optional]
          base: <integer>

          # The weight value designates the relative percentage of the total number of tasks
          # launched that should use the specified capacity provider. The weight value is
          # taken into consideration after the base value, if defined, is satisfied. If no
          # weight value is specified, the default value of 0 is used.
          # [Optional]
          weight: <integer>

      # The network configuration for the task.
      # [Optional]
      network_configuration: <object>

        # The VPC subnets and security groups that are associated with a task. All specified
        # subnets and security groups must be from the same VPC.
        # [Required]
        vpc_configuration: <object>

          # Whether the task's elastic network interface receives a public IP address.
          # [Required]
          assign_public_ip: <boolean>

          # The IDs of the subnets associated with the task. There's a limit of 16 subnets
          # that can be specified and all specified subnets must be from the same VPC.
          # [Required]
          security_groups: array<string>

          # The IDs of the security groups associated with the task. There's a limit of 5
          # security groups that can be specified and all specified security groups must
          # be from the same VPC.
          # [Required]
          subnets: array<string>

  # List of tasks to run after updating services. Same as <tasks.pre>.
  # [Required]
  post: array<object>

Service Options

services: array<object>
    # The name of the service. Assumed convention is that the service
    # name matches the task definition's family name.
    # [Required]
  - name: <string>

    # List of names of the containers in the service's task definition that should have the
    # image tag updated.
    # [Required]
    containers: array<string>

    # Determines whether to force a new deployment of the service. By default, deployments
    # aren't forced. You can use this option to start a new deployment with no service
    # definition changes.
    # [Optional]
    force: <boolean>

    # Maximum duration in minutes to wait for the service to be stable. Defaults to 15
    # minutes.
    # [Optional]
    max_wait: <integer>

AWS Credentials

The AWS SDK used internally uses its default credential chain to find AWS credentials. The SDK detects and uses the built-in providers automatically, without requiring manual configuration. For example, if you use IAM roles for Amazon EC2 instances, it automatically use the instance’s credentials.

IAM Policy

Regardless of the credential setup, the owner of the credentials should have a policy that assigns the required permissions, which are:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:DescribeServices",
                "ecs:DescribeTasks",
                "ecs:RunTask",
                "ecs:UpdateService"
            ],
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "ecs:cluster": "arn:aws:ecs:${Region}:${Account}:cluster/${ClusterName}"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecs:RegisterTaskDefinition",
                "ecs:ListTaskDefinitions",
                "ecs:DescribeTaskDefinition"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": "ecs-tasks.amazonaws.com"
                }
            }
        }
    ]
}

Usage

Deploying

Taking the below definition of an Rails application with a database migration and a web-server:

---
version: v1
cluster: example
tasks:
  pre:
    - family: app-database-migrate
      count: 1
      containers:
        - rails
      launch_type: ec2
services:
  - name: app-web-server
    force: false
    containers:
      - rails

It is easy to deploy a new version of your application using the deploy command:

$ ecs-toolkit deploy --image-tag=49779134ca1dcef21f0b5123d3d5c2f4f47da650
INFO[0000] using config file: .ecs-toolkit.yml          
INFO[0000] reading .ecs-toolkit.yml config file         
INFO[0000] starting rollout of pre-deployment tasks      cluster=example
INFO[0001] building new task definition from app-database-migrate:24  cluster=example task=app-database-migrate
WARN[0001] skipping container image tag update, no changes  cluster=example container=rails task=app-database-migrate
WARN[0001] skipping registering new task definition, no changes  cluster=example task=app-database-migrate
INFO[0001] preparing running task parameters             cluster=example task=app-database-migrate
INFO[0001] no changes to previous task definition, using latest  cluster=example task=app-database-migrate
INFO[0001] running new task, desired count: 1            cluster=example task=app-database-migrate
INFO[0001] watching task [1] ... last status: pending, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0004] watching task [1] ... last status: running, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0007] watching task [1] ... last status: running, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0010] watching task [1] ... last status: running, desired status: running, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0013] watching task [1] ... last status: stopped, desired status: stopped, health: unknown  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0013] successfully stopped task [1], reason: essential container in task exited  cluster=example task=app-database-migrate task-id=87356f4b0da94232b39e3527781d55a2
INFO[0013] tasks ran to completion, desired count: 1     cluster=example task=app-database-migrate
INFO[0013] tasks report - total: 1, successful: 1, failed: 0  cluster=example
INFO[0013] completed rollout of pre-deployment tasks     cluster=example
INFO[0013] starting rollout to services                  cluster=example
INFO[0014] building new task definition from app-web-server:103  cluster=example service=app-web-server
WARN[0014] skipping container image tag update, no changes  cluster=example container=rails service=app-web-server
WARN[0014] skipping registering new task definition, no changes  cluster=example service=app-web-server
INFO[0014] no changes to previous task definition, using latest  cluster=example service=app-web-server
INFO[0015] updated service successfully                  cluster=example service=app-web-server
INFO[0015] watch service rollout progress                cluster=example service=app-web-server
INFO[0015] watching ... service: active, deployment: primary, rollout: 1/1 (0 pending)  cluster=example deployment-id=ecs-svc/6300252591410027253 service=app-web-server
INFO[0015] checking if service is stable                 cluster=example service=app-web-server
INFO[0016] service is stable                             cluster=example service=app-web-server
INFO[0016] services report - total: 1, successful: 1, failed: 0  cluster=example
INFO[0016] completed rollout to services                 cluster=example
WARN[0016] skipping rollout of post-deployment tasks, none found  cluster=example

For more information see ecs-toolkit --help or ecs-toolkit <command> --help.

Inspiration

License

King'ori Maina © 2022. The Apache 2.0 License bundled therein whose main conditions require preservation of copyright and license notices. Contributors provide an express grant of patent rights. Licensed works, modifications, and larger works may be distributed under different terms and without source code.

About

ECS Toolkit is a convenience tool that aims to make it easier to manage an application already running on AWS ECS e.g. deploying within your CI/CI pipeline

Topics

aws ecs

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

5 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases 4

Version 0.2.2 Latest
May 1, 2023
+ 3 releases

Packages

No packages published

Contributors 2

  •  
  •  

Languages