hashi-helper is a tool meant to enable Disaster Recovery and Configuration Management for Consul and Vault clusters, by exposing configuration via a simple to use and share hcl format.
- Requirements
- Building
- Configuration
- Usage
- Templating
- Workflow
- Go 1.11
To build a binary, run the following
# get this repo
go get github.com/seatgeek/hashi-helper
# go to the repo directory
cd $GOPATH/src/github.com/seatgeek/hashi-helper
# build the `hashi-helper` binary
make buildThis will create a hashi-helper binary in your $GOPATH/bin directory.
The following environment variables are required for setting configuration and keys in Consul and Vault.
VAULT_TOKENenvironment variable (preferable a root/admin token)VAULT_ADDRenvironment variable (example:http://127.0.0.1:8200)CONSUL_ADDR_HTTPenvironment variable (example:http://127.0.0.1:8500)
hashi-helper [--global-flags] command [--command-flags]Providing this flag will make hashi-helper only process configuration, and not push any changes to Consul or Vault
This is useful for CI pipelines
A number of key=value pairs to expose as template variables during template rendering.
The flag can be repeated any number of times. The order they are provided in is also the order they are read.
--var key1=value1 --var key2=value2
--var key1=value1 --var key2=value2 --var key1=value2 - key1 will be having the value value2
Aliases: --var key=value
A number of files to parse and expose as template variables template rendering.
The flag can be repeated any number of times. The order they are provided in is also the order they are read.
--variable-file example/conf.hcl --variable-file example/conf.json --variable-file example/conf.yaml
Aliases: --var-file value | --varf value
How many parallel requests to run in parallel against remote servers
Default: 2 * CPU Cores
Environment Key: CONCURRENCY
Debug level of debug, info, warn/warning, error, fatal, panic
Default: info
Environment Key: LOG_LEVEL
One or more directories to recursively scan for hcl configuration files.
The flag can be repeated any number of times. The order they are provided in is also the order they are read.
--config-dir conf.d/
--config-dir conf.d/ --config-dir shared/
Environment Key: CONFIG_DIR
One or more hcl configuration file to parse instead of a directory
The flag can be repeated any number of times. The order they are provided in is also the order they are read.
--config-file conf.d/config.hcl
--config-file conf.d/config.hcl --config-file conf.d/another_config.hcl
Environment key: CONFIG_FILE
The environment to process configuration for
Default: all
Environment Key: ENVIRONMENT
The application to process for
Default: all
Environment Key: APPLICATION
Push all Consul and Vault data to remote servers (same as running vault-push-all and consul-push-all)
Decrypt (or create), open and encrypt the secure HASHI_HELPER_PROFILE_FILE (~/.vault_profiles.pgp) file containing your vault clusters
File format is as described below, a simple yaml file
---
# Sample config (yaml)
#
# all keys are optional
#
profile_name_1:
vault:
server: http://active.vault.service.consul:8200
auth:
token: <your vault token>
unseal_token: <your unseal token>
consul:
server: http://consul.service.consul:8500
auth:
token: <your consul token>
nomad:
server: http://nomad.service.consul:4646
auth:
token: <your nomad token>
profile_name_2:
vault:
server: http://active.vault.service.consul:8200
auth:
method: github
github_token: <your github token>
consul:
server: http://consul.service.consul:8500
auth:
method: vault
creds_path: consul/creds/administrator
nomad:
server: http://nomad.service.consul:4646
auth:
method: vault
creds_path: nomad/creds/administratorDecrypt the HASHI_HELPER_PROFILE_FILE and output bash/zsh compatible commands to set VAULT_ADDR, VAULT_TOKEN, ENVIRONMENT based on the profile you selected.
Example: $(hashi-helper profile-use name_1)
Push all local consul state to remote consul cluster.
Push all service{} stanza to remote Consul cluster
Push all kv{} stanza to remote Consul cluster
Create a Vault token, optionally encrypt it using keybase
--keybaseoptional - can be repeated for multiple recipients, token will be encrypted for all recipients to decrypt. If omitted, token is shown in cleartext in the shell--idoptional - The ID of the client token. Can only be specified by a root token. Otherwise, the token ID is a randomly generated UUID.--display-nameoptional The display name of the token. Defaults to "token".--ttloptional - The TTL period of the token, provided as "1h", where hour is the largest suffix. If not provided, the token is valid for the default lease TTL, or indefinitely if the root policy is used.--periodoptional - If specified, the token will be periodic; it will have no maximum TTL (unless an "explicit-max-ttl" is also set) but every renewal will use the given period. Requires a root/sudo token to use.--orphanWill create the token as orphan--policyoptional - Can be repeated for each policy needed. A list of policies for the token. This must be a subset of the policies belonging to the token making the request, unless root. If not specified, defaults to all the policies of the calling token.
Scan all tokens in the Vault server, optionally tokens matching certain conditions
Filter flags:
--filter-name jose will only match tokens where display name contains jose
--filter-policy root will only match tokens that have the policy root
--filter-path auth/github/login will only match tokens that have the path auth/github/login
--filter-meta-username jippi will only match tokens that have the meta[username] value jippi (GitHub auth backend injects this, as an example)
--filter-orphan will only match tokens that are orphaned
Action flags:
--delete-matches will match all tokens matching the filter flags. You will be asked to verify each token before deleting it.
Print a list of local from conf.d/ (default) or remote secrets from Vault (--remote).
Add --detailed / DETAILED to show secret data rather than just the key names.
NOT IMPLEMENTED YET
Write remote Vault secrets to local disk in conf.d/
Pushes all mounts, policies and secrets to a remote vault server
Write Vault auth {} stanza found in conf.d/ to remote vault server
Mount and configure mount {} stanza found in conf.d/ to remote vault server
Write Vault policy {} stanza found in conf.d/ to remote vault server
Write local secrets to remote Vault instance
Unseal Vault using the raw unseal key from keybase / gpg init/rekey .
The command expect the raw base64encoded unseal key as env VAULT_UNSEAL_KEY or --unseal-key CLI argument
It basically automates echo "$VAULT_UNSEAL_KEY" | base64 -D | keybase pgp decrypt | xargs vault unseal -address http://<IP>:8200
--unseal-key/VAULT_UNSEAL_KEY(default:<empty>) The raw base64encoded unseal key as env or CLI argument--consul-service-name/CONSUL_SERVICE_NAME(default:<empty>) If specified, the tool will try to lookup all vault servers in the configured Consul catalog and unseal all of them. This is the service name, e.g.vault--consul-service-tag/CONSUL_SERVICE_TAG(default:standby) The Consul catalog tag to filter vault instances from ifCONSUL_SERVICE_NAMEis used.--vault-protocol/VAULT_PROTOCOL(default:http) The protocol to use when constructing theVAULT_ADDRvalue when usingCONSUL_SERVICE_NAMEunseal strategy.
All VAULT_* env keys are preserved when using CONSUL_SERVICE_TAG, address is the only field being overwritten per vault instance found in the catalog. So you can still configure TLS and other Vault changes as usual with the environment.
VAULT_UNSEAL_KEY=$token hashi-helper vault-unseal-keybasehashi-helper vault-unseal-keybase --unseal-key=$keyVAULT_CONSUL_SERVICE=vault VAULT_UNSEAL_KEY=$token hashi-helper vault-unseal-keybasehashi-helper vault-unseal-keybase --unseal-key=$key --consul-service-name vault
hashi-helper version >= 2.0 support templating for configuration files.
All configuration files loaded are automatically rendered as a template using go text/template.
This is identical to how hashicorp/consul-template renders content.
Templates use [[ ]] brackes instead of the default {{ }} style to avoid clashes with HCL { } stanza definitions.
You can provide variables for templating through the CLI in various ways. All of these options can be provided any number of times
--variable-file <file> or --var-file <file>
--variable key=value or --var key=value
The tool can load files with extensions .hcl, .yaml/.yml and .json. Variables are loaded in the order they are provided in CLI, so it's possible to cascade / overwrite configuration files by using a specific loading order. --variable-file <file> are loaded before CLI --variable key=value arguments.
consul_domain = "consul"
environment_name = "staging"
environment_tld = "stag"
db_default_ttl = "9h"
db_max_ttl = "72h"
my_key = "hello world"
# a list of things
stuff = ["a", "b", "c"]
here_doc = <<-DOC
something multiline
that will be available
as a single string
DOC---
environment_name: "staging"
environment_tld: "stag"
db_default_ttl: "9h"
db_max_ttl: "72h"
my_key: "hello world"
# a list of things
stuff:
- "a"
- "b"
- "c"
here_doc: |
something multiline
that will be available
as a single string{
"db_default_ttl": "9h",
"db_max_ttl": "72h",
"environment_name": "staging",
"environment_tld": "stag",
"here_doc": "something multiline\nthat will be available\nas a single string",
"my_key":"hello world",
"stuff": [
"a",
"b",
"c"
]
}This tool implement most of the features available in hashicorp/consul-template.
Additionally some useful functions in the context of hashi-helper is available, please see the list further down
The functions currently implemented can be found below.
- base64Decode
- base64Encode
- base64URLDecode
- base64URLEncode
- contains
- containsAll
- containsAny
- containsNone
- containsNotAll
- env
- in
- join
- parseBool
- parseFloat
- parseInt
- parseJSON
- parseUint
- plugin
- regexMatch
- regexReplaceAll
- replaceAll
- scratch
- split
- timestamp
- toJSON
- toJSONPretty
- toLower
- toTitle
- toUpper
- toYAML
- trimSpace
lookup is used to lookup template variables inside a template. If the key do not exist, the template rendering will fail.
[[ lookup "my_key" ]] will output hello-world
lookupDefault is used to lookup template variables inside a template. If the key do not exist, the default (2nd argument) will be returned.
[[ lookupDefault "my_key" "something" ]] will output hello-world
service is used to construct Consul service names programatically.
By default consul is used as the domain, but can be overwritten with a variable consul_domain
[[ service "test" ]] will output test.service.consul
Given --variable consul_domain=test.consul
[[ service "test" ]] will output test.service.test.consul
Also see service documentation above.
[[ serviceWithTag "test" "tag" ]] will output tag.test.service.consul
Given --variable consul_domain=test.consul
[[ serviceWithTag "test" "tag" ]] will output tag.test.service.test.consul
Helper to output a policy path for credentials access.
This is useful for database and rabbitmq access policies.
[[ grantCredentials "db-test" "full" ]] will output
path "db-test/creds/full" {
capabilities = ["read"]
}Helper to output a full policy (with path) for credentials access.
This is useful for database and rabbitmq access policies.
[[ grantCredentialsPolicy "db-test" "full" ]] will output
policy "db-test-full" {
path "db-test/creds/full" {
capabilities = ["read"]
}
}Helper to output a github team to vault policy mapping.
[[ githubAssignTeamPolicy "infra" "infra-policy" ]] will output
secret "/auth/github/map/teams/infra" {
value = "infra-policy"
}Helper to output a ldap group to vault policy mapping.
[[ ldapAssignGroupPolicy "infra" "infra-policy" ]] will output
secret "/auth/ldap/groups/infra" {
value = "infra-policy"
}Please see examples/ folder for a working example on how templates work.
Each file is heavily documented explaining each step of the way.
When in the examples/ folder, you can run the following command to try it out.
hashi-helper \
--lint \
--log-level debug \
--environment staging \
-var-file ./staging/config.var.hcl \
-config-dir ./staging/ \
-config-dir ./shared \
push-allThe following is a sample workflow that may be used for organizations with Consul and Vault clusters in different environments. If your setup deviates from said description, feel free to modify your workflow.
The directory structure is laid out like described below:
/${env}/apps/${app}.hcl(encrypted) Vault secrets or (cleartext) Consul KeyValue for an application in a specific environment./${env}/auth/${name}.hcl(encrypted) Vault auth backends for an specific environment${env}./${env}/consul_services/${type}.hcl(cleartext) List of static Consul services that should be made available in an specific environment${env}./${env}/databases/${name}/_mount.hcl(encrypted) Vault secret backend configuration for an specific mount${name}in${env}./${env}/databases/${name}/*.hcl(cleartext) Vault secret backend configuration for an specific Vault role belonging to mount${name}in${env}.
The following example assumes:
- A service called api-admin in a
productionenvironment - IAM-based authentication to vault
- An elasticache instance called
shared - A mysql instance called
db-apithat should provideread-onlyaccess
Some string will need replacement
The following can be stored in an encrypted file at production/apps/api-admin.hcl.
environment "production" {
# application name must match the file name
application "api-admin" {
# Vault policy granting any user with policy api-admin-read-only read+list access to all secrets
policy "api-admin-read-only" {
path "secret/api-admin/*" {
capabilities = ["read", "list"]
}
}
# an sample secret, will be written to secrets/api-admin/API_URL in Vault
secret "API_URL" {
value = "http://localhost:8181"
}
# an sample secret, will be written to secrets/api-admin/TEST in Vault
secret "TEST" {
value = "http://localhost:8282"
}
# shorthand for API_URL and TEST above
secrets {
API_URL = "http://localhost:8181"
TEST = "http://localhost:8282"
}
}
}environment "production" {
kv "name" "production" {}
# application name must match the file name
application "api-admin" {
# cleartext shorthand configuration for the application, will be written to /api-admin/threads
kv "threads" "10" {}
# cleartext configuration for the application, will be written to /api-admin/config
kv "config" {
value = <<EOF
Some
file
value!
EOF
}
}
}environment "production" {
# The auth "name" can be anything, will be the "path" in auth configuration, e.g. the mount below will
# make the "aws-ec2" secret backend available at "aws-ec2/" in the Vault API and CLI.
#
# The auth stanza maps to the Mount API ( https://www.vaultproject.io/api/system/auth.html#mount-auth-backend )
# API endpoint: /sys/auth/:path (/sys/auth/aws-ec2)
auth "aws-ec2" { # :path
# Type must match the Vault auth types
# based on this type, all config and role config below will map to settings found at https://www.vaultproject.io/docs/auth/aws.html
type = "aws-ec2"
# Client Configuration for the autb backend
#
# maps to the secret backend specific configuration
# in this example it will be https://www.vaultproject.io/docs/auth/aws.html#auth-aws-config-client
# key/value here is arbitrary and backend dependent, matches the Vaults docs 1:1 in keys and values
#
# API endpoint: /auth/:path/config/:config_name (/auth/aws-ec2/config/client)
config "client" { # :config_name
access_key = "XXXX"
secret_key = "YYYY"
max_ttl = "1d"
}
# Auth backend type specific roles
#
# in this case it maps to https://www.vaultproject.io/docs/auth/aws.html#auth-aws-role-role-
# key/value here is arbritary and backend dependent, matches the Vault docs 1:1 in keys and values
#
# API endpoint: /auth/:path/role/:role: (/auth/aws-ec2/role/api-admin-prod)
role "api-admin-prod" { # :role
policies = "global,sample-policy"
max_ttl = "1h"
allow_instance_migration = false
bound_vpc_id = "vpc-XXXXXX"
bound_iam_instance_profile_arn = "arn:aws:iam::XXXXXXX:instance-profile/XXXX"
}
}
auth "sg-github" {
type = "github"
# https://www.vaultproject.io/docs/auth/github.html#generate-a-github-personal-access-token
#
# API endpoint: auth/:name/config (auth/sg-github/config)
config "" {
organization = "seatgeek"
}
}
}The following can be stored in a cleartext file at production/consul_services/cache.hcl
environment "production" {
# service name
service "cache-shared" {
# ID must be unique
id = "cache-shared"
# Pseudo node name to attach the service to (we use "cache" or "rds" depending on service type)
node = "cache"
# The IP or domain the service should resolve to
address = "cache-shared.ang13m.YYYY.use1.cache.amazonaws.com"
# The port the service exposes, used for SRV records and Consul API / Fabio
port = 6379
# Optional list of tags
tags = ["master", "replica"]
# Optional meta keys
meta {
my_key = "my_value"
another_key = "another value"
}
}
}environment "production" {
# the name / path the mount will be mounted at, must be unique for the environment
mount "db-api" {
# the mount backend type to use (see Vault docs)
type = "database"
# optional string
max_lease_ttl = "1h"
# optional string
default_lease_ttl = "1h"
# optional boolean
force_no_cache = true
# mount configuration, see Vault docs for details
config "default" {
plugin_name = "mysql-rds-database-plugin"
connection_url = "xxx:yyy@tcp(zzzz:3306)/"
allowed_roles = "*"
}
}
}# environment name must match the directory name
environment "production" {
# the name *must* match the name from _mount.hcl !
mount "db-api" {
# the role name and configuration
role "read-only" {
# by convention, db_name matches the config{} stanza from the _mount example
db_name = "default"
# How long time a token may be alive without being renwed
default_ttl = "24h"
# How long time a token can life, disregarding renewal timeout
max_ttl = "24h"
# The SQL to execute when creating a user
#
# '{{name}}', '{{password}}' and '{{expiry}}' (used in postgres)
creation_statements = <<-SQL
CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';
GRANT SELECT ON api.* TO '{{name}}'@'%';
SQL
}
}
# policy name, granting users with policy "db-api-read-only" access to create credentials from the Vault mount
# by convention the name is always ${mount_name}-${role_name}
policy "db-api-read-only" {
# the path to allow Vault read from, always ${mount_name}/creds/${role_name}
path "db-api/creds/read-only" {
capabilities = ["read"]
}
}
# This will configure the GitHub team "rds-production-api-read-only" to have the policy "db-api-read-only"
# when they "vault auth"
secret "/auth/github/map/teams/rds-production-api-read-only" {
value = "db-api-read-only"
}
}