gocd-yaml-config-plugin

GoCD server plugin to keep pipelines and environments configuration in source-control in YAML. See this document to find out what are GoCD configuration repositories.

This is second GoCD configuration plugin, which enhances some of shortcomings of JSON configuration plugin

• Format is concise. Unlike JSON, there are no unnecessary quotations ", brackets {} and commas ,
• Tries to enforce correctness where possible. By using maps instead of lists and shorter object graphs.
• Allows to have multiple pipelines and environments in single file. But doesn't force it.
• Comments in configuration files - YAML supports comments, so you can explain why pipeline/environment it is configured like this.

Some highlights

• Shorter syntax for declaring single-job stages
• Very short syntax for declaring tasks with script executor plugin

Setup

If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from the releases page and place it on the GoCD server in plugins/external directory.

Add config-repos element right above first <pipelines />. Then you can add any number of YAML configuration repositories as such:

<config-repos>
  <config-repo pluginId="yaml.config.plugin" id="repo1">
    <git url="https://github.com/tomzo/gocd-yaml-config-example.git" />
  </config-repo>
</config-repos>

In your config repo (tomzo/gocd-yaml-config-example.git in this case), ensure that your gocd yaml config is suffixed with ".gocd.yaml". Any file ending in ".gocd.yaml" is picked up by the plugin. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.

Example

More examples are in test resources.

#ci.gocd.yaml
format_version: 2
environments:
  testing:
    environment_variables:
      DEPLOYMENT: testing
    secure_variables:
      ENV_PASSWORD: "s&Du#@$xsSa"
    pipelines:
      - example-deploy-testing
      - build-testing
pipelines:
  mypipe1: # definition of mypipe1 pipeline
    group: mygroup # note that the group name can contain only of alphanumeric & underscore characters
    label_template: "${mygit[:8]}"
    lock_behavior: none
    parameters: # list of parameters that can be configured for a pipeline
      param1: value1
    materials:
      mygit: # this is the name of material, the name can contain only of alphanumeric & underscore characters
        # keyword git says about type of material and url at once
        git: http://my.example.org/mygit.git
        branch: ci
      myupstream:
        # type is optional here, material type is implied based on presence of pipeline and stage fields
        # type: dependency
        pipeline: pipe2
        stage: test
    stages: # list of stages in order
      - build: # name of stage
          clean_workspace: true
          jobs:
            csharp: # name of the job
              resources:
               - net45
              artifacts:
               - build:
                   source: bin/
                   destination: build
               - test:
                   source: tests/
                   destination: test-reports/
              tabs:
                report: test-reports/index.html
              tasks: # ordered list of tasks to execute in job csharp
               - fetch:
                   pipeline: pipe2
                   stage: build
                   job: test
                   source: test-bin/
                   destination: bin/
               - exec: # indicates type of task
                   command: make
                   arguments:
                    - "VERBOSE=true"
               # shorthand for script-executor plugin
               - script: ./build.sh ci

File pattern

By default GoCD configuration files should end with .gocd.yaml or .gocd.yml.

Specification

See official GoCD XML configuration reference for details about each element. Below is a reference of format supported by this plugin. Feel free to improve it!

1. Format version
2. Environment
3. Environment variables
4. Parameters
5. Pipeline
   • Tabs
   • Tracking tool
   • Timer
   • Locking
6. Stage
   • Approval
7. Job
   • Property
   • Tab
   • Many instances
8. Tasks
   • rake
   • ant
   • nant
   • exec
   • fetch
   • pluggabletask
   • script
9. Materials
   • dependency
   • package
   • git
   • svn
   • perforce
   • tfs
   • hg
   • pluggable scm
   • config repo
10. Secure variables
11. YAML Aliases

Format version

Please note that it is now recommended to declare format_version in each gocd.yaml file. Version 2 was introduced in GoCD v17.12. Currently it is recommended to declare consistent version in all your files:

For GoCD < 17.12:

format_version: 1
pipelines:
  ...
environments:

For GoCD >= 17.12:

format_version: 2
pipelines:
  ...
environments:

Format version V2 only changes the way pipeline locking is configured

Pipeline

A minimal pipeline configuration must contain:

• pipeline name - as a key in hash
• materials - a hash
• stages - a list or a template - as a key in hash

mypipe:
  materials:
    mygit:
      git: http://example.com/mygit.git
  stages:
    - build:
        jobs:
          build:
            tasks:
             - exec:
                 command: make

All elements available on a pipeline object are:

• group
• label_template
• locking
• tracking_tool or mingle
• timer
• environment_variables
• parameters
• secure_variables
• materials
• stages
• template

pipe2:
  group: group1
  label_template: "foo-1.0-${COUNT}"
  lock_behavior: none
  tracking_tool:
    link: "http://your-trackingtool/yourproject/${ID}"
    regex: "evo-(\\d+)"
  timer:
    spec: "0 15 10 * * ? *"
  environment_variables:
    DEPLOYMENT: testing
  secure_variables:
    ENV_PASSWORD: "s&Du#@$xsSa"
  materials:
    ...
  stages:
    ...

Referencing an existing template in a pipeline:

mypipe:
  group: group1
  label_template: "foo-1.0-${COUNT}"
  lock_behavior: none
  parameters:
    param1: value
  materials:
    mygit:
      git: http://example.com/mygit.git
  template: template1

Please note:

• Pipeline declares a group to which it belongs

Tracking tool

tracking_tool:
  link: "http://your-trackingtool/yourproject/${ID}"
  regex: "evo-(\\d+)"

Timer

timer:
  spec: "0 15 10 * * ? *"
  only_on_changes: yes

See XML reference for more information.

Pipeline locking

For GoCD >= 17.12 and format_version: 2:

lock_behavior: none

Where lock_behavior is defined as in GoCD documentation can be one of:

• lockOnFailure - same as locking: true
• unlockWhenFinished
• none - same locking: false

For GoCD < 17.12 and format_version: 1:

locking: on

Where locking is a boolean.

Stage

A minimal stage must contain jobs: element or tasks: in single-job stage case.

build:
  jobs:
    firstJob:
      ...
    secondJob:
      ...

A custom stage:

test:
  fetch_materials: yes
  keep_artifacts: yes
  clean_workspace: yes
  approval:
    type: manual
    roles:
      - manager
    users:
      - john
  environment_variables:
    TEST_NUM: 1
  secure_variables:
    PASSWORD: "!@ESsdD323#sdu"
  jobs:
    one:
      ...
    two:
      ...

Approval

Stage can have approval, which is success by default. There are 2 ways to declare approval:

approval: manual

If you need to set associated users or roles:

approval:
  type: manual
  roles:
    - manager
  users:
    - john

Job

Job is a hash starting with jobs name:

test:
  timeout: 5
  run_instances: 7
  environment_variables:
    LD_LIBRARY_PATH: .
  tabs:
    test: results.xml
  resources:
    - linux
  artifacts:
    - test:
        source: src
        destination: dest
    - build:
        source: bin
  properties:
    perf:
      source: test.xml
      xpath: "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
  tasks:
    ...

Note: timeout is added since 0.2.0 version of yaml plugin

Elastic agent profile id

Job configuration may define elastic agents profile id, as such:

elastic_profile_id: "docker.unit-test"

It MUST NOT be specified along with resources. Available in GoCD server since v16.12.0, yaml plugin 0.4.0.

Run many instances

Part of job object can be number of job to runs:

run_instances: 7

Or to run on all agents:

run_instances: all

Tabs

Tabs are a hash with <tab-name>: <path> pairs. Path should exist in GoCD servers artifacts.

tabs:
  tests: test-reports/index.html
  gauge: functional-reports/index.html

Property

Job can have properties, declared as a hash:

properties:
  cov1: # this is the name of property
    source: test.xml
    xpath: "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
  performance.ind1.mbps:
    source: PerfTestReport.xml
    xpath: "//PerformanceSuiteReport/WriteOnly/MBps"

Single job stage

A common use case is that stage has only one job. This plugin provides a shorthand to declared such stages - just omit the jobs: and job name from configuration tree. You can then declare job and stage options on the same (stage) level:

stages:
  - build:
      approval: manual
      resources:
        - cpp
      tasks:
       - exec:
           command: make

Above configuration declares build stage with build job which executes make task.

Materials

Git

Minimal configuration of a git pipeline material:

• material name is mygit
• git repository url is http://example.com/mygit.git

mygit:
  git: http://example.com/mygit.git

Above can be also written more explicitly:

mygit:
  type: git
  url: http://example.com/mygit.git

More customized git material is possible:

gitMaterial1:
  git: "http://my.git.repository.com"
  branch: feature12
  blacklist:
    - externals/**/*.*
    - tools/**/*.*
  destination: dir1
  auto_update: false
  shallow_clone: true

Since GoCD >= 16.7.0 whitelist is also supported, you can specify whitelist instead of blacklist, as such

gitMaterial1:
  git: "git@my.git.repository.com"
  branch: "feature12"
  whitelist:
    - src/**/*.*

Svn

For details about each option, see GoCD XML reference

svnMaterial1:
  svn: "http://svn"
  username: "user1"
  password: "pass1"
  check_externals: true
  blacklist:
    - tools
    - lib
  destination: destDir1
  auto_update: false

Hg

hgMaterial1:
  hg: repos/myhg
  blacklist:
    - externals
    - tools
  destination: dir1
  auto_update: false

Perforce

p4Material1:
  p4: "host.domain.com:12345"
  username: johndoe
  password: pass
  use_tickets: false
  view: |
    //depot/external... //ws/external...
    //depot/tools... //ws/external...
  blacklist:
    - externals
    - tools
  auto_update: false

Instead of password you can use encrypted_password.

Tfs

TODO: - not supported by yaml plugin yet

Pluggable

myPluggableGit:
  scm: someScmGitRepositoryId
  destination: destinationDir
  blacklist:
    - dir1
    - dir2

Configrepo

This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:

materials:
  foo:
    git: "https://github.com/tomzo/gocd-json-config-example.git",
    branch: ci

And in server XML:

<config-repos>
   <config-repo pluginId="yaml.config.plugin" id="repo1">
     <git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
   </config-repo>
</config-repos>

Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.

Using configrepo material type, above repetition can be avoided, last example can be refactored into:

materials:
  foo:
    type: configrepo

Server interprets configrepo material in this way:

Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of configrepo material.

Dependency

To add a dependency on another pipeline stage:

mydependency:
  pipeline: upstream-pipeline-1
  stage: test

Note: mydependency is the name of material - it must be unique

Package

myapt:
  package: apt-repo-id

Tasks

Every task is a hash starting with its type. Type can be exec, ant, nant, rake, fetch, plugin or script.

<type>:
  <task-prop1>: <prop1-value>

Optionally any task can have run_if and on_cancel.

• run_if is a string. Valid values are passed, failed, any
• on_cancel is a task object. Same rules apply as to tasks described on this page. See rake example.

Exec

exec:
  run_if: any
  working_directory: dir
  command: make
  arguments:
   - -j3
   - docs
   - install

Ant

ant:
  build_file: mybuild.xml
  target: compile
  run_if: passed

Nant

nant:
  run_if: passed
  working_directory: "script/build/123"
  build_file: FilePath
  target: Build
  nant_path: NantExe

Rake

A minimal rake task with default values is very short

rake:

A complete rake example:

rake:
  run_if: passed
  working_directory: sample-project
  build_file: SomeRakefile
  target: build
  on_cancel:
    rake:
      working_directory: sample-project
      build_file: CancelRakefile
      target: cancel

Fetch

fetch:
  run_if: any
  pipeline: myupstream
  stage: upstream_stage
  job: upstream_job
  is_file: yes
  source: result
  destination: test

Plugin

plugin:
  options:
    ConverterType: jsunit
  secure_options:
    password: "ssd#%fFS*!Esx"
  run_if: failed
  configuration:
    id: xunit.converter.task.plugin
    version: 1

Script

Because script-executor plugin requires quite a lot of boiler-plate configuration there is a shorthand for defining tasks with it:

script: ./build.sh compile

Above is equivalent of

plugin:
  options:
    script: ./build.sh compile
  configuration:
    id: script-executor
    version: 1

You can declare multi-line scripts too:

script: |
  ./build.sh compile
  make test

Above executes a 2-line script:

./build.sh compile
make test

If you want to execute a single long line script, but break it into multiple lines in YAML, you can do this with > as such:

script: >
  ./build.sh compile &&
  make test

Above executes a single line script:

./build.sh compile && make test

Environment

NOTE: The agents should be a guid, which is currently impossible to get for user

testing:
  environment_variables:
    DEPLOYMENT: testing
  secure_variables:
    ENV_PASSWORD: "s&Du#@$xsSa"
  pipelines:
    - example-deploy-testing
    - build-testing
  agents:
    - 123

Environment variables

Environment variables can be declared in Environments, Pipelines, Stages and Jobs.

In YAML you have 2 keywords for secure (encrypted) variables and standard variables.

environment_variables:
  DEPLOYMENT: testing
  FOO: bar
secure_variables:
  # this value is encrypted by Go's private key (Note in 16.7.0 there is no easy way to obtain such value yet)
  MY_PASSWORD: "s&Du#@$xsSa"

Parameters

Parameters can be declared at the pipeline level.

parameters:
  param1: value1
  production: no

To generate an encrypted value

For versions of GoCD >= 17.1:

See the encryption API.

For versions of GoCD <= 16.12:

There is no easy way to generate obtain encrypted value from GoServer, alternatively you can login into go-server and execute the following command to generate encrypted value

sudo -u go bash -c 'echo -n 'YOUR-INPUT' | openssl enc -des-cbc -a -iv 0 -K $(cat /etc/go/cipher)'

Boolean values

Among all configuration elements there are boolean values, which can be defined using any of the keywords below (as in yaml specs):

• true - y|Y|yes|Yes|YES|true|True|TRUE|on|On|ON
• false - n|N|no|No|NO|false|False|FALSE|off|Off|OFF

YAML Aliases

YAML Aliases (specification) are supported and provide a way to avoid duplication.

Aliases can be defined anywhere in the configuration as long as they are valid configuration elements.

- exec:
  command: make
  arguments:
   - clean
   - &verbose_arg "VERBOSE=true" # define the alias
- exec:
  command: make
  arguments:
   - world
   - *verbose_arg # use the alias

There is also a dedicated top-level common section which allows you to have all aliases in one place and where you don't need to worry about correct placement within the configuration.

format_version: 2
common:
  verbose_arg: &verbose_arg "VERBOSE=true"
  build_tasks: &build_tasks
    - exec:
        command: make
        arguments:
         - clean
    - exec:
        command: make
        arguments:
         - world
pipelines:
  pipe1:
    stages:
      - build:
          jobs:
            build:
              tasks: *build_tasks
            test:
              tasks:
               - *build_tasks # task list aliases can also be mixed with additional tasks in the same job
               - exec:
                   command: make
                   arguments:
                    - test

Development

Run all tests and create a ready to use jar

./gradlew test fatJar

Tests structure

There are examples of yaml partials and their resulting json to be sent to GoCD server. If something is not working right we can always add a new case covering exact yaml that user has and json that we expect on server side.