Skip to content

Solution creation

Jump to bottom
CARMLPipelinePrincipal edited this page Jul 23, 2022 · 32 revisions

This section shows you how you can orchestrate a deployment using multiple resource modules.

Note: In the below examples, we assume you leverage Bicep as your primary DSL (domain specific language).

Navigation

Upstream workloads

There are several open-source repositories that leverage the CARML library today. Alongside the examples, we provide you with below, the referenced repositories are a good reference on how you can leverage CARML for larger solutions.

Repository Description
AVD Accelerator AVD Accelerator deployment automation to simplify the setup of AVD (Azure Virtual Desktop)
AKS Baseline Automation Repository for the AKS Landing Zone Accelerator program's Automation reference implementation
DevOps Self-Hosted - Create & maintain images with a pipeline using the Azure Image Builder service

- Deploy & maintain Azure DevOps Self-Hosted agent pools with a pipeline using Virtual Machine Scale Set

Orchestration overview

When it comes to deploying multi-module solutions (applications/workloads/environments/landing zone accelerators/etc.), we can differentiate two types of orchestration methods:

  • Template-orchestration: These types of deployments reference individual modules from a 'main' Bicep or ARM/JSON template and use the capabilities of this template to pass parameters & orchestrate the deployments. By default, deployments are run in parallel by the Azure Resource Manager, while accounting for all dependencies defined. With this approach, the deploying pipeline only needs one deployment job that triggers the template's deployment.

    Template orchestration

  • Pipeline-orchestration: This approach uses the platform specific pipeline capabilities (for example, pipeline jobs) to trigger the deployment of individual modules, where each job deploys one module. By defining dependencies in between jobs you can make sure your resources are deployed in order. Parallelization is achieved by using a pool of pipeline agents that run the jobs, while accounting for all dependencies defined.

Both the template-orchestration, as well as pipeline-orchestration may run a validation and subsequent deployment in the same Azure subscription. This subscription should be the subscription where you want to host your production solution. However, you can extend the concept and for example, deploy the solution first to an integration and then a production subscription.

Pipeline orchestration

Publish-location considerations

For your solution, it is recommended to reference modules from a published location, to leverage versioning and avoid the risk of breaking changes.

CARML supports publishing to different locations, either through the use of the CI environment or by locally running the same scripts leveraged by the publishing step of the CI environment pipeline, as explained next.

In either case, you may effectively decide to configure only a subset of publishing locations as per your requirements.

To help you with the decision, the following content provides you with an overview of the possibilities of each target location.

Outline

  • Template Specs

    A Template Spec is an Azure resource with the purpose of storing & referencing Azure Resource Manager (ARM) templates.

    When publishing Bicep modules as Template Specs, the module is compiled - and the resulting ARM template is uploaded as a Template Spec resource version to a Resource Group of your choice. For deployment, it is recommended to apply a template-orchestrated approach. As Bicep supports the Template-Specs as linked templates, this approach enables you to fully utilize Azure's parallel deployment capabilities.

    Note: Even though the published resource is an ARM template, you can reference it in you Bicep template as a remote module like it would be native Bicep. Note: Template Spec names have a maximum of 90 characters

  • Bicep Registry

    A Bicep Registry is an Azure Container Registry that can be used to store & reference Bicep modules.

    For deployment, it is recommended to apply a template-orchestrated approach. As Bicep supports the Bicep registry as linked templates, this approach enables you to fully utilize Azure's parallel deployment capabilities.

  • Azure DevOps Universal Packages

    A Universal Package is a packaged folder in an Azure DevOps artifact feed.

    As such, it contains the content of a CARML module 'as-is', including the template file(s), ReadMe file(s) and test file(s).

    For deployment, it is recommended to use Universal Packages only for a pipeline-orchestrated approach - i.e., each job would download a single package and deploy it.

    Technically, it would be possible to also use Universal Packages for the template-orchestrated approach, by downloading all packages into a specific location first, and then reference them. Given the indirect nature of this approach, this is however not recommended. (:large_orange_diamond:)

    Note: Azure DevOps Universal Packages enforce semver. As such, it is not possible to overwrite an existing version.

Comparison

The following table provides you with a comparison of the locations described above:

Category Feature Template Specs Bicep Registry Universal Packages
Portal/UI
Template can be viewed ✅
Template can be downloaded
Deployment
Supports template-orchestration ✅ ✅ 🔶
Supports pipeline-orchestration ✅ ✅ ✅
Supports single endpoint ✅ ✅
Other
Template can be downloaded/restored via CLI ✅ ✅ ✅
Allows referencing latest minor ✅ ✅
Allows referencing latest major ✅ ✅ ✅

Template-orchestration

The template-orchestrated approach means using a main or so-called master template for deploying resources in Azure. This template will only contain nested deployments, where the modules - instead of directly embedding their content into the master template - will be referenced by the master template.

With this approach, modules need to be stored in an available location, where the Azure Resource Manager (ARM) can access them. This can be achieved by storing the module templates in an accessible location like local, Template Specs or the Bicep Registry.

In an enterprise environment, the recommended approach is to store these master templates in a private environment, only accessible by enterprise resources. Thus, only trusted authorities can have access to these files.

How to start

Once you start building a solution using this library, you may wonder how best to start. Following, you can find some points that can accelerate your experience:

  • Use the VS-Code extension for Bicep to enable DSL-native features such as auto-complete. Metadata implemented in the modules are automatically loaded through the extension.
  • Use the readme
    • If you don't know how to use an object/array parameter, you can check if the module's ReadMe file specifies any 'Parameter Usage' block for the given parameter (example) - or - check the module's Deployment Examples (example).
    • In general, take note of the Deployment Examples specified in each module's ReadMe file, as they provide you with rich & tested examples of how a given module can be deployed (example). An easy way to get started is to copy one of the examples and then adjust it to your needs.
  • Note the outputs that are returned by each module.
    • If an output you need isn't available, you have 2 choices:
      1. Add the missing output to the module
      2. Reference the deployed resource using the existing keyword (Note: You cannot reference the same resource as both a new deployment & existing. To make this work, you have to move the existing reference into it's own .bicep file).

Examples

Referencing local files

The following example shows how you could orchestrate a deployment of multiple resources using local module references. In this example, we will deploy a resource group with a Network Security Group (NSG), and use them in a subsequent VNET deployment.

targetScope = 'subscription'

// ================ //
// Input Parameters //
// ================ //

// RG parameters
@description('Optional. The name of the resource group to deploy')
param resourceGroupName string = 'validation-rg'

@description('Optional. The location to deploy into')
param location string = deployment().location

// =========== //
// Deployments //
// =========== //

// Resource Group
module rg 'modules/Microsoft.Resources/resourceGroups/deploy.bicep' = {
  name: 'registry-rg'
  params: {
    name: resourceGroupName
    location: location
  }
}

// Network Security Group
module nsg 'modules/Microsoft.Network/networkSecurityGroups/deploy.bicep' = {
  name: 'registry-nsg'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: 'defaultNsg'
  }
  dependsOn: [
    rg
  ]
}

// Virtual Network
module vnet 'modules/Microsoft.Network/virtualNetworks/deploy.bicep' = {
  name: 'registry-vnet'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: 'defaultVNET'
    addressPrefixes: [
      '10.0.0.0/16'
    ]
    subnets: [
      {
        name: 'PrimarySubnet'
        addressPrefix: '10.0.0.0/24'
        networkSecurityGroupName: nsg.name
      }
      {
        name: 'SecondarySubnet'
        addressPrefix: '10.0.1.0/24'
        networkSecurityGroupName: nsg.name
      }
    ]
  }
}
Referencing a Private Bicep Registry

The following example shows how you could orchestrate a deployment of multiple resources using modules from a private Bicep Registry. In this example, we will deploy a resource group with a Network Security Group (NSG), and use them in a subsequent VNET deployment.

Note: the preferred method to publish modules to the Bicep registry is to leverage the CI environment provided in this repository. However, this option may not be applicable to all scenarios (ref e.g., the Consume library section). As an alternative, the same Publish-ModuleToPrivateBicepRegistry.ps1 script leveraged by the publishing step of the CI environment pipeline can also be run locally.

targetScope = 'subscription'

// ================ //
// Input Parameters //
// ================ //

// RG parameters
@description('Optional. The name of the resource group to deploy')
param resourceGroupName string = 'validation-rg'

@description('Optional. The location to deploy into')
param location string = deployment().location

// =========== //
// Deployments //
// =========== //

// Resource Group
module rg 'br/modules:microsoft.resources.resourcegroups:1.0.0' = {
  name: 'registry-rg'
  params: {
    name: resourceGroupName
    location: location
  }
}

// Network Security Group
module nsg 'br/modules:microsoft.network.networksecuritygroups:1.0.0' = {
  name: 'registry-nsg'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: 'defaultNsg'
  }
  dependsOn: [
    rg
  ]
}

// Virtual Network
module vnet 'br/modules:microsoft.network.virtualnetworks:1.0.0' = {
  name: 'registry-vnet'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: 'defaultVNET'
    addressPrefixes: [
      '10.0.0.0/16'
    ]
    subnets: [
      {
        name: 'PrimarySubnet'
        addressPrefix: '10.0.0.0/24'
        networkSecurityGroupName: nsg.name
      }
      {
        name: 'SecondarySubnet'
        addressPrefix: '10.0.1.0/24'
        networkSecurityGroupName: nsg.name
      }
    ]
  }
}

The example assumes you are using a bicepconfig.json configuration file like:

{
    "moduleAliases": {
        "br": {
            "modules": {
                "registry": "<registryName>.azurecr.io",
                "modulePath": "bicep/modules"
            }
        }
    }
}
Referencing Template-Specs

The following example shows how you could orchestrate a deployment of multiple resources using template specs. In this example, we will deploy a resource group with a Network Security Group (NSG), and use them in a subsequent VNET deployment.

Note: the preferred method to publish modules to template-specs is to leverage the CI environment provided in this repository. However, this option may not be applicable to all scenarios (ref e.g., the Consume library section). As an alternative, the same Publish-ModuleToTemplateSpec.ps1 script leveraged by the publishing step of the CI environment pipeline can also be run locally.

targetScope = 'subscription'

// ================ //
// Input Parameters //
// ================ //

// RG parameters
@description('Optional. The name of the resource group to deploy')
param resourceGroupName string = 'validation-rg'

@description('Optional. The location to deploy into')
param location string = deployment().location

// =========== //
// Deployments //
// =========== //

// Resource Group
module rg 'ts/modules:microsoft.resources.resourcegroups:1.0.0' = {
  name: 'registry-rg'
  params: {
    name: resourceGroupName
    location: location
  }
}

// Network Security Group
module nsg 'ts/modules:microsoft.network.networksecuritygroups:1.0.0' = {
  name: 'registry-nsg'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: 'defaultNsg'
  }
  dependsOn: [
    rg
  ]
}

// Virtual Network
module vnet 'ts/modules:microsoft.network.virtualnetworks:1.0.0' = {
  name: 'registry-vnet'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: 'defaultVNET'
    addressPrefixes: [
      '10.0.0.0/16'
    ]
    subnets: [
      {
        name: 'PrimarySubnet'
        addressPrefix: '10.0.0.0/24'
        networkSecurityGroupName: nsg.name
      }
      {
        name: 'SecondarySubnet'
        addressPrefix: '10.0.1.0/24'
        networkSecurityGroupName: nsg.name
      }
    ]
  }
}

The example assumes you are using a bicepconfig.json configuration file like:

{
    "moduleAliases": {
        "ts": {
            "modules": {
                "subscription": "<<subscriptionId>>",
                "resourceGroup": "artifacts-rg"
            }
        }
    }
}

Pipeline-orchestration

The modules provided in this repo can be orchestrated to create more complex infrastructures, and as such, reusable solutions or products. To deploy resources, the pipeline-orchestration approach leverages the modules & pipeline templates of the 'ResourceModules' repository. Each pipeline job deploys one instance of a resource and the order of resources deployed in a multi-module solution is controlled by specifying dependencies in the pipeline itself.

GitHub Sample solution for multi-repository approach

Summary

  1. Below, you can find an example which makes use of multiple repositories to orchestrate the deployment (also known as a multi-repository approach) in GitHub.
  2. It fetches the public Azure/ResourceModules repo for consuming bicep modules and uses the parameter files present in the private Contoso/MultiRepoTest repo for deploying infrastructure.
  3. This example is creating a Resource group, an NSG and a VNet -
    1. Job: Deploy multi-repo solution
      1. Checkout 'Azure/ResourceModules' repo at root of the agent
      2. Set environment variables for the agent
      3. Checkout 'contoso/MultiRepoTest' repo containing the parameter files in a nested folder - "MultiRepoTestParentFolder"
      4. Deploy resource group in target Azure subscription
      5. Deploy network security group
      6. Deploy virtual network A

Repo structure

Repository Structure

YAML pipeline

name: 'Multi-Repo solution deployment'

on:
  push:
    branches:
      - main
    paths:
      - 'network-hub-rg/Parameters/**'
      - '.github/workflows/network-hub.yml'

env:
  AZURE_CREDENTIALS: ${{ secrets.AZURE_CREDENTIALS }}
  removeDeployment: false
  variablesPath: 'global.variables.yml'

jobs:
  job_deploy_multi_repo_solution:
    runs-on: ubuntu-20.04
    name: 'Deploy multi-repo solution'
    steps:
      - name: 'Checkout ResourceModules repo at the root location'
        uses: actions/checkout@v2
        with:
          repository: 'Azure/ResourceModules'
          fetch-depth: 0

     - name: Set environment variables
        uses: ./.github/actions/templates/setEnvironmentVariables
        with:
          variablesPath: ${{ env.variablesPath }}

      - name: 'Checkout MultiRepoTest repo in a nested MultiRepoTestParentFolder'
        uses: actions/checkout@v2
        with:
          repository: 'contoso/MultiRepoTest'
          fetch-depth: 0
          path: 'MultiRepoTestParentFolder'

      - name: 'Deploy resource group'
        uses: ./.github/actions/templates/validateModuleDeployment
        with:
          templateFilePath: './modules/Microsoft.Resources/resourceGroups/deploy.bicep'
          parameterFilePath: './MultiRepoTestParentFolder/network-hub-rg/Parameters/ResourceGroup/parameters.json'
          location: '${{ env.defaultLocation }}'
          resourceGroupName: '${{ env.resourceGroupName }}'
          subscriptionId: '${{ secrets.ARM_SUBSCRIPTION_ID }}'
          managementGroupId: '${{ secrets.ARM_MGMTGROUP_ID }}'
          removeDeployment: $(removeDeployment)

      - name: 'Deploy network security group'
        uses: ./.github/actions/templates/validateModuleDeployment
        with:
          templateFilePath: './modules/Microsoft.Network/networkSecurityGroups/deploy.bicep'
          parameterFilePath: './MultiRepoTestParentFolder/network-hub-rg/Parameters/NetworkSecurityGroups/parameters.json'
          location: '${{ env.defaultLocation }}'
          resourceGroupName: '${{ env.resourceGroupName }}'
          subscriptionId: '${{ secrets.ARM_SUBSCRIPTION_ID }}'
          managementGroupId: '${{ secrets.ARM_MGMTGROUP_ID }}'
          removeDeployment: $(removeDeployment)

      - name: 'Deploy virtual network A'
        uses: ./.github/actions/templates/validateModuleDeployment
        with:
          templateFilePath: './modules/Microsoft.Network/virtualNetworks/deploy.bicep'
          parameterFilePath: './MultiRepoTestParentFolder/network-hub-rg/Parameters/VirtualNetwork/vnet-A.parameters.json'
          location: '${{ env.defaultLocation }}'
          resourceGroupName: '${{ env.resourceGroupName }}'
          subscriptionId: '${{ secrets.ARM_SUBSCRIPTION_ID }}'
          managementGroupId: '${{ secrets.ARM_MGMTGROUP_ID }}'
          removeDeployment: $(removeDeployment)

Notes

  1. 'Azure/ResourceModules' repo has been checked out at the root location intentionally because GitHub Actions expect the underlying utility scripts and variables at a specific location.
  2. 'contoso/MultiRepoTest' repo has been checked out in a nested folder, called "MultiRepoTestParentFolder", to distinguish it from the folders of the other repo in the agent, but can also be downloaded at the root location if desired.

This wiki is being actively developed If you discover any documentation bugs or would like to request new content, please raise them as an issue.

Contributions to this wiki are done through the main repo under docs/wiki.

Wiki content

Clone this wiki locally