Skip to content

Latest commit

 

History

History
1002 lines (818 loc) · 91.3 KB

README.md

File metadata and controls

1002 lines (818 loc) · 91.3 KB

IBM Secure Landing Zone module

Graduated (Supported) semantic-release pre-commit latest release Renovate enabled

The landing zone module can be used to create a fully customizable VPC environment within a single region. The five following patterns are starting templates that can be used to get started quickly with Landing Zone. These patterns are located in the patterns directory.

Each of these patterns (except VSI QuickStart) creates the following infrastructure:

  • A resource group for cloud services and for each VPC.
  • Cloud Object Storage instances for flow logs and Activity Tracker
  • Encryption keys in either a Key Protect or Hyper Protect Crypto Services instance
  • A management and workload VPC connected by a transit gateway
  • A flow log collector for each VPC
  • All necessary networking rules to allow communication
  • Virtual Private Endpoint (VPE) for Cloud Object Storage in each VPC
  • A VPN gateway in the management VPC

Each pattern creates the following infrastructure on the VPC:

  • The VPC pattern deploys a simple IBM Cloud VPC infrastructure without any compute resources like VSIs or Red Hat OpenShift clusters
  • The QuickStart VSI pattern deploys edge VPC with one VSI and a jump server VSI in the management VPC
  • The virtual server (VSI) pattern deploys identical virtual servers across the VSI subnet tier in each VPC
  • The Red Hat OpenShift Kubernetes (ROKS) pattern deploys identical clusters across the VSI subnet tier in each VPC
  • The mixed pattern provisions both of these elements
  • Landing zone VPC CRNs can be added to an existing CBR (Context-based restrictions) network zone if the existing CBR zone ID is specified.

For more information about the default configuration, see Default Secure Landing Zone configuration.

VPC pattern QuickStart VSI pattern Virtual server pattern Red Hat OpenShift pattern Mixed pattern
VPC QuickStart VSI VSI ROKS Mixed

Overview

Reference architectures

terraform-ibm-landing-zone

Complete the following steps before you deploy the Secure Landing Zone module.

Set up an IBM Cloud Account

  1. Make sure that you have an IBM Cloud Pay-As-You-Go or Subscription account:

Configure your IBM Cloud account for Secure Landing Zone

  1. Log in to IBM Cloud with the IBMid you used to set up the account. This IBMid user is the account owner and has full IAM access.
  2. Complete the company profile and contact information for the account. This profile is required to stay in compliance with IBM Cloud Financial Service profile.
  3. Enable the Financial Services Validated option for your account.
  4. Enable virtual routing and forwarding (VRF) and service endpoints by creating a support case. Follow the instructions in enabling VRF and service endpoints](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint&interface=ui#vrf).

Set up account access (Cloud IAM)

  1. Create an IBM Cloud API key. The user who owns this key must have the Administrator role.

  2. Require users in your account to use multifactor authentication (MFA).

  3. Set up access groups. User access to IBM Cloud resources is controlled by using the access policies that are assigned to access groups. For IBM Cloud Financial Services validation, all IAM users must not be assigned direct access to any IBM Cloud resources.

    Select All Identity and Access enabled services when you assign access to the group.

(Optional) Set up IBM Cloud Hyper Protect Crypto Services

For Key Management services, you can use IBM Cloud Hyper Protect Crypto Services. Create an instance before you create the Secure Landing Zone.

  1. Create the service instance:

    1. (Optional) Create a resource group for your instance.
    2. On the Hyper Protect Crypto Services (https://cloud.ibm.com/catalog/services/hyper-protect-crypto-services) details page, select a plan.
    3. Complete the required details that are required and click Create.
  2. Initialize Hyper Protect Crypto Services:

    To initialize the provisioned Hyper Protect Crypto Service instance, follow the steps in Getting started with IBM Cloud Hyper Protect Crypto Services.

    For proof-of-technology environments, use the auto-init flag. For more information, see Initializing service instances using recovery crypto units.

Customize your environment

You can customize your environment with Secure Landing Zone in two ways: by using Terraform input variables or by using the override.json file.

Customizing by using Terraform input variables

In the first method, you set a couple of required input variables of your respective pattern, and then provision the environment.

You can find the list of input variables in the variables.tf file of the pattern directory:

Terraform supports multiple ways to set input variables. For more information, see Input Variables in the Terraform language documentation.

For example, you can add more VPCs by adding the name of the new VPC to the vpcs variable in the variables.tf file in your patterns directory.

vpcs  = ["management", "workload", "<ADDITIONAL VPC>"]

You can get more specific after you use this method. Running the Terraform outputs a JSON-based file that you can use in override.json.

Customizing by using the override.json file

The second route is to use the override.json to create a fully customized environment based on the starting template. By default, each pattern's override.json is set to contain the default environment configuration. You can use the override.json in the respective pattern directory by setting the template input override variable to true. Each value in override.json corresponds directly to a variable value from this root module, which each pattern uses to create your environment.

Supported variables

Through the override.json, you can pass any variable or supported optional variable attributes from this root module, which each pattern uses to provision infrastructure. For a complete list of supported variables and attributes, see the variables.tf file.

ℹ️ Tip: You can use the landing zone configuration tool to create the JSON.

Overriding variables

After every execution of terraform apply, a JSON-encoded definition is output. This definition of your environment is based on the defaults for the Landing Zone and any variables that are changed in the override.json file. You can then use the output in the override.json file.

You can redirect the contents between the output lines by running the following commands:

config = <<EOT
EOT

After you replace the contents of the override.json file with your configuration, you can edit the resources within. Make use that you set the template override variable to true as an input variable. For example, within the variables.tf file.

Locally run configurations do not require a Terraform apply command to generate the override.json. To view your current configuration, run the terraform refresh command.

Overriding only some variables

The override.json file does not need to contain all elements. For example,

{
  "enable_transit_gateway": false
}

(Optional) F5 BIG-IP

The F5 BIG-IP Virtual Edition supports setting up a client-to-site full tunnel VPN to connect to your management or edge VPC or a web application firewall (WAF). With this configuration, you can connect to your workload VPC over the public internet.

Through Secure Landing Zone, you can optionally provision the F5 BIG-IP so that you can set up the implemented solution of a client-to-site VPN or web application firewall (WAF). For more information, see Provisioning a F5 BIG-IP host by using Secure Landing Zone.

(Optional) Bastion host by using Teleport

With Teleport, you can configure a virtual server instance in a VPC as a bastion host. Some of Teleport features include single sign-on to access the SSH server, auditing, and recording of your interactive sessions. For more information about Teleport, see the Teleport documentation.

Secure Landing Zone can set up a bastion host that uses Teleport. For more information, see Provisioning a bastion host by using Teleport with Secure Landing Zone.

Module recommendations for more features

Feature Description Module Version
Client-To-Site VPN Create a client-to-site VPN connection between the private VPC network and clients by using Terraform automation that's packaged as a deployable architecture Client-To-Site VPN extension for landing zone >= v1.4.13

Details about the Secure Landing Zone module

VPC

vpc-module

This template supports creating any number of VPCs in a single region. The VPC network and components are created by the Secure Landing Zone VPC module. The VPC components are described in the main.tf file.

vpcs variable

The list of VPCs from the vpcs variable is converted to a map, which supports adding and deleting resources without the need for an update. The VPC network includes the following aspects:

  • VPC
  • Subnets
  • Network ACLs
  • Public gateways
  • VPN gateway and gateway connections

The following example shows the vpc variable type.

  type = list(
    object({
      prefix                      = string            # A unique prefix that will prepend all components in the VPC
      resource_group              = optional(string)  # Name of the resource group to use for VPC. Must by in `var.resource_groups`
      use_manual_address_prefixes = optional(bool)    # Optionally assign prefixes to VPC manually. By default this is false, and prefixes will be created along with subnets
      classic_access              = optional(bool)    # Optionally allow VPC to access classic infrastructure network
      default_network_acl_name    = optional(string)  # Override default ACL name
      default_security_group_name = optional(string)  # Override default VPC security group name
      default_routing_table_name  = optional(string)  # Override default VPC routing table name
      flow_logs_bucket_name       = optional(string)  # Name of COS bucket to use with flowlogs. Must be created by this template

      ##############################################################################
      # Use `address_prefixes` only if `use_manual_address_prefixes` is true
      # otherwise prefixes will not be created. Use only if you need to manage
      # prefixes manually.
      ##############################################################################

      address_prefixes = optional(
        object({
          zone-1 = optional(list(string))
          zone-2 = optional(list(string))
          zone-3 = optional(list(string))
        })
      )

      ##############################################################################

      ##############################################################################
      # List of network ACLs to create with VPC
      ##############################################################################

      network_acls = list(
        object({
          name                = string         # Name of ACL, this can be referenced by subnets to be connected on creation
          add_cluster_rules   = optional(bool) # Automatically add to ACL rules needed to allow cluster provisioning from private service endpoints

          ##############################################################################
          # List of rules to add to the ACL, by default all inbound and outbound traffic
          # will be allowed. By default, ACLs have a limit of 50 rules.
          ##############################################################################

          rules = list(
            object({
              name        = string # Name of ACL rule
              action      = string # Allow or deny traffic
              direction   = string # Inbound or outbound
              destination = string # Destination CIDR block
              source      = string # Source CIDR block

              ##############################################################################
              # Optionally the rule can be created for TCP, UDP, or ICMP traffic.
              # Only ONE of the following blocks can be used in a single ACL rule
              ##############################################################################

              tcp = optional(
                object({
                  port_max        = optional(number)
                  port_min        = optional(number)
                  source_port_max = optional(number)
                  source_port_min = optional(number)
                })
              )
              udp = optional(
                object({
                  port_max        = optional(number)
                  port_min        = optional(number)
                  source_port_max = optional(number)
                  source_port_min = optional(number)
                })
              )
              icmp = optional(
                object({
                  type = optional(number)
                  code = optional(number)
                })
              )
            })
            ##############################################################################
          )
        })
      )

      ##############################################################################


      ##############################################################################
      # Public Gateways
      # For each `zone` that is set to `true`, a public gateway will be created in
      # That zone
      ##############################################################################

      use_public_gateways = object({
        zone-1 = optional(bool)
        zone-2 = optional(bool)
        zone-3 = optional(bool)
      })

      ##############################################################################


      ##############################################################################
      # Object for subnets to be created in each zone, each zone can have any number
      # of subnets
      #
      # Each subnet accepts the four following arguments:
      # * name           - Name of the subnet
      # * cidr           - CIDR block for the subnet
      # * public_gateway - Optionally add a public gateway. This works only if the zone
      #                    for `use_public_gateway` is set to `true`
      # * acl_name       - Name of ACL to be attached. Name must be found in
      #                    `network_acl` object
      ##############################################################################

      subnets = object({
        zone-1 = list(object({
          name           = string
          cidr           = string
          public_gateway = optional(bool)
          acl_name       = string
        }))
        zone-2 = list(object({
          name           = string
          cidr           = string
          public_gateway = optional(bool)
          acl_name       = string
        }))
        zone-3 = list(object({
          name           = string
          cidr           = string
          public_gateway = optional(bool)
          acl_name       = string
        }))
      })

      ##############################################################################

    })
  )
  ##############################################################################

Flow logs

You can add flow log collectors to a VPC by adding the flow_logs_bucket_name parameter to the vpc object. You declare each bucket in the cos variable that manages Cloud Object Storage. For more information about provisioning Cloud Object Storage with this template, see the Cloud Object Storage section.

Transit gateway

You can create a transit gateway that connects any number of VPCs in the same network by setting the enable_transit_gateway variable to true. A connection is created for each VPC that you specify in the transit_gateway_connections variable. You configure the transit gateway resource in the transit_gateway.tf file.

Security groups

You can provision multiple security groups within any of the provisioned VPCs. You configure security group components in the security_groups.tf file.

security_group variable

The security_group variable supports creating security groups dynamically. The list of groups is converted to a map to ensure that changes, updates, and deletions don't affect other resources.

The following example shows the security_group variable type.

  list(
    object({
      name     = string # Name for each security group
      vpc_name = string # The group will be created. Only VPCs from `var.vpc` can be used

      ##############################################################################
      # List of rules to be added to the security group
      ##############################################################################

      rules = list(
        object({
          name      = string # Name of the rule
          direction = string # Inbound or outbound
          source    = string # Source CIDR to allow

          ##############################################################################
          # Optionally, security groups can allow ONE of the following blocks
          # additional rules will have to be created for different types of traffic
          ##############################################################################

          tcp = optional(
            object({
              port_max = number
              port_min = number
            })
          )
          udp = optional(
            object({
              port_max = number
              port_min = number
            })
          )
          icmp = optional(
            object({
              type = number
              code = number
            })
          )
        })

        ##############################################################################

      )

      ##############################################################################
    })
  )

Virtual servers

Virtual servers

This module uses the Cloud Schematics VSI Module to support multiple VSI workloads. The VSI submodule covers the following resources:

  • Virtual server instances
  • Block storage for those instances
  • VPC load balancers for those instances

Virtual server components can be found in virtual_servers.tf

VPC SSH keys

You can use this template to create or return multiple VPC SSH keys by using the ssh_keys variable. For more information about how to locate an SSH key or create one, see SSH keys in the IBM Cloud docs.

ssh_keys variable

Users can add a name and optionally a public key. If public_key is not provided, the SSH key is retrieved by using a data block

  type = list(
    object({
      name           = string
      public_key     = optional(string)
      resource_group = optional(string) # Must be in var.resource_groups
    })
  )

vsi variable

ℹ️ Tip: After the infrastructure is created, changes to the vsi variable won't change the VSI image. The restriction is in place so that you don't inadvertently create an outage or lose data.

The following example shows the vsi virtual server variable type.

list(
    object({
      name            = string                  # Name to be used for each VSI created
      vpc_name        = string                  # Name of VPC from `vpcs` variable
      subnet_names    = list(string)            # Names of subnets where VSI will be provisioned
      ssh_keys        = list(string)            # List of SSH Keys from `var.ssh_keys` to use when provisioning.
      image_name      = string                  # Name of the image for VSI, use `ibmcloud is images` to view
      machine_type    = string                  # Name of machine type. Use `ibmcloud is in-prs` to view
      vsi_per_subnet  = number                  # Number of identical VSI to be created on each subnet
      user_data       = optional(string)        # User data to initialize instance
      resource_group  = optional(string)        # Name of resource group where VSI will be provisioned, must be in `var.resource_groups`
      security_groups = optional(list(string))  # Optional Name of additional security groups from `var.security groups` to add to VSI

      ##############################################################################
      # When creating VSI, users can optionally create a new security group for
      # those instances. These fields function the same as in `var.security_groups`
      ##############################################################################

      security_group  = optional(
        object({
          name = string
          rules = list(
            object({
              name      = string
              direction = string
              source    = string
              tcp = optional(
                object({
                  port_max = number
                  port_min = number
                })
              )
              udp = optional(
                object({
                  port_max = number
                  port_min = number
                })
              )
              icmp = optional(
                object({
                  type = number
                  code = number
                })
              )
            })
          )
        })
      )

      ##############################################################################

      ##############################################################################
      # Optionally block storage volumes can be created. A volume from this list
      # will be created and attached to each VSI
      ##############################################################################

      block_storage_volumes = optional(list(
        object({
          name           = string           # Volume name
          profile        = string           # Profile
          capacity       = optional(number) # Capacity
          iops           = optional(number) # IOPs
          encryption_key = optional(string) # Optionally provide kms key
        })
      ))

      ##############################################################################

      ##############################################################################
      # Any number of VPC Load Balancers
      ##############################################################################

      load_balancers = list(
        object({
          name                    = string # Name of the load balancer
          type                    = string # Can be public or private
          listener_port           = number # Port for front end listener
          listener_protocol       = string # Protocol for listener. Can be `tcp`, `http`, or `https`
          connection_limit        = number # Connection limit
          algorithm               = string # Back end Pool algorithm can only be `round_robin`, `weighted_round_robin`, or `least_connections`.
          protocol                = string # Back End Pool Protocol can only be `http`, `https`, or `tcp`
          health_delay            = number # Health delay for back end pool
          health_retries          = number # Health retries for back end pool
          health_timeout          = number # Health timeout for back end pool
          health_type             = string # Load Balancer Pool Health Check Type can only be `http`, `https`, or `tcp`.
          pool_member_port        = string # Listener port
          idle_connection_timeout = optional(number) # The idle connection timeout of the listener in seconds.

          ##############################################################################
          # A security group can optionally be created and attached to each load
          # balancer
          ##############################################################################

          security_group = optional(
            object({
              name = string
              rules = list(
                object({
                  name      = string
                  direction = string
                  source    = string
                  tcp = optional(
                    object({
                      port_max = number
                      port_min = number
                    })
                  )
                  udp = optional(
                    object({
                      port_max = number
                      port_min = number
                    })
                  )
                  icmp = optional(
                    object({
                      type = number
                      code = number
                    })
                  )
                })
              )
            })
          )

          ##############################################################################
        })
      )

      ##############################################################################

    })
  )

(Optional) Bastion host

You can provision a bastion host that uses Teleport. App ID is used to authenticate users to Teleport. Teleport session recordings are stored in a Cloud Object Storage bucket. You configure the bastion host components in the bastion_host.tf file.

App ID variable

To use the bastion host, either create an App ID instance or use an existing one. If the use_data variable is set to true, an existing App ID instance is used. Set the variable to false create an App ID instance.

object(
  {
    name                = optional(string)         # Name of existing or to be created APP ID instance
    resource_group      = optional(string)         # The resource group of the existing or to be created APP ID instance
    use_data            = optional(bool)           # Bool specifying to use existing or to be created APP ID instance
    keys                = optional(list(string))   # List of App ID resource keys
    use_appid           = bool                     # Bool specifying to connect App ID to bastion host or not
  }
)

Teleport config data variable

The teleport_config block creates a single template for all teleport instances. By using a single template, the values remain hidden.

object(
  {
    teleport_license   = optional(string) # The PEM license file
    https_cert         = optional(string) # The https certificate used by bastion host for teleport
    https_key          = optional(string) # The https private key used by bastion host for teleport
    domain             = optional(string) # The domain of the bastion host
    cos_bucket_name    = optional(string) # Name of the COS bucket to store the session recordings
    cos_key_name       = optional(string) # Name of the COS instance resource key. Must be HMAC credentials
    teleport_version   = optional(string) # Version of Teleport Enterprise to use
    message_of_the_day = optional(string) # Banner message the is exposed to the user at authentication time
    hostname           = optional(string) # The hostname of the bastion host
    app_id_key_name    = optional(string) # Name of APP ID key

    ##############################################################################
    # A list of maps that contain the user email and the role you want to
    # associate with them
    ##############################################################################

    claims_to_roles = optional(
      list(
        object({
          email = string
          roles = list(string)
        })
      )
    )
  }
)

Teleport VSI variable

The following example shows the teleport_vsi variable type.

list(
    object(
      {
        name                            = string           # Name to be used for each teleport VSI created
        vpc_name                        = string           # Name of VPC from `vpcs` variable
        resource_group                  = optional(string) # Name of resource group where the teleport VSI will be provisioned, must be in `var.resource_groups`
        subnet_name                     = string           # Name of the subnet where the teleport VSI will be provisioned
        ssh_keys                        = list(string)     # List of SSH Keys from `var.ssh_keys` to use when provisioning.
        boot_volume_encryption_key_name = string           # Name of boot_volume_encryption_key
        image_name                      = string           # Name of the image for the teleport VSI, use `ibmcloud is images` to view
        machine_type                    = string           # Name of machine type. Use `ibmcloud is in-prs` to view

        ##############################################################################
        # When creating VSI, users can optionally create a new security group for
        # those instances. These fields function the same as in `var.security_groups`
        ##############################################################################

        security_groups = optional(list(string))
        security_group = optional(
          object({
            name = string
            rules = list(
              object({
                name      = string
                direction = string
                source    = string
                tcp = optional(
                  object({
                    port_max = number
                    port_min = number
                  })
                )
                udp = optional(
                  object({
                    port_max = number
                    port_min = number
                  })
                )
                icmp = optional(
                  object({
                    type = number
                    code = number
                  })
                )
              })
            )
          })
        )
        ##############################################################################

      }
    )
  )

Cluster and worker pool

You can create as many iks or openshift clusters and worker pools on VPC. For ROKS clusters, make sure to enable public gateways to allow your cluster to correctly provision ingress application load balancers.

Important: You can't update Red Hat OpenShift cluster nodes by using this module. The Terraform logic ignores updates to prevent possible destructive changes.

The following example shows the cluster variable type.

list(
    object({
      name               = string           # Name of Cluster
      vpc_name           = string           # Name of VPC
      subnet_names       = list(string)     # List of vpc subnets for cluster
      workers_per_subnet = number           # Worker nodes per subnet.
      machine_type       = string           # Worker node flavor
      kube_type          = string           # iks or openshift
      kube_version       = optional(string) # Can be a version from `ibmcloud ks versions`, `latest` or `default`. `null` will use the `default`
      entitlement        = optional(string) # entitlement option for openshift
      pod_subnet         = optional(string) # Portable subnet for pods
      service_subnet     = optional(string) # Portable subnet for services
      resource_group     = string           # Resource Group used for cluster
      cos_name           = optional(string) # Name of COS instance Required only for OpenShift clusters
      kms_config = optional(
        object({
          crk_name         = string         # Name of key
          private_endpoint = optional(bool) # Private endpoint
        })
      )
      worker_pools = optional(
        list(
          object({
            name               = string           # Worker pool name
            vpc_name           = string           # VPC name
            workers_per_subnet = number           # Worker nodes per subnet
            flavor             = string           # Worker node flavor
            subnet_names       = list(string)     # List of vpc subnets for worker pool
            entitlement        = optional(string) # entitlement option for openshift
          })
        )
      )
    })
  )

Virtual private endpoints

Virtual private endpoints can be created for any number of services. Virtual private endpoint components are defined in the vpe.tf file.

IBM Cloud services

You can configure the following IBM Cloud services from this module.

Cloud Object Storage

This module can provision a Cloud Object Storage instance or retrieve an existing one, and then create any number of buckets within the instance.

You define Cloud Object Storage components in the cos.tf file.

VPC placement groups

You can create multiple VPC placement groups in the vpc_placement_groups.tf file. For more information about VPC placement groups, see About placement groups in the IBM Cloud Docs.

Usage

Template for multiple patterns

You can use the modular design of this module to provision architectures for VSI, clusters, or a combination of both. Include a provider block and a copy of the variables.tf file. By referencing this template as a module, you support users who want to add clusters or a vsi by adding the relevant variable block.

VSI example

module "vsi_pattern" {
  source                         = "terraform-ibm-modules/landing-zone/ibm"
  version                        = "latest" # Replace "latest" with a release version to lock into a specific release
  prefix                         = var.prefix
  region                         = var.region
  tags                           = var.tags
  resource_groups                = var.resource_groups
  vpcs                           = var.vpcs
  flow_logs                      = var.flow_logs
  enable_transit_gateway         = var.enable_transit_gateway
  transit_gateway_resource_group = var.transit_gateway_resource_group
  transit_gateway_connections    = var.transit_gateway_connections
  ssh_keys                       = var.ssh_keys
  vsi                            = var.vsi
  security_groups                = var.security_groups
  virtual_private_endpoints      = var.virtual_private_endpoints
  cos                            = var.cos
  service_endpoints              = var.service_endpoints
  key_protect                    = var.key_protect
  atracker                       = var.atracker
}

Cluster and VSI example

module "cluster_vsi_pattern" {
  source                         = "terraform-ibm-modules/landing-zone/ibm"
  version                        = "latest" # Replace "latest" with a release version to lock into a specific release
  prefix                         = var.prefix
  region                         = var.region
  tags                           = var.tags
  resource_groups                = var.resource_groups
  vpcs                           = var.vpcs
  flow_logs                      = var.flow_logs
  enable_transit_gateway         = var.enable_transit_gateway
  transit_gateway_resource_group = var.transit_gateway_resource_group
  transit_gateway_connections    = var.transit_gateway_connections
  ssh_keys                       = var.ssh_keys
  vsi                            = var.vsi
  security_groups                = var.security_groups
  virtual_private_endpoints      = var.virtual_private_endpoints
  cos                            = var.cos
  service_endpoints              = var.service_endpoints
  key_protect                    = var.key_protect
  atracker                       = var.atracker
  clusters                       = var.clusters
  wait_till                      = var.wait_till
}

Cluster example

module "cluster_pattern" {
  source                         = "terraform-ibm-modules/landing-zone/ibm"
  version                        = "latest" # Replace "latest" with a release version to lock into a specific release
  prefix                         = var.prefix
  region                         = var.region
  tags                           = var.tags
  resource_groups                = var.resource_groups
  vpcs                           = var.vpcs
  flow_logs                      = var.flow_logs
  enable_transit_gateway         = var.enable_transit_gateway
  transit_gateway_resource_group = var.transit_gateway_resource_group
  transit_gateway_connections    = var.transit_gateway_connections
  ssh_keys                       = var.ssh_keys
  security_groups                = var.security_groups
  virtual_private_endpoints      = var.virtual_private_endpoints
  cos                            = var.cos
  service_endpoints              = var.service_endpoints
  key_protect                    = var.key_protect
  atracker                       = var.atracker
  clusters                       = var.clusters
  wait_till                      = var.wait_till
}

Requirements

Name Version
terraform >= 1.3
ibm >= 1.68.1, < 2.0.0
random >= 3.4.3, < 4.0.0
time >= 0.9.1, < 1.0.0

Modules

Name Source Version
bastion_host terraform-ibm-modules/landing-zone-vsi/ibm 4.2.0
cluster terraform-ibm-modules/base-ocp-vpc/ibm 3.32.1
dynamic_values ./dynamic_values n/a
f5_vsi terraform-ibm-modules/landing-zone-vsi/ibm 4.2.0
key_management ./kms n/a
placement_group_map ./dynamic_values/config_modules/list_to_map n/a
ssh_keys ./ssh_key n/a
teleport_config ./teleport_config n/a
update_cbr_vpc_zone terraform-ibm-modules/cbr/ibm//modules/cbr-zone-module 1.27.0
vpc terraform-ibm-modules/landing-zone-vpc/ibm 7.19.0
vsi terraform-ibm-modules/landing-zone-vsi/ibm 4.2.0

Resources

Name Type
ibm_appid_redirect_urls.urls resource
ibm_atracker_route.atracker_route resource
ibm_atracker_target.atracker_target resource
ibm_container_addons.addons resource
ibm_container_vpc_cluster.cluster resource
ibm_container_vpc_worker_pool.pool resource
ibm_cos_bucket.buckets resource
ibm_iam_authorization_policy.policy resource
ibm_is_placement_group.placement_group resource
ibm_is_security_group.security_group resource
ibm_is_security_group_rule.security_group_rules resource
ibm_is_subnet_reserved_ip.ip resource
ibm_is_virtual_endpoint_gateway.endpoint_gateway resource
ibm_is_virtual_endpoint_gateway_ip.endpoint_gateway_ip resource
ibm_is_vpn_gateway.gateway resource
ibm_resource_group.resource_groups resource
ibm_resource_instance.appid resource
ibm_resource_instance.cos resource
ibm_resource_key.appid_key resource
ibm_resource_key.key resource
ibm_resource_tag.bucket_tag resource
ibm_resource_tag.cluster_tag resource
ibm_resource_tag.cos_tag resource
ibm_tg_connection.connection resource
ibm_tg_gateway.transit_gateway resource
random_string.random_cos_suffix resource
time_sleep.wait_30_seconds resource
time_sleep.wait_for_authorization_policy resource
time_sleep.wait_for_vpc_creation_data resource
ibm_container_addons.existing_addons data source
ibm_container_cluster_versions.cluster_versions data source
ibm_iam_account_settings.iam_account_settings data source
ibm_is_image.image data source
ibm_is_vpc.vpc data source
ibm_resource_group.resource_groups data source
ibm_resource_instance.appid data source
ibm_resource_instance.cos data source

Inputs

Name Description Type Default Required
appid The App ID instance to be used for the teleport vsi deployments
object({
name = optional(string)
resource_group = optional(string)
use_data = optional(bool)
keys = optional(list(string))
use_appid = bool
})
{
"use_appid": false
}
no
atracker atracker variables
object({
resource_group = string
receive_global_events = bool
collector_bucket_name = string
add_route = bool
})
n/a yes
clusters A list describing clusters workloads to create
list(
object({
name = string # Name of Cluster
vpc_name = string # Name of VPC
subnet_names = list(string) # List of vpc subnets for cluster
workers_per_subnet = number # Worker nodes per subnet.
machine_type = string # Worker node flavor
kube_type = string # iks or openshift
kube_version = optional(string) # Can be a version from ibmcloud ks versions or default
entitlement = optional(string) # entitlement option for openshift
secondary_storage = optional(string) # Secondary storage type
pod_subnet = optional(string) # Portable subnet for pods
service_subnet = optional(string) # Portable subnet for services
resource_group = string # Resource Group used for cluster
cos_name = optional(string) # Name of COS instance Required only for OpenShift clusters
access_tags = optional(list(string), [])
boot_volume_crk_name = optional(string) # Boot volume encryption key name
disable_public_endpoint = optional(bool, true) # disable cluster public, leaving only private endpoint
disable_outbound_traffic_protection = optional(bool, false) # public outbound access from the cluster workers
cluster_force_delete_storage = optional(bool, false) # force the removal of persistent storage associated with the cluster during cluster deletion
operating_system = optional(string, null) #The operating system of the workers in the default worker pool. If no value is specified, the current default version OS will be used. See https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_versions#openshift_versions_available .
kms_wait_for_apply = optional(bool, true) # make terraform wait until KMS is applied to master and it is ready and deployed
verify_cluster_network_readiness = optional(bool, true) # Flag to run a script will run kubectl commands to verify that all worker nodes can communicate successfully with the master. If the runtime does not have access to the kube cluster to run kubectl commands, this should be set to false.
use_ibm_cloud_private_api_endpoints = optional(bool, true) # Flag to force all cluster related api calls to use the IBM Cloud private endpoints.
import_default_worker_pool_on_create = optional(bool) # (Advanced users) Whether to handle the default worker pool as a stand-alone ibm_container_vpc_worker_pool resource on cluster creation. Only set to false if you understand the implications of managing the default worker pool as part of the cluster resource. Set to true to import the default worker pool as a separate resource. Set to false to manage the default worker pool as part of the cluster resource.
allow_default_worker_pool_replacement = optional(bool) # (Advanced users) Set to true to allow the module to recreate a default worker pool. Only use in the case where you are getting an error indicating that the default worker pool cannot be replaced on apply. Once the default worker pool is handled as a stand-alone ibm_container_vpc_worker_pool, if you wish to make any change to the default worker pool which requires the re-creation of the default pool set this variable to true
labels = optional(map(string)) # A list of labels that you want to add to the default worker pool.
addons = optional(object({ # Map of OCP cluster add-on versions to install
debug-tool = optional(string)
image-key-synchronizer = optional(string)
openshift-data-foundation = optional(string)
vpc-file-csi-driver = optional(string)
static-route = optional(string)
cluster-autoscaler = optional(string)
vpc-block-csi-driver = optional(string)
ibm-storage-operator = optional(string)
}), {})
manage_all_addons = optional(bool, false) # Instructs Terraform to manage all cluster addons, even if addons were installed outside of the module. If set to 'true' this module will destroy any addons that were installed by other sources.
kms_config = optional(
object({
crk_name = string # Name of key
private_endpoint = optional(bool) # Private endpoint
})
)
worker_pools = optional(
list(
object({
name = string # Worker pool name
vpc_name = string # VPC name
workers_per_subnet = number # Worker nodes per subnet
flavor = string # Worker node flavor
subnet_names = list(string) # List of vpc subnets for worker pool
entitlement = optional(string) # entitlement option for openshift
secondary_storage = optional(string) # Secondary storage type
boot_volume_crk_name = optional(string) # Boot volume encryption key name
operating_system = optional(string) # The operating system of the workers in the default worker pool. If no value is specified, the current default version OS will be used. See https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_versions#openshift_versions_available .
labels = optional(map(string)) # A list of labels that you want to add to all the worker nodes in the worker pool.
})
)
)
})
)
n/a yes
cos Object describing the cloud object storage instance, buckets, and keys. Set use_data to false to create instance
list(
object({
name = string
use_data = optional(bool)
resource_group = string
plan = optional(string)
random_suffix = optional(bool) # Use a random suffix for COS instance
access_tags = optional(list(string), [])
skip_kms_s2s_auth_policy = optional(bool, false) # skip auth policy between this instance and kms instance, useful if existing resources are used
skip_flowlogs_s2s_auth_policy = optional(bool, false) # skip auth policy between flow logs service and this instance, set to true if this policy is already in place on account
skip_atracker_s2s_auth_policy = optional(bool, false) # skip auth policyt between atracker service and this instance, set to true if this is existing recipient of atracker already
buckets = list(object({
name = string
storage_class = string
endpoint_type = string
force_delete = bool
single_site_location = optional(string)
region_location = optional(string)
cross_region_location = optional(string)
kms_key = optional(string)
access_tags = optional(list(string), [])
allowed_ip = optional(list(string), [])
hard_quota = optional(number)
archive_rule = optional(object({
days = number
enable = bool
rule_id = optional(string)
type = string
}))
expire_rule = optional(object({
days = optional(number)
date = optional(string)
enable = bool
expired_object_delete_marker = optional(string)
prefix = optional(string)
rule_id = optional(string)
}))
activity_tracking = optional(object({
activity_tracker_crn = string
read_data_events = bool
write_data_events = bool
management_events = bool
}))
metrics_monitoring = optional(object({
metrics_monitoring_crn = string
request_metrics_enabled = optional(bool)
usage_metrics_enabled = optional(bool)
}))
}))
keys = optional(
list(object({
name = string
role = string
enable_HMAC = bool
}))
)

})
)
n/a yes
enable_transit_gateway Create transit gateway bool true no
existing_vpc_cbr_zone_id ID of the existing CBR (Context-based restrictions) network zone, with context set to the VPC. This zone is used in a CBR rule, which allows traffic to flow only from the landing zone VPCs to specific cloud services. string null no
f5_template_data Data for all f5 templates
object({
tmos_admin_password = optional(string)
license_type = optional(string)
byol_license_basekey = optional(string)
license_host = optional(string)
license_username = optional(string)
license_password = optional(string)
license_pool = optional(string)
license_sku_keyword_1 = optional(string)
license_sku_keyword_2 = optional(string)
license_unit_of_measure = optional(string)
do_declaration_url = optional(string)
as3_declaration_url = optional(string)
ts_declaration_url = optional(string)
phone_home_url = optional(string)
template_source = optional(string)
template_version = optional(string)
app_id = optional(string)
tgactive_url = optional(string)
tgstandby_url = optional(string)
tgrefresh_url = optional(string)
})
{
"license_type": "none"
}
no
f5_vsi A list describing F5 VSI workloads to create
list(
object({
name = string
vpc_name = string
primary_subnet_name = string
secondary_subnet_names = list(string)
secondary_subnet_security_group_names = list(
object({
group_name = string
interface_name = string
})
)
ssh_keys = list(string)
f5_image_name = string
machine_type = string
resource_group = optional(string)
enable_management_floating_ip = optional(bool)
enable_external_floating_ip = optional(bool)
security_groups = optional(list(string))
boot_volume_encryption_key_name = optional(string)
hostname = string
domain = string
access_tags = optional(list(string), [])
security_group = optional(
object({
name = string
rules = list(
object({
name = string
direction = string
source = string
tcp = optional(
object({
port_max = number
port_min = number
})
)
udp = optional(
object({
port_max = number
port_min = number
})
)
icmp = optional(
object({
type = number
code = number
})
)
})
)
})
)
block_storage_volumes = optional(list(
object({
name = string
profile = string
capacity = optional(number)
iops = optional(number)
encryption_key = optional(string)
})
))
load_balancers = optional(list(
object({
name = string
type = string
listener_port = number
listener_protocol = string
connection_limit = number
algorithm = string
protocol = string
health_delay = number
health_retries = number
health_timeout = number
health_type = string
pool_member_port = string
idle_connection_timeout = optional(number)
security_group = optional(
object({
name = string
rules = list(
object({
name = string
direction = string
source = string
tcp = optional(
object({
port_max = number
port_min = number
})
)
udp = optional(
object({
port_max = number
port_min = number
})
)
icmp = optional(
object({
type = number
code = number
})
)
})
)
})
)
})
))
})
)
[] no
key_management Key Protect instance variables
object({
name = optional(string)
resource_group = optional(string)
use_data = optional(bool)
use_hs_crypto = optional(bool)
access_tags = optional(list(string), [])
service_endpoints = optional(string, "public-and-private")
keys = optional(
list(
object({
name = string
root_key = optional(bool)
payload = optional(string)
key_ring = optional(string) # Any key_ring added will be created
force_delete = optional(bool)
existing_key_crn = optional(string) # CRN of an existing key in the same or different account.
endpoint = optional(string) # can be public or private
iv_value = optional(string) # (Optional, Forces new resource, String) Used with import tokens. The initialization vector (IV) that is generated when you encrypt a nonce. The IV value is required to decrypt the encrypted nonce value that you provide when you make a key import request to the service. To generate an IV, encrypt the nonce by running ibmcloud kp import-token encrypt-nonce. Only for imported root key.
encrypted_nonce = optional(string) # The encrypted nonce value that verifies your request to import a key to Key Protect. This value must be encrypted by using the key that you want to import to the service. To retrieve a nonce, use the ibmcloud kp import-token get command. Then, encrypt the value by running ibmcloud kp import-token encrypt-nonce. Only for imported root key.
policies = optional(
object({
rotation = optional(
object({
interval_month = number
})
)
dual_auth_delete = optional(
object({
enabled = bool
})
)
})
)
})
)
)
})
n/a yes
network_cidr Network CIDR for the VPC. This is used to manage network ACL rules for cluster provisioning. string "10.0.0.0/8" no
prefix A unique identifier for resources that is prepended to resources that are provisioned. Must begin with a lowercase letter and end with a lowercase letter or number. Must be 16 or fewer characters. string n/a yes
region Region where VPC will be created. To find your VPC region, use ibmcloud is regions command to find available regions. string n/a yes
resource_groups Object describing resource groups to create or reference
list(
object({
name = string
create = optional(bool)
use_prefix = optional(bool)
})
)
n/a yes
security_groups Security groups for VPC
list(
object({
name = string
vpc_name = string
resource_group = optional(string)
access_tags = optional(list(string), [])
rules = list(
object({
name = string
direction = string
source = string
tcp = optional(
object({
port_max = number
port_min = number
})
)
udp = optional(
object({
port_max = number
port_min = number
})
)
icmp = optional(
object({
type = number
code = number
})
)
})
)
})
)
[] no
service_endpoints Service endpoints for the App ID resource when created by the module. Can be public, private, or public-and-private string "public-and-private" no
skip_all_s2s_auth_policies Whether to skip the creation of all of the service-to-service authorization policies. If setting to true, policies must be in place on the account before provisioning. bool false no
skip_kms_block_storage_s2s_auth_policy Whether to skip the creation of a service-to-service authorization policy between block storage and the key management service. bool false no
skip_kms_kube_s2s_auth_policy Whether to skip the creation of a service-to-serivce authorization policy between kubernetes and the key management service. bool false no
ssh_keys SSH keys to use to provision a VSI. Must be an RSA key with a key size of either 2048 bits or 4096 bits (recommended). If public_key is not provided, the named key will be looked up from data. If a resource group name is added, it must be included in var.resource_groups. See https://cloud.ibm.com/docs/vpc?topic=vpc-ssh-keys.
list(
object({
name = string
public_key = optional(string)
resource_group = optional(string)
})
)
n/a yes
tags List of resource tags to apply to resources created by this module. list(string) [] no
teleport_config_data Teleport config data. This is used to create a single template for all teleport instances to use. Creating a single template allows for values to remain sensitive
object({
teleport_license = optional(string)
https_cert = optional(string)
https_key = optional(string)
domain = optional(string)
cos_bucket_name = optional(string)
cos_key_name = optional(string)
teleport_version = optional(string)
message_of_the_day = optional(string)
hostname = optional(string)
app_id_key_name = optional(string)
claims_to_roles = optional(
list(
object({
email = string
roles = list(string)
})
)
)
})
null no
teleport_vsi A list of teleport vsi deployments
list(
object(
{
name = string
vpc_name = string
resource_group = optional(string)
subnet_name = string
ssh_keys = list(string)
boot_volume_encryption_key_name = string
image_name = string
machine_type = string
access_tags = optional(list(string), [])
security_groups = optional(list(string))
security_group = optional(
object({
name = string
rules = list(
object({
name = string
direction = string
source = string
tcp = optional(
object({
port_max = number
port_min = number
})
)
udp = optional(
object({
port_max = number
port_min = number
})
)
icmp = optional(
object({
type = number
code = number
})
)
})
)
})
)


}
)
)
[] no
transit_gateway_connections Transit gateway vpc connections. Will only be used if transit gateway is enabled. list(string) n/a yes
transit_gateway_global Connect to the networks outside the associated region. Will only be used if transit gateway is enabled. bool false no
transit_gateway_resource_group Name of resource group to use for transit gateway. Must be included in var.resource_group string n/a yes
virtual_private_endpoints Object describing VPE to be created
list(
object({
service_name = string
service_type = string
resource_group = optional(string)
access_tags = optional(list(string), [])
vpcs = list(
object({
name = string
subnets = list(string)
security_group_name = optional(string)
})
)
})
)
n/a yes
vpc_placement_groups List of VPC placement groups to create
list(
object({
access_tags = optional(list(string), [])
name = string
resource_group = optional(string)
strategy = string
})
)
[] no
vpcs A map describing VPCs to be created in this repo.
list(
object({
prefix = string # VPC prefix
existing_vpc_id = optional(string)
existing_subnets = optional(
list(
object({
id = string
public_gateway = optional(bool, false)
})
)
)
resource_group = optional(string) # Name of the group where VPC will be created
access_tags = optional(list(string), [])
classic_access = optional(bool)
default_network_acl_name = optional(string)
default_security_group_name = optional(string)
clean_default_sg_acl = optional(bool, false)
dns_binding_name = optional(string, null)
dns_instance_name = optional(string, null)
dns_custom_resolver_name = optional(string, null)
dns_location = optional(string, "global")
dns_plan = optional(string, "standard-dns")
existing_dns_instance_id = optional(string, null)
use_existing_dns_instance = optional(bool, false)
enable_hub = optional(bool, false)
skip_spoke_auth_policy = optional(bool, false)
hub_account_id = optional(string, null)
enable_hub_vpc_id = optional(bool, false)
hub_vpc_id = optional(string, null)
enable_hub_vpc_crn = optional(bool, false)
hub_vpc_crn = optional(string, null)
update_delegated_resolver = optional(bool, false)
skip_custom_resolver_hub_creation = optional(bool, false)
resolver_type = optional(string, null)
manual_servers = optional(list(object({
address = string
zone_affinity = optional(string)
})), [])
default_security_group_rules = optional(
list(
object({
name = string
direction = string
remote = string
tcp = optional(
object({
port_max = optional(number)
port_min = optional(number)
})
)
udp = optional(
object({
port_max = optional(number)
port_min = optional(number)
})
)
icmp = optional(
object({
type = optional(number)
code = optional(number)
})
)
})
)
)
default_routing_table_name = optional(string)
flow_logs_bucket_name = optional(string)
address_prefixes = optional(
object({
zone-1 = optional(list(string))
zone-2 = optional(list(string))
zone-3 = optional(list(string))
})
)
network_acls = list(
object({
name = string
add_ibm_cloud_internal_rules = optional(bool)
add_vpc_connectivity_rules = optional(bool)
prepend_ibm_rules = optional(bool)
rules = list(
object({
name = string
action = string
destination = string
direction = string
source = string
tcp = optional(
object({
port_max = optional(number)
port_min = optional(number)
source_port_max = optional(number)
source_port_min = optional(number)
})
)
udp = optional(
object({
port_max = optional(number)
port_min = optional(number)
source_port_max = optional(number)
source_port_min = optional(number)
})
)
icmp = optional(
object({
type = optional(number)
code = optional(number)
})
)
})
)
})
)
use_public_gateways = object({
zone-1 = optional(bool)
zone-2 = optional(bool)
zone-3 = optional(bool)
})
subnets = optional(object({
zone-1 = list(object({
name = string
cidr = string
public_gateway = optional(bool)
acl_name = string
no_addr_prefix = optional(bool, false)
}))
zone-2 = list(object({
name = string
cidr = string
public_gateway = optional(bool)
acl_name = string
no_addr_prefix = optional(bool, false)
}))
zone-3 = list(object({
name = string
cidr = string
public_gateway = optional(bool)
acl_name = string
no_addr_prefix = optional(bool, false)
}))
}))
})
)
n/a yes
vpn_gateways List of VPN Gateways to create.
list(
object({
name = string
vpc_name = string
subnet_name = string # Do not include prefix, use same name as in var.subnets
mode = optional(string)
resource_group = optional(string)
access_tags = optional(list(string), [])
})
)
n/a yes
vsi A list describing VSI workloads to create
list(
object({
name = string
vpc_name = string
subnet_names = list(string)
ssh_keys = list(string)
image_name = string
machine_type = string
vsi_per_subnet = number
user_data = optional(string)
resource_group = optional(string)
enable_floating_ip = optional(bool)
security_groups = optional(list(string))
boot_volume_encryption_key_name = optional(string)
access_tags = optional(list(string), [])
security_group = optional(
object({
name = string
rules = list(
object({
name = string
direction = string
source = string
tcp = optional(
object({
port_max = number
port_min = number
})
)
udp = optional(
object({
port_max = number
port_min = number
})
)
icmp = optional(
object({
type = number
code = number
})
)
})
)
})
)
block_storage_volumes = optional(list(
object({
name = string
profile = string
capacity = optional(number)
iops = optional(number)
encryption_key = optional(string)
})
))
load_balancers = optional(list(
object({
name = string
type = string
listener_port = number
listener_protocol = string
connection_limit = number
algorithm = string
protocol = string
health_delay = number
health_retries = number
health_timeout = number
health_type = string
pool_member_port = string
idle_connection_timeout = optional(number)
security_group = optional(
object({
name = string
rules = list(
object({
name = string
direction = string
source = string
tcp = optional(
object({
port_max = number
port_min = number
})
)
udp = optional(
object({
port_max = number
port_min = number
})
)
icmp = optional(
object({
type = number
code = number
})
)
})
)
})
)
})
))
})
)
n/a yes
wait_till To avoid long wait times when you run your Terraform code, you can specify the stage when you want Terraform to mark the cluster resource creation as completed. Depending on what stage you choose, the cluster creation might not be fully completed and continues to run in the background. However, your Terraform code can continue to run without waiting for the cluster to be fully created. Supported args are MasterNodeReady, OneWorkerNodeReady, and IngressReady string "IngressReady" no

Outputs

Name Description
appid_key_names List of appid key names created
appid_name Name of the appid instance used.
appid_redirect_urls List of appid redirect urls
atracker_route_name Name of atracker route
atracker_target_name Name of atracker target
bastion_host_names List of bastion host names
cluster_data List of cluster data
cluster_names List of create cluster names
cos_bucket_data List of data for COS buckets creaed
cos_bucket_names List of names for COS buckets created
cos_data List of Cloud Object Storage instance data
cos_key_names List of names for created COS keys
cos_names List of Cloud Object Storage instance names
f5_hosts List of bastion host names
fip_vsi_data A list of VSI with name, id, zone, and primary ipv4 address, VPC Name, and floating IP. This list only contains instances with a floating IP attached.
key_management_crn CRN for KMS instance
key_management_guid GUID for KMS instance
key_management_name Name of key management service
key_map Map of ids and keys for keys created
key_rings Key rings created by module
management_cluster_id The id of the management cluster. If the cluster name does not exactly match the prefix-management-cluster pattern it will be null.
management_cluster_name The name of the management cluster. If the cluster name does not exactly match the prefix-management-cluster pattern it will be null.
placement_groups List of placement groups.
resource_group_data List of resource groups data used within landing zone.
resource_group_names List of resource groups names used within landing zone.
security_group_data List of security group data
security_group_names List of security group names
service_authorization_data List of service authorization data
service_authorization_names List of service authorization names
ssh_key_data List of SSH key data
ssh_key_names List of SSH key names
subnet_data List of Subnet data created
subnet_names List of Subnet names created
transit_gateway_data Created transit gateway data
transit_gateway_name Name of created transit gateway
vpc_data List of VPC data
vpc_dns List of VPC DNS details for each of the VPCs.
vpc_names List of VPC names
vpc_resource_list List of VPC with VSI and Cluster deployed on the VPC.
vpe_gateway_data List of VPE gateways data
vpe_gateway_names VPE gateway names
vpn_data List of VPN data
vpn_names List of VPN names
vsi_data A list of VSI with name, id, zone, and primary ipv4 address, VPC Name, and floating IP.
vsi_names List of VSI names
workload_cluster_id The id of the workload cluster. If the cluster name does not exactly match the prefix-workload-cluster pattern it will be null.
workload_cluster_name The name of the workload cluster. If the cluster name does not exactly match the prefix-workload-cluster pattern it will be null.

Contributing

You can report issues and request features for this module in GitHub issues in the module repo. See Report an issue or request a feature.

To set up your local development environment, see Local development setup in the project documentation.