EC2 Metadata Setting not working with latest packer version(1.8.6)

[sumitgaur26]

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

I am using latest version of packer for linux and added below code to my packer.json file to make the IMDSv2 enabled by default.
"metadata_options": {
"http_endpoint": "enabled",
"http_tokens": "required",
"http_put_response_hop_limit": 1
}

As per the packer documentation, using this option should create a AMI which would have IMDSv2 enabled by default when a instance is spin up. But when I am running the packer build the instance created by Packer is having IMDSv2 enabled but the AMI produced is not having it enabled.

Reproduction Steps

Use the metadata setting block in packer.json and see if the AMI created is having IMDSv2 enforced or not

Plugin and Packer version

From packer version
1.8.6

Simplified Packer Buildfile

 "builders": [
    {
      "type": "amazon-ebs",
      "ssh_pty": "true",
      "region": "{{user `region`}}",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "{{user `instance_type`}}",
      "iam_instance_profile": "IAM role",
      "encrypt_boot": false,
      "subnet_filter": {
        "filters": {
          "tag:Name": "us-east-1*"
        },
        "most_free": true,
        "random": false
      },
      "tags": {
        "Base Image Id": "{{user `source_ami`}}",
        "Name": "{{user `ami_name`}}",
        "AWS_STEP_ID": "{{user `AWS_STEP_ID`}}",
        "created_by": "{{user `created_by`}}",
        "Source": "{{user `AWS_STEP_ID`}}",
        "JOB_NAME": "{{user `JOB_NAME`}}",
        "Repo": "{{user `Repo`}}",
        "Playbook": "{{user `playbookname`}}",
        "CommitID": "{{user `commitid`}}"
      },
      "run_tags": {
        "Base Image Id": "{{user `source_ami`}}",
        "Name": "{{user `ami_name`}}",
        "AWS_STEP_ID": "{{user `AWS_STEP_ID`}}",
        "created_by": "{{user `created_by`}}",
        "Source": "{{user `AWS_STEP_ID`}}",
        "JOB_NAME": "{{user `JOB_NAME`}}",
        "Repo": "{{user `Repo`}}",
        "Playbook": "{{user `playbookname`}}",
        "CommitID": "{{user `commitid`}}"
      },
      "ami_name": "{{user `ami_name`}}",
      "ami_description": "{{user `ami_description`}}",
      "metadata_options": {
        "http_endpoint": "enabled",
        "http_tokens": "required",
        "http_put_response_hop_limit": 1
      },
      "temporary_security_group_source_cidrs": [
        "10.0.0.0/8"
      ],
      "associate_public_ip_address": "false",
      "ssh_interface": "private_ip",
      "ssh_username": "{{user `ssh_username`}}",
      "ena_support": true,

      "launch_block_device_mappings": [
        {
          "delete_on_termination": "true",
          "device_name": "/dev/xvda",
          "volume_size": "{{ user `root_volume_size`}}",
          "volume_type": "gp3"
        },
        {
          "delete_on_termination": "true",
          "device_name": "/dev/sdf",
          "volume_size": "55",
          "volume_type": "gp3"
        }
      ]
    }
  ],
  "provisioners": [
    {
      "type": "ansible",
      "playbook_file": "{{user `playbook`}}",
      "user": "{{user `ssh_username`}}"
    }
  ]
}

Operating system and Environment details

Running in a container based on alpine3.7 and amd_64 is the arch.

Log Fragments and crash.log files

No log and nothing failing

Set the env var PACKER_LOG=1 for maximum log detail.

[nywilken]

Hi @sumitgaur26 the provided metadata options specified relate to the instance being built. I believe you need to also specify the "imds_support":"v2.0" in your template to force IMDSv2 on the result AMI.

Please give it a try and let me know if the resulting build is what you are looking for.

[ttrevorr]

There is an error when attempting to use "imds_support":"v2.0" with a policy in place to force only imds v2 instances.
We have a policy in place to only allow instances with this enabled and even setting that option the builder fails with error:
"conditions": {
"items": [
{
"key": "ec2:MetadataHttpTokens",
"values": {
"items": [
{
"value": "required"
}
]
}
}
]
}

Disabling the force imdsV2 policy allows the builder to work correctly. This is on version 1.8.6.

[sumitgaur26]

Hi @nywilken,

I have used the option you suggested, it's not working. When packer launches it has the V2 version enabled but when the AMI is created, that AMI doesn't have the this option enabled

[ostigley]

I'm also having this issue.

[blueprismo]

Same issue, instance_metadata_tags enabled is not reflected

[zachpsurest]

I'm also having this issue. I have tried including "imds_support":"v2.0" but we're using JSON instead of HCL and when I validate the packer JSON it says it's not supported. The version of Packer is v.1.9.4.

[tivanov-qb]

+1 on this issue, hopefully after 3 years or so this will be resolved by the hashicorp turbo top development team

[tivanov-qb]

@nywilken

I'm also having this issue. I have tried including "imds_support":"v2.0" but we're using JSON instead of HCL and when I validate the packer JSON it says it's not supported. The version of Packer is v.1.9.4.

+1

[nywilken]

Hey folks thanks for bubbling up this issue. Looking at the additional comments it looks like we have two issues being reported:

• Instance metadata V2 not enabled on the resulting AMI
• imds_support attribute unsupported error.

Both of which will be resolved by installing the latest version of the Amazon Packer plugin.

For JSON folks running into the issue "imds_support":"v2.0" is not supported you are using an older version of the Amazon plugin bundled into Packer - looking at the Packer CHANGELOG for 1.8.6 to 1.9.4 the Amazon plugin is pinned to version 1.2.1. Meaning that the Packer binary has an old version of the Amazon plugin, which does not contain the latest imds_support attribute for the amazon-ebs builder. The fix for imds_support for AMIs and the EBS builder was added in version 1.2.2.

To resolve your issue I recommend installing the latest version of the Amazon plugin by running

packer plugin install github.com/hashicorp/amazon

You can validate the installed version with the command below

~>  packer plugins installed
~/.packer.d/plugins/github.com/hashicorp/docker/packer-plugin-docker_v1.0.8_x5.0_darwin_amd64
~/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.2.7_x5.0_darwin_amd64

The Packer team has moved away from bundling updated plugins with Packer since 1.8.x in favor of using packer plugins install for JSON users. We called this out in the 1.9.2 Packer release.

---

Once you have the latest version of the Amazon plugin installed please update your configurations either JSON or HCL to include the metadata_options block and the imds_support attribute. Below is a full build template written in HCL.

Example Configuration using HCL and plugin version 1.2.7

packer {
  required_plugins {
    amazon = {
      version = "~>1"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

data "amazon-ami" "ubuntu-amd64" {
  filters = {
    name                = "ubuntu/images/*ubuntu-*-22.04-amd64-server-*"
    root-device-type    = "ebs"
    virtualization-type = "hvm"
  }
  most_recent = true
  owners      = ["099720109477"]
}

locals { timestamp = regex_replace(timestamp(), "[- TZ:]", "") }

source "amazon-ebs" "basic-example" {
  ami_name      = "packer-example-${local.timestamp}"
  communicator  = "ssh"
  instance_type = "t2.micro"
  source_ami    = data.amazon-ami.ubuntu-amd64.id
  ssh_username  = "ubuntu"
  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
  }
  imds_support = "v2.0"

}

build {
  sources = ["source.amazon-ebs.basic-example"]
      # Lets query the instance metadata service running v2
      provisioner "shell" {
        inline = ["TOKEN=`curl -s -X PUT \"http://169.254.169.254/latest/api/token\" -H \"X-aws-ec2-metadata-token-ttl-seconds: 21600\"` && curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -s http://169.254.169.254/latest/meta-data/"]
      }
}

Metadata version Validation Checks

Querying the instance metadata via curl on running instance.

==> amazon-ebs.basic-example: > GET /latest/meta-data/ HTTP/1.1
==> amazon-ebs.basic-example:  Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
   amazon-ebs.basic-example: ami-id
    amazon-ebs.basic-example: ami-launch-index
    amazon-ebs.basic-example: ami-manifest-path
    amazon-ebs.basic-example: block-device-mapping/
    amazon-ebs.basic-example: events/
    amazon-ebs.basic-example: hostname
    amazon-ebs.basic-example: identity-credentials/
    amazon-ebs.basic-example: instance-action
    amazon-ebs.basic-example: instance-id
    amazon-ebs.basic-example: instance-life-cycle
    amazon-ebs.basic-example: instance-type
    amazon-ebs.basic-example: local-hostname
    amazon-ebs.basic-example: local-ipv4
    amazon-ebs.basic-example: mac
    amazon-ebs.basic-example: metrics/
    amazon-ebs.basic-example: network/
    amazon-ebs.basic-example: placement/
    amazon-ebs.basic-example: profile
    amazon-ebs.basic-example: public-hostname
    amazon-ebs.basic-example: public-ipv4
    amazon-ebs.basic-example: public-keys/
    amazon-ebs.basic-example: reservation-id
    amazon-ebs.basic-example: security-groups
    amazon-ebs.basic-example: services/
    amazon-ebs.basic-example: system

Running Describe on created AMI

~>  aws ec2 describe-images --image-ids=ami-0042263517a113537
{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2023-11-03T01:35:41.000Z",
            "ImageId": "ami-0042263517a113537",
            "ImageLocation": "746700064644/packer-example-20231103013415",
            "ImageType": "machine",
            "Public": false,
            "OwnerId": "746700064644",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-0db7ceed7e4b40ea5",
                        "VolumeSize": 8,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                },
                {
                    "DeviceName": "/dev/sdb",
                    "VirtualName": "ephemeral0"
                },
                {
                    "DeviceName": "/dev/sdc",
                    "VirtualName": "ephemeral1"
                }
            ],
            "EnaSupport": true,
            "Hypervisor": "xen",
            "Name": "packer-example-20231103013415",
            "RootDeviceName": "/dev/sda1",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm",
            "ImdsSupport": "v2.0"
        }
    ]
}

[nywilken]

I opened a PR to add a working example of this code to the repository and added a section to the documentation for enabling imdsv2 on the running instance and generated AMI.

[tivanov-qb]

@nywilken thanks for the help. this does not work for me.. i'm using packer json templates and i'm running it from hashicorp/packer docker image.. i just tried using hashicorp/packer:full but it made no diff

example:
image 'hashicorp/packer:full'
args '-v ${WORKSPACE}:/workspace -w /workspace -e PACKER_PLUGIN_PATH=/workspace/.packer.d/plugins --entrypoint='''

13:38:41 Status: Downloaded newer image for hashicorp/packer:full
13:38:41 docker.io/hashicorp/packer:full
13:35:43 1699011343,,ui,error,Error: Failed to prepare build: "amazon-ebs"\n\n1 error occurred:\n * unknown configuration key: '"imds_support"'\n\n\n\n

source doc: https://hub.docker.com/r/hashicorp/packer

[nywilken]

@tivanov-qb thanks for the quick response. It looks like you are overwriting the plugins installed in the container by setting the PACKER_PLUGIN_PATH environment variable.

Did you validate which version of the plugin is being loaded from PACKER_PLUGIN_PATH=/workspace/.packer.d/plugins?

Running the command below should give show what plugins were loaded. If the Amazon plugin is not within the output then you are still using the bundled plugin within the Packer binary.

docker run hashicorp/packer:full -v ${WORKSPACE}:/workspace -w /workspace  -e 
 PACKER_PLUGIN_PATH=/workspace/.packer.d/plugins plugins installed

To confirm the container works I ran a few commands without overriding PACKER_PLUGIN_PATH

Plugins installed into packer:full

~>  docker run -v `pwd`:/workspace -w /workspace hashicorp/packer:full plugins installed
/root/.config/packer/plugins/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.0.3_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/virtualbox/packer-plugin-virtualbox_v1.0.5_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/docker/packer-plugin-docker_v1.0.8_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/qemu/packer-plugin-qemu_v1.0.9_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.0_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/googlecompute/packer-plugin-googlecompute_v1.1.1_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.2.1_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.2.6_x5.0_linux_arm64
/root/.config/packer/plugins/github.com/hashicorp/azure/packer-plugin-azure_v2.0.0_x5.0_linux_arm64

Running a validated on the json templated - available in #429

~>  docker run -v `pwd`:/workspace -w /workspace hashicorp/packer:full validate instance-metadata/ubuntu-imdsv2-enabled.json
The configuration is valid.