CD generation for user-data fails on macos

[viraptor]

Overview of the Issue

The options to create the user-data cd don't work well on macos. The .isos seem to be created just fine and look reasonable, but they're not detected by cloud-init.

Reproduction Steps

Try using a user-data generated on macos with either mkisofs or hdiutil.
The iso is in place, but gets ignored by the image from https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img
I did not dig into the differences too much.

On the other hand, the same user-data packed using cloud-localds on linux gets detected and processed just fine. To test, I replaced hdiutil with:

#!/bin/sh
cp user-data.img $3

and used the image copied from linux.
(update: got the cloud-utils built on macos and that one also gets detected just fine, so it's not a mac/linux issue, but rather cloud-utils vs mkisofs/hdiutil)

Ok, so these are all external tools, why am I reporting it here? Mainly in case this is a general issue with hdiutil and it should be removed from the list. Or maybe there are some options you can use by default to make it work properly.
I can't easily tell what the difference is between the different isos yet. I'll attach the a package with the 2 failing and 1 working version.
isos.tar.gz

Plugin and Packer version

1.8.4

Simplified Packer Buildfile

source "qemu" "local-build" {
  iso_checksum = "none"
  iso_url = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img"
  disk_image = true
  output_directory = "built-images"
  disk_size = "5000M"
  format = "qcow2"
  qemu_binary = "qemu-system-aarch64"
  headless = true
  use_default_display = true
  firmware = "edk2-aarch64-code.fd"
  machine_type = "virt,highmem=off"
  cpu_model = "host"
  accelerator = "hvf"
  ssh_username = "ubuntu"
  ssh_password = "ubuntu"
  cd_files = [ "user-data" ]
  cd_label = "cidata"
}

The user-data contains:

#cloud-config
users:
  - name: ubuntu
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    lock_passwd: False
    plain_text_passwd: "ubuntu"
runcmd:
  - [ sed, "-e", "s/^PasswordAuthentication.*/PasswordAuthentication yes/", "-i", "/etc/ssh/sshd_config" ]
  - [ systemctl, restart, ssh ]

Operating system and Environment details

Darwin Kernel Version 22.3.0, M2, aarch64
mkisofs - 3.02a09
cloud-utils - 0.32-22-g45fe84a5-0ubuntu1
hdiutil - whatever comes with that macos version.

Log Fragments and crash.log files

Nothing relevant beyond the user never being created.

[viraptor]

Looking into this a bit deeper:

cloud-localds.iso (working)

CD-ROM is in ISO 9660 format
System id: LINUX
Volume id: cidata
Volume set id: 
Publisher id: 
Data preparer id: 
Application id: GENISOIMAGE ISO 9660/HFS FILESYSTEM CREATOR (C) 1993 E.YOUNGDALE (C) 1997-2006 J.PEARSON/J.SCHILLING (C) 2006-2007 CDRKIT TEAM
Copyright File id: 
Abstract File id: 
Bibliographic File id: 
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 183

Joliet with UCS level 3 found.
SUSP signatures version 1 found
Rock Ridge signatures version 1 found
Rock Ridge id 'RRIP_1991A'

contents

/META_DAT.;1
/USER_DAT.;1

hdiutil.iso (failing)

CD-ROM is in ISO 9660 format
System id: APPLE INC., TYPE: 0002
Volume id: CIDATA
Volume set id: 
Publisher id: 
Data preparer id: 
Application id: 
Copyright File id: 
Abstract File id: 
Bibliographic File id: 
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 35

Joliet with UCS level 1 found.
SUSP signatures version 1 found
No valid Rock Ridge signature found

contents:

/USER-DATA

mkisofs.iso (failing)

CD-ROM is in ISO 9660 format
System id: Mac OS X
Volume id: cidata
Volume set id: 
Publisher id: 
Data preparer id: 
Application id: MKISOFS ISO9660/HFS/UDF FILESYSTEM BUILDER & CDRECORD CD/DVD/BluRay CREATOR (C) 1993 E.YOUNGDALE (C) 1997 J.PEARSON/J.SCHILLING
Copyright File id: 
Abstract File id: 
Bibliographic File id: 
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 181

Joliet with UCS level 3 found.
No SUSP/Rock Ridge present

contents:

/USER_DAT.;1

Edit: Adding the meta-data file didn't help.

[viraptor]

Success! I tried a few flags and this was failing:

/usr/bin/hdiutil makehybrid -o /tmp/packer3596932755.iso -hfs -joliet -iso -default-volume-name cidata /tmp/packer_to_cdrom670900126

But this works:

/usr/bin/hdiutil makehybrid -o $3 -joliet -iso -default-volume-name cidata $9

It looks like adding -hfs trips up the cloud image... which may make sense in case the module is not loaded at the time? Or some other reason - I didn't dig far enough.

I'll see if I can find the right options for mkisofs as well, but that's not today. (it may just need a -no-hfs)

[viraptor]

Ping @lbajolet-hashicorp since there's at least one fix available.

[lbajolet-hashicorp]

Hi @viraptor!

Thanks for the report; from what I can see, hdiutil is always invoked with -hfs (the code is from the SDK, SEE: https://github.com/hashicorp/packer-plugin-sdk/blob/main/multistep/commonsteps/step_create_cdrom.go#L164), so the resulting image is indeed HFS+.
If you import it from a Linux image later on, this could be a problem as you mention, the driver likely isn't loaded, therefore making it fail when attempting to mount.

Although according to the hdiutil docs, even if there's HFS+, there should also be some ISO9660 and Joliet data, so the image should be mountable even without HFS+ support on the guest OS, so this option should theoretically be harmless. As you point out, I suspect there may be a problem with the cloud-utils here, I do not know what they do, so I'll probably need your help to understand the issue exactly here.
Could you provide some logs to see what's being done with the build here? Preferably with PACKER_LOG=1 so we can have as much information as possible.

Additionally, I see that for the Linux-built ISO, there's Rock Ridge support, while there isn't for those built by macOS. This contradicts the documentation for hdiutil (the -iso option states Generate an ISO9660 Level 2 filesystem with Rock Ridge extensions.). This could be a problem since Rock Ridge adds support for longer filenames and UNIX-like permissions among others.

Removing -hfs from the command-line may not be the solution here, as some users may rely on this, so I'd tread carefully with these options, and since that code lives in the SDK, changes to that code will propagate to all plugins.

If you have other tools like xorriso or mkisofs, they should be used instead of hdiutil if they're installed on your machine since they have precedence over hdiutil when generating the command-line to create the data ISO.

To be frank, I won't be able to test/debug this as I don't have a macOS machine on hand, but we do have mac users on the Packer team, so I think we can manage to test and eventually fix this.

Sorry for the wall of text, I'm trying to figure out what the exact problem is. As mentioned though, could you provide logs of a failing build so we can see what the exact error is, and if possible, the template that you are using to build your image, so that we can replicate this behaviour and try to figure out a fix.

This issue may be an SDK one, but we'll have to figure out to understand what is the problem and the fix before we transfer this issue over, so I'll leave this in this repository in the meantime.

[viraptor]

Ok, good news, at least xorriso "just works":

2023/03/28 15:56:11 packer-plugin-qemu_v1.0.9_x5.0_darwin_arm64 plugin: 2023/03/28 15:56:11 Executing: /nix/store/w5n63bk4j68yfpxkfign1wf7sa91m1va-xorriso-1.5.4.pl02/bin/xorriso [-as genisoimage -rock -joliet -volid cidata -output /tmp/packer1815805504.iso /tmp/packer_to_cdrom3991152261]

mkisofs from cdrtools does too: (cdrkit did not)

2023/03/28 15:59:19 packer-plugin-qemu_v1.0.9_x5.0_darwin_arm64 plugin: 2023/03/28 15:59:19 Executing: /nix/store/79x118zgwxjaslcp5q7bn1vnyf3bl702-cdrtools-3.02a09/bin/mkisofs [-joliet -volid cidata -o /tmp/packer2159155907.iso /tmp/packer_to_cdrom153317239]

hdiutil call producing the failing hfs/iso:

2023/03/28 16:02:02 packer-plugin-qemu_v1.0.9_x5.0_darwin_arm64 plugin: 2023/03/28 16:02:02 Executing: /usr/bin/hdiutil [makehybrid -o /tmp/packer530833760.iso -hfs -joliet -iso -default-volume-name cidata /tmp/packer_to_cdrom453266334]

This fails on ssh connection - the user I expect simply isn't there (cloud-init never sees the provided cidata).

You should be able to replicate this on an ARM macos with:

source "qemu" "local-build" {
  iso_checksum = "none"
  iso_url = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img"
  disk_image = true
  output_directory = "built-images"
  disk_size = "5000M"
  format = "qcow2"
  qemu_binary = "qemu-system-aarch64"
  headless = true
  use_default_display = true
  firmware = "edk2-aarch64-code.fd"
  machine_type = "virt,highmem=off"
  cpu_model = "host"
  accelerator = "hvf"
  ssh_username = "ubuntu"
  ssh_password = "ubuntu"
  cd_files = [ "meta-data", "user-data" ]
  cd_label = "cidata"
}
build {
  sources = ["source.qemu.local-build"]
  provisioner "shell" {
    inline = ["cloud-init status --wait"]
  }
}

(+ user-data like in the description on top)