Deploying OVF from content library results in HTTP error 500

[sofixa]

Terraform Version

0.14.6

vSphere Provider Version

1.24.3

Affected Resource(s)

• vsphere_virtual_machine

Terraform Configuration Files

resource "vsphere_virtual_machine" "vm" {
  name             = var.name
  num_cpus         = var.cpu
  memory           = var.memory * 1024
  resource_pool_id = data.vsphere_compute_cluster.compute_cluster.resource_pool_id
  datastore_id     = data.vsphere_datastore.datastore[0].id
  host_system_id   = var.host_system_id
  folder           = var.folder
  annotation       = var.notes

  dynamic "disk" {
    for_each = var.disk_size
    content {
      label            = "${var.name}-${disk.key}.vmdk"
      size             = disk.value
      eagerly_scrub    = var.eagerly_scrub
      thin_provisioned = var.thin_provisioned
      unit_number      = disk.key
    }
  }

  clone {
    template_uuid = data.vsphere_content_library_item.image.id
  }

  lifecycle {
    ignore_changes = [clone[0].template_uuid, disk[0].key]
  }

  cpu_hot_add_enabled         = true
  memory_hot_add_enabled      = true
  wait_for_guest_net_routable = false

  dynamic "network_interface" {
    for_each = data.vsphere_network.network
    content {
      network_id   = network_interface.value.id
      adapter_type = var.network_interface_type
    }
  }
  custom_attributes = {
    (data.vsphere_custom_attribute.project_id.id) = (var.project_id)
  }

  extra_config = {
    "guestinfo.vmname" = var.name
  }
}

Debug Output

https://gist.github.com/sofixa/4cd7740a8628d5a43ec4db5fb75a5594
Note: this is with a custom build of the provider with extra logs and most notably the deployment spec of the template.

Expected Behavior

The content library OVF template should have been successfully deployed since there's nothing weird or special about it. The exact same template, with almost the same options gets deployed just fine with PowerCLI or govc, so it's something in terraform ( the differences are in the number of network interfaces and disks since we can't tell PowerCLI or govc to make multiple of them). The gist contains the specs generated by govc and terraform.

Actual Behavior

Error: POST https://sv-test.ad/rest/com/vmware/vcenter/ovf/library-item/id:d18e2fba-16cb-4d83-9ca7-85b360f07d16?~action=deploy: 500 Server Error

As per usual, a completely useless error from VMware. After going through the logs, there doesn't seem to be anything suspicious.

Steps to Reproduce

Try deploying a content library OVF template??? Not really sure what's causing the issue

How can we proceed about debugging this further? I found this issue in govmomi which seemed like a potential cause, but it was fixed in ~0.23, and when i rebuilt the provider with govmomi v0.24.0 the problem is still there, and govc 0.21 doesn't have any issues, so it's not that.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[sofixa]

So, after some more digging, we've narrowed down the cause for this problem.

terraform-provider-vsphere/vsphere/internal/helper/contentlibrary/content_library_helper.go

Lines 553 to 563 in 9d5570c

	// MapNetworkDevices maps NICs defined in the OVF to networks..
	func MapNetworkDevices(d *schema.ResourceData) []vcenter.NetworkMapping {
	nm := []vcenter.NetworkMapping{}
	nics := d.Get("network_interface").([]interface{})
	for _, di := range nics {
	dm := di.(map[string]interface{})["ovf_mapping"].(string)
	dd := di.(map[string]interface{})["network_id"].(string)
	nm = append(nm, vcenter.NetworkMapping{Key: dm, Value: dd})
	}
	return nm
	}

ovf_mapping is used for the key of the storage_mapping struct ( spec here ). If it isn't set ( why would it be, the terraform docs mention it once and it says (optional), terraform sends an empty key, like so:

 NetworkMappings: ([]vcenter.NetworkMapping) (len=2 cap=2) {
    (vcenter.NetworkMapping) {
     Key: (string) "",
     Value: (string) (len=15) "dvportgroup-128"
    },

which the vCenter doesn't appreciate, with the resulting vapi log:

com.vmware.vapi.bindings.convert.ConverterException: Could not convert field 'network_mappings' of structure 'com.vmware.vcenter.ovf.library_item.resource_pool_deployment_spec'
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:89)
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:33)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:281)
    at com.vmware.vapi.bindings.type.StructType.accept(StructType.java:372)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:305)
    at com.vmware.vapi.bindings.type.TypeReference.accept(TypeReference.java:20)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:661)
    at com.vmware.vapi.internal.bindings.StructValueExtractor.valueForField(StructValueExtractor.java:46)
    at com.vmware.vcenter.ovf.LibraryItemApiInterface$DeployApiMethod.doInvoke(LibraryItemApiInterface.java:44)
    at com.vmware.vapi.internal.bindings.ApiMethodSkeleton.invoke(ApiMethodSkeleton.java:175)
    at com.vmware.vapi.provider.ApiMethodBasedApiInterface.invoke(ApiMethodBasedApiInterface.java:82)
    at com.vmware.vapi.provider.local.LocalProvider.invokeMethodInt(LocalProvider.java:471)
    at com.vmware.vapi.provider.local.LocalProvider.invoke(LocalProvider.java:290)
    at com.vmware.vapi.admin.interposer.impl.Invoker.execute(Invoker.java:46)
    at com.vmware.vapi.admin.interposer.impl.PreInterposerHandler.execute(PreInterposerHandler.java:57)
    at com.vmware.vapi.admin.interposer.impl.VetoInterposerHandler.execute(VetoInterposerHandler.java:51)
    at com.vmware.vapi.admin.impl.InterposerImpl.invoke(InterposerImpl.java:277)
    at com.vmware.vdcs.activation.ActivationFilter.invoke(ActivationFilter.java:123)
    at com.vmware.vapi.core.DecoratorApiProvider.invoke(DecoratorApiProvider.java:37)
    at com.vmware.vsphere.common.impl.SecurityContextInterceptorProvider.invoke(SecurityContextInterceptorProvider.java:72)
    at com.vmware.vapi.cis.authz.impl.AuthorizationFilter.invoke(AuthorizationFilter.java:219)
    at com.vmware.vapi.provider.introspection.ErrorAugmentingFilter.invoke(ErrorAugmentingFilter.java:74)
    at com.vmware.vapi.security.AuthenticationFilter$1.setResult(AuthenticationFilter.java:180)
    at com.vmware.vapi.security.AuthenticationFilter$1.setResult(AuthenticationFilter.java:166)
    at com.vmware.vsphere.common.sessions.impl.SessionAuthnHandlerImpl.authenticate(SessionAuthnHandlerImpl.java:42)
    at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:165)
    at com.vmware.vapi.core.DecoratorApiProvider.invoke(DecoratorApiProvider.java:37)
    at com.vmware.vsphere.vcde.diagnostics.interceptor.ApiDiagnosticsInterceptorProvider.invoke(ApiDiagnosticsInterceptorProvider.java:50)
    at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processApiRequest(JsonServerConnection.java:275)
    at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:200)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:124)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:92)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1627)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.bindings.convert.ConverterException: Element already present in the map.
    at com.vmware.vapi.internal.bindings.convert.impl.JavaUtilMapMapConverter.fromValue(JavaUtilMapMapConverter.java:62)
    at com.vmware.vapi.internal.bindings.convert.impl.JavaUtilMapMapConverter.fromValue(JavaUtilMapMapConverter.java:30)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:264)
    at com.vmware.vapi.bindings.type.MapType.accept(MapType.java:42)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:661)
    at com.vmware.vapi.internal.bindings.convert.impl.NullableReferenceOptionalConverter.fromValue(NullableReferenceOptionalConverter.java:33)
    at com.vmware.vapi.internal.bindings.convert.impl.NullableReferenceOptionalConverter.fromValue(NullableReferenceOptionalConverter.java:23)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:239)
    at com.vmware.vapi.bindings.type.OptionalType.accept(OptionalType.java:24)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:661)
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:86)
    ... 51 more

Once we know that, it's easy, just set ovf_mapping on the network_interface block:

network_interface {
   network_id     = data.vsphere_network.network_bkp.id
   adapter_type   = "vmxnet3"
   ovf_mapping    = "nic0"
  }
  network_interface {
   network_id     = data.vsphere_network.network_adm.id
   adapter_type   = "vmxnet3"
   ovf_mapping    = "nic1"
  }

IMHO, at the very least, that needs to be documented much better. I'd even go as far as setting a default value if ovf_mapping is empty.

It's even worse with storage:

terraform-provider-vsphere/vsphere/internal/helper/contentlibrary/content_library_helper.go

Lines 534 to 551 in 9d5570c

	// MapStorageDevices maps disks defined in the OVF to datastores.
	func MapStorageDevices(d *schema.ResourceData) []vcenter.StorageMapping {
	sm := []vcenter.StorageMapping{}
	disks := d.Get("disk").([]interface{})
	for _, di := range disks {
	dm := di.(map[string]interface{})["ovf_mapping"].(string)
	dd := di.(map[string]interface{})["datastore_id"].(string)
	if dd == "<computed>" || dd == "" {
	dd = d.Get("datastore_id").(string)
	}
	dp := di.(map[string]interface{})["storage_policy_id"].(string)
	if dp == "" {
	dp = d.Get("storage_policy_id").(string)
	}
	sm = append(sm, vcenter.StorageMapping{Key: dm, Value: vcenter.StorageGroupMapping{Type: "DATASTORE", DatastoreID: dd, StorageProfileID: dp}})
	}
	return sm
	}

ovf_mapping is used for the key, but there is no such thing :

terraform-provider-vsphere/vsphere/internal/virtualdevice/virtual_machine_disk_subresource.go

Lines 77 to 235 in 9d5570c

	func DiskSubresourceSchema() map[string]*schema.Schema {
	s := map[string]*schema.Schema{
	// VirtualDiskFlatVer2BackingInfo
	"datastore_id": {
	Type: schema.TypeString,
	Optional: true,
	Computed: true,
	ConflictsWith: []string{"datastore_cluster_id"},
	Description: "The datastore ID for this virtual disk, if different than the virtual machine.",
	},
	"name": {
	Type: schema.TypeString,
	Optional: true,
	Description: "The file name of the disk. This can be either a name or path relative to the root of the datastore. If simply a name, the disk is located with the virtual machine.",
	ValidateFunc: func(v interface{}, _ string) ([]string, []error) {
	if path.Ext(v.(string)) != ".vmdk" {
	return nil, []error{fmt.Errorf("disk name %s must end in .vmdk", v.(string))}
	}
	return nil, nil
	},
	Deprecated: diskNameDeprecationNotice,
	},
	"path": {
	Type: schema.TypeString,
	Optional: true,
	Computed: true,
	ConflictsWith: []string{"datastore_cluster_id"},
	Description: "The full path of the virtual disk. This can only be provided if attach is set to true, otherwise it is a read-only value.",
	ValidateFunc: func(v interface{}, _ string) ([]string, []error) {
	if path.Ext(v.(string)) != ".vmdk" {
	return nil, []error{fmt.Errorf("disk path %s must end in .vmdk", v.(string))}
	}
	return nil, nil
	},
	},
	"disk_mode": {
	Type: schema.TypeString,
	Optional: true,
	Default: string(types.VirtualDiskModePersistent),
	Description: "The mode of this this virtual disk for purposes of writes and snapshotting. Can be one of append, independent_nonpersistent, independent_persistent, nonpersistent, persistent, or undoable.",
	ValidateFunc: validation.StringInSlice(diskSubresourceModeAllowedValues, false),
	},
	"eagerly_scrub": {
	Type: schema.TypeBool,
	Optional: true,
	Default: false,
	Description: "The virtual disk file zeroing policy when thin_provision is not true. The default is false, which lazily-zeros the disk, speeding up thick-provisioned disk creation time.",
	},
	"disk_sharing": {
	Type: schema.TypeString,
	Optional: true,
	Default: string(types.VirtualDiskSharingSharingNone),
	Description: "The sharing mode of this virtual disk. Can be one of sharingMultiWriter or sharingNone.",
	ValidateFunc: validation.StringInSlice(diskSubresourceSharingAllowedValues, false),
	},
	"thin_provisioned": {
	Type: schema.TypeBool,
	Optional: true,
	Default: true,
	Description: "If true, this disk is thin provisioned, with space for the file being allocated on an as-needed basis.",
	},
	"write_through": {
	Type: schema.TypeBool,
	Optional: true,
	Default: false,
	Description: "If true, writes for this disk are sent directly to the filesystem immediately instead of being buffered.",
	},
	"uuid": {
	Type: schema.TypeString,
	Computed: true,
	Description: "The UUID of the virtual disk.",
	},
	// StorageIOAllocationInfo
	"io_limit": {
	Type: schema.TypeInt,
	Optional: true,
	Default: -1,
	Description: "The upper limit of IOPS that this disk can use.",
	ValidateFunc: validation.IntAtLeast(-1),
	},
	"io_reservation": {
	Type: schema.TypeInt,
	Optional: true,
	Default: 0,
	Description: "The I/O guarantee that this disk has, in IOPS.",
	ValidateFunc: validation.IntAtLeast(0),
	},
	"io_share_level": {
	Type: schema.TypeString,
	Optional: true,
	Default: string(types.SharesLevelNormal),
	Description: "The share allocation level for this disk. Can be one of low, normal, high, or custom.",
	ValidateFunc: validation.StringInSlice(sharesLevelAllowedValues, false),
	},
	"io_share_count": {
	Type: schema.TypeInt,
	Optional: true,
	Default: 0,
	Description: "The share count for this disk when the share level is custom.",
	ValidateFunc: validation.IntAtLeast(0),
	},
	// VirtualDisk
	"size": {
	Type: schema.TypeInt,
	Optional: true,
	Description: "The size of the disk, in GB.",
	ValidateFunc: validation.IntAtLeast(1),
	},
	// Complex terraform-local things
	"label": {
	Type: schema.TypeString,
	Optional: true,
	Description: "A unique label for this disk.",
	ValidateFunc: func(v interface{}, _ string) ([]string, []error) {
	if strings.HasPrefix(v.(string), diskOrphanedPrefix) {
	return nil, []error{fmt.Errorf("disk label %q cannot start with %q", v.(string), diskOrphanedPrefix)}
	}
	return nil, nil
	},
	},
	"unit_number": {
	Type: schema.TypeInt,
	Optional: true,
	Default: 0,
	Description: "The unique device number for this disk. This number determines where on the SCSI bus this device will be attached.",
	ValidateFunc: validation.IntBetween(0, 59),
	},
	"keep_on_remove": {
	Type: schema.TypeBool,
	Optional: true,
	Default: false,
	Description: "Set to true to keep the underlying VMDK file when removing this virtual disk from configuration.",
	},
	"attach": {
	Type: schema.TypeBool,
	Optional: true,
	Default: false,
	ConflictsWith: []string{"datastore_cluster_id"},
	Description: "If this is true, the disk is attached instead of created. Implies keep_on_remove.",
	},
	"storage_policy_id": {
	Type: schema.TypeString,
	Optional: true,
	Computed: true,
	Description: "The ID of the storage policy to assign to the virtual disk in VM.",
	},
	"controller_type": {
	Type: schema.TypeString,
	Default: "scsi",
	Optional: true,
	Description: "The type of controller the disk should be connected to. Must be 'scsi', 'sata', or 'ide'.",
	},
	}
	structure.MergeSchema(s, subresourceSchema())
	return s
	}

How can we proceed? I'm open to writing a PR with the documentation change and sane defaults for network ovf_mapping, but i have no idea about storage mapping.

[slarimore02]

I'm running into this same issue. I'm trying to deploy an OVF from a content library and when I have 1 NIC specified it deploys normally. When I add additional NICs it fails with the same 500 error response from vSphere.

[clementbernet]

Did you specify ovf_mapping for each NIC?
By using ovf_mapping = var.ovf_map[network_interface.key] in the dynamic network_interface mentioned above we are not running the same problem anymore.

We use it with the variable below:

variable "ovf_map" {
  type    = list(string)
  default = ["eth0", "eth1", "eth2", "eth3"]
}

It should work even if you're not using a dynamic for your NIC by just specifying eth0, eth1 anyway.

[tenthirtyam]

Per the VMware vSphere product documentation, which compares the use of VM Templates vs OVF Templates in Content Library:

"During the deployment of an OVF template, only guest OS customization is supported. Hardware customization is not supported."

Reference: The VM Template as a Content Library Item

Hardware guest customization for OVF templates would need to be supported in vSphere to enable this capability.

Regards,
Ryan

Similar to #1441

cc @appilon and @iBrandyJackson for review and closure.

[appilon]

Closing as per @tenthirtyam recommendation

[sofixa]

@appilon @tenthirtyam I don't see the link between hardware customisation and the fact that it's pretty non-obvious how to deploy an OVF template because terraform has optional parameters that are in fact required by VMware ( and rather useless i might say). Having that clearer in the docs and/or having defaults on the terraform side would make for an easier UX.

[sofixa]

So i checked the linked issue and it's an unrelated problem :) Manually mapping nics and disks fixes the 500 from this one. @appilon could you please reopen this one?

[tenthirtyam]

@appilon, we can retest and pick it up in a doc update in the least. @sofixa, feel free to suggest a resolution in a pull request.

Ryan

[appilon]

I'll re-open and have it close once we at least provide a documentation update

[lado936]

Setting ovf_mapping for network_interface is not fixing whole issue, as ovf_mapping parameter is set to force new resource creation, so if i need to add another interface after server was created, it just tries to destroy whole resource and recreate vm again
#1313