packer/image build as part of Azure GenerateResourceandImages.ps fails on private IP

[greadtm]

I have had the process to generate a windows2022 image from the GenerateResourceandImage process for a few years. (part of the git-imagerunners)
The VNET we used in Azure was open and connections to packer to pull down the bits worked fine.

I have a new resource group in Azure that has a defaulting VNET. I have relaxed the service endpoints so there is no keyvault firewall.
I created a new VNET/subnet to look like the one we were using.
I setup the packer.json to specify, the VNET name, VNET subnet name, and build image resource group.

When I run the release to execute the process, I get the following error:

Getting the certificate's URL ...
2024-02-27T20:39:54.8581459Z ==> azure-arm.image: -> Key Vault Name : 'pkrkvztifzt2p81'
2024-02-27T20:39:54.8583086Z ==> azure-arm.image: -> Key Vault Secret Name : 'packerKeyVaultSecret'
2024-02-27T20:39:54.9360793Z ==> azure-arm.image: ERROR: -> Forbidden : Client address is not authorized and caller is not a trusted service.
2024-02-27T20:39:54.9361545Z ==> azure-arm.image: Client address: x.x.x.x from unknown subnet
2024-02-27T20:39:54.9364202Z ==> azure-arm.image: Caller: appid=***;oid=29e1fbed-ab6d-4741-8d3f-48590f5943c0;iss=https://sts.windows.net/***/
2024-02-27T20:39:54.9364737Z ==> azure-arm.image: Vault: pkrkvztifzt2p81;location=eastus2

where x.x.x.x=private IP address of our on prem(in Azure) build agent.

Do we know how to resolve this, and why its looking at the private IP vs public when the VNET is open?