"LoadImage error 800000000000000E (Not Found)" on VirtualBox starting from release 1.4

[ntqbit]

Operating system

Windows 10 x64 22H2 (19045.2006)

Issue description

Hello. First of all, thank you for the great project you made!

EfiGuard outputs an error "LoadImage error 800000000000000E (Not Found)" (see the attachments) during boot.
The issue happens only with the latest at the moment release 1.4, while with 1.3 everything works perfectly.
Maybe 1.4 introduced a bug?

VirtualBox version: 7.0.12

Steps to reproduce

1. mountvol X: /S
2. X:, cd EFI/Boot
3. copy ...\EfiGuard\EFI\Boot\Loader.efi .
4. copy ...\EfiGuard\EFI\Boot\EfiGuardDxe.efi .
5. Create EFI boot entry. Tried with efibootmgr: efibootmgr -c -d /dev/sda -L "EfiGuard" -l \\EFI\\Boot\\Loader.efi,
   and EasyUefi.
6. Reboot, observe error.

Logs

$ EfiDSEDix.exe -i

SystemBootEnvironmentInformation:
        - BootIdentifier: {f0b6dc23-93ac-11ee-9f8c-948dbd404809}
        - FirmwareType: UEFI
        - BootFlags: 0x0

SystemModuleInformation:
        - Kernel: ntoskrnl.exe (\SystemRoot\system32\ntoskrnl.exe)

SystemCodeIntegrityInformation:
        - IntegrityOptions: 0x0001
           0x0001: CODEINTEGRITY_OPTION_ENABLED

SystemKernelDebuggerInformation:
        - KernelDebuggerEnabled: 0
        - KernelDebuggerNotPresent: 1

SystemKernelDebuggerInformationEx:
        - DebuggerAllowed: 0
        - DebuggerEnabled: 0
        - DebuggerPresent: 0

SharedUserData->KdDebuggerEnabled: 0x00

SystemKernelDebuggerFlags: 0x00

SystemCodeIntegrityPolicyInformation:
        - Options: 0x40000000
        - HVCIOptions: 0x0000

SystemIsolatedUserModeInformation:
        - SecureKernelRunning: 0
        - HvciEnabled: 0
        - HvciStrictMode: 0
        - DebugEnabled: 0
        - FirmwarePageProtection: 0
        - EncryptionKeyAvailable: 0
        - TrustletRunning: 0
        - HvciDisableAllowed: 0

Attachments

Error:

EFI entry:

[Mattiwatti]

Hi, sorry for the delay in response.

I thought I knew what the cause of this issue was after reading your post, but this turned out to be a different bug (in VirtualBox, not in EfiGuard, but triggered by a change I was testing in response to Microsoft's new UEFI image signing requirements). In short: I remembered VirtualBox failing to load EFI images with IMAGE_DLLCHARACTERISTICS_NX_COMPAT set as per the new policy, but I just double checked this, and it seems I discovered this in time before the v1.4 release, because neither EFI image has this flag set in the v1.4 release ZIP.

So the above is almost definitely unrelated, I'm just mentioning it for completeness' sake and because it's the closest I've come to (unintentionally) reproducing this.

---

Using VirtualBox v7.0.12, EfiGuard v1.4 and a Windows 10/11 VM (builds 14393 and 22621 - I don't have a 19045 VM on hand currently, but I can retry with that if you expect it will make a difference), I can not reproduce this using your steps above. I also don't see anything unusual in your screenshot that would lead to this error.

Can you attach the .vbox configuration file for your VM, as well as any .nvram file (if one exists) in the same directory? That way I can hopefully recreate your exact VM setup more closely in order to reproduce this. (I'd love it even more if you could simply upload the entire VM including the VDI/VMDK somewhere for me, but I understand that is probably an unreasonable request due to privacy and/or bandwidth related reasons...)

[ntqbit]

@Mattiwatti
Hi. I decided to upload the entire VM with the reproducible problem.

I created a new VM, installed the same windows operating system from the same ISO I used to install the system where I first encountered the problem, installed EfiGuard on it, and I could NOT reproduce the problem.
Everything worked as it should.

I still have the VM where I encountered the problem. After I installed the 1.4 efiguard version instead of previously installed 1.3 I confirm that the issue is still present on that VM, and the system cannot boot.

Since I could not find any distinction between a fresh windows installation and the system where I encounter the problem I decided to make a full clone of the VM and upload it instead.
However, when I made a full clone of the VM with the problem, the issue was partially gone. "Partially" because some errors were still there, but efiguard now could successfully boot the system. No changes were done after clone.
I still wonder how just making a full clone of the VM may influence boot process.

Unfortunately, I can't upload the original VM with the un-bootable efiguard 1.4, but I upload the cloned VM with some errors present, but bootable. Maybe the origin of those errors will give a hint about the source of the problem.

Here are some boot log screenshots from different VMs / efiguard versions:
EfiGuard 1.3 on any system with any configuration (boots successfully):

EfiGuard 1.4 on a fresh windows 19045 installation (boots successfully):

EfiGuard 1.4 on the original VM, not a clone (boot fails):

EfiGuard 1.4 on the full clone of the original VM without any other changes, the VM I upload (boots successfully):

And the uploaded VM (15 GB uncompressed): mega

[Mattiwatti]

Thank you so much for taking the effort to upload the entire VM for me. This made all the difference, because without it I would never have been able to reproduce this (even though I've personally seen this happen in VirtualBox - but just once and never again). The fact that the error is not fatal in the cloned VM is not a problem since the underlying issue is the same.

This took me a very long time to debug, because I assumed the issue must naturally be a regression in EfiGuard v1.4, because after all, as you said (and I reproduced), the issue does not occur when using the v1.3 release instead. Specifically, it is Loader.efi that is responsible for this. In theory, updating only EfiGuardDxe.efi to v1.4 should work, though I haven't bothered to verify this.

Apart from not making much logical sense (there were no significant changes to the loader in v1.4 that would have caused something like this), I also manually verified the commits from v1.3 to v1.4 using git bisect and found that there was not a single 'good' commit in this entire range. In other words: if I recompile EfiGuard v1.3 today, the bug still occurs when using the resulting 'new' v1.3 in your VM!

At this point I think it's reasonable to say that the bug can't possibly be in EfiGuard itself, and has to be in some other component that is part of the build process. In theory I suppose this could have been a very strange compiler bug, but I have enough experience with EDK2 by now to know better than this. And indeed, not long after this I managed to build a working binary (both v1.3 and v1.4) simply by switching to an older version of the EDK2 build environment.

So to clarify: the real reason that you (and now I as well) are seeing this bug in the v1.4 release is because EfiGuard v1.4 was compiled with a different (apparently worse...) version of EDK2 than EfiGuard v1.3 was.

---

Please confirm that attached Loader.efi fixes the issue in your VM as it does for me:
Loader.efi-b8d787c3.zip

This is the loader from the current latest EfiGuard commit, compiled with EDK2 edk2-stable202208.

I've attached a version of EfiGuardDxe.efi compiled using this same EDK2 version below as well just in case, but note that I don't believe there are any issues in EfiGuardDxe.efi v1.4 related to this regression in EDK2, so please try updating only the loader first and let me know if your experience confirms this.
EfiGuardDxe.efi-b8d787c3.zip

---

Misc. notes:

I haven't investigated the underlying issue in EDK2 in much detail, other than taking a quick peek at the two v1.3 binaries and concluding that it's probably not something simple, as there were no obvious issues in the 'broken' v1.3 as far as I could tell, other than exhibiting this bug at runtime of course. I may still end up hunting down the EDK2 regression with git bisect, but probably not anytime soon. This would involve bisecting a range of somewhere in the hundreds or maybe thousands of commits, and I'm not that fond of EDK2 to be perfectly honest.

Another interesting thing to note is that the issue does not occur in the development version of VirtualBox compiled from trunk! I had to install VirtualBox 7.0.14 in order to reproduce this. After that, if I then upgrade VirtualBox to the current trunk revision, the same VM no longer exhibits the problem. So whatever the issue is precisely, I think it's reasonable to assume it has to be related to either VirtualBox or EDK2, or maybe the combination of the two. (VirtualBox uses EDK2 OVMF for its EFI firmware).

[ntqbit]

I have updated EfiGuardDxe.efi to the version from the 1.4 release, and the Loader.efi you provided in the original VM that previously failed to boot with a fatal error, and I confirm that there are no errors anymore and the system boots successfully.

I'm pleased to hear that the approximate origin of the problem is found.
I'm not closing the issue as the problem is not yet found with the new version of EDK2.
If you believe the issue is resolved, feel free to close it.

Thank you for your invaluable work!

[Mattiwatti]

Indeed, the issue is still not resolved in the current edk2 master branch. So while this is not technically a bug in EfiGuard, I will leave the issue open in case it helps anyone else running into this.

I do have some news to share re: this issue: I finally found the time to identify the guilty commit in edk2 using git bisect, and I can now say with certainty that 502c01c5028038e4e6b4512e9c66be0ec4d11492 is the commit that introduced this issue (more context: TianoCore bug #3937 / EDK2 PR #3572).

Unfortunately, merely identifying the bad commit in this case still doesn't really provide me with any clues as to how or why this commit is causing EfiGuard to be miscompiled, as all this commit does is add a new value to EFI_MEMORY_TYPE in UefiMultiPhase.h, without (intentionally) changing any C code.

So the next step would be to do a comparison of the code generated for Loader.efi by the compiler using edk2 at the last known good commit, 9b64811, vs the code generated when using edk2 with the same compiler at the first known "bad" commit, 502c01c. To be continued...

---

For anyone reading this who simply wants a quick and reliable workaround for this issue: compile EfiGuard using EDK2 tag edk2-stable202208. This is the most recent stable tag of EDK2 that does not include the problematic commit.