By default, "custom boot objects" (created using kmutil create)
can't be used in macOS Virtualization framework guest VMs. Here I show
how to get around this Apple design flaw. Note that Virtualization
framework macOS guests can only be created on Apple Silicon Macs.
To proceed, you'll need to work through my Running Third Party Kernel
Extensions on Virtualization Framework macOS Guest
VMs. Many
of the steps you need to perform here come from that document: You
need to make all the changes it describes to "stage 0", "stage 1" and
"stage 2" iBoot modules. But you won't (necessarily) need to patch the
kernel cache (kernelcache).
Custom boot objects are useful if you need to add kernel extensions to, or remove them from, the standard set of kernel extensions built into the kernel cache (the boot kernel collection). You'd also use them if you were building and installing your own kernel from source, as per Building XNU for macOS 11.2 (Intel + Apple Silicon). Of course this is much more difficult, and I won't be covering it here.
The only requirement for building a custom boot object (as of macOS 13) it that you install (in your guest VM) a Kernel Debug Kit for that version of macOS. Apple doesn't provide KDKs for all versions of macOS, so you'll need to install a version of macOS, in your guest VM, for which a KDK is available.
The standard procedure for custom boot objects is to create one using
kmutil create, then boot into the recovery partition and run kmutil configure-boot to "install it". With the correct KDK, kmutil create
works even if you haven't patched anything in your guest VM. And
kmutil configure-boot -v /Volumes/Macintosh\ HD -c custom-kernelcache appears to work. But the VM always refuses to boot
from the custom kernelcache.
Patching the three iBoot modules allows kmutil configure-boot to
work properly for most purposes.
But if you want to load third party kernel extensions, you need to create the custom kernelcache in a particular way, described below. Then follow my Running Third Party Kernel Extensions document's instructions to patch it and copy it to its final destination.
Here's a way to create a custom boot object, in *.im4p format, that
adds the AppleARMWatchdogTimer kext to the guest VM's boot kernel
collection. It assumes the guest VM is running macOS 14.4.
kmutil create -a arm64e -z -V release -n boot -B kernelcache.im4p.org --img4-encode -k /Library/Developer/KDKs/KDK_14.4_23E214.kdk/System/Library/Kernels/kernel.release.vmapple -x -b com.apple.driver.AppleARMWatchdogTimer $(kmutil inspect -V release --no-header | awk '{print " -b "$1; }')
Make sure you've also found (and copied) the "orginal" kernelcache
file, so that you can generate kernelcache.im4m.org from it:
img4tool -e -m kernelcache.im4m.org kernelcache.org
Then use the "new" kernelcache.im4p.org file in the following
command, and proceed from there.
img4tool -e -o kernelcache.bin.org kernelcache.im4p.org
As I say
here,
the failures in _validate_acm_context(), which need to be patched in
the kernel cache's AppleVPBootPolicy kext, may be caused by a timing
problem. This may be connected to the absence of the
AppleARMWatchdogTimer kext from the guest VM's kernel
cache. Here
and
here
I talked about trying (and failing) to add this kext to the guest VM's
kernel cache.
Well, I was doing it wrong. The method I describe here works just
fine. But my custom kernel (with AppleARMWatchdogTimer.kext added)
still isn't enough, unpatched, to allow loading of third party kexts.
The AppleARMWatchdogTimer kext loads automatically on bare metal. It
implements an IOWatchdog service, whose client (in userspace) is
watchdogd. In userspace an IOWatchdogUserClient object is
implemented using a hardware wdt device. (See the results of ioreg -p IOService -n "IOWatchdogUserClient" -w 0 -r -t and ioreg -p IODeviceTree -n "wdt" -w 0 -r -t.)
Unlike on bare metal, the AppleARMWatchdogTimer kext doesn't load
automatically on a macOS guest VM, even when it's present in the
kernel cache. Both the service client and the device are absent from
userspace. watchdogd dies when it can't find the IOWatchdog
service:
watchdogd: 2977082625: failed to discover watchdog KEXT service
watchdogd: 2977098166: detected virtual machine environment and no watchdog KEXT found, exiting...
And, when trying to load a third party kext, kcgend complains about
not being able to find the IOWatchdog service:
kcgend: Could not find IOWatchdog
It's possible to load the AppleARMWatchdogTimer kext "by hand",
using kmutil load -b com.apple.driver.AppleARMWatchdogTimer. Then
you can explicitly load watchdogd using sudo launchctl -p system/com.apple.watchdogd. But watchdogd doesn't stay loaded. It
logs the "failed to discover watchdog KEXT service" message and dies.
It seems that Apple's Virtualization framework doesn't emulate the
wdt hardware. It's not in the guest VM's devicetree file, though
it is in my Macmini9,1's devicetree file. I suspect Apple won't be
able to fix the _validate_acm_context() errors without implementing
this device. I can't prove it, but I think the circumstantial evidence
is pretty compelling.
The devicetree.img4 file exists in the file system. It can be found
the same way as the iBoot.img4 file. Once you have it, run the
following commands to extract its binary:
img4tool -e -p devicetree.im4p devicetree.img4
img4tool -e -o devicetree.bin devicetree.im4p
Then run the following command to see its contents in human-readable format:
ipsw dtree devicetree.bin
ipsw is available via
Homebrew. To install it do brew install ipsw.
A devicetree describes the hardware components of a particular computer. It's needed on operating systems, like macOS on Apple Silicon, that don't have ways to discover the hardware they're running on.
awesome. thanks.