Guide for using Apple Vision Pro as HDMI display

HDMI on Apple Vision Pro.md

Displaying HDMI sources on Apple Vision Pro

While it's possible to stream most content to Apple Vision Pro directly over the internet, having the ability to use Apple Vision Pro as an HDMI display can still be useful.

Since Apple Vision Pro does not support connecting to an HDMI input directly or using an HDMI capture card, we have to be a little creative to make this work. NDI provides the ability to stream HDMI content over a local network with really low latency, and it works great with Apple Vision Pro.

This page shows the setup I’m using.

What’s needed

• HDMI NDI Encoder
  • Personally, I recommend the Kiloview N40, as it supports streaming up to 4K60 while being fan-less, and can use USB-C as a power input.
  • Kiloview N60 comes with more features, but it’s larger and has a fan.
• Vxio app
  • I made this one, you can also use other NDI monitor apps as well.
  • Separately, Finn made Castaway that makes it possible to use a USB Capture Card + Mac/iPad instead of an NDI encoder. It's a more cost effective option for most people I imagine.

Things to consider

• To ensure high video quality, an NDI stream typically consumes significantly more bandwidth than a typical H.264/H.265 stream. Before purchasing the encoder, make sure your network can support data transfers at that speed.
  • For 4K60 stream, the bandwidth required is typically around 250~300mbps. If you're having issue to stream smoothly, try go to the encoder's settings page and reduce the streaming quality.
  • You can test this with some free software NDI encoders.
• Since this setup requires encode and decode the video content, it is not latency free. From my experience, I typically see 3~4 frames (~50ms, same as moonlight based on my measurement) of delay from the content source.

Setup

Setting up an NDI encoder is pretty straightforward. Just connect the device to your local network via Ethernet, supply power, and connect the HDMI input to the encoder.

If the visionOS shows the Local Network permission dialog, and after granting the permission the app still shows searching, you may need to force close the app, and reopen it.

After that, you should be able to see the encoder show up as a source in the Vxio app. Select the source, and you'll get your HDMI display 🎉

Misc

Adjusting audio queue size

Depending on the NDI encoder, the default audio queue size might be too small for some devices. In that situation, try increasing the audio queue size so that the audio no longer experiences abrupt stops between samples.

Disable Multi-TCP

If your Wi-Fi channel is congested, you can try force N40 to use UDP instead of Multi-TCP. In my experience, UDP performs a lot better in those network condition. You can do so from the N40's configuration UI, select NDI Connection -> Multi-TCP Disallowed.

Portable setup

I was able to make this setup portable by getting one of those mini router (GL.iNet Beryl AX) with a LAN port.

#AppleVisionPro #visionOS

Can't watch 3D? SBS?

You can I can see option when you click the source and hold it like on the screenshot here. I have never tried, but I can see there.

Oh, thanks. I forgot about that feature. I tried "retro", it's cool.

I'll share my experience. I created a virtual 8K monitor on a Windows PC (I think 4K is enough). MPC-BE with MadVR. 2020 color, HDR. Vixo 2020 RealityKit. HDR works, the colors are correct, 90Hz, no artifacts are visible, NDI is pleasing. I miss the IMAX cinema mode and gamma settings; I would like to make some movies a little brighter.

Is anyone having thermal issues with a M5 Vision Pro? I'm not 100% on the cause but playing 4K50fps HDR content @~250mbit (N60 standard settings) seems to work fine for 10-15 minutes. From 15 minutes onwards I'm getting stuttering. If I switch to something like Sky Sports News with a scroll bar it's notably jumpy not smooth.
Is there anyway to get the app or something else to display if it's dropping frames due to throttling to confirm if this is the problem? Thanks!

Is anyone having thermal issues with a M5 Vision Pro? I'm not 100% on the cause but playing 4K50fps HDR content @~250mbit (N60 standard settings) seems to work fine for 10-15 minutes. From 15 minutes onwards I'm getting stuttering. If I switch to something like Sky Sports News with a scroll bar it's notably jumpy not smooth. Is there anyway to get the app or something else to display if it's dropping frames due to throttling to confirm if this is the problem? Thanks!

Generally is a thermal issue of Apple Vision Pro, if the whole device is hot it will throttle to cool it down and one way of cooling down is to dim displays which kills HDR effect. But is the same effect Apple doing with all devices, eg. iPhones dim a screen as well when they run hot.

Vxio generally using a lot of cpu, around 200% cpu, and AVP as whole using around 400% cpu when this is working.
The only way is too limit what AVP doing, just kill all other apps, widgets, etc. than is generally stable image.
And to be frank is great they put M5 chip but they don't have enough cooling to keep all system stable under load.

Update and some information I hope others might find useful when trying to achieve smooth motion playback using a Kiloview N60.

1. If you're using the GL.iNet Beryl AX login go the admin area go to System > Overview then click the fan icon next to CPU Average Load. Change the default of 75degrees up to 90. This seems to have stopped the intermittent jitter I was getting which I was blaming on the Vision Pro thermal throttling but I now suspect was interference caused by the access points fan switching on and off. I've never seem temps exceed 80 degrees even in a warm room so I don't think it needs the fan.

2. I'm based in the UK so mostly play 50fps and 25fps content. This is a problem for the Vision Pro. From the specs at https://www.apple.com/uk/apple-vision-pro/specs/ it states:
   "Supported refresh rates: 90Hz, 96Hz, 100Hz, 120Hz. Supports playback multiples of 24 fps and 30 fps for judder‑free video"

It seems that AVF is a clock-scheduled pipeline. It's built to play timed media, so it takes each frame's presentation timestamp and schedules it against a timebase. This probably works fine if it has enough frames to schedule the inevitable repeat due to the frame mismatch but I suspect due to the extremely small (if any) buffer with NDI it can't do this correctly which results in a lot of judder. This is potentially fine for content in a multiple of 24 or 30fps (I haven't checked) but looks awful on other refresh rates.

RealityKit is a render-loop, latest-frame pipeline. I assume you push NDI frames into a texture, and RealityKit just draws whatever the current texture is on every compositor vsync, at the display's native rate. There's no rigid "hold this frame for exactly X ms" schedule to violate, so arrival jitter gets absorbed — the freshest available frame is simply shown each refresh. The non-integer cadence is still technically there, but spread continuously rather than as discrete drop/dup events, so it's imperceptible. It's essentially the way a game engine presents: vsync-locked, triple-buffered feel, no scheduler second-guessing the timing.

Switching my 4k, 50FPS HDR (HLG) content to RealityKit eliminates the judder and gets me nice smooth playback.

This leaves me with 1 final problem I'm struggling to solve and would be grateful if anyone could offer advice. Some content is 4K SDR (Rec 709) @50FPS. I can't get RealityKit to play this correctly (I just get a black / empty window). If I select RealityKit then any of the HDR modes e.g. PQ / HLG it displays a lovely smooth feed, but with incorrect colours. Is there anyway to play a Rec709 feed with RealityKit? Thanks again!