Monday, 17 June, 2024 UTC
Summary
AirPlay Mirroring to Apple Vision Pro with visionOS 2 dramatically expands the headset's capabilities, and new features better integrate your iPad, Mac, keyboard and controllers too.
The first VR headset to ship after Facebook’s acquisition of Oculus in 2014 was the Samsung Gear VR Innovator Edition, using the Note 4 smartphone as its display, processing center, and battery. Gear VR got through multiple generations before proving free wasn't cheap enough.
Later, the HTC Vive powered over PC by SteamVR promised to pair with your phone to keep you connected while in VR. Most people who owned a Vive didn’t know this feature was there, and there are few accounts of it being used successfully by anyone.
Post-iPhonePrevious phone-in-VR efforts are lost to history because neither unholy matchup of Samsung and Facebook-purchased Oculus nor HTC and Valve had enough platform control to keep you connected to your most personal computer while in VR.
In the case of Gear VR and the cardboard boxes shipped by the likes of Nintendo, Google, and the New York Times, you could (and still can!) enjoy a simple head-tracked game like Proton Pulse inside a literal cardboard box – just as you can today on a $3500 Apple Vision Pro.
0:00
/1:14
1×
Playing Call of Duty Warzone in VR via iPhone AirPlay Mirroring.
What’s changed is that 10 years of progress in software and hardware integrated vertically by Apple makes iPhone a Swiss Army Knife to visionOS. With visionOS 2, nearby devices running macOS, iOS, and iPadOS become parallel computing tools to flip out and use when necessary.
0:00
/0:07
1×
In visionOS 2, for example, when you connect one of Apple's keyboards to the headset it'll breakthrough your immersive environment and become another tracked object you can use.
As another example, you can look at the existing Apple Watch which already includes a remote camera that gives wrist-based viewfinding and record buttons for iPhone. The feature lets photographers and videographers time or angle their shots just right, without requiring the iPhone’s screen be directly visible for framing. Add visionOS to the mix and you can go one step further, with a pair of buttons to press on your wrist to activate the phone’s camera for photo or video recording.
Taken altogether, Vision Pro with visionOS 2 becomes a more capable director unlocking Apple’s other devices in slightly new ways. One hand can frame the iPhone camera shot, the wrist buttons are there for controls, and a virtual viewfinder floats wherever you look to show you how your shot looks from any angle you choose.
That’s the difference of a decade and tens of billions of dollars in investment to make this shiny face computer. Apple’s initial visionOS 1 operating system launched in February alongside the headset and proved that, with next-generation resolution and OLED contrast as its backdrop, a VR system could actually stand on its own as an infinite canvas for your media.
visionOS 1 started to tell this story by offering the Apple TV and Apple Immersive experiences as its primary showpieces, with classic gaming emulators suddenly added to Apple storefronts shortly after initial Mac support for mirroring your computer into the headset. visionOS 2 makes things even more interesting by bringing your iPhone and other devices in via AirPlay Mirroring.
Starting with visionOS 2, a docked iPhone charging via magnetic connection can mirror its widgets even when locked. And when unlocked? You can view the iPhone on a gigantic screen, or a tiny screen hovering right above your keyboard — it's your choice.
0:00
/0:08
1×
iPhone mirroring in full VR is far more than a minor improvement here. And it works for iPad too.
Overall, AirPlay Mirroring points to the reason Facebook bought Oculus in 2014 and it is shipping from Apple within the first six months of entering the market in 2024. I've supervised a child playing in Quest 3 from an Apple Vision Pro by casting their view from a web browser via Mac Virtual Display. With iPhone mirroring, I might be able to supervise two Quest 3 headsets this way, assuming that much low latency high bandwidth data streaming across my network doesn't melt my Wi-Fi router. In the other direction, Meta is asking Apple to let Quest be an AirPlay receiver.
The featured image on this post was captured straight from Apple Vision Pro showing five concurrent windows:
- Stardew Valley purchased from Steam running locally on a Mac Mini via Mac Virtual Display.
- RetroArch running on the headset.
- Delta emulator running on the headset.
- ALVR connecting in the background to a local VR-ready PC.
- iPhone standby window mirroring clock and calendar widgets while locked and charging on a dock.
Every iPhone made for the last four years can be mounted for charging this way using MagSafe. A decade ago, you’d have to give up the juice necessary to keep your constant companion going just to enjoy a few minutes of uncomfortable VR in which you can no longer use the phone as a phone.
Roughly a decade on from those false starts and, rather than requiring a PC or using a phone, Apple chose to ship its first VR headset with a phone-sized battery. David Heaney, in his definitive hardware review, rightly criticized the dangling cord that battery forces on Vision Pro buyers. With visionOS 2 beta running on my headset, I understand why Apple launched the device without widgets for a clock or calendar. That's because, to visionOS, iPhone is just another window taking you away from Bora Bora.
0:00
/0:15
1×
Clip from Apple showing how the upcoming Ultrawide mode on Mac Virtual Display should work.
visionOS recognizes that, in trusting your sense of sight to Apple, everything in your home is of little use to use in VR if you can't actually use it at least as well as without the headset. That means, for example, forthcoming Mac Virtual Display improvements will give you a high-resolution ultrawide display wherever you are.
Even the iPhone that built Apple's spaceship headquarters is little more than a widget to visionOS informing you of the happenings outside virtual reality. You can set up the visionOS widgets, for instance, to quickly tell you the air quality outside the house and temperature outside your climate-controlled environment. Now, you can keep an eye on that DoorDash order on your iPhone without taking the headset off just like Wade Watts accomplished as a Gunter in Ernest Cline's original Ready Player One novel.
And when it comes to gamepads, tilting the analog stick on a gamepad in visionOS 2 now navigates system menus. Having even a single Joy-Con on hand hanging from a wrist-strap gives you the basic inputs for many platformer games while simultaneously navigating the OS more easily. You can rearrange apps more quickly, for instance, by using the stick to move through the app pages.
visionOS 2 Officially Announced For Apple Vision Pro
visionOS 2 introduces a number of new features and improvements across the operating system.
UploadVRDavid Heaney
There's much more to look at in visionOS 2 including much-needed improvements to hand tracking, and we'll keep covering Apple's spatial computing operating system as it continues to evolve.
Ethics Disclosure: Ian is married to an Apple employee.