The Vision Pro has brought augmented and virtual reality to mainstream attention once again, and Apple has already started using its wide product ecosystem to make the headset seem more appealing. Case in point: the company is betting that spatial video will become everyone’s favorite way of reliving past memories. So in this article, let’s unpack what spatial video is, how it works, and where you can experience it for yourself.
What is spatial video?
Spatial video is the latest term for 3D video that has become popular thanks to Apple’s Vision Pro headset. Compared to a typical two-dimensional video, spatial videos can only be viewed with a VR headset since they deliver different images to each eye. You see, we view the world with two eyes spaced slightly apart. Our brain combines these views to judge depth and how far away objects are.
Think back to watching 3D movies with the blue and red spectacles you got at the cinema. These glasses polarize or block certain kinds of light, allowing different images to reach each eye. This tricks our brains into perceiving depth even where none really exists. Spatial video is an extension of this same principle, but using a headset instead of two projectors. It has also become easier than ever for anyone to record and play back their own 3D content. More on how to record spatial video in the next section.
Similar to how we need both of our eyes to perceive our surroundings, a spatial video needs to record from two cameras simultaneously. This is not a novel concept as true 3D movies have used a dual-camera system during the production process for decades. However, this kind of “stereoscopic” capture was relatively inaccessible to ordinary people like you and me until recently.
How does spatial video work?
The term “spatial” may confuse some, but you cannot walk around in 3D space or interact with the video in any way. It’s also different from 360-degree video as it does not envelop your surroundings.
Spatial videos do offer a more realistic and immersive experience, but it’s still a flat projection inside a VR headset. Moreover, the effect can range from subtle to impressive. As with 3D video from years prior, the depth effect is most pronounced with near-by objects rather than distant ones. This means you’re better off recording intimate moments with loved ones in this format. Landscapes and city skylines won’t benefit much from the technology.
The 3D effect of spatial videos will seem subtle in many cases, despite the optimistic name.
In late 2023, Apple announced that owners of the iPhone 15 Pro and 15 Pro Max would be able to record spatial video without any additional hardware. Of course, an iPhone display cannot showcase the depth data captured in a spatial video. But the good news is that you can record spatial videos today and have them ready for any upcoming VR/AR headset you may buy in the future.
The iPhone 15 Pro models can only record spatial video in landscape mode because Apple cleverly uses the primary and ultrawide cameras to capture depth information. The slight gap between these two sensors replicates how our eyes view the world.
Does Android support spatial video? What is MV-HEVC?
Robert Triggs / Android Authority
Currently, no Android phones are capable of recording spatial video. Additionally, Apple has developed a new storage-efficient way to record spatial video called multiview High Efficiency Video Coding or MV-HEVC. It’s an extension of the HEVC codec that most smartphones already use to record standard video. However, it’s unclear if Android smartphones will adopt MV-HEVC either.
Instead of storing two different video feeds, MV-HEVC only records the difference between the two cameras. This way, 2D video players like your iPhone or Android smartphone can ignore the depth information while reading the file. On the other hand, 3D video apps made for a VR headset can calculate two different views for each eye using the base 2D image and accompanying depth data.
How do I watch a spatial video?
The best way to watch spatial videos is with a VR headset like the Apple Vision Pro or Meta Quest 3. The former offers the most seamless way to watch spatial videos at the moment as it can sync videos with your iPhone wirelessly via iCloud.
On the Meta Quest 3 or other headsets with spatial video support, you’ll need to manually upload videos to the headset. For example, Meta’s official smartphone companion app allows you to upload spatial videos. You’ll then need to find the video in the headset’s Files app and play it back from there.
Spatial videos undoubtedly let you experience memories in a whole new light, but it’s still not enough to convince many first-time headset buyers. Still, even if you’re not going to spend $3500 on the Vision Pro today, I’d recommend recording spatial videos if you own an iPhone 15 Pro. There’s no downside, aside from slightly larger file sizes, and you’ll reap the benefits someday in the future.
