Apple's AR Strategy Is Starting To Come Together

Category: 
Apple's AR Strategy Is Starting To Come Together
June 18, 2018

Apple's AR updates in iOS 12 are paving the way for where headsets could go next.

 

The future of augmented reality headgear and smartglasses is still very much in flux. The long-awaited Magic Leap might emerge this year. Microsoft's Hololenshas hung in a state of enterprise limbo -- one from which it may finally emerge next year. Little smartglasses like ODG's might get better with Qualcomm's next chips, but don't expect miracles.

 

Then there's Apple. As CNET reported in April, Apple is working on a powerful headset capable of both AR and VR. Whether that version is a "what if" prototype or something akin to what Apple may ship in 2 years is anyone's guess. But you don't need to wait until 2020: Apple's plans for virtual magic are playing out in real-time, right now, on the iPads and iPhones that your currently own.

 

Apple showcased its upgraded augmented reality toolkit, ARKit 2, alongside iOS 12at its Worldwide Developers Conference earlier this month. And it already has a surprising number of key upgrades that vastly improves how iOS can handle augmented reality. These bits and pieces, combined, are a roadmap for where AR needs to head if it's to move from nerdy plaything to Fortnite-level mass market adoption.

 

Apple's AR doesn't live on a headset (yet) but according to Apple, that doesn't matter. "We think the big deal right now is we've got it on hundreds of millions of devices, iPhones and iPads," Apple's Greg Joswiak, Vice President of iPhone and iPad product marketing, told CNET. "We think that's an unbelievably great place to start because a lot of us are already carrying iPhones in our pockets."

Playing AR slingshot games at a table with two people, which means holding up a phone/tablet.
James Martin/CNET

 

Multiplayer shared worlds

If a set of layers on top of our world are going to be a part of our future, then everyone needs to be able to see them. Shared AR worlds are a relatively new thing: Google demoed its first multiplayer AR apps a month ago at its own developer conference, and Apple's multiplayer support in ARKit 2 does similar things.

 

My first hands-on experience of multiplayer AR in iOS 12 was really impressive, although holding an iPad upright for a long time can get tiring. Same-room gaming in a real space feels completely fascinating, but this also opens up collaborative projects or persistent virtual objects that many people could visit and interact with. For now, it's Lego kits on tables that blend physical pieces and virtual ones. Think shared augmented-reality site-specific theater pieces of the kind that William Gibson dreamed of years ago. Or the next wave of holograms in classrooms. Or experimental art projects, such as Google's group AR doodle app that's already live.

 

Object detection, with pop-up info

Going back to Google Glass, the future fantasy of magic glasses is that they'll somehow show head-up annotations to things seen in the real world. Google Lens, a part of Google's Android Oreo OS last year and Android P this year, can recognize objects through the camera and automatically search for related information. ARKit 2 can also be used to not just see objects, but to pin information to them. Maybe it's purchase information, or someone's name floating over their head, or the name of a dinosaur, or player stats hovering over athletes at a future sports event. The above demo by a developer shows promise.

 

Face and eye tracking

Eye tracking is coming to VR, allowing better graphics and more ways to control things with simple eye movements -- or even make direct eye contact for shockingly intimate social experiences. In AR, it could be used to control hovering interfaces, change events based on emotions or expressions, or map avatars to do things puppeted by facial expressions.

 

ARKit 2 can track eye movement using the iPhone X's front-facing TrueDepth camera, which will also likely end up on Apple's other iPhones arriving later this year, and maybe Apple's next iPad Pros. The results, based on developer experiments seen on Twitter, are already impressive. This could be a test run to evolve where Apple's future eye-tracking tech goes next. Maybe it'll be in headsets eventually. Or it'll be used to find ways to not just read what we're looking at and make eye-controlled hands-free interfaces, but turn our expressions and emotions into information. Or it could help make a whole new wave of Memoji-like avatar puppets.

Related articles

VRrOOm Wechat