A Pokémon Go! player hunts Pokémon in Tokyo, Japan. Photograph: Aflo/Rex/Shutterstock
While Dorothy, blue-skirted and pigtailed, clutching a wicker basket and a bewildered dog under her arm, surveys the weird flowers and pygmy huts around her, she’s sure of just one thing: she’s not in Kansas any more. L Frank Baum’s character was, it turns out, born slightly too early. In 1901, a year after the publication of The Wonderful Wizard of Oz, Baum wrote The Master Key, a novel credited with the invention of augmented reality via a pair of imagined spectacles that could map information on to whatever or whomever its wearer looked at. Had Dorothy owned a pair, she might have learned that she’d been whisked to Oz, or that her new friend the Tin Man was in need of a heart, even, perhaps, that a wicked witch is burned not by fire, but by water.
It was almost a century until Baum’s invention gained a label. In a 1992 research paper, the Boeing engineers Thomas Caudell and David Mizell described a pair of “see-thru virtual reality goggles”, a device that would enhance the vision of factory workers with the complicated task of piecing together a jumbo jet’s nests of internal wiring with dynamically changing labels and information. Caudell termed this principle of annotating the seen world “augmented reality”, thereby formalising for Silicon Valley’s mavens and investors a fresh and unplundered field of technological opportunity, one that would eventually lead to the invention of Google Glass, a pair of information-spewing spectacles built, unbelievably, to Baum’s century-old definition.
Who wouldn’t like a natural history programme where the dinosaurs roam across your room?
If the breathless press releases are to be believed, 2016 has been the year of virtual reality – a discarded technology that, thanks to the revolution in low-cost components heralded by smartphones, is currently enjoying a lavish revival. Sony, Facebook and HTC have all launched expensive, miraculous visors that transport us into fabricated realities, fooling the brain into believing that, like Dorothy, we’ve been bodily transported to another place. At its best, VR allows us to visit places too remote, too dangerous or too expensive to otherwise reach. Documentarians are using the nascent medium to allow us to experience life through the eyes of another. But VR technology also has numerous drawbacks. As well as the cost (just 3% of Britons are expected to invest in a serious headset this year, the cheapest of which costs £350, even without the PlayStation needed to run it) there’s also the sense of vulnerability that wearers experience while sitting blind in front of the television, as well as the perils of bumping into furniture. As one Microsoft spokesperson put it, VR users are “best advised to stay seated or keep still to avoid collisions”.
AR, while not having quite the same transporting effects, gets around this issue by layering its simulated elements over the real world. Through a pair of AR glasses, or a smartphone’s camera, you are able to see both the elbow-y chair and teetering lamp in front of you, and the data that overlays it. Microsoft’s apparent scepticism about VR is understandable; the company has instead focused millions of R&D dollars into AR (or, as the company prefers to term it, “mixed reality”) via its HoloLens, a see-through PC-in-a-visor that launches in November for the precise and gargantuan cost of £2,719.
Star Chart lets you annotate the night sky using GPS positioning on your smartphone. Photograph: Escape Velocity Ltd
Microsoft believes that HoloLens is the next evolution for AR, technology that we have seen for years whenever, for example, a weather reporter gestures toward a virtual map behind them in a television studio, or a football pundit offers an assessment of the starting 11 in a football match, as digital versions of the players magically appear in front of them on the desk. In HoloLens the data isn’t merely layered on top of the world you’re looking at. It’s conjured and anchored into the environment around you. In this way, you can play Minecraft on the dining room table, walking around the room in order to get a different perspective on your creation. For Microsoft, it’s a way to pull the software on which we rely from our smartphone screens into the room, a holographic conjuring act that, as the spokesperson puts it, enables us to “interact with them in the same way we interact with physical objects”.
For Caudell, who coined the term AR, the technology, which he believed would be adopted across industries such as architecture, automotive design, building construction and even medicine never quite took off in the ways he expected. “There were technical and cultural issues at that time that kept it in the research lab,” he says. “In that sense it was probably ahead of its time.” The pervasiveness and power of smartphones has, however, “drastically opened up the market,” he says.
Meanwhile, the success of AR games such as Pokémon Go!, which was downloaded more than 100 million times in its first month, reportedly earning $10m per day at the height of its popularity, has attracted widespread attention and investment. “It’s hard to say whether all the attention is warranted,” says Caudell. “But the potential for benefit is huge.”
At the BBC, which has been using AR elements in its programming for more than a decade, the next generation of technology is opening up extraordinary possibilities. “It’s becoming possible to break down all the aspects of a traditional programme – like a presenter, an overlay and even the sound – and treat them as separate objects,” explains Graham Thomas, from the corporation’s R&D department. This could enable, Thomas claims, new kinds of TV programming. “It might be as simple as swapping out the usual presenter of a show for one who uses sign language, or automatically editing a news report to fit the time you have available. Or it might allow you to create AR experiences by reusing the objects from a TV show in a novel way. Who wouldn’t like a natural history programme where the dinosaurs roam across your coffee table?”
This kind of speculative pitching may sell AR to venture capitalists (according to Digi-Capital, investment in VR and AR in the 12 months from March 2015 to March 2016 exceeded $1.7bn) but consumers soon tire of gimmickry. “As with any emerging technology, the experiences need to be compelling enough once the novelty has worn off,” says Thomas.
Besides, simplicity, rather than ambition, is where AR has proved itself. Snapchat, the social media platform beloved of tweens, where your messages self-destruct after a few seconds, has produced perhaps the most widely used AR app in the world via its novelty camera filters which, for example, add dog ears and a lolling tongue in real time to the camera feed of one’s face.
For Alex Fleetwood, the designer of Beasts of Balance, a forthcoming boardgame that uses AR to complement the play experience, Snapchat’s AR filters show how the technology can remain compelling, at scale. “Adding fun things to video has become a natural extension of putting emoji at the end of your DMs,” he says.
While some point to HoloLens and MagicLeap (another AR headset that accounts for the majority of that $1.7bn in investment) as the future of AR, for Fleetwood Snapchat’s lighter, less invasive approach is a better vision. “I believe in AR much more than VR, which in my view really is a bubble. But we need to get away from the idea that AR means wearing a computer on your face. Smartphones have already intervened in our social dynamics in ways we’re still coming to terms with. I think faces are going to be a big no-go area for most people.”
For AR’s architects, the possibilities of further democratising information, and placing relevant facts and directions on to the real world, may be enough to convince us to move technology from our palms to our faces. “I consider accurate, readily available information a human right,” says Caudell. “Maybe AR technology will help spread the information riches to everyone. But, yes, as with any invasive technology, I remain concerned about the unintended consequences.” As with so many technological breakthroughs and refinements in the past decade, humanity’s engineers are realising science fiction’s blueprints, without reconciling them with science fiction’s accompanying warnings.
In Baum’s The Master Key, which was subtitled with a phrase that offers a rather delicious description of Silicon Valley – An Electrical Fairy Tale, Founded Upon the Mysteries of Electricity and the Optimism of Its Devotees – the protagonist is given the magical spectacles by a demon. After a fortnight’s adventuring, he concludes that neither he nor the world is ready for the specs. On the third week, he returns the invention until, he says, that time when humankind knows how to use them.
“There are competing notions of what AR is going to be,” says Fleetwood. “There are the people who think we’re going to be living in some mass consensual Neal Stephenson novel. I’m very against that. I hope we’ll see those companies that put design and empathy at the heart of what they do with AR winning out.”
From star-gazing to bone-labelling... 10 apps that enhance reality
Quiver lets characters from children’s colouring pages come to life.
This crowdfunded adaptor attaches to your smartphone, allowing you to instantly print out 2x3in photos. Also when you take a picture, the Prynt app automatically records a digital video and creates a link to that photo on your phone.
This CBBC programme, broadcast between 2004 and 2010, provided an early large-scale glimpse into the potential of AR. Children could create digital creatures on a website that were then pitted against each other in a TV studio, while the audience watched via a graphics system that mixed the creatures into the real world. There is talk at the BBC of a similar programme returning, one that potentially allows children to pit their creatures against each other on the kitchen table or in the playground.
Most Lego stores now feature widescreen TV on the wall, relaying a continual live feed of the view directly in front of the screen. If you step into the camera’s field with a box of Lego, a virtual, animated representation of the kit inside the box will come to life in front you.
This simple app lets you see what a tattoo will look like on your body by superimposing the image on to the camera feed from your smartphone. The app comes with a range of tattoo designs that can be edited and shared with friends. It’s also possible to upload your own designs to see how well they suit you.
5. Star Chart
An elegant and hugely popular AR app for smartphones that annotates the night sky. Simply angle your phone toward the heavens and, by using GPS positioning, the app will label the stars, moons and planets above you. A quicker and potentially more effective way of learning about our galaxy than any text book.
In the past year, the social media platform, whose videos are reportedly watched more than 6bn times a day, subtly shifted its focus on to AR through a selection of lenses. These lenses modify the phone’s camera feed in humorous ways, swapping people’s faces, “embiggening” their eyes, allowing them, for example, to puke rainbows. Snapchat became the first company to make a profit from AR on a large scale by selling and allowing companies to sponsor lenses.
7. Google Translate
Google’s revelatory translation technology, underpinned by computer learning to constantly improve its interpretation, can now be used to annotate foreign text on the fly, using your phone’s camera feed. While the technology isn’t great at handling reams of text, particularly in non-roman scripts, it can prove invaluable at handling signage.
Perhaps the closest to Tom Caudell’s original vision [see main story] for AR, ARnatomy is able to identify and digitally label bones and muscles for medical students.
It uses the camera to identify replicas of human bones, and when the user places a bone in front of the camera, it identifies which bone it is and adds visuals pointing to the parts of the bone, such as where muscles attach.
SmartSpecs enhance the visual appearance of everyday objects to augment vision for partially sighted people. By a combination of a 3D camera and Android-powered software, the specs highlight the edges and features of nearby objects – from walls, tables, doorways, signposts, buggies, and even faces – to enhance visibility.
This relatively simple yet effective use of AR brings colouring books to life. Print out the black and white drawings, colour them in in the traditional manner. When you view the completed picture through your phone’s camera using the Quiver app the picture comes to animated life on screen. Touch the screen and you can interact and play games with the character you have created.