Is Apple working on a virtual, or augmented, reality device? Recent comments made by Apple CEO Tim Cook, along with a report claiming that Apple is developing augmented reality glasses point towards a brand new wearable.
There are multiple rumours that Apple is developing a virtual reality (VR), or augmented reality (AR) device, going back well over a year. Is Apple making a VR headset, and what would an Apple VR headset look like? If the Apple VR device is real, when can we expect an Apple virtual reality device to be announced? In this feature, we are going to look at all the latest Apple VR rumours, patents and leaked images - including the remarks that Tim Cook made during an interview in October 2016.
There's no smoke without fire, and Apple has filed patents and hired VR experts. So Apple is clearly investigating, and building prototype VR or AR devices. And with Google, Microsoft and even Facebook also making VR and AR devices, it seems like it's a party Apple can't afford to miss.
Apple VR: What is virtual and augmented reality?
First, a quick refresher on terms. Virtual reality devices are headsets worn like glasses, but inside a VR headset are screens that display a 3D image. The VR headset tracks your head movement, and the 3D image moves according. This makes it appear as if you are wholly inside a 3D 'virtual' world.
Virtual reality hit the headlines in the 1990s when video games companies like Sega and Nintendo attempted to create and sell VR products to the public. The technology wasn't ready then, but a more recent attempt called The Rift developed by Oculus VR in 2012 was developed with modern 3D technology and development has been wowing people ever since. Eventually, FaceBook bought Oculus VR in 2014 for over $2b, and the VR headset is now available to buy in the UK.
Augmented reality, on the other hand, is where the glasses are see-through and you can still see the world around you, but an image is displayed in front of your eyes. Augmented reality also hit the headlines in recent years, first thanks to Google Glass (which displays 2D images) and more recently with an impressive demonstration by Microsoft for an upcoming product it is calling Hololens that embeds 3D images in the world around you.
Meanwhile, a rather mysterious company called Magic Leap is also working on a headset utilising Mixed Reality Digital Lightfields, allowing the headset to generate contextual digital lightfield content that can interact with the world around you. Not much is known about the company, and it's said that only a handful of people have actually seen the headset - but judging by demo showcased by the company (which can be seen below), it's looks impressive to say the least.
Apple is known to be a trailblazer, but it's still part of the Silicon Valley tech industry (albeit a more secretive one). If Google, Facebook and Microsoft are all working on VR and AR solutions, you can be your bottom dollar that Apple also has a prototype in its labs.
Apple AR rumours: Apple is working on augmented reality glasses
According to a rumour first picked up by Bloomberg, Apple's next project will be a Google Glass-style pair of augmented reality glasses and not a VR headset as originally believed. According to the news outlet, Apple has even gone as far as to discuss the project with its hardware suppliers, and even order a small number of "near-eye displays" for internal testing displays.
If Apple's internal testing goes well and it moves on from the early stages of development, Bloomberg's sources claim that the glasses will wirelessly connect to an iPhone and will display not only information but images and other data directly in front of the wearer's field of view. It's said that the glasses won't be made available until at least 2018, if ever, as Apple is notoriously secretive about its product development and the project could be canned at any moment.
This, along with Apple CEO Tim Cook's recent comments about the future of augmented reality, suggest that Apple is gearing up for its next big product in the wearables industry - Apple Glasses. There are other rumours supporting the theory too (which can be found below) and for those more interested in the idea of a VR headset, there are plenty of rumours supporting that theory too.
Apple has a 'secret team' working on mobile AR solution
As well as working on a full-blown pair of augmented reality glasses, Apple is also rumoured to be working on integrating the same augmented reality technology directly into the iPhone camera app. The claim comes from a Business Insider source that claims that the effort involves teams from several acquired startups, and this reflects Apple's short-term desire to put AR into the hands of consumers. Sure, the AR glasses are coming, but integrating it into a smartphone that millions of people already have will introduce them to the concept of AR, and may make them more likely to buy the glasses once they go on sale.
The source also claims that Apple has recently hired an expert in head-mounted displays (HMDs), and that this showcases Apple's long-term AR/glasses initiative. Apple adding AR capabilities to the iPhone should allow consumers to point the phone at a real-world object, like a statue or a car, and be offered contextual information based on it overlaid onto the real world.
Tim Cook hints at Apple's AR focus
Apple CEO Tim Cook has said in an August 2016 interview with The Washington Post that Apple is "doing a lot of things" in the augmented reality space, contrary to the popular belief that Apple was working on a virtual reality headset. He didn't stop there though, as the Apple boss went on to claim that he thinks of it as a "core technology" for the fruit company.
"I think AR is extremely interesting and sort of a core technology" Cook stated during the interview. "So yes, it's something we're doing a lot of things on behind that curtain we talked about."
Following on from these comments, Cook took part in a rare interview session at Sen. Orrin Hatch's Utah Tech Tour in Salt Lake City in October 2016 where he again discussed augmented reality, although this time he gave us a better idea on what Apple may be working on.
"AR I think is going to become really big," said Cook. "VR, I think, is not gonna be that big, compared to AR … How long will it take? AR gonna take a little while, because there’s some really hard technology challenges there. But it will happen. It will happen in a big way. And we will wonder, when it does [happen], how we lived without it. Kind of how we wonder how we lived without our [smartphones] today."
It's interesting that Apple is focusing more on augmented reality than virtual reality, although it does make sense - especially with the booming popularity of apps that feature the technology like Pokemon GO, and of course the fact that its rival Microsoft is working on its own AR headset, the Microsoft HoloLens. Cook see's it as becoming something that a huge part of the population will use on a daily basis, going on to state that he thinks "that a significant part of the population, of developed companies, and eventually all countries, will have AR experiences every day".
iOS code hints at AR integration
It seems that Apple might've been working on VR/AR integration for quite some time, as iOS developer Steve Troughton-Smith uncovered some "interesting" features that he stumbled across in the iOS code. He revealed his findings via a tweet, which can be found below:
As you can see from the tweet and attached media, the code hints at a head-mounted display - and this is nothing new. According to Troughton-Smith, the code has been in place since iOS 9 and has been temporarily abandoned by Apple. "I think the kind of VR you build for something dumb and simplistic like Google Cardboard is very different from what real VR needs. I imagine they got to the point where they realized they were going down the wrong path if they want to do this seriously," he said, speaking to Business Insider.
Apple VR Rumours: Apple is working on a virtual reality headset
Of course, if Apple isn't working on an augmented reality headset, chances are that it'll fall back on a virtual reality headset, much like its main competitor Samsung. There are quite a few rumours supporting that theory too, which started way before the news of Google Glass-like AR glasses started doing the rounds.
Apple hires virtual reality developers
If we cast our minds back to November 2014, some may remember that Apple posted a job listing for app engineers experienced with 3D graphics, virtual realityand augmented reality. The job posting was quickly removed, but it specifically asked for developers to build “high-performance apps that integrate with Virtual Reality systems for prototyping and user testing.”
Since then, Apple has made a number of relatively high-profile hires for its virtual reality project, including Microsoft’s Nick Thompson, an engineer that worked on the HoloLens audio hardware for three years. Apple also hired Bennett Wilburn, another Microsoft employee, who focused on “machine learning technology for human activity recognition” with his previous experience including the likes of Lytro and Huawei.
Along with ex-Microsoft employees, Apple has also hired Doug Bowman, a man considered to be a leader in the field of 3D user interfaces. His previous experience includes working as a computer science professor at Virginia Tech, as well as being the head of the school’s Center for Human-Computer Interaction. Lastly, we have Graham Myhre, a man that previously worked at Lytro developing specialised lenses and sensors, but is now “investigating new display and optical technologies for future generations of Apple products” – at least that’s what his LinkedIn says anyway.
As well as this, a recent report from the Financial Times claims that Apple has actually successfully developed a virtual reality prototype, and is actively testing it behind closed doors - no doubt by the spree of VR-related hires at Apple over the past few months. Though a prototype is floating around, the FT source doesn't go into detail about what we can expect from Apple's seemingly upcoming headset.
As first reported by Business Insider, Apple has also hired Zeyu Li, who previously worked at Magic Leap the AR startup. Yury Petrov was also hired, a man who had previously worked on Oculus.
Apple has since made another high-level hire for its AR/VR venture - this time from the company that provides the graphics processor for the iPhone 7. Apple confirmed back in March 2016 that it was interested in buying Imagination Technology, but that never panned out - and it looks like it didn't really matter, as the company appears to have got what it wanted.
The company has hired several senior employees from the company, with the most notable being ex-Imagination Technology COO John Metcalfe. According to his LinkedIn profile, he has been working at Apple since July 2016, leaving IT after almost 20 years. It's speculated that Metcalfe and others are working on a new GPU to power the AR/VR experience that Apple is planning, although there's no new information beyond that.
Apple buys a number of VR-related companies
One of Apple’s biggest and most popular acquisitions was back in 2013 when the company bought PrimeSense, the company that developed the original Xbox Kinect, for a whopping $345 million. That’s a huge amount of money to buy a company for technology that might not be used, in our opinion anyway. While many first assumed that it’d tie in with Apple’s rumoured TV set (or the Apple TV itself), it seems more suited to a possible VR/AR headset.
It didn’t stop there either, as 2015 was a busy year for Apple – and we’re not talking about the number of new products. The first acquisition of the year was Metaio, a German company that Apple acquired back in May 2015. Metaio was a relatively well-known company with regards to augmented reality, powering many of the popular AR applications being used today – including Ikea’s virtual catalog, and Ferarri’s AR showroom app. Since being bought out by Apple, Metaio has gone quiet – heading to its website will offer you a brief overview of the company, a contact email address and not much else. Mysterious.
Along with acquiring Metaio, Apple also acquired Faceshift in November 2015. While it isn’t an AR-specific company like Metaio, Faceshift created the motion-capture technology that was used in the recent Star Wars film. The firm is known for developing software that allows CGI-animated 3D characters to ‘copy’ the facial expressions of the actor. While this could be linked to a virtual/augmented reality headset, it could also be used on FaceTime – think Snapchat-esque lenses.
Most recently, Apple has confirmed the acquisition of Flyby Media, a company well-known for developing technology that allows smartphones to 'see' the world around them. Notably, the company has worked with Google in the past to develop the image-recognition abilities found in Google's Project Tango. Though the website is now down (typical of an Apple acquisition), Flyby Media described itself as "dedicated to building new technology that can elevate, rather than replace, our real-world experiences." and boasted a team with knowledge in the areas of large-scale SLAM, indoor navigation, sensor fusion, image recognition as well as 3D tracking.
Tim Cook discusses VR at Apple quarterly earnings call
While people have been speculating that Apple will be making a virtual reality headset for well over a year now, the company hasn’t officially acknowledged its interest in the VR market – until now. During Apple’s quarterly earnings conference call in January 2016 (where Apple announced its biggest quarterly profit ever, $18.4 billion) Apple CEO Tim Cook fielded a question regarding virtual reality. His response? “In terms of VR, I don't think it's a niche. It's really cool and has some interesting applications.”
Okay, so it’s not quite confirmation that the company is actively developing a virtual reality headset, but let’s look at the past. Back in 2014, before the announcement of the new Apple TV, Tim Cook was quoted saying that TV was an area that Apple had a “great interest in” – a year later, we had a completely new Apple TV boasting its own App Store and gaming capabilities. Cook also called the idea of smartwatches “interesting” back in 2013, way before the launch of the Apple Watch. So as we said, it’s not confirmation, but it gives us a good idea at what to expect in the (possibly near) future.
Apple sends employees to specialist VR lab
If Apple isn't developing a virtual reality headset, then why has it sent employees to Stanford University's Virtual Human Interaction Lab several times over the course of three months? As first reported by the Wall Street Journal, Director Jeremy Bailenson made the remark regarding Apple visiting the centre during a Wall Street Journal conference in California. The Virtual Human Interaction Lab based at Stanford University is the first place that businesses will go before developing a virtual reality headset - Mark Zuckerburg, CEO of Facebook even went along shortly before it acquired Oculus.
Speaking at the event, Bailenson remarked: "Apple hasn't come to my lab in 13 years - except they've come three times in the last three months. They come and they don't say a word, but there's a data point for you." As ever, Apple is keeping tight-lipped on the subject but this, along with comments made by Tim Cook claiming that VR isn't a niche and that it has interesting applications suggests that an Apple-branded VR headset is in the works, although it could still be a way away.
Apple VR/AR headset patents
Way back in December 2013 an Apple Patent (USPTO 8,605,008) surfaced. This patent described a head-mounted display. The abstract talks about "A goggle system for providing a personal media viewing experience to a user is provided. The goggle system may include an outer cover, a mid-frame, optical components for generating the media display, and a lens on which the generated media displayed is provided to the user."
Three years later, in October 2016, another Apple patent appeared online depicting the same head mounted display as above, albeit a more refined version. The accompanying text also gives us a better idea of how the VR headset would work: "A head-mounted device that is worn on a user’s head and configured to integrate with a cellular telephone that is removable, the head-mounted device comprising: a frame that is configured to physically receive and carry the cellular telephone, wherein the frame places a display screen of the cellular telephone in front of the user’s eyes; and an optical subassembly configured to receive at least one image frame from the display screen of the cellular telephone, wherein the optical subassembly is interposed between the display screen and the user’s eyes."
It's interesting to note the likeliness between Apple's VR headset and a standard pair of glasses, as most VR headsets resemble a ski mask and block out all vision from the sides of your eyes for a more immersive experience. Of course, it's only a patent and the design is likely to change again before it's released - whenever that may be.
Apple VR/AR design rumours: what will Apple's virtual- or augmented reality headset look like?
According to Apple's patent, "the goggle system may resemble ski or motorcycle goggles. To enhance the user's comfort, the goggle system may include breathable components, including for example breathable foam that rests against the user's face, and may allow the user to move the display generation components for alignment with the user's eyes. In some embodiments, the goggle system may include data processing circuitry operative to adjust left and right images generated by the optical components to display 3-D media or account for a user's eyesight limitations"
The patent was filed in May 2007 before the iPhone was even released.
Apple VR/AR UK release date rumours: When will Apple release its upcoming headset?
Assuming that Apple is working on a VR headset, and is going to announce one at some point, when would be a likely time?
While other companies seem to be making public headway with their VR offerings, nothing is truly available to the public yet. Oculus Rift will come first, with pre-orders shipping in March, with the HTC Vive Pre to follow suit soon after. Hololens is being given to developers for testing and probably won't get a public release until at least later in 2016, and development on Google Glass has been halted.
Apple traditionally announces devices it can sell in the same year. If there is to be an Apple VR announcement from Apple, we'll hear more rumours as time progresses. We're hoping to hear more information about the headset this year, but there is no word yet on a possible release date - we can only speculate at it coming out in 2017-2018.