Head of audio products Mirek Stiles on fusing visuals to our brain with 3D sound.
Yet another promising crowdfund project bit the dust this week - the 3D audio, made-for-VR Ossic X headphones which raised over $3.2 million on Kickstarter and Indiegogo. A healthy haul of 23,000 orders were placed online but in the end Ossic was only able to deliver 250 units.
It seems, though, that this latest failure is more damning of how difficult it is for innovative startups to succeed - Creative's similar Super X-Fi headphones cost $100m+ to develop - than the potential for spatial audio, where sounds are placed and reproduced not just in stereo but in 3D space including above, below and behind you. The impact spatial audio could have on the sense of immersion while wearing VR headsets and future augmented/mixed reality eyewear could be enormous. That's one of the reasons why in early 2016 Ossic was one of the first batch to join Abbey Road Studios' Red incubator program, for innovative music tech startups.
"Stereo was introduced to the public in 1958 and it's still the standard," says Mirek Stiles, head of audio products at Abbey Road. "It didn't catch on in a big way until probably the early 70's, a good twenty year gap there where a lot of people were still listening to mono. And it just feels like maybe it's time for something else."
That something else is in development at Facebook, at Netflix, even at secretive mixed reality startup Magic Leap. The new, standalone Oculus Go headset, which is both beginner friendly and affordable, has integrated spatial audio via built-in speakers which is impressive considering this is Facebook's 'budget' headset. But most virtual reality apps, games and experiences are still only available in stereo so it's still very early days.
So what's so exciting about spatial audio, in the words of someone who isn't trying to shift you a shiny new VR headset? "Sound, I think, is so important," says Stiles. "It's the fusion between the images and the mind to really make the experience fully enveloping. As in the real world, our field of vision is 110 degrees at the most and we rely on sound to tell us where we're supposed to be looking, what we're supposed to be interacting with."
The push for total immersion might differ slightly but this idea of a powerful narrative aid can apply to augmented and mixed reality experiences too: "For augmented and mixed reality, spatial audio is just as important as for VR. Everyone is taking it seriously. It's one thing to see a dinosaur walk across your desk but if there's no audio there to fuse those images to your brain, it's not going to have as big an impact as if you have audio to tell you there's something arriving on your right hand side, it's walking across the desk in front of you, going to the left and now it's walking behind you."
One challenge ahead is binaural head measurements, figuring out how to measure users' head and ear shapes so that that any hardware, such as 3D audio headphones, can then accurately calibrate spatial audio experiences to our individual profiles.
Oculus' chief scientist Michael Abrash predicted that by 2021, we'll be able to "quickly and easily generate a personalised head related transfer function - or HRTF - in the comfort of your own home". IDA Audio can now do HRTF modelling via photos of ears and videos of your whole head uploaded by the customer - instead of visiting an anechoic chamber. But Stiles thinks that ultimately Facebook's machine learning algorithms will figure out our measurements from social media photographs.
"It does make a massive difference. I think it will put a lot more faith in the experience," says Stiles. "Because at the moment, you'll hear something differently to how I hear it. You might get a great sense of sound above you where I might not get such a good sense above me, but I might get more of a sense behind me than you do. It's a little messy at the moment but that will be resolved."
How to record music for virtual reality
So the big players in VR and AR are prioritising sound and companies like Creative are stepping in to fill the Ossic-shaped hole in 3D audio headphones. What are we going to be listening to?
Abbey Road Studios has done a few experiments with recording spatial audio so far including a project with York University's Dr. Gavin Kearney designed to explore how best to record music for virtual reality. When it comes to all the software and hardware on offer, Stiles says spatial audio is "like the Wild West" right now.
"Even people like these amazing academic researchers, we're all still learning and we're all still trying to work things out," he says. "We took a band and put it in one of our studios, Studio Three [famously favoured by Pink Floyd] and set up a 360 degree camera. Obviously you can use tools in post production to place things around you but if you go back one step, you can try to get the best from the source material. We used all sorts of different microphone arrays, surround sound microphones, ambisonic microphones, traditional microphone techniques."
The team then created a VR experience - that's sadly not currently for public consumption - and did blind tests in which users could indicate which setups gave the best sense of space, envelopment and "directional activity" as they turned and moved their heads.
There are further collaborations planned with York University for the future and the studio is preparing as more and more artists, producers and engineers are asking about spatial audio, including game and film composer Stephen Barton whose scores and credits include Call of Duty 4: Modern Warfare, Titanfall and Gone Baby Gone. "I'm working a lot with Stephen Barton and he's really into this as well," says Stiles. "We've done some collaborations in Studio One with the LSO, again trying out different microphone arrays, different ambisonic microphones."
In fact, the leap to spatial audio could be just as intriguing for the people creating the sounds as those experiencing them. "From an artistic point of view, you say to an artist, 'look, you can now massively expand the canvas which you can paint with', that's getting people excited," he says. "You can have sounds above you, below you, behind you, anywhere in between, different distances, you're not confined. Stereo over headphones is quite unnatural in a way. So I think it's giving a lot of artists and content creators an opportunity to really expand on the way they think about sound and how they present their art."