Still from 'Beethoven's Fifth' VR experienceJESSICA BRILLHART
VR video installation uses filmmaking and immersive sound to explore human perception.
For those who have been in the VR industry for a few years, Jessica Brillhart is a name that needs no introduction. Her work – both as a creator and thinker – were formative in the early growth of VR video. Her work as Google’s Principal VR filmmaker led her to produce standout experiences like World Tour and Resonance, while her writing offered vital insights that few others were sharing in a public way – be that thinking about audience participation in VR (she popularized the term “Points of Interest” as a way of reframing the director’s role in a 360 experience) or lessons learned in editing projects prior to the ubiquity of streamlined toolsets.
Since then, she’s created her own company, Vrai Pictures, a VR/AI company based in New York. Her experience Beethoven’s Fifth was on display in Montreal at The New Storytellers 5, an event curated by Phi Centre and Future of StoryTelling (FoST). The experience explores the eponymous symphony in relation to its inclusion on the Voyager Golden Record, the repository of sounds and images sent out into space on the Voyager probes.
At NS5, Brillhart and I sat down to discuss translating the iconic piece to VR, using audio to drive experience, and storytelling in 360:
As fate would have it, Brillhart was already in touch with the Philharmonic Orchestra in London when she was first tasked with bringing the Golden Record to life in VR.
“I felt that if the content was meant to tell the aliens about what it was like to be on earth -- and surely we’re aliens on our own planet -- VR was a way that we could reconnect with each other,” Brillhart said. “I was talking to the Philharmoic Orchestra in London already and I slid them this huge list of classical music and said, ‘Pick one and maybe we that would be how we work together’…and they immediately got back to me and said, ‘We actually have a recording on the record it was ‘Beethoven’s Fifth.’’
But Brillhart immediately recognized this serendipity as a creative challenge to address.
“I’m just like, oh no, because Beethoven’s Fifth is like candy for classical music, it’s like the ear worm…the everyone learns in school; it’s the one thing that everyone has made like VR about it like,” Brillhart said. “I [couldn’t] not do it but I [had] to think of a different way of approaching it.”
So she strategized with Esa-Pekka Salonen, the conductor of the Philharmonic Orchestra in London.
“[Beethoven] was a complicated man, he was not happy, and a lot of it had to do with the fact that he was going deaf,” Brillhart said. “So it became more about Beethoven and his disability which then became a lot more about just the act of hearing; and I thought that was really on point with what the golden record was trying to do.”
So for the VR rendition of Beethoven’s Fifth, Brillhart explored the eponymous [score] as an act of “human perception” in relation to the expanse of space – and the human endeavor to explore it.
“It leverages human perception but rethinks it so it can include sounds of interstellar space,” Brillhart said. “I use audio to transport you to space, so the idea of going from a spatial mix back to a mono mix and then back to spatial has been something I’m really excited about.”
For the less technical: this alternation between spatial and mono audio mixes has the effect of switching between a “three-dimensional” sound (what you hear is reflective of where you are positioned in relation to sound sources in the experience) and one-dimensional (you’re receiving one “positionless” audio feed). The former feels truer to reality, while the latter is more disembodied and abstract – it’s how she is able to incorporate a subtle ringing throughout the piece that she calls “tasteful tinnitus,” reflective of both Beethoven’s deafness and the endless expanse of outer space.
And, with the install at Phi Centre (and other showcases), this audio component took a more physical shape in the form of SubPac backpacks, which translate the sound into your body. Brillhart came to this aspect through experimentation.
“I plugged in a SubPac just to the mix…and it was like hearing some voice from another dimension,” Brillhart said.
The cumulative effect is a piece that uses the very idea of sound – and the different ways we perceive it – as an experiential, storytelling device, giving participants a bodily “narrative” of the symphony. Alongside that narrative, though, Brillhart’s expertise and trained eye for rich 360 filmmaking is clear. Deft camera pushes and cuts throughout the space allow viewers to not only behold the symphony in its sum, but also the individual parts – the small talk among friends, the interplay between conductor and orchestra. And because of the directionless nature of the 360 camera, the musicians exhibit a type of naturalism impossible with 2D cameras. It’s a singular experience that feels ultimately more intimate than even physically being there.
“That’s the replayability aspect, right…the ‘rewatchability,’” Brillhart said. “I watch it and it’s new every time; I like those sorts of experiences that aren’t so black and white that kind of play that line.”
Where a 2D film would demand that the creator select shots that drive a linear narrative (even a subtle one), the 360 film format allows an ambient story to emerge – one that will be different for each viewer depending on the subject(s) they choose to watch. Over the past half-decade, Brillhart has realized that this format is actually more intuitive to her natural tendencies as a storyteller.
“I think more in three-dimensional cypher type things than I do a straightforward thing,” Brillhart said.