If the promise of VR is to create realistic immersive experiences, we need realistic immersive audio to match the visuals.
That means investing time in tools that enable creators to bring more content with spatialized sound to the public. Below are some lessons I learned designing the end-to-end experience for Facebook Spatial Workstation, a new software suite for professional sound designers to create spatial audio for 360 video and cinematic VR that enables audio sources to be added and positioned in virtual space after filming.
Hearing is one of the fundamental senses through which we perceive the world. In order to create engaging, memorable and immersive experiences for 360 Media and Cinematic VR, we have to ensure that sounds feel real and authentic. If the audio doesn’t match the geometry or the environment around us, the immersion breaks.
With 360 or spatial audio, the sound feels like it’s coming from a certain direction in space, like it would in real life. It feels natural. Our brains are amazing at calculating the distances and determining where each element is coming from in three dimensional space. Using sound cues, we can even quickly and unconsciously identify the size and type of space we’re in. Are we in small room or a concert hall? Is the object below or above? In front or behind?
To craft believable 360 experiences, the immersive graphics need equally immersive 360 audio that replicates the natural listening experience. As you look around inside the 360 video, sounds need to react and reposition themselves depending on where the objects are located on the screen or virtual space.
Until now, there haven’t been any reliable end-to-end tools for creating such experiences and bringing them at scale to an audience. That’s why our team recently launched new professional tools and rendering technology that makes high-quality spatial audio possible for large-scale consumption.
Spatializer and Control plugins: before (left) and after (right)
Disclaimer: I’m not an expert in spatial audio but working closely with Spatial Workstation over the last couple of months, here are some of the things I learned as I designed the end-to-end experience for this software suite.
1.Understand the Context
People problem: “I work for hours in the dark studio with bright screens and my eyes get tired.”
At the beginning of the design process, I took things back to fundamentals by interviewing sound designers and observing their workflow and design space. I found out that recording studios are often low-lit, so managing legibility was one of my main concerns. In addition, looking into an overly bright screen while working in the dark is like staring into a flashlight: your eyes perceive the darkness around you, but you are looking directly into a light source, which can cause lots of eye strain.
To address legibility and reduce eye strain, we chose a darker scheme for our software suite. With the help of my team we defined consistent patterns for the interactive elements and different button states. For example, white is only used for interactive components and affordances such as the handle on sliders, dropdowns or text inputs, whereas blue is used for feedback.
We used colors actively to identify the change of state or progress status. For instance, the color of the UI changes as you switch between different channels in the Spatializer plugin to indicate which source you are currently manipulating.
Lastly, we also ran all the colors through color accessibility software to make sure we maintain a necessary contrast ratio between background and foreground colors.
2. Observe the workflow
People problem: “In my work, I use both hardware and digital software, multiple plugins from different companies….often I have several open at the same time and pinned on the screen.”
In their workflow, sounds designers often switch back and forth between hardware (knobs and sliders) and digital software. In addition to a DAW (Digital Audio Workstation) they also have several plugins open at the same time, as there is currently no single end-to-end pipeline for spatial audio production.
Therefore the design patterns that we chose aim to reduce cognitive load and ease the transition between different tools and modes. Without making the design skeuomorphic, we borrowed metaphors from the physical world such as a slider on mixing board and gave the UI elements a clean and minimalist look.
3. What’s important?
People problem: “Lots of sound work is in fine-tuning.”
I also learned that sound design is all about details — adjusting and refining the audio until it sounds just right. Designers use both hardware keyboard (lots of knobs) and digital tools (more knobs!) to do precise and detail-oriented work. Since skeuomorphic design is prevalent in the existing audio tools, knobs become as ubiquitous in software as they are in hardware. However, turning a digital knob on the screen can cause lots of physical strain on the wrist as the mouse or touchpad are not optimized for performing circular motion.
Massive audio plugin
To provide the level of precision needed without causing strain on the wrist, I worked on the design for the UI slider that consists of individual bars to give a feel of precision while still being easy to use.
4. Create a moodboard!
People problem: “All the tools I use look very different.”
There are thousands, if not tens of thousands, of sound plugins in the market and multiple DAWs that all have very distinct looks. How do you design a plugin that, on the one hand fits in well with multiple different combinations of plugins, while on the other hand crafts a recognizably Facebook experience?
One of the most helpful tools in defining the visual language of the new product is to create a moodboard, a collection of inspirational images. Visual mapping is a particularly effective way to learn about existing industry tools. Through that exercise, I discovered some common patterns among industry tools such as use of dark scheme, active use of color and shadows and prevalent skeuomorphism of UI components such as sliders and knobs.
My team and I then used those patterns to define the design guidelines for Spatial Workstation.
5. Dig deep into technical dimensions of the tools
Without understanding the workflow and technical aspects of the tool, then my design solutions would be limited to visual refresh.
While I’m not an expert in spatial audio, I found it necessary to learn the process of spatial audio design first before designing the experience for others. I had to understand how to go step by step from recording the sound, to positioning it in 360 space, and encoding and publishing my 360 video with spatial audio on the platform.
Going through the full tool workflow myself (with many tries and troubleshooting) helped me discover multiple pain points and usability problems, and made me more empathetic to our users.
We launched the new re-designed Spatial Workstation in December 2016 and got very positive feedback from the audio community. There is even more work to do to improve the tools workflow and usability, and we hope that with these new tools available to content creators and designers we’ll see more and more 360 content with spatial audio coming to all the platforms.
In the end I want to share with you the great video by SoKrispyMedia and hope it will leave you wanting to create your own spatial audio.
Please don’t hesitate to reach out to me if you would like to chat. I would love to learn about you and about your experiences designing for VR and spatial audio.