Yes: even more than the iPhone.
Steve Jobs is known for many things, but arguably his most profound impact on technology was his approach to design: an emphasis on friendly, elegant minimalism. By studying what made a given piece of technology complicated, he believed it was possible to counteract those aspects to create fun, intuitive products that belied the complexities under the hood.
With the launch of the iPhone in 2007, this ethos took shape in a broader way. The iPhone delivered a new, mainstream infrastructure for engagement with the digital world, and in turn revolutionized a raft of industries—notably among them User Experience and User Interface Design.
Because design is rooted in the way consumers interact with products, it fundamentally depends on how technology evolves. By creating a device that let users interact with the digital world in a more tactile way, Apple actually changed the shape of what that the digital world (and ultimately, reality) became. All of a sudden, companies needed professionals who understood how to “communicate” in this new medium, and this only became truer as new smartphones -- and then tablets -- entered the picture.
A decade later, we find ourselves at the cusp of a new era of digital communication an order of magnitude greater than the mobile revolution: immersive technology (XR), a category that includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) -- among any of the other "R’s." We’ve already begun to see how XR will revolutionize industries as far-reaching as medicine and real estate, but one of the deepest impacts will be felt in UX/UI.
No longer will consumers just use fingers to interact with a 2D window into cyberspace; XR opens us up to the full 3D reality of the digital world, which we’ll interact with using our whole bodies—marking the single biggest growth opportunity in the history of design.
For the skeptics in the room: XR technology will undoubtedly take some time to reach the desired intersection of form, function, and cost. It's still early days, but we're already witnessing its march into the mainstream. To many of us, it's a foregone conclusion that XR will become the predominant computing platform—it's just a question of when.
When it indeed does suffuse every aspect of our daily existence—via lightweight, low-friction hardware—what will the fields of UX/UI entail then? What strategies will have the greatest impact?
We’re only just beginning to discuss which tactics will stand the test of time, but there’s one thing we already know for sure—a sort of thesis for UX/UI in XR: successful efforts will be those that exhibit an intimate understanding of storytelling principles—not necessarily in terms of deploying linear narrative, but in activating the narrative aspects of our minds with spatial information.
The Narrative Potential of Space
Storytelling has always been an integral aspect of design, but in spatializing our media, XR demands an understanding of storytelling at a deeper, more fundamental level than ever before. Story is how we make sense of space in time. It's how we record our memories.
For the past century, our visual content—film, television, advertisements, web—has existed in 2D. We flattened our 3D world into two dimensions to communicate, and this language was ultimately codified into UX/UI. On your favorite map application, you use a simulated overhead view to find your way to that new coffee shop on the other side of town. Along the way, you toggle the dimensions and compass for clarity, but there's still sometimes room for confusion; you're abstracting a 3D experience to 2D space.
When we ultimately have lightweight AR glasses or contacts, the directions will appear around you, within your world (let's not even get into the added content that might exist alongside it). And this is just one example; it will apply to virtually every existing industry.
As we move through space, we assemble otherwise random data into chronological memories. These amass into stories. Because we're creatures who evolved needing an intimate understanding of space, it's the most intuitive way we learn and transmit experience. It's why the "memory palace" is one of the most effective and time-tested mnemonic devices in history.
This is a paradigm shift in storytelling akin to that of the discovery of fire: we're transitioning from what I call the "Teller-Listener Paradigm" to the "Builder-Participator Paradigm" (if you you want to dive deeper into that, read my pieces in VRScout and HuffPost).
With VR and AR, all of 3D space is now a palette. These perceptual opportunities constitute “Narrative Potential,” the chance to fill the space with information that will kickstart our brains’ native storytelling impulses.
I describe this concept at greater length here, but the following excerpt elaborates on how this notion:
Ask any architect or interior designer: every space tells a story. When you walk into a typical classroom, what’s communicated to you? Orderly rows of desks indicate that a group of people will all focus in one direction, rather than talk as a group. Fluorescent lighting and bookshelves imply a space intended for focus and scholarship. These embedded details drive us to make automatic assumptions about what to expect, how we act and ultimately become our “story” of that space in time.
If, as a VR experience designer, I seat you in front of a table where a glass vase is positioned precariously close to the edge, I’m tapping narrative potential by making you imagine it falling and shattering around you. This is true even if it remains still.
In other words, in VR/AR, space is story.
The designer’s job in creating immersive experiences that are rewarding for audiences—be they apps, advertisements, web sites, or otherwise—is to know how to reverse engineer story from space. Without this basic sense of narrative potential, a designer is dead in the digital water. It won’t be long before most of the materials we currently conceive of as 2D exist as 3D spaces. What might the homepage of this publication look like as a webVR site? What features of physical space would communicate the brand you’re currently witnessing as a 2D space? No matter what, your audience will leave with a story: what will it be?
What’s more, spaces, and the objects in them, drive different kinds of embodiment. In Story Unframed, Robin Hunicke explains that the way we hold objects influences engagement before we’ve even used them. She gives the example of a hammer vs. a pencil; the former being a “power grip” and the latter a “precision grip.”
When you move your arm along in a precision grip, like if you just pretend that you’re painting a line in space with your fingers, the parts of your body that get activated at most are your triceps, biceps, and your shoulder. When you grab the invisible hammer and then strike with the hammer—even if you just grab the hammer before striking—your entire upper torso and your back get engaged. It’s a fist. The physical feeling of the power grip immediately sends signals to your body that you are going to do something with force. Whereas the precision grip sends the signal to your body that you were going to do something delicate. It’s the feeling of fist versus conductor, fist versus paintbrush, fist versus chopstick. That feeling is so important.
How your hands look in an immersive experience will also shape your interaction. While creating Luna, her company’s flagship VR game, Hunicke tested different types of hands to best suit the goals of the game. It’s a story about growth; your aim is to help a little bird retrieve each piece of the waning moon and put it back together. Each level mirrors one of the stages of grief. Through testing, she realized that framing players’ hands as curvilinear, beak-like, almost floral shapes induced a precision grip that instantly communicated that players were meant to use careful, nurturing movements rather than exert force.
Though this is set in the context of a game, it’s not hard to imagine how UX/UI designers will need to carry these insights into any type of XR. Leap Motion, for example, has made a clear case that digital embodiment applies to immersive media where your hand is itself the interaction device. This becomes even truer when we imagine the applications of eye-tracking, haptics, and other related interaction technologies.
What does this mean for the industry?
It’s true: designers are going to have to create and learn an entirely new rulebook, with no small share of dead ends along the way, but the bright side is that UX and UI as professions will experience a surge in growth on a scale beyond what we witnessed in mobile. This revolution positions their skillset as one of the most invaluable of any company in any industry; XR will supplant mobile and desktop to become our predominant form of computing and communication, meaning virtually everyone will need an XR strategy.
Those who start learning how to design for immersion now will be poised to meaningfully contribute to this emerging language. Best practices in usability methodologies, information architecture, and user conditioning in XR are just a few of the spaces ripe for exploration right now. It's very likely that new subsets of the industry will emerge, and the early adopters will become the pioneers who shape them in their infancy.
We’ll bring with us many lessons from 2D design, and of course, there will still be a healthy place within immersive reality for 2D media—some things will still make more sense as two-dimensional panes—but it’s important that we have these discussions with native thinking in mind. Successful cinema isn’t “theatre on film,” for example; it’s a whole medium unto itself, with its own language and rules. As we graduate from 2D media to XR, we must try to imagine what aspects make it a unique language and design toward them.
VR and AR constitute the embodiment of exponential digital reality—both overlaid on the real world (AR) and transporting you beyond it entirely (VR). Buckminster Fuller famously said, “If you want to teach people a new way of thinking, don’t bother trying to teach them. Instead, give them a tool, the use of which will lead to new ways of thinking.” These emerging tools mark a complete shift in our perception of reality; they in fact demand us to interrogate what reality even is. They elicit new types of thinking, and the net sum over time is beyond anything we can currently imagine. Uber wasn’t built in a day—the “killer apps” of XR won’t be either.
But what we know for sure is this: designers will quite literally create the future of reality. Let’s just hope it’s a good story.