AUSTIN, Texas—As has become standard at my job, I recently strapped on a virtual reality headset to go somewhere far, far away. On this day, it was a trip to outer space, with pit stops on a moon's surface and a sci-fi-styled briefing room, to hear about future expeditions to Mars.
I should have been looking out at the virtual planets, solar-sail rigs, and plans for human colonies. But I kept focusing my attention somewhere else: Buzz Aldrin standing next to me.
"You can look at Buzz from any angle," a demo representative said while I was wearing an HTC Vive VR headset. It was true. I could get on my toes and look at the top of his head, crouch and look up at his chin, or sidestep and see his face and body from any angle.
This wasn't the result of 360-degree video trickery, where a warped panorama might stretch around my field of view, nor was Aldrin made of blocky polygons. Every step I took, this VR system restitched Aldrin's appearance, and the image was based on an array of dozens of cameras that had captured his likeness from all angles and sides.
The effect is achieved by a process known as "volumetric capture." Up until now, real-time 3D imaging hasn't placed much stress on this resource-hungry technique. As tempted as I was to stick my nose up at its gimmicky nature, a mind-blowing demo (not the one with Aldrin) changed my tune. Virtual reality offers a particular use case that breathes new life into the concept—and in a scratch-my-back kind of way, VR has good reason to embrace the standard, however proprietary it might be.
Image 1: Flat images of VR demos are never very good at representing what they contain. This is the view when looking up at a solar-sail device flying over your head on its way to Mars.
Image 2: This awkwardly zoomed shot represents the stitched-together nature of 8i's volumetric rendering system. It's a little fuzzy, even in motion, but the effect is very impressive, especially as you move around the man.
Image 3: Buzz's Mars colony dreams, in giant-dome form.
Image 4: Quick pit-stop on a moon. Sup, moon man?
A startup imaging company called 8i has been publicly crowing about its volumetric capture system for over a year, and it has used festivals like Sundance, film properties like Mad Max, and stars like Jon Hamm to excite VR newbies. Put this headset on and look-ee-here, you're really next to other people! Neat.
As they've said during other public demos, company representatives clarified to Ars that its human VR subjects are captured by a multi-camera array in a green-screen room. Lenses are arranged above, below, and all around the subject in question, and their motion is limited; you won't get the walk-and-dance results of, say, a flatter hologram projection like Tupac or Hatsune Miku. Instead, you get a convincing render of a person (with some compression-related splotching) that works no matter what angle you see them at. Sadly, all of my prying technical questions—are you using voxels, how do you make the image refresh efficiently in a VR display, do the cameras employ infrared capture, what GPU attributes are best for this kind of rendering—were waved aside.
Instead, I was offered something very weird: a chance to hang out with tiny Buzz Aldrin.
One rep held up his smartphone, which included 8i's "augmented reality" collection of captured actors and celebrities, and pointed its camera at the other end of the room. Aldrin appeared as if he were standing in the room (with the usual AR jitters from smartphone sensing, much like in Pokémon Go), and the famed astronaut started shouting, "get your ass to Mars!" I was asked if I wanted to stand with Buzz. I countered: how small can you make him?
8i's proprietary capture system, used here to put tiny Buzz Aldrin in author Sam Machkovech's hands. | 8i
Minutes later, I was posing with an itsy-bitsy, teeny-tiny American hero in the palm of my hands. (The Rick and Morty fanboy in me spent the rest of the day shouting, "I'm Tiny Buzz!")
8i's promotional track record seems to revolve around these kinds of moments—where people gush less about the tech and more about the "ooh neato" effect of a celebrity rendered in unusual ways. I laughed when I was sent a laggy video of the moment, then figured I'd file the curio into one of our usual conference photo galleries and be done with it.
Then my South By Southwest changed.
New perspective after After Solitary
A few days later, I bumped into Alex Schwartz, the VR evangelist from Owlchemy Studios (makers of the brilliant Job Simulator). His face had the telltale "raccoon eyes" indentation from wearing too many VR headsets, and that's because he had just finished testing and sampling the festival's VR award nominees as one of that category's judges. Anything good? I asked.
Before I'd even finished my question, Schwartz said two words: After Solitary.
Image 1: Kenneth Moore stands in his cell. In the room-scale VR version of After Solitary, you stand with him, too.
Image 2: Moore shows off a "fishing" technique used to share items between people in nearby solitary cells.
Image 3: You get to stand near him and watch him attempt to connect to the outside world right here. I cannot even imagine the PTSD flare-ups he must have felt recreating this moment.
This documentary, which later won the festival's "room-scale film" award, also utilizes 8i's volumetric capture technology, but there's no Jon Hamm, no Hollywood production value. It is a cold, dark, and cramped experience. And it has completely flipped my expectations of VR documentaries.
After Solitary stars Kenneth Moore, a former inmate at a correctional facility in Maine who eventually took center-stage in conversations about deplorable prison conditions. Moore's case was particularly intense, as he was subjected to lengthy stints in solitary confinement and inhumane punishment at the hands of guards at the "Supermax" facility.
Some documentaries want you to engage with their human subjects by stitching together a story that arouses empathy. But After Solitary's subject is quick to tell viewers that he wasn't a model citizen in prison. He attacked guards. Tried to set fires.
Instead, the film uses its setting, and your movement within that setting, to tell its incredible story of isolation and captivity. You wake up within the documentary, which currently requires a room-scale VR system like the HTC Vive or Oculus Touch, as Moore's cellmate. The tiny quarters of an average room-scale VR rig match up with how small Moore's solitary confinement cell was in prison—a room he spent more than five years in throughout nearly two decades of imprisonment.
Developers confirmed to Ars that they pre-baked as many convincing real-life details into the room rendering, particularly its light and shadow effects, to devote more GPU horsepower to Moore's volumetric presentation. The result is astounding, and not just because it runs at a smooth 90 frames-per-second refresh. As Moore recalls his most despairing moments in solitary—pulling hairs out of his arms and head, using his own blood to write song lyrics on the walls—you can walk right up to him and look at his face, his tattooed hands, his lumbering frame.
I realized as I stood with the prisoner that the scale of VR settings becomes all the more convincing and emotionally affecting when a believable human is there with you. Later, I was warped outside, looking through Moore's window, and he looked back at me, telling me about how he'd tape a piece of paper to the window sometimes. This required guards to bust in and drag him out, he explained, since it was against the rules. It was his way of just getting out for a minute.
Then a video screen appeared where his window should be, and I was shown a video of a brutal and inhumane punishment he suffered for doing this paper trick once: being dragged by guards, then disrobed and strapped to a toilet with a gag in his mouth.
The ending wouldn’t be the same on a flat screen
Depressing and heart-wrenching don't begin to explain my feelings about this film. But so much of that is because of the technology employed here, along with After Solitary's filmmakers (who produced this for PBS' Frontline) really respecting what it means to be a viewer within a VR experience.
Theatrical principles are used to incredible effect. Important text always appears in multiple places in the film's rooms so that room-scale viewers never have to hunt for it. Light and darkness do the same thing to frame where viewers should direct their eyes. Activity is limited so that the act of being within the scene is less about making viewers the cinematographers and more about letting them take their time to understand how the setting impacted the man who lived within it for so long.
All of this makes its ending so harrowing that I struggle to type about it, even though on paper, it seems so simple and pedestrian: you're transported to Moore's current, real-life bedroom. You stand next to the man while he sits on a large bed and explains how tough it has been to transition to life outside. Within the room, as he starts explaining things, I noticed that, yeah, this bedroom is about as big as his cell. Then he says it: this is his home. He doesn't leave this room. I've been inside of his cell, and inside of his bedroom, and I realize that his life will forever be more comfortable inside these tiny spaces. Seeing him next to me—his face, his tattoos, his vacant stare as he talks at the wall about his life—frames it all in a way that may never leave me.
Moore doesn't have to list off what he does or doesn't deserve, or rattle off statistics or analysis about prison culture, to convince me of anything. I've been with him. No human should ever go through what he did, I think.
This is the beginning of a new wave of VR documentaries and journalism: to take human stories, captured with realistic environs and people, and share them in such a way that doesn't leave us sitting with poor, Cardboard-powered 360-degree videos. Those panoramas force us to work, hunt, and peck for stories and insight. After Solitary's directors and crew clearly worked hard to brush away as many barriers as possible to let the setting and sensation do the storytelling.
"Show, don't tell" just got a lot more interesting, and I hope that 8i is careful to open its platform up to filmmakers and creators with similar storytelling aspirations. It's fun to say that 8i's tech can put me side-by-side with Buzz Aldrin to imagine life on Mars, but I'd much rather the tech show me more life on Earth.