A scene from Save Every Breath: The Dunkirk VR Experience
Academy Award-nominated director Christopher Nolan has referred to his film Dunkirk in IMAX as “virtual reality without the goggles,” so when it came time to build Save Every Breath: The Dunkirk VR Experience, the team at Practical Magic knew that the stakes were higher than usual.
And as Practical Magic’s Matt Lewis began working with Adobe’s Director of Immersive, Chris Bobotis, they realized that their task stretched past the challenge of creating a tie-in worthy of a supremely immersive Academy Award-nominated Best Picture, all the way into the future of user interfaces, the role of community in storytelling, and the new ways that young creators are driving technology.
When Practical Magic was brought on board to find a way to make Save Every Breath: The Dunkirk VR Experience happen, they had a number of unique challenges, including the need to use large-format cinema cameras for the VR experience. As the company’s co-founder, Matt is used to solving problems for studios after a long career putting out fires in technology situations.
To make the project possible, Practical Magic used Adobe Premiere and After Effects with support offered by Chris Bobotis and his team. This was in fact one of Chris’s first projects after Adobe acquired his company, Mettle, whose products including the Skybox Suite had become key players in solving some of VR’s stickiest workflow problems.
“Matt couldn't disclose what he was on," Chris told me, "but he reached out and put forth some challenges and asked if we could work closely and prioritize to accommodate this property. I didn't even know what it was, but I got very excited."
Matt described working with Nolan: “When I sat down with him in person, he said that he wanted to give everyone a breath -- and never quite let them have a moment to exhale. So you get these moments of suspense in the sea, on land and in the air. You're waiting for the enemy approaching, but it never quite gets there. I think that's what we accomplished with Save Every Breath: the sense of impending danger.”
I recently got to speak with both Matt and Chris about building the VR experience, how Adobe’s extensible and democratized tools made Save Every Breath: The Dunkirk VR Experience possible, and how VR is a perfect medium for emerging talent.
Creative COW: Can you tell me more about how the project came to you? What were the challenges and what was unique about how you handled it?
Matt Lewis: It was a project that came in with some urgency. When you've got a director like Christopher Nolan, you can't show up with a bunch of GoPros. You have to do things better, for all the right reasons — he’s demanding. So the things they wanted to do in terms of visual quality, it was difficult to find a way to do that.
Within a couple of months, the project came together from concept to delivery, which is an extraordinarily short amount of time for a VR project. We got pulled into figure out how to do it on a technical basis, to take the creative idea and mold it into something that's actually shootable. The hype [of VR’s potential] has to go away at that point and we become tacticians: nobody cares what's going to be possible in 2 years, what's possible in the next 3 months?
Practical Magic's Matt Lewis
How did Adobe’s tools play into that?
Matt Lewis: Both Premiere and After Effects played major roles in ways that are really not obvious to people. I would bet most people would assume we used Premiere just to cut it and After Effects to do visual effects. But we actually did a lot of unusual things with Creative Cloud. One of those things being we used Premiere to drive motion control.
We had to repeat these shots over and over again, but we had to do it in a VR sense. Because Premiere is extensible, we wrote a piece of software between Premiere and the motion control to allow us to use cinema cameras to shoot multiple passes of every individual actor, so we never had to stitch anything together. So that was actually being driven by Premiere, which is not a use case that has ever been done before — but is possible. And we wouldn't have been able to do the show without it.
Hearing about different use cases through Premiere’s extensibility is fascinating. It seems like a growing focus for Adobe is just to provide as many tools in as many ways as possible, especially in VR.
Chris Bobotis: I think the biggest and best stories for Adobe are that we have a lot of great tools, and my job is to bring as many of these tools into immersive support moving forward. Our customers are storytellers, they're well versed in cinematic production, and we made it all easy wtih 360. It's just a new medium. It's an amazing new medium.
Chris Bobotis, Adobe Director of Immersive
Matt Lewis: And from a workflow perspective, it helps to have a giant venn diagram of tools that overlap each other. We look closely at Premiere and After Effects, we use them for non-obvious uses and see where they overlap at that level, and we have partnerships and relationships with Adobe. If you're a developer and you're developing with Adobe, you have access to people in Adobe that can give you real time help to do new things. It's not a big black box, you actually access smart people and you can get some real assistance. We're using Adobe at the absolute edge of what it's capable of doing because they're with us.
Chris Bobotis: The extensibility is a huge factor as well. The third party community is a big part of the story. They enrich the whole experience. And one of the things that I make effort to do is allow for even more extensibility. A lot of it we do foundationally in the products moving forward, and we’re bringing it forward to the public SDK level as quickly as I possibly can.
And that'll just allow you to do even more. So it's not just third party developers, but it's developers like Practical Magic who are well versed in sripting languages, coding languages, who can create bespoke solutions for themselves and their clients and go way beyond what we can possibly imagine. That's a big part of my job.
What are some of the biggest challenges to the VR workflow?
Matt Lewis: Workflow for doing anything in VR is really difficult for people. It's not like going from black and white to color, it's like going from color to warp speed. People think they've been through this: like look, I made the transition from HD, the transition to 4K, and I'm making a transition to HDR. But this is not the same thing. This isn't like I'm going to incremementally adjust my hardware, learn some new color spaces, and get a new tape deck. This is rethinking all the way from square one, from the story to the screen, every concievable thing you're going to do.
Like putting titles on the screen [in VR] -- I layered it in and it was warped. There are a lot of tasks that seem very simple, like putting text on the screen, but how the hell do you do a lower third in VR? What is a lower third in VR? I can't bake that in because if I put on a subject and turn, it stays on you when I turn away and I forget who's talking. So if you're trying to explain to me ‘hey turn around and look at this thing’ and you're the expert, and I'm looking away and I can't see your lower third, that's a problem we all have right now.
It'll get solved soon. But that's just the simplest, 9am day one example of how the workflow is different. Everything you think you know is different. Trying to take artists, editors, workflow people, anybody and put them in the chair? The minute you see someone who has never worked outside a rectangle go to start on it, they just get quiet. It's like a child picking up a Lego for the first time. It's an absolutely fascinating process for most people. Seeing the sense of realization on someone's face when they start think oh, I need to think beyond the edge of the frame.
Chris Bobotis: I think that's the real challenge. When that frame disappears and it's all around you, it gives you a great opportunity to reinvent storytelling. And I think that's what I'd love to bring to the medium. Have you forget about the technicalities and really focus more on creating stories within the new medium.
Matt Lewis: What is the lifespan of a rectangular frame? How long are we still looking at a rectangle at our desk?
Chris Bobotis: I don't think it'll go away anytime soon, I think it'll be a combination of augmented reality or mixed reality — having tactile movement and decision making.
Do creators need to learn how to tell new stories with this kind of experience?
Matt Lewis: I’m learning again all the time. Myself and everyone else, we’re in kindergarten. We really are. Everyone who thinks we know how to do this, we're all wrong. I don't think I know how to do this. There is no right way to do it yet, because the question hasn't been answered yet. The potential is so far beyond what anyone can imagine. I think if we thought we figured it out, we'd be selling VR short. I'm constantly relearning and rethinking and changing my mind about how to tell stories in VR.
Chris Bobotis: I've been around since 1984, I've seen a lot of workflows. I was pretty jaded, I've seen everything. When I saw 360, I realized I knew nothing. It was very exciting. You get to reinvent it. Anyone who tells you they know how to do this, they don't know. There's a lot of trial and error, blood sweat and tears, but when it comes together it speaks for itself.
Matt Lewis: When any new media is created, you still talk about it years later. We still talk about Trip to the Moon. You look back at these seminal pieces, and someone cracked it. Someone makes the first Wizard of Oz — that hasn't happened for VR yet.
And if that isn't enough of a reason to get into something creative, I don't know what is. The fact that that thing hasn't happened yet, everyone in this room should have goosebumps. Somebody in this room might actually be the person who is going to make that ,and that's incredibly exciting. The big thing is in front of you and you know it hasn't happened yet. Is today the day I'm going to look at the news and someone has done it?
How might this translate to the actual experience of creating the content, like in editing?
Matt Lewis: If I was wearing a mixed reality headset, still editing in my rectangle on an iMac on my desk, if I could back up for a second and my bin went into the air and every shot I needed popped out and I could look around tap something and it all disappeared, I think it would change the concept of what we think about what a bin is.
Chris Bobotis: As we're continuing to move forward with UX, the mouse has been a thing that has served its purpose and will continue to serve its purpose because ergonomically it's comfortable, while putting my hands in the air is not. But if I've got my hands up working on something tactile, it's a different experience.
A lot of our discussion has been on re-learning, but what about young people coming up right now for which 360 has always been something attainable?
Matt Lewis: You can make an argument that because the technology is so new, it's an advantage for people who are just getting into the business. They're going to learn the toolsets from the ground up in a very different way. They're looking at it as the only thing they've ever known. They're going to develop the foundation of knowledge and, these young smart kids, they might be able to learn it faster. They're going to come in the door saying I've only done this a year or two, but I know how to use tools that haven't existed very long.
If you can consume that knowledge and get good at it, you're going to bring value to companies you would have had a harder time getting into. I pulled a kid right out of Columbia College onto Dunkirk. I was teaching a class and he mentioned something about playing around with Skybox. He was at my office four hours later working on Dunkirk because that's how hard it is for me to find talent for VR. He worked on Dunkirk for three weeks. He's like 20 and it was a great thing. And it was because he had the skillset and was able to learn quickly on the job.
The same tools I used on Dunkirk, a kid can have those for $20 a month. The same stuff we were using with Christopher Nolan. I didn't have any special thing, I had Adobe software and ingenuity. And we obviously had the project and budget, but we couldn't just light that money on fire in the parking lot and sprinkle magic dust on it — you have to apply talent to it. It's amazing that the tools have become so democratized that you can actually get the same tools the studios are using to make your passion project.
Chris Bobotis: It's all about accessibility, not just from a price standpoint but more so from a usability standpoint. That's the real challenge.