Tim Sweeney On VR And Civilization

Tim Sweeney On VR And Civilization
January 10, 2017

The next few years of the video game industry will determine nothing less than the future of human civilization, Tim Sweeney told an audience of developers this fall at the annual Steam Dev Days conference. So, game designers: No pressure!


As the founder of Epic Games, Sweeney is the creator of the ubiquitous Unreal Engine that powers thousands of games on PC, console and mobile, and the chief executive of the studio that made Gears of War. Now, like so many technologists, he thinks virtual reality will transform not just his 25-year-old business but the entire world.


By the end of the next decade, Sweeney believes, VR will be as popular and convenient as the smartphone is today. For games, he says, that will mean a global platform for immersive, photorealistic virtual worlds instead of a global market for tile-matching games. Game developers, who know how to create three-dimensional interactive entertainment, will be at the center of this effort to digitize "the entire real world," he says.


In a 13-minute peroration at the end of his keynote address, Sweeney compared VR to teleportation and world-historical events like the dawn of agriculture. "The next stage of mankind's development," he said, "is the pervasive ability of anybody to connect with anybody else to live a life that's unconstrained by physical goods."


That doesn't mean we are about to enter a utopia. There will be crime, vandalism, and harassment in VR, he said. But the dystopic threat Sweeney truly fears is that VR will evolve into a closed platform like Facebook or the iPhone, rather than the open Web.


"It would be really tragic if we let the future metaverse, that binds all humanity together into shared online environments, were a closed platform controlled by a giant corporation," Sweeney said. "As always, they'd use it to spam you with advertising, they'd use it to gather information about your private life and sell it to the highest bidder, and they'd act as the universal intermediary between all users, content creators, and transactions, ensuring that everything has to be approved by them."


I flew to North Carolina before Christmas to ask Sweeney to elaborate on his vision of the future. We talked for three hours over two days, at a downtown restaurant in Raleigh where he declined vegetables ("they're what food eats," he said), and then again at Epic's campus in nearby Cary, which includes the expected architectural touches: a rock climbing wall, a giant slide, and a towering sculpture of Marcus Fenix.


I hear you don't play video games?

I've always been much more of a programmer than a gamer.


So what led you to found a game studio?

Real-time graphics, to me, is the most exciting problem in computing. Because you not only have to simulate reality, but you also have to do it efficiently. And because the hardware is constantly getting faster because of Moore's Law, there are endless opportunities to invent new techniques that weren't possible before and to make them practical, and to build a really unique piece of software in the industry.


That's the Unreal Engine.

We got calls from people who wanted to license our engine before we even knew we had one. It turned out we built a useful thing, and so we built a business around it. It was like 1996, after we started hyping the original Unreal with screenshots.


It's been one of the keys to Epic's survival. If you look at the company's history, we've survived several really harsh transitions, from PC to console retail games, and from console retail games back to PC online games. It's the fact that we have this engine business operating right beside our game business that's provided stability. We can go through an industry transition for a couple years without having to make money from a game. We can survive and fund the company and come out successfully on the other side.

Epic's first game 'ZZT' (1991). Epic Games


Without that, I think we probably would have gone under, sometime over the past 25 years. There are very few companies from 1991 that are still around today in the game industry.


And to you, real-time 3D rendering is the art, whether it creates a video game or a car showroom or an architectural model.

Exactly. Real-time rendering tied into real-time input and processing and simulation to create the believable sensation that you're in a game world. That's the apex achievement of the game industry.


Pixar has a farm of 30,000 processors running 24 hours a day to render 1.5 million pixels per frame for its movies. Or so I learned at an exhibition last year at Boston's Museum of Science. The average Pixar frame takes 29 hours to render. Your engine has to do that kind of work on the fly on a $300 box in someone's living room.

I think you could render the original Toy Story movie at very close to its rendering quality in real time now. At some point it's just not going to be worth bothering with pre-rendering. You'll be so much more productive to build your movie in real time. And once it's running in real time in a computer, you can strap on a VR headset and actually be immersed in it. You move your head around and see the actual 3D objects from different points of view. Maybe you add bits of interactivity to it. I think it's going to create this entirely new type of medium that is somewhere in between a video game and a movie. It's much more scripted and planned-out than a game is now, but it's also more interactive than a movie.


Do you think we're in the beginning of another seismic transition, a shift that will produce some winners and lots of losers, companies that go under?

All of the retail console sales data this holiday season indicated that the console market is going through a really significant downturn. I don't think it's seasonal. I think what we're really facing is that young new gamers aren't going into consoles. And so there's this audience of increasingly aged gamers like a lot of us at Epic, who are sticking with it – but some are dropping out. Ultimately, the games that you can get and watch continually improve over the course of many years are far more engaging and reliable than this model of releasing a new game every Christmas and trying to convince everybody to buy it again.


It's going to be a real task for the colossal industry giants, who are currently retail console developers and publishers, to make the transition. I'm very thankful at Epic we recognized this coming about four years ago and now have Paragon operating and Fortnite coming soon and other projects in the works. It will be very hard for a company with a team of 1,000 people releasing new editions of a game every year to transition out of that and into a new model completely.


Why are young people not playing console games?

Young people are growing up in mobile. They're playing games in different ways, usually shorter play sessions spread throughout the day. It's not the kind of experience where you sit down for four hours and play a game. And the gamers who are getting into that are moving more and more into PC, I think.


PC is becoming a more convenient way of playing. And also these PC-centric games that are operated continually are becoming a bigger and bigger factor in hardcore gaming: League of LegendsDotaOverwatch.


Another factor is that laptops don't suck for gaming anymore. They're actually pretty good. It's a much more mobile device than a console that is always tethered to a TV. And what percentage of people have exclusive control of a TV all to themselves? If you're in a family, you probably have competition with other people. A PC, by default, is yours.


These long-term, permanent trends in PC and console gaming are overlapping with the early stages of VR. Even though VR, for the next three years, won't be big enough to make a big dent in it, the future of the industry is going to be shaped by it.

'Gears of War 2' (2008). Microsoft


Were you always interested in VR, or did this fascination start more recently?

In the early 1990s, there were VR headsets that could render Doom-level quality. It was like the original Palm Pilot, another thing I fooled around with for a few days and then ignored. It was technology that didn't work with the level of hardware that we had then. But those old stupid ideas that didn't work suddenly become mass-market products. If you look at the transition from the Palm Pilot to the iPhone, it was a sucky product for 20 years and then suddenly it was great. There wasn't really a point where it was becoming slightly more viable over time.


And now, the exciting part, is that over the next 12 years we're going to see VR scale down from a huge helmet to something the size of your glasses, which has a display for each eye that's higher quality than any display you can buy now, and cheaper, because it uses very little material. And that's going to revolutionize all forms of entertainment. Instead of having televisions and monitors and smartphone screens, you're going to have this VR device to project imagery wherever you want. It's going to occupy 140 degrees of your field of view--far, far higher quality and more immersive than the best PC entertainment experience you can get today.


People who can just barely afford a smartphone today in Africa or the Middle East will be able to buy augmented reality glasses and have a better entertainment experience than you can buy today for any price. You could build an IMAX theater, and it won't be as good as what you will be able to get at consumer prices within 12 years.


Within 12 years? Will these be untethered?

You probably will have something resembling Oakley sunglasses on your head. Really light, really convenient, a battery just providing power for the display. They'll be tethered to probably a brick in your pocket, something that looks like an iPhone today but doesn't have a display, just has the computing hardware and the battery. Because you're going to need a lot of power to make all of this a reality. It's going to take some serious battery.


Moore's Law says it's going to happen. Even if hardware performance only doubles every three years, we'll be at desktop levels of PC performance on mobile form factors in 12 years. That's going to be scary.


Why will it be scary?

Imagine a kid growing up in 12 years. They're going to have a completely different view of what's real and what's virtual.


It's hard to even contemplate the social implications of something like this. You'll be able to combine computer imagery with your actual view of the real world, in a way that's completely seamless.


We'll recreate humans. You'll have inward-facing cameras capturing your facial motion and outward-facing cameras capturing your body motion. You'll be able to see anyone, doing their natural motions, wherever they are. Do anything but touch them. Elon Musk is building the Hyperloop, but we don't need that. We're going to have teleportation, with this technology.


Do you really think we'll be able to simulate believable humans in 15 years?


The great thing about VR is we don't need to simulate them. We'll have cameras pointing at actual people, picking up their facial motion and their body motion. We'll simply be able to sample that data, reconstruct it with computer models, and display it to you.


So it's a problem of motion capture rather than artificial intelligence. That's a completely solved problem. It's completely solvable by Moore's Law with known algorithms and technology that we have today and doesn't require any radical new invention.

'Robo Recall,' Epic's foray into VR for Oculus Rift. Epic Games


But you just spent $15 million to preserve 7,000 acres of North Carolina wilderness. If we're going to be able to teleport into the metaverse, why bother?

Well, the real world is still out there, right? It's not going away. But I do think that is going to be a really interesting thing in the history of civilization. For the first time you'll be able to interact with anybody, anywhere, on a really deeply personal level without traveling.


You might have half as much air travel in 15 years as you have now. You might have a great reduction in road traffic. Unless your job involves touching people, you can simulate everything you do in a real-world work environment and telecommute through VR and you're missing absolutely nothing. You can sit in a conference room with a bunch of people and have deeply personal interactions, see their body language and their faces. You don't need to be there.


A decade ago, the economist Edward Castronova predicted massive economic dislocation caused by widespread "emigration" to virtual worlds, during the heyday of EverQuest. But that didn't happen. Or think of Second Life, which turned out to be an unnecessarily literal translation of what you can do more easily just by visiting Amazon and clicking a button. Why will it be different this time?


Those forms of virtual worlds were only good enough for gaming. That's why games took over all of those spaces. Pure games like World of Warcraft took over the social experiences like Second Life. Gaming has always been such a cool diversion that it's been good enough for any version of technology. The early 1980s arcade games, you couldn't make a computer out of them, you couldn't make a virtual movie, but they were good enough for games. Any technology is good enough for games.


What the metaverse needs in order to take off is a level of technology that's so good you could actually have a believable conversation with another human being, seeing their face and their emotions and their gestures, in a way that's so plausibly close to the real world that you don't mind the differences. That's going to take some time. We're not there yet.


It was 15 years from the debut of the consumer Internet to the explosion of social networks. You could have created an Internet-based social network in 1993, and it would have been fine. It could have taken off. Nobody figured it out. It might be like that with the metaverse. We might find that actually we had the components we needed to do it today.


So the technology will be here in 12 years, but we won't have the software to fulfill this vision that you have?

You could have that software in two or three years. I think in 12 years, this will be done. We'll be in this new world. And the market leaders will already have been determined. They might be Facebook and Google and Microsoft, but they might be a completely new set of companies. I think in 12 years, you have a billion users, and it's on the trajectory to reach everybody who has a smartphone today. I think there are 2.8 billion smartphone users, worldwide. Almost half of humanity. That's pretty good.


In your keynote address to the Steam Dev Days conference in October, you talked about your vision of VR as a parallel universe that is a shared virtual space, largely created by users. Why isn't that just the Internet?

The Internet started in the 1970s, and by the time it came to consumers, you had a global network providing communication to everybody in the country, and the ability of everybody to communicate with everybody else. What's next was the web. Now you have the ability for serious professionals to create content that is available to anybody in the world. The next stage beyond that was social networking, which made it very easy to exchange information with people. But all of that information was text and GIFs of cats doing funny things. It's still a very low-fidelity medium.


At the same time you had the Internet used for video games. You could play multiplayer video games, shoot people, play with mouse and keyboard, and have an interactive experience in a shared world. But with very, very limited input and output. A medium for gaming but not really for communication.

The next step, which VR and AR will enable, brings all that together. Basically, bringing the real-time 3D fidelity of a game world with easy social interactivity with other people – bringing the ability to express a full range of emotions, to talk, to see other people's faces and body language, all of the nonverbal aspects of communication. I think also to build content, like users do in Minecraft now, but on a much larger scale and in a more immersive way.


I think that goes way beyond anything that exists now. Like the web before it, it builds on top of the building blocks we already have.


There's been a lot of talk about filter bubbles and echo chambers since the election. Couldn't the metaverse just make that worse? We could all live in literally separate realities. If the reality I choose to live in doesn't reflect the physical world, maybe it doesn't matter, because the virtual reality I've constructed is satisfying to me.

We have a new generation of monopolists developing. That's kind of scary. Already, Facebook and Twitter want to censor certain types of news from certain websites, because we claim they're fake. But what we really mean is we don't agree with them. You have the Internet locked down in certain countries, like the great firewall of China. It might become increasingly hard, or even impossible, for users in two different countries to even communicate. That's really a terrifying thought. I think it's a real concern for humanity, that we should all be concerned about, fighting to keep the thing open.


The technology is so powerful, though, that whenever companies start to abuse their positions, there are correcting forces that come into play.


For example, given Microsoft's diminished status, the idea that the Justice Department was prosecuting the company in the 1990s as a monopoly for tying Internet Explorer to Windows seems almost ludicrous in retrospect.

I think it's ludicrous that they're not prosecuting more of the tech companies right now for their practices. There's a lot going on that's wrong. Apple has a monopoly on iOS hardware. That's fine. But they shouldn't be able to tie that market to a monopoly on distributing software, on collecting in-app revenue from software. I should be able to go to a web page and download a new Epic game to my phone without Apple's approval. I should be able to use Confederate flags, if that was our design choice, in our product.


All these hardware companies – Google does that. Microsoft is trying to do it with the Windows Store. They're in the early stages. There should be laws and principles against that. All these platforms should have to be open. If the platform maker wants to provide their own store, they can do that. They can curate it, censor it all they want. But it cannot be the only store. That is not an acceptable outcome in a democracy.


But Oculus, right now, is following the iOS model.

Yes. I think it's the wrong model. When you install the Oculus drivers, by default you can only use the Oculus store. You have to rummage through the menu and turn that off if you want to run Steam. Which everybody does. It's just alienating and sends the wrong message to developers. It's telling developers: "You're on notice here. We're going to dominate this thing. And your freedom is going to expire at some point." It's a terrible precedent to set. I argued passionately against it.


But ultimately, the open platforms will win. They're going to have a much better selection of software. HTC Vive is a completely open platform. And other headsets are coming that will be completely open. HTC Vive is outselling Oculus 2-to-1 worldwide. I think that trend will continue.

'Jazz Jackrabbit' (1994). Epic Games


Any software that requires human communication is completely dysfunctional if it's locked to a platform. And everything in VR and AR will be socially centric. Communicating with other people is an integral part of the experience.


Look at the top mobile communication apps. They're all multiplatform: Instagram, Facebook. They are only useful if you can communicate with all of your friends. If Apple had the most awesome chat platform ever, but it only ran on Apple devices, guess what? It wouldn't succeed. Because half of your friends have Android devices.


But right now, Epic is making a closed-platform game, Robo Recall, for Oculus.


Yes. It's funded by Oculus. It was a budget that could never be funded just on the basis of sales. So that enabled us to do some cool things. My view is that the Oculus store, which is an awesome store, should run on all PC and VR devices.


Oculus would do best if they tried to bring users into their store by supporting HTC Vive and Oculus Rift and any other PC hardware that comes out. I think if they don't do that, they're going to pretty quickly fail, because you're not going to want to buy a multiplayer game that you can't play with half of your VR friends.


Have you been disappointed by the sales of VR so far?


It's pretty much what we expected – a little more than half a million PC units; I don't know the PSVR numbers; several million smartphone VR units. That's a good number to start with.


The very first year of the personal computer revolution, when the Apple II shipped, there were 23,000 computers sold. I think VR is going to build up similarly slowly, starting with this early adopter, hardcore gamer base. It's going to be nurtured there for years before it goes really mainstream.


I think the mainstream version of this product does have to be like sunglasses. The helmet version is for serious people. The entirety of the console market – all console gamers are potential VR buyers – that's 100 million people, maybe, in the U.S. and Europe. And then there's 100 million, maybe, in Asia who are not console gamers but do play hardcore PC games like League of Legends. So there's 200 million helmet-wearing VR users worldwide. I think that's where that market tops out.


I think the market will grow by a factor of two or three or four, every year, until it goes from this current 500,000 to 200 million.


And then it really starts to go on a larger exponential growth curve, and reach everybody, including mainstream consumers, when it's in the AR form factor.


At Oculus, John Carmack is focused on mobile VR. But you think PC is going win in the short-term?


Carmack and I, we almost always agree on these core premises about the industry. But he has a very different view than I do. I think his view is that the convenience of mobile will more than make up for its shortcomings in graphics fidelity. My view is that creating a convincing sense of realism is so important that mobile VR won't be able to take off until mobile hardware has really improved considerably.


So far, the market share numbers would suggest that mobile VR is winning by a landslide. You have several times more Samsung Gear VR units out in the market than Oculus Rifts. But if you look at software sales, they tell the opposite story. Software revenue per user is at least 10 times higher on the PC platforms than on the smartphone platforms. It's so much higher that even though the mobile user base is so much larger, there's still more money to be made in the PC and console business.

I think that squares with our view that VR will be a high-end platform: higher fidelity, higher immersion than any platform before it, and so it will be dominated by high-quality games with high-end graphics. Which is the opposite of mobile.


We'll see. It will take a decade for that to play out.


Ten years is a long time. What was the world like in 2006? There was no iPhone.


That's right! No iPhone! What did we do?


Speaking of which, I read Anna Anthropy's book about ZZT, your first game, which came out in 1991. She talked about the age of shareware, when you would mail people disks with the promise of the full game after they sent in money. It was the same business model as free-to-play on the iPhone.


Exactly. You give people something and they love it, they're going to spend money on more. We've been in that business, I guess, for 25 years now.


All of Epic's early games were broken up into three episodes. We gave out the first episode of the game for free. It would end with an advertisement for the subsequent ones and ask you to send in your money by mailing a check. And we would copy a floppy disk and mail you a floppy disk. That's how Epic's early business developed.


That got to a pretty decent scale. By the time we were shipping Epic Pinball, Epic overall was making $50,000 to $100,000 a month. That was enough to fund our early development efforts of Unreal.


With the Xbox 360, Epic decided to bet the company on consoles. Now you're back to developing for PCs. What changed?


It was a funny transition. We built PC games for all of Epic's history: Jazz JackrabbitJill of the JungleEpic Pinball, and then Unreal. But before Steam, PC game revenue started falling off a cliff, right around the time BitTorrent and Napster came out. Piracy became really easy and accessible with broadband Internet. Our competitors who were shipping games were almost going out of business, because for every copy they sold there would be like 10 pirated copies. As a result, PC was not growing as a viable platform for the scale of game that we wanted to build. We wanted to build games like Gears of War.


So we started to work on Xbox 360 and focused on console for that whole time frame, from 2006 to 2012. That was a good decision. We were able to make games on console that sold vastly more than they could have sold on PC. Even now, Gears of War, several games in the franchise sold more than 6 million copies. I don't think there's a single game on Steam that sold 6 million copies.


No game on Steam sold 6 million copies back then, or now?


Now? Good question. It's unclear.


A funny thing happened in the console market, though. Budgets were being bid up. The first Gears of War cost $12 million for us to make. And it made us $100 million in profit. So that was awesome. But by Gears of War: Judgment, the game cost about $60 million to build, and made about $100 million still.

'Unreal' (1998). Epic Games


We saw as you moved to this new console generation, the Xbox One and PlayStation 4, we could expect our costs to double again. And the user base wasn't going to double. It was going to go back to zero and then have to rebuild. We felt we would be of questionable viability as a standalone developer in Triple A.


But do you think the PC will always be the main device for gaming? History suggests that computing keeps spreading into new machines – our phones, boxes under our TVs – rather than remaining centralized in a home PC.

I was reading Neal Stephenson's Snow Crash, and one of the funny things about that is a brilliant sci-fi author envisions what the future of VR will be like – 40 years into the future, from the time when he wrote it – but the thing is, when they're going to coordinate how they're going to meet together in the metaverse, they run to a phone booth and call the other person. He didn't think of smartphones at that point in time. I think we're probably making similar mistakes to that, when we're trying to project current experiences with games into VR. Some of our ideas will work, and some will fail in hilarious ways.


I think it's a very transitory period we're in right now. Smartphones, because they're so much more convenient than the PC, have grown enormously. But they've also meant a major step back in graphics quality. Because instead of dealing with a big screen filling most of your field of view, you're dealing with a tiny screen and a tiny keyboard and no controller. And so it's really been a step backward for gaming in a lot of ways, even though it's a bigger platform than anything before it. It's led to a downward trend in game innovation, game size, game depth.


I think that's going to be completely reversed as you move to augmented reality. When your smartphone is replaced by these glasses, that can provide a more immersive experience, then you're going to go back to serious, in-depth, awesome games.


What do you think of augmented reality?

AR is a much harder technology problem because you have to merge your actual view of the real world together with computer objects, which is easy to do to a rough approximation as HoloLens does. The end goal will be to display a seamless merging of computer images with the real world.


I think it's going to be years before AR really takes off – maybe five, six, seven. We're not talking a lifetime. VR is going to lead the way for the next several years, and the key innovations are going to happen in VR before the AR revolution starts.


The device for a billion users is unquestionably AR. That's the mass-market consumer product that is analogous to the iPhone today. VR isn't that. VR is like a super hardcore, badass PC.

Before we get to sunglasses, how long until we reach a PlayStation 4 size audience for VR?

I think you need one or two hardware generations of iteration to get to that point. The funny thing to realize about VR is that not a single unique hardware component has been designed for VR yet and shipped. All of the displays in HTC Vive, Oculus, and Sony's platform are smartphone displays that have been repurposed for VR. All of the cameras are smartphone sensors. We're reusing a lot of components that have been designed for some other purpose. It turns out they're not nearly optimal for VR.


I think as we see the next generation of VR hardware coming out, and the generation after that, you're going to see these custom, purpose-built components for VR that dramatically improve the experience. I think that's the point where you have hardware that's suitable for the market for 200 million gamer VR devices.


How often do you use VR?

Once a week or so. I don't have a dedicated VR setup in my house. Mostly I'm playtesting things, or seeing what our partners are working on. Like I said, I've never really been much of a gamer. I have a clinical separation from it, like a doctor would have with a patient.


You have said that whoever controls this new platform will be more powerful than any currently existing government. Do you really believe that?

Yes. I bet in 20 years, we're going to live a very large fraction of our lives in the metaverse. Right now we're just typing stuff to each other in social media. Just imagine, if you telecommute, all of your work will be conducted through VR and AR. If there is one corporation that controls and accesses everybody's data stream, then they have complete insight into every aspect of everybody's lives.


That's really dangerous. That company, and any intelligence agencies and governments that it feeds into, will have the power to blackmail anybody. Because everybody has something to hide. Pervasive information collection is a really dangerous factor for a democracy.


That just makes it all the more important to ensure that this new medium is built in a decentralized way, where people can communicate with each other without surveillance, using technologies like end-to-end encryption. And that it's open to basically any company that would want to participate, just like the web is today. Any company can run a website. Any company should be able to host their part of the metaverse, so they can control that exclusively and nobody else can dictate terms to them. That decentralization is going to be the key to keeping it robust and secure and preserving everybody's rights.


This interview has been edited and condensed.

Related articles

VRrOOm Wechat