Kiwi Company Aims To Recreate The Entire Planet In VR

Category: 
Kiwi Company Aims To Recreate The Entire Planet In VR
May 13, 2020
VR Egypt: the famous Tutankhamun mask is one of the things Reality Virtual has re-created in virtual reality.
 PHOTO BY CHRISTIAN WAGNER ON UNSPLASH

 

A New Zealand company wants to recreate the planet in virtual reality. They plan to enlist you and me to help them do it. And, they just got a major grant from Epic Games, makers of the hugely popular game Fortnite, to help them do it.

 

At least they’re not dreaming small.

 

That’s a major task, and huge, well-funded global giants like Google are already working on it. So they’re using AI to do the heavy lifting, or at least most of it, and a crowdsourced approach to getting the required images and data.

 

“Basically we’re looking at open sourcing mass adoption of photographs from parties on the ground,” Reality Virtual’s chief creative visionary officer Simon Che de Boer told me in a recent TechFirst podcast. “And essentially automating the process mostly for AI so we can actually quickly create these environments successively, without the kind of huge overhead and the huge amount of labor involved.”

 

de Boer has already done this for global landmarks like the Egyptian resting place of Queen Nefertari.

 

He took about 4,000 photographs for the Nefertari project, and it took six weeks for a team of three to turn them into a VR experience. (“It was just crazy ... five AK-47 guys above guarding the place and me just in this amazing tomb for eight hours taking the photos.”) Tutankhamun — the famous King Tut — was next, followed by The Homestead, an art gallery in New Zealand, the country’s Parliament House debating chamber, and the Large Hadron Collider.

 

Clearly, if you have your eyes set on digitizing the whole world, you can’t spend six weeks on just a few rooms.

 

Especially since most volumetric video — video with dimensional data so you can move around inside the stream — can run to gigabytes of data per minute.

 

Without artificial intelligence and smart data compression, it wouldn’t be be possible at all.

 

So Reality Virtual is working on AI to automate “retopology,” the process of converting high-resolution photos into a mesh of polygons the company can create a 3-D experience with. When complete, that will reduce the human labor required by a factor of six. Then there are a variety of post-processing requirements to get lighting right. Everything is detailed, processor-intensive, and at the edge of what’s possible.

 

“Most of these processes ... we’re having to deal with 24 billion points of detail,” de Boer says.

Part of the Queen Nefertari VR experience.
 REALITY VIRTUAL

 

While AI is making that easier, it’s still not as quick as it needs to be. de Boer says the development team is still working on photogrammetry, the process of acquiring a point cloud from photographs and using GANS (generative adversarial networks) to get faster and faster.

 

A couple orders of magnitude of less work is probably closing in on what the company needs.

 

And while the company continues to work on that side, another critical piece is raw material for the AI: original photos of basically every place on the planet.

 

But not just photos of the outside of cities and landmarks, like Google has done for Google Earth VR. Reality Virtual needs inside major landmarks and buildings. And it needs higher quality photos with a greater level of detail.

 

And, of course, a rights management system for all those photos.

 

“We’re talking about like an artist’s rights management system,” de Boer says. “Kind of like I guess where you get paid royalties, like if you do an album you get paid a check occasionally for how much airplay you get. So we really want to get people on the ground, taking the photos and having their location and their geotag, all their EXIF data embedded into it. So if a studio, or a university, or polytechnic, or studio for virtual studio production, or any kind of educational facility wants to then use these environments, essentially the original artists who took the original photos actually gets a cut of it from the licensing.”

 

Achieving this might seem impossible, and, honestly, it probably is.

 

But open source projects have resulted in operating systems that run most of the world’s servers, most of the world’s phones, and the biggest and most extensive encyclopedia in the history of the world. So Reality Virtual might have a shot at it, if it can galvanize enough attention.

 

In the meantime, another outcome of Reality Virtual’s developing technology is real-time virtual reality for web conferences, a project the company calls DeepMirror.

 

Imagine Zoom, but you feel like you’re actually in the same space as the person you’re talking to. And while you are looking at a camera, not your colleagues’ eyes (and she’s doing the same) the software adjusts for camera and eye position automatically. In other words, you can achieve eye-to-eye contact in live, real-time, 3D conferencing.

 

This is interesting, although it requires VR for web conferencing

 

All of these projects, of course, will require funding. Which is why the Epic Games grant is so important.

 

de Boer can’t reveal the size of the Epic Games grant, but he did say “we can now afford a house in Aukland.” At a median house price of about $830,000 in New Zealand Dollars, that’s in and around $500,000 USD. It’s a major boost to a small studio that has been financing its technology development by doing customer projects, but it won’t be anywhere close to what Reality Virtual needs to digitize a city like Aukland, never mind a country or even the entire planet.

 

That cost, for data storage alone, would be multiple orders of magnitude more.

Related articles

VRrOOm Wechat