Last week, Zalando held its annual Hack Week which as you might guess is a week long hackathon during which teams can brainstorm and work on innovative ideas that could be beneficial for the company. This year I went for something totally geeky compared to last year which had me working on a refugee related-project. The name of it is VRify.
VRify’s purpose is to create an immersive conference call experience that allows you to communicate with your remote colleagues more efficiently. It brings geographically dispersed team-members together by utilizing cutting edge VR and AR technology to create virtual meetings with realistic 3D avatars and real-time audio. For the scope of this article, we will only focus on the 3D avatar part which from now on we will refer to as holograms. So how can someone do that?
First things first, we need to digitize people somehow and this is possible using 3D scanning. The scans produced will be the starting point for our holograms. Using a special sensor called Structure and an iPad we went scanning people and create 3D models out of them. Apart from the hardware we also needed some software for that. After running some experiments with Skanect we decided to go for itSeez3D for it’s easy of use and quality.
The scanning process looked like this
Scanning takes almost 1 minute
and as you can see the results were stunning.
Fully textured model
Filters → Normals, Curvature and Orientation → Compute Face Normals
and then re-export the .obj. At this point, we are ready to import into Unity and have a perfect looking model.
Our 3D model in Unity
Having the model in Unity is the moment when we start building our virtual experience. To make things a bit more impressive, though, we will be doing it with Augmented Reality putting our 3D models into the real, physical world as holograms using Microsoft HoloLens.
HoloLens is an AR or Mixed Reality if you prefer, set of goggles that allows you to pin and see virtual holograms or 2D UWP apps into real space. Out of the many cool things that it can do are:
- * View your favorite TV shows or movies on a virtual big ass screen
- * or even better play Super Mario on the wall using an Xbox One S controller.
But in our case, we will use its power for the sake of productivity. In order to develop applications for HoloLens you first need to install the tools and then configure Unity and Visual Studio. We managed to build our own HoloLens app called ARify and pin and project our previously scanned 3D model in the real world as a hologram. The result? Almost scary…
Our real life person with his new twin brother
Walking around the hologram
Turns out we managed to bring the 3D scanning quality into HoloLens quite nicely. Resolution was adequate and the overall feeling quite realistic. The next step would be to try to rig and animate the model which we also managed to do.
Model animation (without textures applied)
We also managed to get real-time audio going on between two users of the app. We also started looking into animating lips and audio-lip syncing. However, as I wrote in the beginning, this part is beyond the scope of this article.
The project had been worked on, in the context of Zalando’s Hack Week. We made a first step towards building a real-time holographic conference experience. The results surpassed our expectations, super-ignited our excitement and definitely demonstrated how nicely 3D scanning and AR technologies play together.