The current standards for Virtual Reality videos and livestreams usually include a person walking around, waving their arms aimlessly, with a shaky first-person view from the player’s head. The main issue with this perspective is that first, the head is a terrible camera, and second, translating the concept that the two floating controllers represent the player’s hands isn’t as intuitive as you’d think.
My objective is to fix that with Mixed Reality videos; so, what are Mixed Reality videos?
MR videos are essentially combining the “spectator cam” from VR games with a greenscreen video of the player. The end result is something quite amazing:
Mixed Reality videos make VR games a lot more fun to watch. In a way, watching people play games can be more fun than playing them yourself. There are thousands of “Let’s Play” style videos where the player records their gameplay and commentary for the audience to interact with. This is such a successful model that the most famous Youtuber, Pewdiepie has earned anestimated $124 million since 2010. The bottom line is, people like to engage with personalities and feel a sense of friendship and connection, no matter how minute that might be.
Virtual Reality games are great at conveying players’ emotions, so seeing them physically move and dodge conveys the largest amount of implicit emotion.
For gamers, the ability to save, share, and stream videos from within a game have catapulted indie games into the limelight through the nature of viral videos. Goat Simulator, a parody game, rose to an incredibly high popularity because of the amount of video content that game generated through the streaming community. Silly games make for silly experiences to watch.
Experimental Filmmaking / Instant CG
James Cameron’s Avatar was groundbreaking for a multitude of reasons, however one technology that was developed during the course of the film was called SIMUL-CAM. Since most of the film was shot in greenscreen studios, the actors and directors needed a better way of visualizing what the final CG render would look like. What they developed was a 3D tracked camera that had a live feed of what the CG + real actor renders looked like.
This key piece of technology is now available to anyone with an HTC Vive.
What’s still needed:
Currently, only Unity-based HTC Vive games support Mixed Reality mode, but we’ve heard promises for support from both Epic’s Unreal Engine, as well as games on Oculus Rift. The fact that Mixed Reality mode exists by default in any Unity Vive experience is something that most developers don’t know about. It works….most of the time, but here are a few things that are still missing…:
- Toggle to turn MR mode on/off
- An easier way of calibrating VR and Camera feeds to match the controllers.
- Assigning the 3rd controller as the MR camera for good
- Support for multiple MR camera rigs
- Fixing the First Person View mode (and adding smoothing) in most Unity games to have a different viewpoint to transition to. (some games have it optimized, most don’t)
- Rending the MR comp in engine, rather rendering it multiple times for post composition.
- A model representing the camera model (to not bump into it), as well as a way to see the FOV and framing of the camera to keep yourself in the shot.
- A real time feed of the MR comp while in VR.
- A way to zoom! A real time FOV adjuster of the virtual camera to match the camera’s zoom.
- A way to read Facebook Live comments in VR
Want to shoot Mixed Reality footage of your experience? Checklist:
- You’re using the SteamVR Camera prefab
- Script your shots, and direct your actors. If the talent isn’t emotive, then the experience doesn’t look as “real”
- If controller models don’t need to be there, have an option to toggle them off (if they’re not guns).
- If you have an HMD model in your game, it’ most likely should be toggled off as well.
- EXTRA PRO TIP: have a way to rotate the world around the player to set up shots without having to recalibrate the room and change the front/back settings. Pinch to Zoom/Scale works great for some shots(See Tiltbrush, or City VR above), but having a keyboard command to rotate the environment would make things much easier!
Video tutorial: https://www.youtube.com/watch?v=IZI7-3rK-X4
Text Guides to setting up an MR studio:
High Quality MR Trailer writeup: http://www.kertgartner.com/making-mixed-reality-vr-trailers-and-videos/
Behind the Scenes of Making an MR Trailer:https://www.youtube.com/watch?v=b8bSeLca130