Read any extensive coverage of virtual reality and undoubtedly you’ll come across the word “immersion” at least once or twice. That’s a relatively vague term for the feeling of presence within the virtual world, but it’s as close as we’ll get to providing a singular point of reference for gauging the success of a VR developer in crafting a believable experience.
Most developers utilize similar technique to achieve immersion. They work within the bounds of existing virtual reality headsets and craft small-scale, high-fidelity scenes, with binaural audio. If they do a good enough job, you may find yourself wandering into the uncanny valley. It’s rare to climb the rise on the other side — yet that’s exactly what Awake from virtual reality production house Start VR is attempting to do.
Its roughly 20-minute first episode is a lightly interactive cinematic VR experience that delivers a captivating story around the theme of lucid dreaming. But as intriguing as the narrative is, it’s the volumetric capture technique used to craft it that is Awake‘s most striking element.
The end result is characters with incredibly realistic facial expressions, motions, and gestures. This is no ping-pong-ball motion capture. It’s something else entirely.
**Mild spoilers for Awake’s story may be present in the following interview.**
So long ping pong balls, hello holograms
“The technique that Awake was specifically written around is volumetric video,” explained creator and director Martin Taylor. “So it’s a new type of performance capture that takes the full actor’s performance including all of the wardrobe and the full expression and captures them as a filmic hologram.”
Volumetric capture wasn’t something Start VR could perform alone. It needed Microsoft’s help.
This is very different from the motion capture techniques used to craft most in-game characters for both virtual reality experiences and more traditional 2D games. While those use samples of movements to craft the 3D models that are skinned to bring virtual characters to life, volumetric capture records the entire person. Inside and out, so to speak.
“It’s using a special rig with 106 cameras with depth sensors all pointing towards a cylindrical volume in the center and up to two actors can stand in that volume and deliver their performances,” Taylor told Digital Trends. “All of these cameras and sensors capture that, and lots of processing and software stitches it together into a completely believable and solid digital copy of the performance.”
That gives the developers not only an accurate 3D representation of where those actors are and how they’re posed at any particular time, but a near photo-realistic video impression of them too.
“The raw version of the captures are a huge series of OBJ models which are all uniquely different from one another and then a video texture that’s wrapped over the top,” Taylor said.
Early pioneers in its usage, volumetric capture wasn’t something Start VR could perform alone. It needed Microsoft’s help.
The right dance partner
In one of the more uplifting segments from Awake: Episode One, our protagonist, Harry, shares a dance with his partner with the image splashed across the in-game wall as if from some unseen projector. Just as those characters found each other within the game’s universe, Start VR required a specific partner to help create the volumetric performances that form the centerpiece of Awake’s experience.
“We did some very crude early tests with this technology with a Microsoft Kinect and a Canon 7D camera joined together and the results were comically crude,” Taylor explained. “But there was just something in the experience of being inside VR and knowing you’re looking at a real person […] It took you that bit closer to getting across the uncanny valley to give you a very connected experience. It was a really exciting moment and very early on in our business.”
Initially Start VR looked into developing its own volumetric capture rig, but the complexities of managing such a facility, as well as the up front cost involved in acquiring and configuring so many cameras and depth sensors was too much to consider.
“I thought there was no way that we would be able to pull this off without this level of grunt.”
The team spent a couple of years looking at local suppliers and studios with their own volumetric capture set ups. None of them could do what Start VR wanted, which was to not only record performances, but be able to re-light them after recording. One of the most unique aspects of Awake is that the player, although limited in interactive possibilities, can release small motes of light from their virtual hands, which can enlighten shadowed areas of the experience.
The one company who would make this possible, as well as deliver the kind of motion capture system Start VR needed, was Microsoft.
“It wasn’t until we turned up at their studio that we realized what was required to make this work,” Taylor continued. “The amount of computers, raw processing power, hardware and infrastructure made us think. Immediately I thought there was no way that we would be able to pull this off without this level of grunt.”
From stars to screens
Once the actors’ performances have been captured, bringing them into the game environment was no simple feat. Start VR blazed a trail in software as well as hardware.
“The other big challenge was [building] the experience in Unreal engine,” Taylor explained. “The plugin didn’t exist to accept dynamic lighting. We had to do basic functions like create an in point and an out point. If two characters were filmed in the same volume we had to split them apart and put them on different sides of the room.”
“We’re definitely pushing the engine to do things it doesn’t natively do. We’re working on it daily.”
Once that was solved and the experience was starting to come together, the next big problem was optimization. Start VR worked closely with Nvidia to get the experience running smoothly on a GTX 1080 Ti graphics card and then began to work its way down to see what kind of cuts could be made to allow the experience to run on weaker hardware.
Awake currently offers two modes of play. The high fidelity option is by far the more enjoyable experience, though Start VR suggests that a GTX 1080 as a minimum required specification. We managed to get it working on an AMD Radeon Fury X, but frame rates did stutter at times. The standard option is much more forgiving, but the visuals do take a significant hit.
This may not always be the case with this sort of technology, though. Start VR CEO, Kain Tietzel told Digital Trends that he believes Nvidia may leverage bespoke hardware (perhaps akin to the Tensor and RT cores found in its RTX graphics cards) to offer improved performance for volumetric capture video.
“No one has really written anything specific to handle this kind of rendering,” he said. “As demand increases and the production starts to become more common place I imagine we’ll start seeing Nvidia having dedicated hardware acceleration to make the performances greater and more realistic.”
Install size was also a major concern for Start VR.
“The raw capture is 10GB per second of raw volumetric capture so there’s a lot of work that goes into squeezing 15 minutes into a 10GB [download],” Taylor said. “It’s using the latest and greatest of all the plugins within Unreal Engine. We’re definitely pushing the engine to do things it doesn’t natively do. We’re working on it daily.”
You’re not a participant in this world. You’re a voyeur. For Start VR, that was the plan all along.
Start VR could opt for even higher fidelity models than it uses now. Earlier implementations of volumetric capture like Factory 42’s “Hold the World“ educational experience hosted by David Attenborough, look better than Awake in some senses. That’s because they are delivering a mere three degrees of motion virtual reality experience. They’re also focusing all of the attention on the face and upper body.
“You could push the polygon count we allow [in Awake] just into the face and then it becomes super high resolution,” Taylor explained. “There are some tradeoffs we’re choosing there. We’ve chosen to go for entire body captures because we want people to be able to see each part of the character, walk around them. I’ve seen people rolling around on the floor to see if the characters are persistent.”
Playing with holograms
After all the effort and special consideration needed to craft Awake’s volumetric experience, the end result is something very different from what most gamers will be used to. The “texture” of the characters vary from extremely detailed on their faces, to roughshod and blurred in harder to reach areas. The mix of extremely realistic motion and somewhat blocky models reminded us of Rockstar’s 2011 release, LA Noire with its unique visual style. Or as if someone had wrapped a UHD video over an original Xbox character’s shell.
It’s far from perfect, with the technology struggling in particular with gaps between fingers and hairs, and a slightly blocky outline of the characters. But it’s captivating in a manner that is rare to find in virtual or any other kind of simulated reality experience, and it holds huge potential for a different kind of interactive experience.
While screenshots of Awake may have an uncanny valley vibe to them, when you’re actually in Awake’s world, there’s something about the volumetric capture that hurdles that depression entirely. The actors don’t look perfect, but they look real, and that’s something that cannot be said for even the best motion captured CGI models.
“Even understanding how we light and direct performances will change as well.”
That immersion in the virtual world is also somehow enhanced by forcing you to take a step back from it. You’re not a participant in this world. You’re a voyeur. For Start VR, that was the plan all along.
“[Other filmmakers] always feel compelled to have the performers in the scene look at you, the camera, and I don’t know if that necessarily works all the time,” Tietzel told Digital Trends. “If you’re constantly treated like you have to be looked at I think it changes the nature of the performance and I think it’s rare that you actually see believable performances acting to a camera and you’re meant to react to them.”
That’s not to say that you never feel involved in the scene in Awake. There are moments, specifically crafted ones, where you end up in the line of sight of a character, giving you jolts of wonder as to whether they can see you or not. It doesn’t pull you out of the experience, but reminds you of your voyeuristic roll in the scene. If anything, it enhances how personal the story is, despite its hints at a more expansive narrative beyond this first episode.
(Lucid) dreaming of the future
Start VR has eight episodes planned out for the Awake series already, with Taylor having already spent years crafting a story that is both personal and expansive. As those episodes roll out we can expect lots of improvements, we’re told.
“The technology, the accuracy, the frame rate is going to change dramatically,” Tietzel told Digital Trends. “Even understanding how we light and direct performances – what’s the best way to get performances out of the actors in that space will change as well. This is a learning experience.”
“So there’s moments where the character reaches out to give you something. Do you feel compelled to try and take that off them?”
As well as expanding their own expertise with directing actors and VR game development, Start VR is also working with Microsoft to continue enhancing the volumetric capture technique to bring new wrinkles to the virtual canvas.
“Having the character notice you is one way to tell a story and already I think there are variations coming up where we can add in eyeline,” Taylor highlighted. “We’re working with Microsoft to control the character’s head movement to be able to turn and lock on to you up to about 30 degrees either side of the recorded position […] Also compelling people to do reactive things. So there’s moments where the character reaches out to give you something. Do you feel compelled to try and take that off them? It’s those different ways of connecting with a character rather than just looking at them that we’re trying.”
Start VR also plans to experiment with multiple environments (Awake: Episode One takes place almost exclusively in one. Maybe one and a half), but it’s beyond the scope of virtual reality which Taylor and Tietzel feel their experiments with volumetric capture will be most readily felt.
“We’re already exploring what it might mean to bring this kind of capture into augmented reality, and real life performances in the real world are the next step,” Tietzel said. “Early tests in this area have delivered some fascinating results.”
These experiences and Start VR’s first episode of Awake highlight the exciting potential of volumetric capture. They sidestep the performance hiccups of technologies like ray tracing for realistic in-game lighting and deliver something that is believable up close, even if it still has a few blemishes.
Awake: Episode One is an exciting first step in a narrative journey, but it’s an even greater leap for the technological enhancement that helped birth it into the world. We’re excited for what Awake’s story and the capture technologies it champions may evolve into, and imagine we’ll be seeing a lot more of them both in the future.
Awake: Episode One is available on Steam and Viveport with a price of $8.
