The AAA games industry has a clear interest in adopting more cinematic techniques. The most lauded titles of recent years — God of War (2018) and Red Dead Redemption 2 are among them — all seem like interactive movies in some respect. Many people point to aesthetics, like the continuous shot that frames Kratos, as clear examples of “cinematic” approaches to game design. And they are right to do so.
But the truth is that modern AAA production itself sometimes bears an even stronger, and less discussed, resemblance to cinema. Such is the case of New Tales from the Borderlands. Digital Trends sat down with James Lopez, Gearbox Software’s head of production, and Amelie Brouillette, Gearbox Quebec’s associate producer, to dig deeper into the team’s first attempt at performance capture — a modern technique that adds filmic detail to the already immersive world of Borderlands.
Note: This interview has been condensed and edited for length and clarity.
James Lopez: A really great way [to add weight and emotion to character interactions is] with performance capture, which is a much more improved version of mocap [motion capture]. So whereas mocap is body movements — but that’s it — performance capture takes the body movement and also the face of the actor and the words that they’re saying as they’re [speaking] — so the entire performance. That’s where [the name] comes from. And so it’s a lot more intense than doing mocap.
Digital Trends: With The Last Of Us Part I, we saw Naughty Dog utilize modern technology to bring new levels of emotional nuance out of its characters. But that game is predicated on photorealism. What value is there to performance capture in an experience more so about comical expression, like Borderlands?
Lopez: With something like The Last of Us, that sort of performance capture is the obvious choice, especially because it’s such an emotional game because it’s already photorealistic. Even with all the other stuff they have in there, it’s still grounded in photorealism. So everything looks natural, everything looks human. Even the monsters look human in their own way. So it’s a great choice.
Something that we definitely were a little concerned about with New Tales is that the Borderlands art style itself is ageless because of the sort of comic book art style. It ages really well. But it also is not photorealistic.
The franchise originally was! With Borderlands 1, one of the earlier versions of it that was never released went photorealistic, and what we found was that just didn’t really work for the attitude we were trying to convey and [for] selling the characters. So we went more stylistic … Then with the previous Tales title, a lot of that emotion that you see is because someone spent dozens of hours handkeying the big sequences to kind of sell that.
And we knew that we couldn’t do it that way [for New Tales], but we did want to sell that emotion. And so for us, performance capture was a sort of interesting challenge because we knew that it was going to be a key to the success of the story, but also our art style initially didn’t quite lend to that. So we kind of had to make some nuanced changes to support that.
Amelie Brouillette: Usually, when you look at the characters of the Borderlands series, they have [exaggerated] proportions, like big legs and small torsos. We play with that. That’s really useful for gameplay because it [sets us apart] from other games. Well, for [performance capture], we had a challenge because we have to follow the proportions of the actors. So we changed our rigs to adapt to different actors. For example, [one character was] supposed to be smaller, but the actor was so good we had to change the whole character. We were, of course, inspired by the first version. But we adapted his height.
To give people a better sense of how performance capture shapes design and informs production, what were some key takeaways from learning how to use this technology?
Brouillette: It seems so simple, but yet it is not. For example, there is a scene where there are monsters and aliens. We have technical guys running around with a pool noodle on set to fake its tentacles, and we had to build stalagmites and stalactites from wood to be sure that their interactions were OK. And I think for that scene, we shot the whole day with seven, eight actors – our maximum allowed for COVID.
So for about five minutes of the game, we shot eight hours with multiple [variations] … we had a Tediore Guard, and we shot them walking, running, dying multiple times, and we multiplied that for all the Tediore Guards that are involved in that sequence. That’s the first takeaway.
The second one? Everything can be done. We had stunts on set. Usually, we just decided what was most cost-efficient and, of course, what was safe, but we could mocap everything. It just depended on what we wanted to do, and that’s pretty interesting and [exciting] for the team to have this possibility with that technology.
Lopez: I’ll also add that one of the strengths about performance capture is that once you have what you want to shoot and once you’ve done the prep, it’s much faster in general to shoot with performance capture than doing motion capture and adding everything else in afterward — because you do like 80% of it in one swoop, which can save a lot of time.
But the trick is that you have to lock your script really early for the shoot because for one scene, there’s probably a couple of months of preparation time. I get a script from our writing team to Amelie, who needs time to get the concept artist to create storyboards that help visualize where things are.
And what that does is it gives the actors an idea of what we’re going for because they see motion on the storyboard. They go, “Oh, OK, this is where they’re angry, or this is where they’re desperate.” But this is also valuable information for the stage crew, who are marking the floor with these little different-colored pieces of tape that tell them where a table or wall is because there’s not really that many props on stage. You’re trying to capture the actors from all angles.
And so, we’re [working] off the storyboard, but we’re also making sure that we have all these other moving parts ready so that when the actors show up to shoot, they can just shoot, which is still, like Amelie mentioned, several takes in the same day, sometimes across multiple days or even a week.
So it can be really difficult to adjust the story once you’re in the shoot phase because you need lead time for it. We’re doing focus testing while we’re shooting and hoping it tests well, because we don’t really have time to go back and change that. Also, these are working actors, and we don’t own the stage. So both the stage and the actors are also trying to make sure that they’re staying as busy as possible when we don’t need them. And then we’re like, “Heeeeyyyy, can we use you for another week?” And they’re like, “Oh, I’ve already booked another thing,” and we’re like, “Cool. Neat.”
So you need a considerable amount of lead time on your story. I’ll say that in the industry at large, we’re not really great at locking stories a year-plus in advance for this sort of thing. So there’s a lot of learning there to be had.
Brouillette: This is not how we usually work because there is a lot of back-and-forth, and we improve the story along the way. But as you said, when we are on shoot, we’re kind of printing the book that’s going to be on the shelf. We could adapt, but our options are very limited.
There’s something cinematic about how performance capture directs game production to be more like a movie set. Despite its challenges, was this new creative approach personally enjoyable?
Brouillette: Well, absolutely! For me, it was kind of the cherry on top of everything … Before going on set, I have to work so much and really intensely to look at all the details and ask the team everything. So once I’m on set working with super-great actors and a technical team, that’s absolutely a joy. And also, there is the rush of being live, [having] to be efficient with our time, as James said.
We had very little time, sometimes, to shoot some things [and we’d say], “OK guys, let’s work together!” … And we were, I don’t know, maybe 20 people on set all the time trying to build something on time with everything in it. It was really, really awesome to have the chance to do that.
Lopez: I have kind of the cheap seats on this because I got to observe a new process, but I personally did not have to solve [its problems]. So I got to show up and watch people rehearse or watch people set the stage and go, “Oh, that’s really interesting. That’s how they do that.” And I go, “Alright, good luck Amelie, have a good shoot. I’m out of here!” Because I’m kind of a jerk. [laughs]
But it’s always definitely neat to see new things. We’re trying to keep up with the times. There are plenty of games that will never need performance capture, but especially as the technology keeps getting more realistic and the consoles become capable of so much more data [being displayed] on screen, I think performance capture is going to get bigger and bigger. And so I’m glad that we’re keeping up with all that because it would be really easy for it to get away from us.
From the player’s perspective, it’s exciting to see how this immersive technology can deepen our connections to narrative storytelling in games. Do you anticipate performance capture becoming a staple technique for Gearbox Software?
Lopez: The Gearbox motto, internally anyway, is that we’re here to entertain the world. And I think we’ve done a pretty good job until now. But part of that is understanding what the world is into. How can we represent that in our art and how can we reach out further? … We have a lot of story-driven games and we’re always looking for the best way to tell those stories and to make sure [we’re] not just entertaining, but also moving our players. So I think this will be around for a while.
As we wrap up, are there any stories from the set that you’d like to share with the Digital Trends audience?
Lopez: When you’re making a game, you start off with this really big-picture view of the whole thing. But eventually, as a project producer, once you’re handing that off to the team to make it, you’re always kind of maintaining this big-picture view, but it also means you start to lose sight of the smaller stuff. And some of that can be spontaneous takes that happen onstage.
And so, there was some ad-libbing that I saw in the final product that I was not aware of, that I was like, “Oh, that’s really fun.” Because I don’t remember that in the script, and it was like, “Oh yeah, the cinematics director just ad-libbed some of that on stage.”
I love that there’s always room for a little extra creativity and trying to get a little extra fun out there … There’s a scene where these Tediore soldiers are trying to follow one of the other soldiers, and they can’t remember his name. And so they shout out a couple of names thinking like, “I don’t know, is it Doug? Is it Anthony? And they call him Doug-Anthony.” And that’s a joke to some of the writers that were on the project, because one of the lead writer’s names is Doug. One of the other writers’ names is Anthony. And so it’s just this sort of ad-lib nonsense that was definitely not in the script, but is much better than what we had in the script.
Brouillette: I have this same example of the Tediore guys. We worked with mostly the same actors for the Tediores. That’s kind of the point of the joke, like they die several times. They [had] very good ideas … So sometimes they’re like, “Hey sorry, Mr. Writer, I have an idea! What do you think about that?” [and the writer answers], “Yeah, Good! Cool! I’m already adding that to the script.”
So last-minute changes before the shoot? Yeah, really tough. But during the shoot, if that doesn’t touch the continuity of the story, we could have a lot of fun. And the actors really enjoyed that opportunity to be a little bit more free than usual.
New Tales from the Borderlands is now available for all major platforms, including PlayStation 5, Xbox Series X, and the Nintendo Switch.
