Making young Furiosa

How Rising Sun Pictures employed machine learning techniques to translate the look of Anya Taylor-Joy’s Furiosa to the actor playing her younger self, Alyla Browne, while preserving the original performance.

When audience members began seeing George Miller’s Furiosa, many very quickly observed that the young Furiosa (played by Alyla Browne) had a remarkably similar appearance to the older Furiosa (Anya Taylor-Joy). It was not just incredible casting, but in fact also a deliberate decision from the director to use the latest in visual effects and machine learning techniques to bridge the gap between the two ‘Furiosas’.

Effectively, Taylor-Joy’s facial characteristics were able to be translated onto Browne at different percentages as she aged, in a completely seamless way. What was important, however, was the preservation of Browne’s original performance, something that was completely kept in the final frames.

In press interviews around the film’s release Taylor-Joy has repeatedly praised the face replacement visual effects work. On ‘The Kelly Clarkson Show’, the actor said, “George Miller had the idea very early on. The audience was already getting used to a new Furiosa. He wanted the transition from both actors playing her to be seamless. It’s wild to see.”

The VFX work was spearheaded by Rising Sun Pictures (RSP) utilizing their REVIZE™ machine learning workflows. The team was led by visual effects supervisor Guido Wolter in collaboration with production visual effects supervisor Andrew Jackson. RSP produced approximately 150 shots of young Furiosa, some as standalone VFX scenes and others were outputs that were shared with other vendors.

Here’s how Rising Sun Pictures did it.

A hybrid

Rising Sun Pictures began by studying the facial features of Anya Taylor-Joy. They analyzed photographic references provided by the filmmakers of Taylor-Joy at a similar age to Browne. “We needed to create a hybrid of the two actresses,” says Wolter.

Creating the hybrid would be challenging for several reasons, not least of which were the different facial characteristics of each actor. “Taylor-Joy has different facial proportions than Browne,” notes Wolter. “To start, we simply took Taylor-Joy ’s face and put it on Browne’s face, which did not look right for many reasons–it was an adult face on a kid. But this was just the beginning.”

Wolter and his team then imagined the translation to be more about a change over time. “We treated it like a virtual slider. We would try to find the sweet spot as to where the hybrid should be. We found that it worked at around 70 to 80% Taylor-Joy, and 20 to 30% of Browne, and that it would change as Furiosa grew older.”

For Rising Sun Pictures’ REVIZE™ process to work effectively, the visual effects studio needed to train a machine learning model that would produce their hybrid of young Furiosa. RSP was provided access to Furiosa rushes featuring footage of Taylor-Joy and Browne, in addition to controlled capture data. This was acquired using a multi-camera rig and varying lighting situations tailored to match the on-set conditions.

Terabytes of data: the models

Using the supplied footage of Taylor-Joy and Browne, RSP began the lengthy process of training a machine learning model to create the hybrid. Data management was key, says Wolter. “Over the past few years, we have been building tools to make data management more efficient and distributable. It’s not a hurdle to be underestimated, because you end up with terabytes of data.”

Wolter continues. “We intermittently looked at how the model was progressing so we could direct it to a specific end goal. This could be achieved through several approaches, including data curation and filtering. We built a broad base and then started specializing for per shot application.”

In addition to the hybrid model, RSP created other utility models for artists so creative iterations could be made throughout the process. Some of these included enhancing skin detail for surfacing, face segmentation for localized mattes and the ability to correct eye direction. “At the end of the day,” says Wolter, “it’s a stack of models that we combine together to achieve a desired outcome.”

During the training process, RSP found that developing a model that accounted for the difference in the distance between Taylor-Joy and Browne’s eyes was particularly tricky. “When you start moving eyes, the eyelines and convergence points change,” explains Wolter. “If I look at an object close by, but then change the distance between my eyes but my eyeline stays the same, I’m now looking at something that’s a meter or so behind or in front. Your convergence changes because your interocular changes. So, we had to develop tools to deal with this challenge at pixel precision. We created models that gave our artists the ability to alter that aspect and have creative control.”

Wolter says an initial challenge machine learning artists encountered, was how to creatively direct the model. “How do we have creative input and build intuition on a process utilising a model that is essentially a black box? How do the artists change their problem-solving approaches to compliment the model? We solved these challenges by developing a number of tricks to guide the process. It’s never just as simple as pushing a button, there’s a real creative, collaborative and problem-solving aspect to machine learning.”

Alyla Browne as Young Furiosa in Warner Bros. Pictures’ action adventure “FURIOSA: A MAD MAX SAGA,” a Warner Bros. Pictures release. Photo courtesy of Warner Bros. Pictures Using the model

Combining the machine learning models of the two actresses to achieve a result that was both natural and convincing took considerable refinement.

Wolter advises that the machine learning output could be delivered to compositing as either a baked image output, or as a model that could be used natively in Nuke via the inference node. “We wanted Browne’s performance to be completely reflected in our results. In Nuke we could always look back to the scan and compare it. In fact, the objective was for George Miller and the studio to see if we could one-to-one transpose Browne ‘s emotions onto our hybrid model and support the narrative. Machine learning was not used to cut corners, it was used to enhance the storytelling.”

RSP worked hard to get more functional control into the hand of their artists. “It’s one thing to create an output from machine learning alone, but it’s just as important to distribute that output to a large team of artists and enable them to apply their artistry to the final product,” explains 2D machine learning supervisor Rob Beveridge. “For this project, we built tools that allowed artists to tilt the balance one way or the other, a little more of Taylor-Joy, or a little more of Browne, depending on the shot.”

There is more to the process than waiting for algorithms to churn out results. “Varieties of human artistry are required to bring the illusion to life,” observes compositing supervisor Darwin Go. “It still needs to be assembled and composited. It takes artists with diverse skill sets, some who are versed in the subtleties of performance others who have mastered the nuances of lighting. We had a really strong team and delegated tasks with some artists focusing on facial features, others on closeups, others on details of performance. If it was a dramatic moment, we’d have artists making sure that the head aligned to the body movement in a natural way or that the motion blur was correct.”

A lot of refinement occurred directly in Nuke. “Using the hybrid,” says Wolter, “we were able to reveal and peel back details from the plate that you couldn’t do on a normal face replacement. If you bring too much detail back from the plate, you’ll see more Browne. You want to make sure to keep the established look of the hybrid intact. Guiding that creative process was a tough nut to crack considering the volume of shots.”

A successful young Furiosa, right from the beginning

The film essentially opens with a scene featuring young Furiosa. As the camera zooms down into the place of abundance, you see her picking a peach from a tree. “That is one of my favourite shots,” states Wolter. “These shots were the first ones we received and focused on. My other highlight was when young Furiosa is at the meeting with Immortan Joe. That sequence turned out really well, she looks so angelic and equally irate in that sequence.”

“Alyla Browne is a wonderful actress, and it was essential to transpose 100 percent of her performance to our hybrid,” concludes Wolter. “Much of her emotion comes through nonverbal expressions and our team spent a lot of time making sure that we hit every single subtle cue. It was a formidable challenge both from a machine learning and storytelling point of view. I think we were very successful in getting our hybrid to exactly mirror the outstanding performance delivered by Alyla Browne.”

Subscribe (for FREE) to the VFX newsletter