Machine Learning: changing the game of Motion Capture

The use of Motion Capture (MoCap) is no anomaly in the world of VFX. For years, the technique has seen actors suiting up in professional spandex and painting dots on their faces, all to capture their expressions and movements so they can live on through their animated counterparts. But now, thanks to machine learning, this process may be set to change. 

As we have touched on before, machine learning teaches computer systems to make decisions from large datasets. The technique has proved itself to solve the more difficult problems of the VFX industry, helping artists to achieve results they couldn’t before in a faster and more effective way. 

With this in mind, it becomes easier to see why machine learning techniques are being applied to motion capture. Where previously MoCap involved more tedious tasks, like artists painting out head rigs, the use of machine learning promises to make the process more efficient and whilst it’s a relatively new concept, there are a number of VFX companies turning to machine learning and uncovering its possibilities.

Putting wheels in motion

One of the companies pioneering this use of machine learning, along with artificial intelligence, is DeepMotion. Together, these two strands of tech aim to solve the problems that can occur when using MoCap—like the extensive equipment which is often needed—instead advocating the use of markerless motion capture, allowing for intelligent and realistic animations. They also further enable more interactive character creation and the expansion of pipelines of animation and other simulation data.

RADiCAL is another company using machine learning and AI to improve motion capture. However, their focus is on detecting and reconstructing 3D human motion from 2D content, all without the use of the conventional MoCap suits and hardware. What’s more, the team is also making headway with the use of cloud in the motion capture sphere. 

By sending in and uploading a video of a person to their cloud system, RADiCAL can send back a motion-captured animation of the movements in the video. Now, of course, for more amateur content, this brings promises of realistic motion and animation creation. But for blockbusters or TV shows, this merely scratches the surface of what might be possible in the future. 

While the work of these companies is still in development, the possibilities they present for the VFX industry are monumental. Not only could it allow for more dynamic methods of motion capture, but also a reduction in the amount of equipment that is needed to do so. Plus, these higher resolution capture techniques and sophisticated processing algorithms are likely to increase the quality of work being produced and the predictability of output. 

Move like nobody's watching

As mentioned previously, while motion capture remains one of the more popular sources of motion data, it typically requires a vast amount of equipment. This is one of the main reasons there has been a shift over to markerless motion capture technology, with it even making an appearance in more high profile films like The Irishman and its use of deepfake technology. 

And even though this was a successful use of the technique, it required heavy-duty camera set-ups and laborious work from the VFX team. However, research coming out of the University of California explores the application of deep reinforcement learning as a way to capture motion data. This particular branch of machine learning focuses on training AI models to learn from past experiences and aims to diminish the intense use of equipment.

Using the concept, the research team discovered that AI models could learn from any number of reference materials they’ve been given, including more limited sources like YouTube videos. This opens up the possibility for faster and easier designs of characters, alongside a more efficient way to clean-up analog-style noise—all based on what the model learned about the way the human body moves.

In fact, the de-noising of more obscured MoCap data is becoming more widely used within the industry, with it being used to create Thanos in Avengers: Infinity War. As was clear for those eagle-eyed fans, the overall look of Thanos changes from when he is first seen in The Avengers end-credits scene, right the way through to Infinity War. This is largely due to the difference in how Digital Domain used motion capture to create the character. 

In Infinity War, the team wanted to see more of Josh Brolin’s expressions in Thanos to create a realistic and believable character. To do this, Digital Domain used a new machine learning software called Masquerade. Tracking dots were put on Brolin’s face, as is not uncommon with MoCap, and then recorded by a pair of HD cameras. These recordings were then sent through the machine learning algorithm which contained a large database of high-res facial scans of Brolin’s face. 

Masquerade then took this low-res mesh and figured out what the best solution for a high-res face shape would be. Once the team had the solution, they could tweak it however necessary to make it look just right. This proved to be a massive time saver for the artists working on the project as it meant they didn’t have to manually adjust every inch of the CG Thanos, which is often a somewhat tedious task.

The future of Motion Capture 

One of the main challenges of machine learning is its potential to remove artists from the creative and technical processes of production, a danger we explore in this article. But when we look at how machine learning works in tandem with motion capture, it’s clear that the two don’t adhere to this. 

Instead, they work harmoniously together—something happily experienced by Digital Domain in the team’s creation of Thanos. 

In this instance, machine learning improved how artists work with motion capture, allowing them to achieve their desired results faster and more efficiently. It seems that as machine learning continues to develop, not only will it pave the way for more advanced and dynamic use of motion capture, but also extend further into other areas of the VFX industry and continue to improve artist quality of life.