How data analysis is improving matchmoving

In filmmaking, complex techniques are driven by complex ambitions. This is particularly true of matchmoving, which allows computer generated imagery (CGI) to be inserted into live-action footage while keeping scale and motion correct.     

Matchmoving has been born from a natural progression of filmmakers wanting animated characters to interact in live-action setting. From the likes of Pete’s Dragon and Who Framed Roger Rabbit? in the 1970s and ‘80s, the CGI matchmoving of today’s movies is now so commonplace that it almost goes unnoticed by audiences. 

However, as any filmmaker will tell you, the process can be time consuming and fraught with frustration. What if there was a way to improve this through technology?

One man who has tried to do just that is Alastair Barber, whose work with DNEG and the University of Bath focused on improving the matchmoving process using data analysis and algorithms.

Room for improvement

One reason that matchmoving is so frustrating is because tracking camera placement within a scene is typically a manual process. Barber’s research suggests that matchmoving can take more than 5% of the total time spent on the entire visual effects (VFX) production pipeline. Even when using software designed to take up most of the grunt work, artists will be required to scale manually. 

“I wanted to find a better way to track how the camera actually moves,” Barber told Foundry Trends. “I thought initially about getting hardware on set that could read what was happening, but a film set is a pretty hostile environment. You’ve got equipment moving at all times, and crews who don’t have time to faff about with tech they’ve never tested before.”

Clearly, an alternative method was required. Step forward DNEG, the 20-year-old London VFX and computer animation studio, responsible for Academy Award-winning work on Inception, Interstellar, Ex Machina and Blade Runner 2049. 

“DNEG has been around for some time,” said Barber. “I was pretty sure the estimates on camera tracking it had collected over those years were fairly accurate, but when I started to dig deeper into it I realised how much data we really had to work with.”

Accelerating through algorithms 

Creating algorithms using this vast data set has lead to drastic improvements in the matchmoving process, but first Barber had to work out what to feed in. “We had to pick apart things on set,” he explained. “What type of lens is on the camera, how fast the camera is moving, what’s going on in the scene - basically, anything we could measure.”

He also considered the needs of the VFX artists themselves. ‘Streamlining’ - the process of getting different types of software to talk to each other - was found to be in demand. “We wanted everything to be ready to go so that the artist could start working straight away,” said Barber.  When an artist has easy access to detailed and relevant on-set data, Barber says the process takes place 20% faster.

Alternative applications

The idea of using data sets to improve studio efficiency does not stop at matchmoving. Barber believes rotoscoping is the next part of the pipeline where improvements can be made. “It’s very similar to matchmoving,” he explained, “in the sense that you have a task that has to be repeated across all footage but will vary in terms of difficulty based on what’s in the picture. There’s a lot of potential for further applications based on data we already have.”

However, for wider adoption to take place, studios will have to convince clients to let them delve into their data. Barber is optimistic that this shouldn’t be too difficult. “A lot of this comes down to the relationship between client and studio. If a studio has good access to what is happening on set, it’s easier to explain what they need and why without causing alarm.”

With metadata on shots able to be stored with relative ease, there’s no reason why the VFX pipeline can’t be continuously improved using algorithms and workflows that can take advantage of this data. Data holds exciting possibilities for studios of all sizes to boost efficiency and save valuable time.