Data aggregation in VFX: Impossible dream or untapped opportunity

Data aggregation has been transformative for industries as broad as pharmaceuticals and media and entertainment. Amazon, once an online bookshop but now with its fingers in almost every pie imaginable, was one of the first businesses to aggregate data from a network of vendors to support future transactions - offering customers recommendations based on what other people had purchased.

Today, most online businesses similarly use data to optimize the customer experience. Expedia recommends holidays, Netflix tells you what shows you’ll enjoy, and Spotify suggests other songs you might like. But such use of data hasn’t really had a significant impact on the world of visual effects (VFX).

One of the big challenges for VFX studios, which we’ve looked at in detail before, is effective resource planning from project to project; deciding when to invest in additional rendering capacity while avoiding a scenario where machines are left idle for weeks on end. We’ve also discussed how migration to the cloud can help.

What role, then, could data sharing and aggregation play in project planning and execution?

Studios usually exist independently from one another. They use different processes, passing on or absorbing different costs based on what’s worked for them in the past. There’s no magic spreadsheet that will suck in project details and spit out the perfect cost-benefit option.

But what if studios could draw insight from others, learning from their failures and successes to better manage their own work? What if a central repository of past project data could be drawn on for insight, or to feed an algorithm fuelled by machine learning? The accepted way of doing things would change dramatically.

The glaringly obvious sticking point, of course, is whether said studios would indeed be happy to share information with their fiercest rivals for the ‘greater good’. This is an industry that has been burned by very public data leaks in years gone by, and one shrouded in secrecy when it comes to new work.

Perhaps we’re being fanciful, and perhaps it’s a hurdle that will prove just too big to ever be overcome. There are, however, companies trying to bridge the gap by offering new ways to more accurately make predictions for project resourcing.

Gareth Williams is founder and chief executive of YellowDog, a British startup offering cloud rendering for VFX, animation, and CGI studios. YellowDog’s ‘limitless compute’ gives users access to as much rendering capacity as they need, without shelling out for on-prem server farms.

YellowDog also helps studios plan resource by predicting the time it will take to render a piece of work, based on available compute - giving them the insight they need to meet all-important client deadlines.

“One of the big challenges is that VFX work is so competitive and there are such low margins,” Williams told us. "So a studio’s pipeline - and its efficiency for clients - is often its competitive edge. It’s hard to imagine a future where studios effectively give that competitive edge away.”

Williams continued: “It’s not just about rendering quickly but rendering to a deadline. People just want to know what they need to put in place in order to have something finished by a specific point.

“We’ve developed machine-learning algorithms that allow us to look at render jobs before they’ve started, analyse any restraints and tell customers with a degree of accuracy when the rendering will be finished.”

To do that, YellowDog does need an element of data aggregation. It takes information from projects uploaded by its customers, assessing more than 20 data points from each scene - such as resolution, lighting and texture - to inform its models and algorithms.

And Williams believes people are happy to share project data to improve the service, so long as it’s anonymised and does not give away any competitive advantage. “It all hinges on what the data is used for,” he added.

The industry is clearly some way off data aggregation Utopia. There remain significant challenges to overcome - not least studio’s willingness to share with competitors - but there’s no doubt the sharing of information between studios has the potential to transform production pipelines as we know them.

Want to find out more about the potential impact of cloud tech on the VFX industry? Check out our article that looks at how cloud could play a transformative role in helping studios plan project costs.