A digested guide to real-time production

There’s a discipline that’s creating an audible buzz across multiple industries at the moment, that’s both the subject of serious whitepapers and excited YouTube comments: real-time production.

Yet with all the discussion – including from us at Foundry Trends – it can be difficult to work out what actually constitutes production in “real-time”, and what techniques being touted as revolutionary are currently achievable.

Reaching nirvana 

Before we get serious, let’s allow ourselves to get carried away a bit. What could the "nirvana" for real-time production look like?

Better pre-visualization than ever before, with directors able to experiment with their centrepiece sequences before getting anywhere near the set - a cost-effective way to create and iterate without having to commit the studio’s budget to several reshoots.

Directors filming their cast against fully realised virtual sets, allowing them to – as they would on a physical set – frame, and re-frame in the search for the perfect shot. Directions can be relayed instantly, via a quick check of a monitor, where actors will see themselves acting within a perfectly rendered world, as a perfectly rendered character.

Post-production could be changed forever, with much of the VFX work having taken place before shooting has even taken place – allowing for a greater level of continuity throughout the pipeline, from initial idea to final cut. 

But if this is the ideal for where we’d like real-time production to get, where are we now? At this moment in time, does real-time production actually exist?

Real-time philosophy

If “real-time production” is achieved when the time it takes to create new images – no matter how technically complex – is virtually nil, arguably you could say that right now we’re still some way off.

However, constant innovation means we’re getting closer. And we can identify tangible ways real-time production will bring significant benefits to three stages of the production process: pre-vis, on-set, and post-production. 

Advanced pre-visualization

From its initial development in the late 1980s, pre-vis has grown into an essential part of the film production pipeline. Yet the historically lightweight nature of the simple assets used as part of pre-vis has arguably held it back. Directors have been happy to use less detailed models, with primitive animations, because of the speed of planning they allow.

But advances in real-time pre-vis mean that directors don’t have to compromise quality for much longer. They can inform their shoots in high fidelity, with real-time playback, and high-quality assets that can be used throughout the entire production process - whether on set or in post. 

An early example of this was seen in the filming of Avatar, for which James Cameron built software so he could visualize what his actors would look like interacting in a space as 10-foot-tall aliens. The ability to do this instantaneously, without needing to send the scene away for a day’s rendering, shows how far the technology has come.

Real-time production enables the convergence of previously disparate processes to allow for more joined-up filmmaking. We’ll come back to the concept of on-set virtual production shortly, but an example of real-time techniques blending pre-vis and on-set can be seen in the making of Rogue One: A Star Wars Story. In this clip, we can see how director Gareth Edwards used a virtual camera in order to line up the shots for the film’s climatic space battle, which had already been animated in pre-vis. 

On-set, virtual production

The use of virtual production on-set is the aspect of real-time production technology that is currently capturing the industry’s imagination. And why not? The prospect of giving directors and actors access to quick and accurate feedback when performing against green screens or interacting with CGI characters and environments is certainly exciting, and could mean the end of hastily arranged (not to mention costly) re-shoots that arise from problems encountered in post.

Some filmmakers are already using this technique. John Favreau’s Jungle Book, shot entirely against green screen with only one physical human present – the prodigious young actor Neel Sethi as Mowgli – used it in abundance. Skip to 8:30 in this video and you can see the technology in use: as Neel walks through a green-screen sound stage, the monitor shows him wandering through a landscape abundant with wildlife.

However, as the video also shows, the quality of the CGI models used within virtual production is still relatively crude – showing that while real-time production is currently achievable, it is not yet able to produce high-quality film visuals.

The end of post-production?

Less eye-catching, perhaps, yet certainly more significant in its impact to the industry, is the potential real-time production holds to flip the entire film production pipeline on its head – possibly even spelling the end of post-production as we know it. 

For real-time production to work effectively, the vast majority of visual effects work may have to be completed before filming begins. Only that way will scenes shot in real-time be considered of high enough quality for inclusion within the final cut – or treated as ‘final pixels’. 

Over the past 100 years, we’ve seen the focus of manpower on films shift from pre-production – in the form of set building, carpentry and other skilled trades – to post-production. The adoption of real-time technologies could see that reversed once again.

The Human Race project, from Chevrolet and Unreal, shows how this nascent idea is starting to take off. A purpose-built physical car chassis – called Blackbird – onto which a huge range of pre-built photorealistic bodies can be placed, was filmed racing through the streets of California. 

Depending on the whims and tastes of the filmmaker, the car model, colour, trim and other accessories featured in the clip can be changed in seconds – with the footage rendered in real-time to include their options. 

Works in theory, but in practice…

Real-time production, on paper, looks set to do exciting things. Yet, as the late baseball legend Yogi Berra once said: In theory, there is no difference between theory and practice. But in practice there is.

As we’ve alluded to, there are significant barriers that real-time production needs to overcome before it can be considered a serious contender to the traditional production pipeline; from technical developments, to sceptical directors. 

Foundry Trends will be exploring these, and taking a closer look at the studios propelling the techniques advancement, in the coming weeks.