Lighting in visual effects: crafting CG cinematography
Technological advancement in visual effects (vfx) capabilities means that directors and cinematographers have an ever-increasing arsenal of filmmaking techniques at their fingertips.
In turn, films are increasingly relying on special effects and computer-generated imagery (CG) to varying degrees, either for practical reasons—such as shooting in one place and making it look like another, to save on budget—or to go above and beyond the realms of possibility, to delight and amaze its audience.
After all, with films looking more complex than ever—bolstered by breathtaking entire CG environments and characters such as Paddington, Transformers, Detective Pikachu and so on—audiences have come to expect more with every new blockbuster release.
Yet, usually, the end goal in both cases is to attain what pre-CG films, or films without CG, intrinsically enjoy: photorealism. The idea here is that CG elements should appear as realistic as possible, so as not to break the audience’s immersion and interrupt the story. The challenge here then becomes one of aligning the final vfx shot with what the director of photography (DP) had in mind live on-set.
DP’s and cinematographers make creative decisions on how best to guide the viewer’s eye within any given frame, using appropriate lighting equipment and techniques. Whilst the application of their method varies between sets and projects, the kinds of questions they ask when deciding how light should be applied follow a fairly consistent pattern.
Whose perspective are we seeing the scene through—and how does lighting portray what they see? Which emotions are being expressed, and how should these best be conveyed through things like light color, placement or intensity? Which props and scene elements should be emphasised—and how does lighting combine with lens and aperture choice create the final image?
Further, how does the light affect the tone of the scene? Horror films typically use sharp lighting at crazy angles with strong contrast to heighten any sense of dread or foreboding, whilst a romantic scene will feature lighting that’s the opposite.
Below, we explore how these creative decisions are expressed in CG-heavy films, and how the DP’s creative vision is carried through to post-production.
Creativity is key
We’ve already seen how lighting artists working in CG animation enjoy a great deal of freedom and flexibility when bringing their creative vision to life. Lighters have more freedom to be creative and build on a scene, since they’re not bound by the physics of light.
Lighting crews working in vfx typically express their creativity in a different way. Whilst both CG and vfx lighting artists act like digital cinematographers, the latter has the additional caveat of working against a plate, which presents its own unique challenges.
Each lighting artist has to look at the plate they’re working with and put themselves in the cinematographer’s shoes on-set—understanding how they work, deconstructing their techniques and then replicating these digitally, since any CG work done in post-production has to match what’s already in the plate. Otherwise, the end result runs the risk of appearing unnatural and out of place.
The human eye is extremely adept at noticing when something looks ‘off’. If, for instance, the shadows present in a scene from a CG object do not match well enough to real shadows from on-set lighting, then the overall effect falls apart, along with any creative intent that the cinematographer, working on-set, wished to impart.
To avoid this, Gary Jones, Foundry’s Associate Product Manager for Katana, recalls his approach toward lighting a scene when working as a lighting artist at MPC: “There’s a couple of things I’d begin by thinking about. One of the things I liked to do is look at the previous shot: where the previous light was, where the eye was, what the focus of the composition was, then work out what the mood of the new shot is going to be. What are you trying to highlight? What are you trying to bring out of the story?
“You’re being a cinematographer,” he continues. “You’re thinking about composition. You’re thinking about ‘What’s the story I want to tell?”. It’s all training for being a cinematographer and the tricks that they do as part of visual storytelling.”
Put into practice
Gary’s sentiments above are often caveated by a key consideration that varies from scene to scene and film to film, which is: How much of the cinematography is driven by vfx?
Put another way, this becomes a tale of two extremes. On the one hand, there’s the idea that any vfx work should look so integrated that it follows all the cues in the plate, vs the opposite end of the spectrum whereby vfx completely drives the final frame. In both cases, the goal is still photorealism, but how much is open to interpretation as part of the vfx process?
Taking Life of Pi as an example, the majority of scenes involved the actor, on a boat, on a motion platform, each set against a blue screen. The stunning backdrops and fearsome tiger Richard Parker—both integral to the film and plot—would then later be made entirely CG. Here, most of the cinematography decisions driven by lighting would necessarily be vfx work, taking place during post-production.
The Jungle Book tells a similar story, with much of the CG informing the lighting on-set which, again, involved one actor in front of a green screen. Speaking of his time working as lighting artist on the film, Gary comments: “Mowgli was interacting with real-world trees, so we still had to make sure the lighting matched to the plate, even if the plate was tiny.”
“But we certainly have more creative freedom because the majority of the environment is CG, so you’re asking things like ‘Where does the shadow for this off-screen, fake tree fall on the fake ground in the background?’ You have more control there, for sure.”
In contrast, look development and lighting artists involved in the creation of Will Smith’s digital twin in Gemini Man were tasked with making a digital double fit into a frame that was already filmed—so all of the cinematography decisions were made on-set.
Any vfx work done had to match these—especially since Weta Digital’s goal was to “finally cross the uncanny valley and bring to screen a totally emotive digital human”, as Guy Williams, VFX Supervisor for Gemini Man, tells us in a previous article. In order to do this, matching CG with the plate, and preserving the creative cinematography decisions made on-set, would be pivotal in order to make Junior look as photorealistic as possible
“We made sure to protect the lighting as best as possible,” Guys tells us, speaking of the team’s process. “First, we used an actor to always play the part of Junior so we had good lighting reference. If Will was performing the part in situ, we still had the stand-in performer on hand to step in for a reference pass. From there, we used HDRI stills to capture all of the lighting reference for each take. We also used a spectrometer to sample all of the lights in the light kit so that all of our lighting could be done in wavelength instead of RGB.”
But what challenges did the Weta Digital team encounter as part of this?
“A lot of this approach is standard for Weta Digital so it didn’t require a lot of new thought,” Guys explains. “The frequency factors weren't new, but the extent that we used it was, so we had to solve some issues there. We had to rewrite some of our tools to allow us to pass spectral information straight into the renderer.
Another interesting thing we had to consider was that we couldn’t capture all of the spectral information. You can measure lights, but you can't measure the spectral values of every bounce coming in. We figured that the direct path lighting was the main contributor to the equation, so we rationalized that capturing it would be good enough. It's all value added so even getting it mostly right is a win.”
He continues: “On top of this, incident light coming in has a relatively smooth spectral curve so the wavelength math behaves rather well with the defaults we use, so the problem was even more mitigated.”
Helping the team navigate these challenges is Weta Digital’s PhysLight technology—a suite of tools, which, when coupled with the studio’s Manuka spectral renderer, makes any vfx work match the plate photography and lighting as closely as possible for a phenomenal sense of photorealism.
Watch PhysLight and Manuka in action on award-winning War for the Planet of the Apes—Weta Digital’s first full PhysLight production.
Through a film, darkly
Releases like Sin City pay direct homage to the Film Noir genre discussed in The Art of Lighting: A History Through Time.
Based on the neo-noir graphic comics by Frank Miller, Sin City and its sequel Sin City: A Dame to Kill For use low-key lighting, marked by extreme use of deep shadows and very high contrasts between the brightest and darkest parts of a scene, throughout both films.
The benefits of this are immediately tangible for the cinematographer in conveying to the audience a sense of moody mystery and intrigue—or smoke and mirrors—that typify films of this genre. Details can be obscured or highlighted through savvy use of contrasting light to either deceive the audience or reveal an ‘a-ha!’ moment.
Yet whilst much of this lighting is done on-set, many of the actors still worked in front of a green screen—so a lot of cinematography was driven by vfx across both films. This recalls the lighting work involved in the 2D animated classic Bambi as discussed in the aforementioned article—whereby the majority of the lighting was driven by the painted background, and not the characters. In a similar vein, films like Sin City usually enjoy more cinematographic cues in their CG environments rather than on the actors.
Completed by Prime Focus World, Sin City: A Dame to Kill For involved 2,282 visual effects shots with a vfx crew of 702 and a stereo team comprising 1,502 artists. Almost everything in the film was digitally crafted—including 65 locations, 1,572 assets and a number of props. The lighting artists certainly had their work cut out here, since each one of those CG elements accurately had to accurately represent and reflect the unique style that the films have come to be known for.
A blend of something beautiful
Known for their out-of-the-ordinary visual effects, LAIKA blend stop motion with CG to create a unique form of 3D animation that gives each of their productions a distinct and unmistakable style.In our previous article, we explored how films like Kubo and the Two Strings compile puppetry, vfx and on-set techniques together to create something incredible. This same technique was used by LAIKA on The Boxtrolls, most notably during the ballroom dance scene in which the hero characters are stop-motion controlled physical puppets, surrounded by 50-60 CG dancers.
The challenge here lies in integrating the CG with the plate in a way that not only gives the audience a seamless visual experience, but also preserves the team’s creative vision as expressed on-set. Practical lights and cameras come into their own here, making use of the full suite of real-world cinematography techniques such as lens choice, light intensity and placement, framing and so on to capture the look and feel of the scene.
When this same scene is sent to post-production, the vfx artists tasked with expanding the world beyond the physical realm then have solid reference points to preserve the artistic intention all the way through. The resultant cinematography of the scene comes from the cohesion of the footage shot on camera combined with the vfx work to create something truly breathtaking.
Practical lighting
Using practical lights on-set, as in the examples above, is a fundamental tool in every cinematographer's arsenal in not only capturing the tone of the scene in the moment—and, by extension, on camera—but also to help sell composites in post-production. The end result in CG-heavy films becomes that much more believable when the visual effects artists have a concrete reference point to draw from.
Practical lights are traditionally defined as any light where the source is in frame. Often, this means light built into set locations, such as televisions, desk and floor lamps, and ceiling features. Outside, practical lights may involve car and street lamps, strings of Christmas lights—the list goes on.
The benefits of practical lighting are tangible, not least because practical lights are bound by real-world physics. Their rays wrap around corners, for example, creating subtle shadows that would be very tricky to replicate digitally in a natural way.
Using practical lighting to help carry cinematography techniques through to post-production and into a film’s visual effects is just one part of a high-level trend that the industry is experiencing—one marked by exciting burgeoning tech that aims to bring each stage of the production pipeline closer together.
A match made in heaven
It’s clear that a technically challenging career is par for the course for lighting artists working in visual effects.
Not only do you have to match the CG to a plate seamlessly, but you also have to preserve the cinematography and creative decisions as they were made live on-set whilst doing so.
Yet, as Jordan Thistlewood, Foundry’s Director of Product and former lighting artist explains, it’s not all bad: “One of the beauties of lighting is this visceral response when you’ve made the last piece of the puzzle fit.”
He continues: “When what you’re working on stops being CG and instead becomes part of the story—the point where something clicks and becomes part of a plate. That’s a magical moment for lighting artists.”
The technology and techniques explored in this article go some way in fueling these magic moments experienced by lighting artists, by giving them a better idea of what the creative intent is behind any scene that makes it way to post-production.
And as this technology advances, artists are given more scope to fulfil the DP’s creative vision through a cohesion of cinematography principles and practices. Foundry’s own efforts in making this a reality for artists involve game-changing development plans for lighting tool Katana’s upcoming 4.0 release, due later this year.
In the next and final article in this series, we sit down with Jordan and Gary to explore how Katana 4.0 aims to make these magic moments happen more easily, and more often. Major improvements to Katana’s lighting UX are just a fraction of some of the updates planned, built around how cinematographers think, so lighting artists can express their creativity in the same way.
Stay tuned for expert insight into what the future of Katana looks like as the ultimate digital cinematography platform for look development and lighting artists everywhere.
