Future proof your content creation with machine learning
Machine Learning (ML) is impacting the VFX, and other industries, in a number of different ways. More recently, one of the areas in which it has started to come into fruition is content creation.
The recent progress of deep learning techniques allows hours of manual and laborious content creation work to be done in minutes. This isn’t just true for the VFX industry but in applications such as gaming, Virtual Reality (VR), advertising and even retail.
In this article we take a deep dive into some of the areas in VFX content creation in which ML is having the biggest impact, and how it’s transforming the way we make content—from image style transfer and 3D terrain modeling to awe-inspiring animation.
Creating art with tech
As humans, we have perfected the art of creating unique visual content, whether that be through paintings and art, or through more technical mediums like film and VR. It seems only natural, then, that we have begun to use ML and other neural networks to experiment with how we can create images.
One technique that is having a big impact is style transfer. This uses an algorithm which drives the combination of two images, one for content and another for style—usually a painting. Then, using convolutional neural networks, this creates a new output image. So if you’ve ever wanted to be part of a Van Gogh painting, now you can by simply applying your favorite style to your next selfie.
This concept can produce some intriguing results, especially as the algorithm allows for the manipulation of the weight of both the style and content. It also gives creators the ability to add multiple style images which gives the final output more depth and layers and has been known to create images that can be easily mistaken as a piece of original artwork.
While a lot of the advancements happening around style transfer relate to static images, there has been progress seen in using the algorithm on films as well. But this doesn’t come without complications. When the algorithm was first tested on video, the results were less than successful and saw inconsistencies in the film. The style would slip in and out on a frame-by-frame basis—it would be there one frame and gone the next.
However, thanks to the innovations in machine learning, a research team in Germany managed to develop a method that stops discontinuities seen when applying style transfer to a video. The method penalizes any deviations it detects between two frames, whilst following the movement elements of the video. This produces a more finalized video at the end, and preserves the appearance of the image that was originally inputted.
Whilst a step forward, this method does have its drawbacks, in that it proves to be a long and tedious process with the style-transfer taking several minutes per frame. While there may still be a way to go in terms of developing this method, style transfer as a whole presents groundbreaking possibilities for the VFX industry. Particularly the ability for machines to understand what an image is and the details that go into making it. This could save artists huge amounts of time when creating lots of different assets for a scene, like a street of houses, and further alleviate the time they spend on manual tasks.
While style transfer takes your current work and makes it look different, what about when you want to ideate for a more creative exploration?
NVIDIA, whose GPUs power machine learning inference and training, is also at the forefront of procedural content generation with their creation of GauGAN, which takes sketches and turns them into images. The machine learning system allows artists to use a paintbrush tool to design their own landscapes. Users can create rough sketches using this tool, which are then turned into breathtakingly realistic images. Not only does it aim to save artists time, but also elicits creativity, presenting them with environments that they may not have thought of, or previously not had the time to create.
Moving mountains
Machine learning has also been breaking ground in 3D content creation. More specifically in 3D procedural content generation, which sees the creation of textures and 3D models for games or animation—something we discuss in greater detail here. It uses data algorithms like deep learning to create these, instead of an artist having to do it manually.
A common problem that occurs when using 3D procedural content creation is the generation of terrain. This can often be a tedious process which involves creating thousands of different elements for the terrain environment, like rocks, mountains, valleys, etc. However, innovations happening in ML and neural networks aim to reduce some of the more time consuming and tedious elements of this.
Montreal Institute of Learning Algorithms in Canada have joined forces with NASA to take 3D terrain content creation even further. Using deep learning, the researchers have managed to generate realistic landscapes using satellite images of Earth.
The system consists of two deep learning machines, one that generates new terrain and a second that evaluates the results and provides feedback to the first. The first machine then uses this feedback to produce another set of landscapes, and so on. The idea being that the machines learn what an ideal piece of landscape should look like and can then create something from what they’ve learnt.
The team even went so far as to train the machine to produce terrain of the Earth, using NASA’s Visible Earth program, which has created a detailed map of our planet including data on its height, shape and color, to provide the machine learning system with images. While the process proved slow at first, eventually after many iterations the machines were able to produce terrain that looked like Earth.
As a 3D modeler or creative, you often have to create several different versions of a scene or a shot until you get the perfect one. This process can take hours, meticulously poring over all the individual elements. But, with NVIDIA GPUs and programs like NVIDIA GauGAN or the Montreal Institute’s, artists will be able to save time when creating these different environments. If a machine can produce a background image or a landscape quickly, then an artist can spend more time crafting the more prominent or important aspects of their shot.
Turning Animation into gold
As with so many things, there are parts of an animator’s job which can be tedious. But, with the recent advancements in machine learning, Jiayi Chong, a former Pixar animator, has created Midas Creature, which intends to revolutionize the animation industry.
Midas Creature uses a variety of tools that aims to further the goals of animators. Using deep learning, it fine-tunes complex secondary motion and takes control of the more technical details animators often have to endure while animating. The software aims to aid artist creativity, rather than overshadowing them.
Alongside this, deep learning has also begun making advancements with rigging in animation. Creating new characters is often one of the more challenging jobs of an animator, but with ML it is possible to automate and speed up the rigging process and even suggest rigs for new characters. Using Generative Adversarial Networks (GANs), machine learning can come up with new sample characters by learning from ones that have been fed into the GANs.
The future is content
Of course, there is still much debate around machine learning and artistry—a topic we’ve discussed before. However, the machine learning innovations we have discussed above all aim to heighten the creativity of artists and help them, rather than diminish their need. They also look to alleviate more manual and tedious tasks, giving artists more time to be creative and concentrate on the more important aspects.
ML is set to become more and more prominent in the VFX industry as it continues to advance. It aims to give artists the toolkits they need to do less drudge work, whilst working in harmony with creators. These innovations show what an exciting time it is to be working in the VFX industry and mastering these skills—who knows what the future of machine learning could hold. Right now, it’s changing the way we make and consume content forever.
Stay up-to-date with the latest Machine Learning trends with our free quarterly newsletter.
