We’re not referring to a simple palette swap, either; we’re talking about eliminating roadsides lined with snow and replacing them with grassy banks. The only thing that’s missing is a family out having a barbecue.
“The goal of our research is to give machines the ability to create or ‘imagine’ scenes on their own,” Ming-Yu Liu, a senior research scientist at Nvidia, told Digital Trends. “This is a difficult challenge, because most A.I. today require you to have images as training data that exactly correspond for both the input and target image. Let’s say you wanted an A.I. that could turn a driving video from night into day, or convert a sunny day into a rainy day. Today, you would need to record video of that street during both daytime and nighttime, shot from exactly the same location, with the objects — vehicles, trees, pedestrians — in exactly the same location. In contrast, our method just needs a set of daytime images and another set of nighttime images for training, and these images can be taken in different cities or countries. Without the requirement of corresponding images, collecting data for training our model is much easier.”
To create their image-altering tool, the Nvidia researchers developed a novel neural network design to achieve unsupervised image-to-image translation. The algorithm and its source code is described in a paper available here. While it’s certainly an impressive tech demo, Liu points out that it has numerous real-world applications. For example, it could be immensely useful in video editing work. However, Nvidia has a much more immediate application in mind: Training self-driving cars.
“We did this research to help train self-driving cars under different weather and lighting conditions,” he said. “You can shorten training time for self-driving cars by teaching them with simulation. Using our technique, we can convert daytime to nighttime video, add rain or snow, and use that to help train self-driving cars [to deal with a wide range of scenarios they might face].”
Next up, the team wants to work to improve the robustness of the technology, while finding even better ways of improving data efficiency — allowing them to train their neural network with less data.
