Moving photos aren’t anything new — just look at the millions of GIFs floating around Facebook. But all those animations are generally made from short videos or a sequence of still photos. Facebook and the researchers from Tel Aviv University set out to create an animation using only a single photo. (A Google researcher recently took on a project with a similar objective, creating a scenic video using artificial intelligence and a single photo as a starting point.)
The research team built software that maps a facial expression on a reference video and applies it to the still photo. The system uses several facial markers and watches how the expression changes in the reference video. Using similar facial markers on the image, the program then animates the photo using the same motions.
The result is an emoji-like animation of the still photo. But, a few things were still missing — literally. In a still photo where the expression didn’t include teeth, for example, mapping the photo to a smiling animation creates an odd pink space where the teeth should be. The team then took that coarse animation and created an algorithm to fill in the missing details. Of course, those teeth probably don’t match the teeth of the person in the photo, but the researchers say that since people don’t generally identify someone by their teeth, adding in the missing detail created better results than without the “false” teeth.
The team then further refined the results to add fine details to enhance the expression and remove anomalies, like wrinkles created from the animation that were not in the original photo.
Animating a still photo could create a number of different apps for creating animations with little data, like making the Mona Lisa smile. But as the researchers demonstrate in the project’s video, the tool could also create responsive photos on social media. For example, when a follower clicks the like reaction on Facebook, the image could animate with a smile while other reactions, like sad and angry, could have animations matching those emotions.
The resulting animations aren’t perfect with a few oddities to the animation, but further refinement could bring the feature mainstream. The team plans to continue developing the program by allowing the software to choose from a library of reference videos based on the sample that closely matches the facial features for more accurate animations, as well as expanding the work with 3D methods. Paired with the patent for recognizing emotions through a webcam, tracking facial expressions appears to be an area of interest for the social media company’s research.
The project will be presented at Sigggraph Asia in November.
