The previous auto crop feature worked by using face detection to keep faces in the frame. When no faces were detected in the image, the software would simply crop the preview at the center, while a click on the image allowing users to see the entire shot. Twitter says the crop option without faces would often lead to awkward crops, while sometimes the software didn’t correctly identify faces.
To fix those awkwardly cropped previews, Twitter engineers used what’s called salient image maps to train a neural network. Salient maps use eye trackers to determine the areas of an image that most catch the viewer’s eye. Earlier research in the area showed that viewers tend to focus on faces, text, animals, objects, and areas with high contrast.
Twitter used that earlier data to train the program to understand which areas of the image are the most important. Using that data, the program can recognize those features and make that auto crop in a place that will leave the most visual areas inside the crop.
But Twitter wasn’t done — while saliency software works well, it’s also slow, which would have prevented tweets from being posted in real time. To solve the awkward crops problem without a slowdown, Twitter refined the program again using two different techniques that improved the speed tenfold. The first trained a smaller network using that first good but slow program in order to speed up those crops. Next, the software engineers determined a number of visual points to map on each image, effectively removing the smaller, less important visual cues while keeping the largest areas intact.
The resulting software allows images to post in real time, but with better crops. In a group of before and after pictures, Twitter shows images with faces that the earlier system wouldn’t detect properly cropped to face rather than feet. Other examples show images of objects that were cut out in the first program because they didn’t sit in the middle of the image, but were more appropriately cropped using the updated algorithms. Another example shows the program recognizing text and adjusting the crop to include a sign.
The updated cropping algorithm is already rolling out globally on both iOS and Android apps as well as Twitter.com.
