Scientists believe the solution to bridging this gap is deep learning, a type of artificial intelligence that uses neural networks to analyze and process images. What are neural networks, you ask? Neural networks are brain-like computer systems designed to pick out and create patterns based on inputs.
In an effort to get around the current limitation of image recognition techniques, scientists have started work on a program that will be able to take a doodle or sketch of an image and find a matching photography – a visual search engine, if you will.
One of the main reasons doodles have stumped image recognition programs in the past is that many times, the drawings aren’t perfect representations. Rather than being lifelike recordings of what it is we’re doodling, sketches and drawings are merely how we visualize something.
These less-than-accurate representations, be it on purpose or due to lack of drawing skills, often leads to dramatized proportions and missing elements that might not prevent humans from recognizing the image, but causes problems for computers. For example, that stick figure you drew in third grade likely wouldn’t be recognized as a human by an image recognition program, as its body is nothing more than a few lines placed below a circle – not the lifelike proportions of the human form.
While still in its infancy, New Scientists notes the program has already proven viable. Using more than 600 individuals, scientists displayed a random image for only two seconds and asked the participant to sketch the pictured object from memory. These resulting sketches were put into one neural network while the images shown to the participants were put into another. The two networks then worked together to analyze the sketches and match them to their original picture.
In the initial testing phase, the program was able to match the doodles up to their original image 37 percent of the time. Seemingly unimpressive, it’s important to note that humans only matched the sketches and images correctly 54 percent of the time. While not conclusive, these preliminary numbers show that with a little tweaking and more inputs, the program could very will surpass humans capabilities to match sketches to photos.
Where would this technology be useful though? The possibilities are limitless, really. Looking at it from a consumer standpoint, this tech could be used to create an app that lets you search for images and artwork based on doodles of your own. Imagine being able to draw up a little sketch, scan it with your phone, and find similar artwork to be inspired by or to compare your own work to.
From a more commercial standpoint, one of the most useful features of this technology would be to help police identify criminals by comparing sketch artists renderings to mugshot databases.
More thorough findings and information on the system is set to be released at the SIGGRAPH conference in Anaheim, California, in July.
