The downside to computer programs is they lack the ability to interpret. A tiny human can look at a picture of a chicken and a cartoon of a chicken and know that's a chicken while a computer program cannot.

Researchers from Queen Mary University of London believe they have overcome one obstacle and have built the first computer program that can recognize hand-drawn sketches better than humans. They call it Sketch-a-Net and in their tests it is capable of correctly identifying the subject of sketches 74.9 percent of the time compared to humans that only managed a success rate of 73.1 percent.

As sketching becomes more relevant with the increase in the use of touchscreens, the development could provide a foundation for new ways to interact with computers because touchscreens could understand what you are drawing enabling you to retrieve a specific image by drawing it with your fingers, which is more natural than keyword searches for finding items such as furniture or fashion accessories. 


Sketch-a-Net could successfully identify a seagull, pigeon, flying bird and standing bird better than humans. Credit: Mathias Eitz, James Hays and Marc Alexa

The research also showed that the program performed better at determining finer details in sketches. For example, it was able to successfully distinguish the specific bird variants ‘seagull’, ‘flying-bird’, ‘standing-bird’ and ‘pigeon’ with 42.5 percent accuracy compared to humans that only achieved 24.8 percent.

Sketches are very intuitive to humans and have been used as a communication tool for thousands of years but recogniszing free-hand sketches is challenging because they are abstract, varied and consist of black and white lines rather than colored pixels like a photo. Solving sketch recognition will lead to a greater scientific understanding of visual perception.

Sketch-a-Net is a ‘deep neural network’ – a type of computer program designed to emulate the processing of the human brain. It is particularly successful because it accommodates the unique characteristics of sketches, particularly the order the strokes were drawn. This was information that was previously ignored but is especially important for understanding drawings on touchscreens.

Timothy Hospedales, co-author of the study and Lecturer in the School of Electronic Engineering and Computer Science, QMUL, said, “It’s exciting that our computer program can solve the task even better than humans can. Sketches are an interesting area to study because they have been used since pre-historic times for communication and now, with the increase in use of touchscreens, they are becoming a much more common communication tool again. This could really have a huge impact for areas such as police forensics, touchscreen use and image retrieval, and ultimately will help us get to the bottom of visual understanding.”

Citation: 'Sketch-a-Net that Beats Humans' by Q. YU, Y. YANG, Y. SONG, T. XIANG AND T. HOSPEDALES will be presented at British Machine Vision Conference 2015 in September 2015