ChemInk

Abstract

We describe a new sketch recognition framework for chemical structure drawings that combines multiple levels of visual features using a jointly trained conditional random field. This joint model of appearance at different levels of detail makes our framework less sensitive to noise and drawing variations, improving accuracy and robustness. In addition, we present a novel learning-based approach to corner detection that achieves nearly perfect accuracy in our domain. The result is a recognizer that is better able to handle the wide range of drawing styles found in messy freehand sketches. Our system handles both graphics and text, producing a complete molecular structure as output. It works in real time, providing visual feedback about the recognition progress. On a dataset of chemical drawings our system achieved an accuracy rate of 97.4%, an improvement over the best reported results in literature. A preliminary user study also showed that participants were on average over twice as fast using our sketch-based system compared to ChemDraw, a popular CAD-based tool for authoring chemical diagrams. This was the case even though most of the users had years of experience using ChemDraw and little or no experience using Tablet PCs.

Mechanix: A natural sketch interface tool for teaching truss analysis and free-body diagrams

“…The SUS is simply a questionnaire where students can record and indicate their perceived usability of the two programs. Similar to the method used in a study by Ouyang and Davis (2011) where they compared a new program, ChemInk (a natural real-time recognition system for chemical drawings), with ChemDraw (a popular CAD-based tool for authoring chemical diagrams), we will ask the students to use both programs (WinTruss and Mechanix) during the focus groups, instead of just using Mechanix. We will directly measure the time differences in using both programs.…”

“…In teaching science and engineering, visual and verbal coordination is also essential. According to the dual coding theory of cognition (Sadoski, 2001; Ouyang & Davis, 2011), the coordination of verbal and nonverbal information facilitates learning because there are separate complementary cognitive channels for each type of representation. Therefore, when instruction is coordinated between verbal and nonverbal, a learner can process more information simultaneously because the size of one's working memory is typically the limiting factor in learning (Baddeley, 1986); effective instruction must not overload cognitive capacity.…”

“…When the intrinsic cognitive load of instruction is large (i.e., a very hard task), as is often the case with engineering concepts, it is critical that the presentation format of material, or extrinsic cognitive load, be well designed. In this case, the visual sketching exploits two nonverbal channels (Ouyang & Davis, 2011): visual learning via the actual sketches and kinesthetic learning via the process of sketching. This will theoretically maximize learning.…”

A Novel Method for Recognizing Sketched Objects by Learning Their Geometrical Features

“…LA Viola et al [6] investigated recognition of mathematical sketches. Ouyang et al [7] addressed chemical structure drawings recognition. Cao et al [8] proposed a low dimensional symmetry aware flip invariant descriptor for sketches.…”

“…Although sketch recognition has been studied for more than twenty years, the focus in previous research has been mostly on producing methods that can identify geometric shapes separately. Most of the existed work was limited to some narrow domains, such as chemical drawings (Ouyang and Davis) [7], diagrams (Peterson et al) [14] , or simple hand-drawn shapes such as circles, rectangles, and triangles (Paulson and Hammond [15] ; Manuel J. Fonseca Joaquim A. Jorge [16] . Yongxin & Timothy developed the first deep neural network model (sketch image retrieval system) for sketch classification and it has outperformed state-of-the-art results in the sketch benchmark of [17].…”