Reverse‐Engineering Visualizations: Recovering Visual Encodings from Chart Images

Abstract

We investigate how to automatically recover visual encodings from a chart image, primarily using inferred text elements. We contribute an end‐to‐end pipeline which takes a bitmap image as input and returns a visual encoding specification as output. We present a text analysis pipeline which detects text elements in a chart, classifies their role (e.g., chart title, x‐axis label, y‐axis title, etc.), and recovers the text content using optical character recognition. We also train a Convolutional Neural Network for mark type classification. Using the identified text elements and graphical mark type, we can then infer the encoding specification of an input chart image. We evaluate our techniques on three chart corpora: a set of automatically labeled charts generated using Vega, charts from the Quartz news website, and charts extracted from academic papers. We demonstrate accurate automatic inference of text elements, mark types, and chart specifications across a variety of input chart types.

Visualizing for the Non‐Visual: Enabling the Visually Impaired to Use Visualization

“…Poco and Heer proposed an automatic pipeline for extracting a visual encoding specification given a raster chart image [PH17]. Poco et al focused on recovering the color mappings of charts that include a color legend [PMH18].…”

“…We adopt Convolutional Neural Networks (CNNs) [KSH12, HZRS16] as a classification model, which have shown impressive performance on image classification tasks. Whereas previous studies [PH17, JKS*17] used AlexNet [KSH12] and GoogLeNet [SLJ*15] for classification, we use residual networks [HZRS16] that yield state‐of‐the‐art performance in most computer vision tasks. Specifically, we employ existing Resnet trained on the Imagenet dataset [RDS*15] and append a global average pooling layer before the last fully connected layer.…”

“…For text role classification, we adopt the method introduced by Poco and Heer [PH17]. They classify the text of charts into title, y ‐tick, y ‐label, x ‐tick, y ‐label, legend label, legend title, and text label using support vector machines (SVMs).…”

“…To compare our model with previous work, we also validate our classification model on the ReVision [SKC*11] dataset, which includes 2,084 images in 10 classes. Following the test protocol used by Poco and Heer [PH17], we split the ReVision dataset into 75% and 25% for training and evaluation sets. Our classifier also shows state‐of‐the‐art performance in classifying 10 chart types in the ReVision dataset as shown in Table .…”

“…To compare our model with previous work, we also validate our classification model on the ReVision [SKC * 11] dataset, which includes 2,084 images in 10 classes. Following the test protocol used by Poco and Heer[PH17]…”