A Real-World Approach on the Problem of Chart Recognition Using Classification, Detection and Perspective Correction

Abstract

Data charts are widely used in our daily lives, being present in regular media, such as newspapers, magazines, web pages, books, and many others. In general, a well-constructed data chart leads to an intuitive understanding of its underlying data. In the same way, when data charts have wrong design choices, a redesign of these representations might be needed. However, in most cases, these charts are shown as a static image, which means that the original data are not usually available. Therefore, automatic methods could be applied to extract the underlying data from the chart images to allow these changes. The task of recognizing charts and extracting data from them is complex, largely due to the variety of chart types and their visual characteristics. Other features in real-world images that can make this task difficult are photo distortions, noise, alignment, etc. Two computer vision techniques that can assist this task and have been little explored in this context are perspective detection and correction. These methods transform a distorted and noisy chart in a clear chart, with its type ready for data extraction or other uses. This paper proposes a classification, detection, and perspective correction process that is suitable for real-world usage, when considering the data used for training a state-of-the-art model for the extraction of a chart in real-world photography. The results showed that, with slight changes, chart recognition methods are now ready for real-world charts, when taking time and accuracy into consideration.

A Multi-Purpose Shallow Convolutional Neural Network for Chart Images

“…CNNs can also be used out-of-the-box; some are available as pre-trained models. Pre-trained models are trained on large datasets (e.g., ImageNet [ 32 ]), and to use them, only the final layers need to be changed and retrained as in Reverse-Engineering Visualizations [ 4 ], FigureSeer [ 33 ], Chart-Text [ 34 ], VizByWiki [ 24 ], Visualizing for the Non-Visual [ 25 ], DocFigure [ 26 ], by Huang [ 35 ] and by Arujo et al [ 36 ]. Common to both architectures is the average classification accuracy, in the range from 90% to 100%.…”

Огляд Інструментів Ocr Для Завдання Розпізнавання Таблиць І Графіків У Документах

“…There is a lack of works in the literature linking real-world photos with the task of labeling charts before labeling. There are many issues to solve, such as locating charts in images and removing camera distortions [20].…”

Challenges in chart image classification

“…This section first evaluates and compare the performances of nine popularly used CNN based models in chart classification namely AlexNet [16], VGG-16 [3,10,19], VGG-19 [2], ResNet-50 [5,18], ResNet-101 [8], ResNet-152 [2], Inception-v3 [14], Inception-v4 [5] and Xception [2]. These classifiers are built on our dataset using randomly selected five-fold cross validation.…”

“…For example, the study in [10] reports to obtain 89% accuracy, whereas study in [3] reports to have obtained only 80% over the same dataset using the same model. Some study uses a real dataset as small as 129 samples [11], and some study uses real dataset as large as 21,099 samples [2]. Motivated by the inconsistencies in observations reported in various studies, this paper aims at identifying potential challenges in building automatic chart classification system.…”