Intrinsic Decomposition of Document Images In-the-Wild

Abstract

Automatic document content processing is affected by artifacts caused by the shape of the paper, non-uniform and diverse color of lighting conditions. Fully-supervised methods on real data are impossible due to the large amount of data needed. Hence, the current state of the art deep learning models are trained on fully or partially synthetic images. However, document shadow or shading removal results still suffer because: (a) prior methods rely on uniformity of local color statistics, which limit their application on real-scenarios with complex document shapes and textures and; (b) synthetic or hybrid datasets with non-realistic, simulated lighting conditions are used to train the models. In this paper we tackle these problems with our two main contributions. First, a physically constrained learning-based method that directly estimates document reflectance based on intrinsic image formation which generalizes to challenging illumination conditions. Second, a new dataset that clearly improves previous synthetic ones, by adding a large range of realistic shading and diverse multi-illuminant conditions, uniquely customized to deal with documents in-the-wild. The proposed architecture works in two steps. First, a white balancing module neutralizes the color of the illumination on the input image. Based on the proposed multi-illuminant dataset we achieve a good white-balancing in really difficult conditions. Second, the shading separation module accurately disentangles the shading and paper material in a self-supervised manner where only the synthetic texture is used as a weak training signal (obviating the need for very costly ground truth with disentangled versions of shading and reflectance). The proposed approach leads to significant generalization of document reflectance estimation in real scenes with challenging illumination. We extensively evaluate on the real benchmark datasets available for intrinsic image decomposition and document shadow removal tasks. Our reflectance estimation scheme, when used as a pre-processing step of an OCR pipeline, shows a 26% improvement of character error rate (CER), thus, proving the practical applicability. The data and code will be available at: https://github.com/cvlab-stonybrook/DocIIW.

The Barcelona Historical Marriage Database and the Baix Llobregat Demographic Database. From Algorithms for Handwriting Recognition to Individual-Level Demographic and Socioeconomic Data

“…However, this kind of approach consists in large deep learning architectures that need many labelled documents to train. Since labelled data is costly to produce and barely available, the generation of "realistic" synthetic documents (Das et al, 2020) to increase the amount of training data is worth exploring. All in all, the automatic extraction of information from images of population documents have shown to speed up the data entry process, although the performance of such techniques is not perfect, so a manual validation is still needed.…”

DocScanner: Robust Document Image Rectification with Progressive Learning

“…Recently, deep learning has been introduced to document image rectification with promising performance as well as a significant reduction in computational cost. In deep learning based methods [13], [14], [15], [16], [17], [18], [19], document image rectification is approached by directly regressing a dense 2D vector field (warping flow) that samples the pixels from the distorted images to the rectified ones. However, these methods still suffer from two non-trivial issues.…”