Dewarping Document Image by Displacement Flow Estimation with Fully Convolutional Network

Abstract

As camera-based documents are increasingly used, the rectification of distorted document images becomes a need to improve the recognition performance. In this paper, we propose a novel framework for both rectifying distorted document image and removing background finely, by estimating pixel-wise displacements using a fully convolutional network (FCN). The document image is rectified by transformation according to the displacements of pixels. The FCN is trained by regressing displacements of synthesized distorted documents, and to control the smoothness of displacements, we propose a Local Smooth Constraint (LSC) in regularization. Our approach is easy to implement and consumes moderate computing resource. Experiments proved that our approach can dewarp document images effectively under various geometric distortions, and has achieved the state-of-the-art performance in terms of local details and overall effect. Our code and trained models are available at https://github.com/gwxie/Dewarping-Document-Image-By-Displacement-Flow-Estimation.

DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction

“…Then, we use image similarity and OCR accuracy to evaluate the performance of illumination correction. To be specific, for pixel alignment, we use Local Distortion (LD) [43] as recommended in [7,22,41] to evaluate the geometric distortion of rectified images. For image similarity, we use Multi-Scale Structural SIMilarity (MS-SSIM) [39] as previous works [7,22,41] suggest.…”

“…To be specific, for pixel alignment, we use Local Distortion (LD) [43] as recommended in [7,22,41] to evaluate the geometric distortion of rectified images. For image similarity, we use Multi-Scale Structural SIMilarity (MS-SSIM) [39] as previous works [7,22,41] suggest. For OCR, following [7,22], we choose Edit Distance (ED) [17] and Character Error Rate (CER) to evaluate the capacity on text recognition.…”

DocScanner: Robust Document Image Rectification with Progressive Learning

“…However, the estimation and subsequent stitching of the warping flow patches heavily increase the computational cost. More recently, based on Fully Convolutional Network [49], Xie et al [16] perform a foreground/background classification as a post-processing to refine the predicted forward warping flow on boundary regions of the document.…”

“…Recently, deep learning has been introduced to document image rectification with promising performance as well as a significant reduction in computational cost. In deep learning based methods [13], [14], [15], [16], [17], [18], [19], document image rectification is approached by directly regressing a dense 2D vector field (warping flow) that samples the pixels from the distorted images to the rectified ones. However, these methods still suffer from two non-trivial issues.…”

“…First, a common problem in the current learning-based approaches [13], [14], [15], [16], [19] is that the boundary regions cannot be well rectified. In fact, one of the reasons is that many of these methods [13], [14], [16] directly take the whole distorted image as input to rectification networks (pipeline (I) in Figure 1), which involves extra implicit learning to identify the foreground document for predicting the rectified document. As a result, the rectified documents often struggle with incomplete or redundant boundaries, which cause further geometric distortion to the nearby contents.…”