EmmDocClassifier: Efficient Multimodal Document Image Classifier for Scarce Data

Abstract

Document classification is one of the most critical steps in the document analysis pipeline. There are two types of approaches for document classification, known as image-based and multimodal approaches. Image-based document classification approaches are solely based on the inherent visual cues of the document images. In contrast, the multimodal approach co-learns the visual and textual features, and it has proved to be more effective. Nonetheless, these approaches require a huge amount of data. This paper presents a novel approach for document classification that works with a small amount of data and outperforms other approaches. The proposed approach incorporates a hierarchical attention network (HAN) for the textual stream and the EfficientNet-B0 for the image stream. The hierarchical attention network in the textual stream uses dynamic word embedding through fine-tuned BERT. HAN incorporates both the word level and sentence level features. While earlier approaches rely on training on a large corpus (RVL-CDIP), we show that our approach works with a small amount of data (Tobacco-3482). To this end, we trained the neural network at Tobacco-3482 from scratch. Therefore, we outperform the state-of-the-art by obtaining an accuracy of 90.3%. This results in a relative error reduction rate of 7.9%.

DocXClassifier: Towards a Robust and Interpretable Deep Neural Network for Document Image Classification

“…For example, the two classes Presentation and Scientific Report have an overlap of 3-4%. This finding is similar to that reported by Kanchi et al (2022) [45,Fig. 9] on their multimodal approach.…”

“…For example, the Scientific class is mainly confused with the Report and News classes, which makes perfect sense since these classes usually have similar visual semantics. This is again very similar to the results of Kanchi et al (2022) [45,Fig. 10] who found a large overlap between the Scientific and Report classes.…”

“…The lighter variant DocXClassifier- B performed slightly worse than the multimodal approach mentioned above, but still outperformed all existing imagebased approaches by a wide margin. With only ImageNet pre-training, we achieved an accuracy of 90.14% on the Tobacco3482 dataset, which is not only the highest reported image-based classification accuracy, but also comparable to the recently presented multimodal approach [45] based on the combination of EfficientNet and Hierarchical Attention VOLUME 4, 2016 Networks, which achieved an accuracy of 90.3%. We also analyzed the class distribution of the DocXClassifier-XL with RVL-CDIP pre-training on the To-bacco3482 dataset, as shown in Fig.…”

DocXClassifier: Towards an Interpretable Deep Convolutional Neural Network for Document Image Classification

“…For example, the two classes Presentation and Scientific Report have an overlap of 3-4%. This finding is similar to that reported by Kanchi et al (2022) [48,Fig. 9] on their multimodal approach.…”

“…It is interesting to note that even our lightest variant DocXClassifier-B achieved a comparable accuracy of 94.00%, and performed significantly better than all existing image-based models as well as some of the more sophisticated multimodal approaches [35], [46], [47], thus representing a good trade-off between accuracy and computational cost. It is important to note that two of the best performing multimodal solutions, those of Kanchi et al (2022) [48] and Bakkali et al (2020) [17], simply combined ConvNetbased visual backbones (EfficientNet and NasNet, respectively) with a Transformer-based textual backbone (BERT) to achieve extraordinary improvements in document classification. We suspect that using our improved ConvNet models as visual backbones in such multimodal approaches could lead to even better results.…”