A Few Shot Multi-representation Approach for N-Gram Spotting in Historical Manuscripts

Biswas, Sanket; Souibgui, Mohamed Ali; Bensalah, Asma; Lladós, Josep; Fornés, Alícia; Marcelli, Angelo

doi:10.1007/978-3-031-21648-0_1

Cited by 2 publications

(1 citation statement)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Later, and following the success of other computer vision tasks, machine learning-based techniques (in particular convolutional neural networks (CNNs)) are now dominating the field of word retrieval [9,10,11,12,13,14]. But, despite the significant improvement in the performance of these models over classical handcrafted techniques, they have their associated shortcomings.…”

Section: Introductionmentioning

confidence: 99%

ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical Handwritten Documents

Jemni¹,

Ammar²,

Souibgui³

et al. 2023

Preprint

View full text Add to dashboard Cite

Keyword spotting (KWS) in historical documents is an important tool for the initial exploration of digitized collections. Nowadays, the most efficient KWS methods are relying on machine learning techniques that require a large amount of annotated training data. However, in the case of historical manuscripts, there is a lack of annotated corpus for training. To handle the data scarcity issue, we investigate the merits of the self-supervised learning to extract useful representations of the input data without relying on human annotations and then using these representations in the downstream task. We propose ST-KeyS, a masked auto-encoder model based on vision transformers where the pretraining stage is based on the mask-and-predict paradigm, without the need of labeled data. In the fine-tuning stage, the pre-trained encoder is integrated into a siamese neural network model that is fine-tuned to improve feature embedding from the input images. We further improve the image representation using pyramidal histogram of characters (PHOC) embedding to create and exploit an intermediate representation of images based on text attributes. In an exhaustive experimental evaluation on three widely used benchmark datasets (Botany, Alvermann Konzilsprotokolle and George Washington), the proposed approach outperforms state-of-the-art methods trained on the same datasets.

show abstract