Deep Learning for Text Detection and Recognition in Complex Engineering Diagrams

Abstract

Engineering drawings such as Piping and Instrumentation Diagrams contain a vast amount of text data which is essential to identify shapes, pipeline activities, tags, amongst others. These diagrams are often stored in undigitised format, such as paper copy, meaning the information contained within the diagrams is not readily accessible to inspect and use for further data analytics. In this paper, we make use of the benefits of recent deep learning advances by selecting models for both text detection and text recognition, and apply them to the digitisation of text from within real world complex engineering diagrams. Evaluation of such deep learning methods on a dataset of Piping and Instrumentation Diagrams from the Oil & Gas industry showed promising results on detecting and recognising text, without the need for pre-processing steps in complex engineering diagrams.

A review of deep learning methods for digitisation of complex documents and engineering diagrams

“…The reviewed literature is listed by application and extracted data type in Table 1. Amongst these applications, there has been a considerable research focus on P&IDs (Rahul et al 2019;Sinha et al 2019;Yu et al 2019;Mani et al 2020;Gao et al 2020;Elyan et al 2020a;Moreno-García et al 2020;Jamieson et al 2020;Nurminen et al 2020;Paliwal et al 2021a;Moon et al 2021;Kim et al 2021b;Stinner et al 2021;Paliwal et al 2021b;Toral et al 2021;Bhanbhro et al 2022;Hantach et al 2021). Another research area is architecture diagram digitisation (Ziran and Marinai 2018;Zhao et al 2020;Rezvanifar et al 2020;Kim et al 2021a;Renton et al 2021;Jakubik et al 2022).…”

“…Most of the P&ID digitisation literature focussed on the extraction of specific data types (Sinha et al 2019;Gao et al 2020;Elyan et al 2020a;Jamieson et al 2020;Nurminen et al 2020;Moon et al 2021;Kim et al 2021b;Stinner et al 2021;Paliwal et al 2021b;Toral et al 2021). There is a particular focus on P&ID symbols (Elyan et al 2020a;Nurminen et al 2020;Paliwal et al 2021b).…”

“…Meanwhile, Paliwal et al (2021b) used a graph-based approach for symbol recognition. Other studies focussed on the text (Jamieson et al 2020;Francois et al 2022) or connectors (Moon et al 2021). Studies that presented methods for multiple element types were also seen (Gao et al 2020;Stinner et al 2021).…”

“…Each diagram typically contains numerous text strings. For example, Jamieson et al (2020) used 172 P&IDs and reported on average 415 text instances per diagram, whilst Francois et al (2022) used 330 engineering documents and reported on average 440 text boxes. Unlike text in documents with a specific format, text in complex diagrams can be present anywhere in the drawing (Francois et al 2022), including within symbols (Mani et al 2020).…”

Optical character recognition on engineering drawings to achieve automation in production quality control

“…Previous studies, such as those by Moreno-García et al ( 2017) and Jamieson et al (2020), have made significant contributions to the field through the implementation of heuristics for segmentation in P&ID diagrams and the application of advanced deep learning techniques in OCR for raster diagrams, respectively. Other research, such as that by Rahul et al (2019), Kang et al (2019), andMani et al (2020), has focused on the complete digitization of P&ID diagrams, with a strong emphasis on the detection and recognition of symbols, text, and connections, utilizing a combination of image processing techniques, heuristics, and deep learning methods.…”