Multimodal Classification of Safety-Report Observations

Abstract

Modern businesses are obligated to conform to regulations to prevent physical injuries and ill health for anyone present on a site under their responsibility, such as customers, employees and visitors. Safety officers (SOs) are engineers, who perform site audits to businesses, record observations regarding possible safety issues and make appropriate recommendations. In this work, we develop a multimodal machine-learning architecture for the analysis and categorization of safety observations, given textual descriptions and images taken from the location sites. For this, we utilize a new multimodal dataset, Safety4All, which contains 5344 safety-related observations created by 86 SOs in 486 sites. An observation consists of a short issue description, written by the SOs, accompanied with images where the issue is shown, relevant metadata and a priority score. Our proposed architecture is based on the joint fine tuning of large pretrained language and image neural network models. Specifically, we propose the use of a joint task and contrastive loss, which aligns the text and vision representations in a joint multimodal space. The contrastive loss ensures that inter-modality representation distances are maintained, so that vision and language representations for similar samples are close in the shared multimodal space. We evaluate the proposed model on three tasks, namely, priority classification of input observations, observation assessment and observation categorization. Our experiments show that inspection scene images and textual descriptions provide complementary information, signifying the importance of both modalities. Furthermore, the use of the joint contrastive loss produces strong multimodal representations and outperforms a baseline simple model in tasks fusion. In addition, we train and release a large transformer-based language model for the Greek language based on the Electra architecture.

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…[169] has 2 articles in the medical domain. In total, 60 articles were found related to the domain agnostic domain, where, 16 were in classification [170]- [185], 9 were in detection [186]- [194], 11 were in analysis [195]- [205], 8 were in recognition [206]- [213], 9 were in prediction [214]- [222], 1 was in language processing [223], 2 were in image processing [224], [225], 1 was in image retrieval [47], 2 were in integration [226], [227] and 1 was in segmentation [228]. Of 43 articles in the human activity domain, 20 were in recognition [229]- [248], 8 were in detection [249]- [256], 5 were in classification [257]- [261], 4 were in analysis [262]- [265], 3 were in identification [266]- [268] and 1 was in comparison [269], monitoring [270] and assessment [271].…”

“…[138], [139], [160], [176], [178], [179], [185]- [187], [198], [202], [216], [217], [220], [221], [230], [236], [257], [266], [267], [272], [275], [301], [317], [345], [348], [352], [356], [378], [407], [408], [418], [419], 10 [49], [112], [113], [128], [193], [203], [227], [249], [293], [383] , and 14 [73], [94],…”

“…Figure 11 shows that neural networks (NNs), a subtype of joint representation, have more consideration than others to represent data. The use of NNs to represent visual, audio and text data is increasing [179], [185], [186], [216]. Graphical models frame representation using implicit random variables where probabilistic graphical models such as deep Boltzmann machines (DBM) [113] are used.…”

Autonomous Systems Functional Safety Overview with Multimodality and Explainability Perspectives

“…Multimodal representation represents data using information from multiple entities (Paraskevopoulos et al, 2022). Joint representations combine the unimodal signals into the same representation space, while coordinated representations process unimodal signals distinctly, but enforce some similarity constraints on them to bring them to a coordinated space.…”

“…Work proposed by Arrieta et al, 2020, Coulibaly et al, 2022, Gohel et al, 2016, and Khan & Vice, 2022 deals with some AI and Explainability without any functional safety consideration. Contributions dealing with some safety, incorporating AI, particularly multimodality such as Paraskevopoulos et al, 2022 have no hardware failure consideration. This paper is organized as follows.…”