Automatic Workflow Monitoring in Industrial Environments

Abstract

Abstract. Robust automatic workflow monitoring using visual sensors in industrial environments is still an unsolved problem. This is mainly due to the difficulties of recording data in work settings and the environmental conditions (large occlusions, similar background/foreground) which do not allow object detection/tracking algorithms to perform robustly. Hence approaches analysing trajectories are limited in such environments. However, workflow monitoring is especially needed due to quality and safety requirements. In this paper we propose a robust approach for workflow classification in industrial environments. The proposed approach consists of a robust scene descriptor and an efficient time series analysis method. Experimental results on a challenging car manufacturing dataset showed that the proposed scene descriptor is able to detect both human and machinery related motion robustly and the used time series analysis method can classify tasks in a given workflow automatically.

Temporal Relations in Videos for Unsupervised Activity Analysis

“…This data has previously been used for learning spatio-temporal scene topics [4]. As car and pedestrian flow patterns are of importance in this scene, we apply the motion features proposed in [17] and extract the motion in 18 × 18 pixel patches. To additionally encode the motion direction, a forgetting rate of 0.95 is applied.…”

On hierarchical modelling of motion for workflow analysis from overhead view

“…We will outline the conditions which would need to be satisfied for this approach to be applicable to a new problem and provide guidelines for how to do so. In addition to what has been offered in [19,20] our approach enables us to (i) localize the tasks in image frame and (ii) offer a stage by stage commentary-like report of the task as it happens.…”

“…Manufacturing [7,9,19,20] and office [3,10,17], environments, medical operating rooms [12], TV studios [13] and smart homes for the elderly [8] are amongst the settings wherein 'action' perception and understanding from video streams had been studied. Pinhanez and Bobick [13] were among the first to analyse the actions within a video stream.…”

“…Voulodimos et al [20] and Oliver et al [11] used HMMs as a means for low level sensory data or high level decision fusion. Criticizing HMMs as requiring well-defined states and robust features, Veres et al [19] opt for a neural network: they used an echo state network to learn the patterns within an annotated global scene descriptor. Shi et al [17] proposed the use of propagation networks to handle parallel tracks within one action.…”

“…The other point of concern is the spatial segmentation of actions. The complex and cluttered environment which is analysed in this paper, poses an overwhelming detection and tracking problem as was experiences by [7,19,20]. They have thus opted for a global scene descriptor and abandoned the problem of spatial segmentation.…”