Attention-based encoder-decoder networks for workflow recognition

Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models

“…In the context of manufacturing, a limited number of papers were found, with notable works being [6,7,[33][34][35][36][37] The application of classic machine learning models was predominant in these papers, where models like Hidden Markov Models [6,34,35] or Support Vector Machines [7] were employed following the manual extraction of features. Makantasis et al [33] applied a deep learning model based on a 2D convolutional neural network and multi-layer perceptron, using manually created features with the Motion History Image algorithm.…”

“…Makantasis et al [33] applied a deep learning model based on a 2D convolutional neural network and multi-layer perceptron, using manually created features with the Motion History Image algorithm. Zhang et al [36] proposed an encoder-decoder framework for workflow recognition using a 3D CNN, transforming the activations of the last convolutional layer into clip-level representations, which were then fed into an LSTM network with an attention mechanism for enhanced recognition. In paper [37], a system comprising three stages was developed: spatial feature extraction using a Vectors Assembly Graph (VAG) and graph networks from RGB-D video frames; contact force feature extraction via a sliding window technique; and action segmentation through a multi-stage temporal convolution network (MS-TCN) that combines these features.…”

“…Rude et al [6] used a dataset from [39], collected with a depth sensor for painting manufactured parts, recording data over a single workday with two different workers. The XIOLIFT dataset was utilized in [36], capturing data from six production lines in an elevator factory over one working month. A newly created dataset of 11 assembly actions performed in laboratory conditions, featuring RGB-D videos and contact force data, was used in paper [37].…”