Progress Estimation and Phase Detection for Sequential Processes

Abstract

Process modeling and understanding are fundamental for advanced human-computer interfaces and automation systems. Most recent research has focused on activity recognition, but little has been done on sensor-based detection of process progress. We introduce a real-time, sensor-based system for modeling, recognizing and estimating the progress of a work process. We implemented a multimodal deep learning structure to extract the relevant spatio-temporal features from multiple sensory inputs and used a novel deep regression structure for overall completeness estimation. Using process completeness estimation with a Gaussian mixture model, our system can predict the phase for sequential processes. The performance speed, calculated using completeness estimation, allows online estimation of the remaining time. To train our system, we introduced a novel rectified hyperbolic tangent (rtanh) activation function and conditional loss. Our system was tested on data obtained from the medical process (trauma resuscitation) and sports events (Olympic swimming competition). Our system outperformed the existing trauma-resuscitation phase detectors with a phase detection accuracy of over 86%, an F1-score of 0.67, a completeness estimation error of under 12.6%, and a remaining-time estimation error of less than 7.5 minutes. For the Olympic swimming dataset, our system achieved an accuracy of 88%, an F1-score of 0.58, a completeness estimation error of 6.3% and a remaining-time estimation error of 2.9 minutes.

RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations

“…1. This approach has been utilized to estimate the remaining duration of activities in [15]. We compare to this approach in order to investigate whether the derived RSD is better than the direct RSD estimation.…”

Region-based Activity Recognition Using Conditional GAN

“…Previous approaches have introduced additional sensors to enrich the phase information. For example, the sound captured by the microphone array proved useful for resuscitation phase detection [15, 17]. Our research directly generated the team role masks and not only outperformed previous work on the same dataset but also provided team-role masks that can be used in other applications.…”

“…Instead of using the difference of adjacent frames as an indicator, we selected the frames based on the resuscitation process phase. The resuscitation phases are the major steps of the process, with each trauma resuscitation contains six phases: pre-arrival, patient-arrival, primary, secondary, post-secondary and patient-leave [17]. We randomly selected five consecutive frames within each phase in 20 videos and labeled four roles of the trauma team (Table 2) and the patient bed (where the resuscitation activities occur) with categorical labels (Table 2).…”

Am I Done? Predicting Action Progress in Videos

“…Here the goal is to accurately detect, as soon as possible, when an action has started and when it has finished, but they do not have a model to estimate the progress. Some similarities are shared with the task of activity recognition [30,54] where the goal is to detect which high-level phase is currently in progress in a long procedure composed by multiple actions. Differently, action progress focus on such actions which can be made of one or many movements but can be detected by visual means only.…”