Workflow monitoring based on 3D motion features

Abstract

Activity recognition has primarily addressed the identification of either actions or well-defined interactions among objects in a scene. In this work, we extend the scope to the study of workflow monitoring. In a workflow, ordered groups of activities (phases) with different durations take place in a constrained environment and create temporal patterns across the workflow instances. We address the problem of recognizing phases, based on exemplary recordings. We propose to use Workflow-HMMs, a form of HMMs augmented with phase probability variables that model the complete workflow process. This model takes into account the full temporal context which improves on-line recognition of the phases, especially in case of partial labeling. Targeted applications are workflow monitoring in hospitals and factories, where common action recognition approaches are difficult to apply. To avoid interfering with the normal workflow, we capture the activity of a room with a multiple-camera system. Additionally, we propose to rely on real-time low-level features (3D motion flow) to maintain a generic approach. We demonstrate our methods on sequences from medical procedures performed in a mock-up operating room. The sequences follow a complex workflow, containing various alternatives.

Surgical Task and Skill Classification from Eye Tracking and Tool Motion in Minimally Invasive Surgery

“…They have been assessed in many studies by either tracking the surgeon's body motion in the operating room [1] or hand motion while performing a specific surgical task [2][3] [4]. The Imperial College Surgical Assessment Device (ICSAD) system tracks the surgeon's hand motions during a surgery using electromagnetic (EM) markers [4].…”

Real-Time Range Imaging in Health Care: A Survey

“…One example of using range images for workflow recognition has been shown by Padoy et al, using a real-time 3-D reconstruction of a simulated OR that was obtained from a multi-camera system [67]. Based on the 3-D reconstruction, the motion flow of staff and objects inside the OR was computed and a statistical model was generated.…”

On hierarchical modelling of motion for workflow analysis from overhead view

“…Considering the sequential nature of the patterns, using a hidden Markov model is a justified and popular choice. Padoy et al [12] studied the workflows in a simulated operating theater. They introduced the Workflow-HMM which is a form of two-layered hierarchical HMM.…”

“…Manufacturing [7,9,19,20] and office [3,10,17], environments, medical operating rooms [12], TV studios [13] and smart homes for the elderly [8] are amongst the settings wherein 'action' perception and understanding from video streams had been studied. Pinhanez and Bobick [13] were among the first to analyse the actions within a video stream.…”