Multi-view Surgical Video Action Detection via Mixed Global View Attention

Schmidt, Adam; Sharghi, Aidean; Haugerud, Helene; Oh, Daniel S.; Mohareri, Omid

doi:10.1007/978-3-030-87202-1_60

Cited by 14 publications

(17 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these settings, action recognition is typically performed on short image sequences, referred to as "clips", which average only 10 seconds in length for Kinetics [17]. Surgical workflow analysis from room cameras is currently limited to one large dataset, which is not publicly available [30,28]. Despite publishing both RGB and depth sequences, the MVOR [31] dataset consists of 732 frames at relatively low FPS, making video action recognition infeasible.…”

Section: Related Workmentioning

confidence: 99%

“…In the medical domain, events of interest may appear indistinguishable on such a short timescale, requiring the aggregation of additional temporal information. Cliplevel architectures, however, remain relevant even for longer videos, as they can be similarly combined with LSTMs to aggregate both short and long-term image features [28]. Extracting clip features can provide performance benefits as they reduce dimensionality compared to image level features [30].…”

Section: Related Workmentioning

confidence: 99%

“…3). This distinguishes our acquisitions from existing surgical action recognition datasets, which span many hours without interruption [32,28]. For instance, patient-in averages slightly over 30 seconds in length, making the long-term accumulation of temporal features challenging.…”

Section: Datasetmentioning

confidence: 99%

“…Classifying procedures obtained from OR room cameras has become a topic of interest in the past several years [28,30]. Temporally aggregating features extracted from video frames enables an understanding of complex workflow tasks that single image analysis alone could not achieve.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Know your sensORs -- A Modality Study For Surgical Action Classification

Bastian¹,

Czempiel²,

Heiliger³

et al. 2022

Preprint

View full text Add to dashboard Cite

The surgical operating room (OR) presents many opportunities for automation and optimization. Videos from various sources in the OR are becoming increasingly available. The medical community seeks to leverage this wealth of data to develop automated methods to advance interventional care, lower costs, and improve overall patient outcomes. Existing datasets from OR room cameras are thus far limited in size or modalities acquired, leaving it unclear which sensor modalities are best suited for tasks such as recognizing surgical action from videos. This study demonstrates that surgical action recognition performance can vary depending on the image modalities used. We perform a methodical analysis on several commonly available sensor modalities, presenting two fusion approaches that improve classification performance. The analyses are carried out on a set of multi-view RGB-D video recordings of 18 laparoscopic procedures.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Know your sensORs -- A Modality Study For Surgical Action Classification

Bastian¹,

Czempiel²,

Heiliger³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Several works have used multi-angle photography during MIS, with the internal camera capturing the tools being used, and the other cameras identifying the area surrounding the surgery. Using such systems enables detection of automatic activity [15], and inoperative errors and distractions, as well as the evaluation of skill level [16,17]. In another open surgery study, multiple cameras were placed in the surgical lamp [18,19].…”

Section: Introductionmentioning

confidence: 99%

Open surgery tool classification and hand utilization using a multi-camera system

Basiev¹,

Goldbraikh²,

Pugh³

et al. 2021

Preprint

View full text Add to dashboard Cite

The goal of this work is to use multi-camera video to classify open surgery tools as well as identify which tool is held in each hand. Multi-camera systems help prevent occlusions in open surgery video data. Furthermore, combining multiple views such as a Top-view camera covering the full operative field and a Close-up camera focusing on hand motion and anatomy, may provide a more comprehensive view of the surgical workflow. However, multi-camera data fusion poses a new challenge: a tool may be visible in one camera and not the other. Thus, we defined the global ground truth as the tools being used regardless their visibility. Therefore, tools that are out of the image should be remembered for extensive periods of time while the system responds quickly to changes visible in the video. Methods: Participants (n=48) performed a simulated open bowel repair. A Top-view and a Close-up cameras were used. YOLOv5 was used for tool and hand detection. A high frequency LSTM with a 1 second window at 30 frames per second (fps) and a low frequency LSTM with a 40 second window at 3 fps were used for spatial, temporal, and multi-camera integration. Results: The accuracy and F1 of the six systems were: Top-view

show abstract

Multi-modal Unsupervised Pre-training for Surgical Operating Room Workflow Analysis

Jamal

Mohareri

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Multi-view Surgical Video Action Detection via Mixed Global View Attention

Cited by 14 publications

References 25 publications

Know your sensORs -- A Modality Study For Surgical Action Classification

Know your sensORs -- A Modality Study For Surgical Action Classification

Open surgery tool classification and hand utilization using a multi-camera system

Multi-modal Unsupervised Pre-training for Surgical Operating Room Workflow Analysis

Contact Info

Product

Resources

About