SwiftNet: Real-time Video Object Segmentation

Abstract

Robotic-assisted tracheal intubation requires the robot to distinguish anatomical features like an experienced physician using deep-learning techniques. However, real datasets of oropharyngeal organs are limited due to patient privacy issues, making it challenging to train deep-learning models for accurate image segmentation. We hereby consider generating a new data modality through a virtual environment to assist the training process. Specifically, this work introduces a virtual dataset generated by the Simulation Open Framework Architecture (SOFA) framework to overcome the limited availability of actual endoscopic images. We also propose a domain adaptive Sim-to-Real method for oropharyngeal organ image segmentation, which employs an image blending strategy called IoU-Ranking Blend (IRB) and style-transfer techniques to address discrepancies between datasets. Experimental results demonstrate the superior performance of the proposed approach with domain adaptive models, improving segmentation accuracy and training stability. In the practical application, the trained segmentation model holds great promise for robotassisted intubation surgery and intelligent surgical navigation.

Deep learning for video object segmentation: a review

“…SwiftNet (Wang et al 2021a) improves STM in memory management and encoder architecture. The method utilises a similar strategy to AFB-URR to build the memory bank, but different update triggers are used.…”

Siamese Network with Interactive Transformer for Video Object Segmentation

“…FEELVOS (Voigtlaender et al 2019) proposes a global and a local pixel-level matching mechanism to gather information from the first and previous frames, respectively. Recently, the STM network (Oh et al 2019) is proposed to propagate the non-local object information, which has been a solid baseline in VOS task for its simple architecture and competitive performance (Seong, Hyun, and Kim 2020;Wang et al 2021). GC (Li, Shen, and Shan 2020) improves the STM architecture by only using a fixed-size feature representation and updates a global context to guide the segmentation of current frame.…”

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

“…Newly inferred frames can be added to the memory, and then the algorithm propagates forward in time. Derivatives either apply STM at other tasks [20,54], improve the training data or augmentation policy [20,21], augment the memory readout process [15,20,21,23,24], use optical flow [25], or reduce the size of the memory bank by limiting its growth [22,26].…”

“…Most current methods either fit a model using the initial segmentation [4,5,6,7,8] or leverage temporal propagation [9,10,11,12,13,14,15], particularly with spatio-temporal matching [16,17,18,19,20,21,22,23]. Space-Time Memory networks [17] are especially popular recently due to its high performance and simplicity -many variants [21,15,22,20,23,24,25,26], including competitions' winners [27,28], have been developed to improve the speed, reduce memory usage, or to regularize the memory readout process of STM.…”