Towards Holistic Surgical Scene Understanding

MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation

“…In the same way, we design class-specific score thresholds to counter the high inter-class score variability caused by the class frequency miss-balance. The following section explains how we use the video analysis methodology of [33] to improve our segment classification.…”

“…Temporal Consistency: Following TAPIR [33], our temporal consistency module uses a Multi-Scale Vision Transformer (MViT) [31] as a backbone for video analysis. TAPIR uses a time window centered on a keyframe to compute global spatio-temporal features that encode the complex temporal context of the middle frame.…”

“…For the temporal consistency module, we use TAPIR's official implementation [33] pretrained in Kinetics 400 [35]. We train the model for 20 epochs in a single NVIDIA Quadro RTX 8000 GPU with a window size of 8 frames, a stride of 1 frame, a batch size of 12, and an ADAMW optimizer.…”

“…However, this method uses a pixel-classification modality that does not differ among instances. Recently, TAPIR [33] addressed these problems and posed a fully transformer-based model for multi-level surgical workflow analysis that incorporates Deformable DETR [28] as a box proposal network for instrument detection.…”

“…MATIS utilizes Mask2Former [29] with a Swin Transformer backbone [22] as an instance segmentation baseline to leverage its multi-scale deformable attention pixel decoder and masked attention mechanisms. Furthermore, MATIS uses a TAPIRlike [33] architecture to include temporal information and classify the regions proposed by its masked attention baseline. In summary, our key contributions are:…”