X-Net: A Binocular Summation Network for Foreground Segmentation

Abstract

In foreground segmentation, it is challenging to construct an effective background model to learn the spatial-temporal representation of the background. Recently, deep learning-based background models (DBMs) with the capability of extracting high-level features have shown remarkable performance. However, the existing state-of-the-art DBMs deal with video segmentation as single-image segmentation and ignore temporal cues in video sequences. To exploit temporal data sufficiently, this paper proposes a multi-input multi-output (MIMO) DBM framework for the first time, which is partially inspired by the binocular summation effect in human eyes. Our framework is an X-shaped network which allows the DBM to track temporal changes in a video sequence. Moreover, each output branch of our model could receive visual signals from two similar input frames simultaneously like the binocular summation mechanism. In addition, our model can be trained end-to-end using only a few training examples without any postprocessing. We evaluate our method on the largest dataset for change detection (CDnet 2014) and achieve the state-of-the-art performance by an average overall F-Measure of 0.9920. INDEX TERMS Foreground segmentation, background subtraction, deep learning, focal loss, binocular summation.

An Empirical Review of Deep Learning Frameworks for Change Detection: Model Design, Experimental Frameworks, Challenges and Research Needs

“…1) Pretraining and Finetuning: To take advantage of the foundational CNN architectures trained over large-scale image datasets, several studies have proposed the use of pretrained blocks or layers to enhance the representation capability of the CD models. The feature learning capability of off-theshelf CNN models such as VGG16, ResNet50, GoogleNet, DeepLab and ResNet18 have been successfully adapted for change detection in [69], [71], [74], [81], [83], [84], [89]- [91], [136]- [142]. Chen et al [69] designed an attention ConvL-STM to model pixel-wise changes over time.…”

“…Thus, it contains both the categorylevel semantics and finegrained details. Similarly, Zhang et al [136] augment a U-Net shaped architecture to detect the pixel-level change. The pretrained weights of VGG16 has been most widely used due to its simplicity and flexibility to alter the intermediate layers for enhance the encoder-decoder for change detection [71], [83], [84], [89]- [91], [136], [137], [139], [140].…”

“…Similarly, Zhang et al [136] augment a U-Net shaped architecture to detect the pixel-level change. The pretrained weights of VGG16 has been most widely used due to its simplicity and flexibility to alter the intermediate layers for enhance the encoder-decoder for change detection [71], [83], [84], [89]- [91], [136], [137], [139], [140]. Ou et al [138] designed the CD network with ResNet18 and encoder-decoder structure in order to retain the low-level features through a shallow architecture.…”

“…The existing deep learning CD methods have employed diverse network inputs to train the models. We can categorize them into networks with: single frame [67], [69], [71], [82]- [84], [90], [138]- [140], [143]- [148], 2 frames [66], [68], [70], [85], [87], [89], [92], [136], [149]- [156], 3-10 frames [20], [74], [80], [91], [141], [157], [158], 11-30 frames [81], [86], [137], [142], [159]- [161], and 50 frames [19], [72]. The methods with single frame input primarily rely on the availability of certain labeled frames in a video.…”

“…The methods with 2 frames as input aims to detect the changing pixels between them. Usually, these methods [66], [68], [70], [85], [87], [89], [92], [136], [149]- [156] first compute a background with the help of a traditional background subtraction method. Thereafter, the current frame and the background image is fed to the network to produce Fig.…”