Spatio-Temporal Self-Attention Network for Fire Detection and Segmentation in Video Surveillance

Abstract

Convolutional Neural Network (CNN) based approaches are popular for various image/video related tasks due to their state-of-the-art performance. However, for problems like object detection and segmentation, CNNs still suffer from objects with arbitrary shapes or sizes, occlusions, and varying viewpoints. This problem makes it mostly unsuitable for fire detection and segmentation since flames can have an unpredictable scale and shape. In this paper, we propose a method that detects and segments fireregions with special considerations of their arbitrary sizes and shapes. Specifically, our approach uses a self-attention mechanism to augment spatial characteristics with temporal features, allowing the network to reduce its reliance on spatial factors like shape or size and take advantage of robust spatial-temporal dependencies. As a whole, our pipeline has two stages: In the first stage, we take out region proposals using Spatial-Temporal features, and in the second stage, we classify whether each region proposal is flame or not. Due to the scarcity of generous fire datasets, we adopt a transfer learning strategy to pre-train our classifier with the ImageNet dataset. Additionally, our Spatial-Temporal Network only requires semi-supervision, where it only needs one ground-truth segmentation mask per frame-sequence input. The experimental results of our proposed method significantly outperform the state-of-the-art fire detection with a 2 ∼ 4% relative enhancement in F1-score for large scale fires and a nearly ∼ 60% relative improvement for small fires at a very early stage.

Comparative Research on Forest Fire Image Segmentation Algorithms Based on Fully Convolutional Neural Networks

“…In general, these methods not only require a large amount of resource input, but also are severely limited by the weather and environmental conditions, making it difficult to ensure the quality of forest fire monitoring. With the advancement in image-processing technology, in recent years, authorities have increased the use of unmanned aerial vehicles (UAVs) and surveillance for aerial monitoring of forest areas [9,10], collecting and real-timeprocessing fire images for earlier warning and intervention. In order to identify forest fire areas in the images, flame detection techniques are introduced in some studies.…”

“…Zhang et al [23] developed an effective SqueezeNet-based asymmetric encoder-decoder U-shape architecture, mainly functioning as an extractor and a discriminator of a forest fire, which is relatively reliable with good accuracy and prediction time. M Shahid et al [10] introduced the spatio-temporal network and proposed a two-stage cascaded architecture to improve accuracy, achieving better performance than the existing state-of-the-art methods. In general, these works only analyze from the perspective of a single model and backbone network, lacking comparisons between models, and therefore cannot compare the performance of different models in forest fire segmentation tasks.…”

Dynamic attention network for flame detection

“…Attention mechanisms have shown excellent performance in a wide range of computer vision tasks, such as flame detection [28]. We find recent studies only focus on channel or spatial attentions for flame detection [29–32]. Based on this deficiency, we introduce a novel mechanism named dynamic attention inspired by [33], which separately deploys weights on particular level‐wise and spatial‐wise dimensions.…”

“…Dynamic attention strategy for scale and spatial locations: Attention mechanisms have shown excellent performance in a wide range of computer vision tasks, such as flame detection [28]. We find recent studies only focus on channel or spatial attentions for flame detection [29][30][31][32].…”

Forest Fire Smoke Detection Based on Deep Learning Approaches and Unmanned Aerial Vehicle Images

“…Fires can negatively impact ocean infrastructure, coral reefs, and other underwater environments due to the way in which they can change biomass accumulation in ecosystems and disturb the natural cycle of water [ 20 ]. Also detrimental to the well-being of humans and animals is the fact that smoke from forest fires may greatly limit the production of photosynthesis [ 21 ]. The Amazon jungle, for instance, is home to an astonishing number of both plant and animal species, many of which have not been thoroughly described.…”