Multimodal Fusion of Sentinel 1 Images and Social Media Data for Snow Depth Estimation

Abstract

Recent developments in remote sensing have shown that snow depth can be estimated accurately on a global scale using satellite images through cross-polarization and copolarization backscatter measurements. This method does however have some limitations in low-land areas with dense forest coverage and shallow snow, that is often found nearby urban areas. In these areas, citizen observations can be fused with satellite-based estimations to deliver more accurate solutions. To that end, we use snow-related tweets that have been annotated by artificial intelligence (AI) methods and are introduced in a novel neural network model, aiming to increase the estimation accuracy of the state-of-the-art remote sensing method. The proposed model combines the estimated snow depth from Sentinel 1 images with the number of Twitter posts and Twitter images that are semantically relevant to snow. The use of instant social media data for purposes of snow depth estimation is investigated, validated and tested in Finland. Our results show that this approach does improve the snow depth estimation, highlighting its potential for use in civil protection agencies in managing snow conditions.

Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications

“…Others used NNs to learn new features, e.g. for bands in optical-MS images [19], or for image and text data [174].…”

“…Meanwhile, active-based views are the quintessential views chosen to complement (and improve by fusion) the optical in classification tasks with MV models, e.g. by using SAR view [1], [15], [33], [34], [36], [99], [113], [120], [124], [126], [143], [163], [173], [174], [188], [190], [193], [199]- [202] or LiDAR view [94], [139]. Furthermore, the DSM view has been widely used together with the optical view [17]- [19], [55], [58], [64], [65], [76], [80], [84], [85], [88], [89], [92], [104], [123], [184], where on some occasions is a LiDAR-derived DSM [32], [34], [35], [59], [62], [81], [86], [87], [93], …”

Earthquakes: From Twitter Detection to EO Data Processing

“…Furthermore, Panteras and Cervone [13] suggested that the increased temporal resolution of crowdsourced data can partially compensate for the limitations of satellite data and thus presented a geostatistical analysis of combined satellite and Twitter data for the delineation of flood extent. More recently, social media data (Twitter text and images) have been fused with Sentinel-1 images on the snow depth estimation problem [14]. The use of relevant text and images that contain the concept "snow" has shown that it can improve the estimated snow depth, exploiting the massive ground observations from in situ social media posts.…”

“…4, we can also see a detected event that was not clearly identified by the daily count of tweets and matches with a magnitude 5.2 earthquake southeast of Chitose, Hokkaido, on January 27, 2021. 14 The output of the data pipeline 1 is two different heat maps, which can be used to visually check the geo-distribution of tweets for each event similar to Fig. 2, and a CSV file containing the locations and timestamps of the detected events.…”

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

“…It is worth mentioning that authors propose a novel solution to deal with the tricky situation where one modality data is missing during the testing phase.Besides, other kinds of GBD are also widely used in the fusion task. Taxi trajectory data and user visit data are utilized to identify the urban functional areas(Qian et al, 2020;Cao et al, 2020) Mantsis et al (2022). use the snow-related twitters along with Sentinel-1 images to realize snow depth estimation Lu et al (2022).…”