A Generalized Multimodal Deep Learning Model for Early Crop Yield Prediction

Kaur, Arshveer; Goyal, Poonam; Sharma, Kartik; Sharma, Lakshay; Goyal, Navneet

doi:10.1109/bigdata55660.2022.10020917

Cited by 4 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(ii) our baselines for single modality SITS. (iii) our baselines on high temporal SITS generated using generative fusion models Histogram models: We considered four existing CYP models CNN (Sun et al), CNN+GP (Sun et al), CNN+LSTM (Sun et al 2020), and CYN (Kaur et al 2022) working on histogram TS to compare with the proposed PatchNet. CNN and CNN+GP use only surface reflectance data and did not exploit the temporal dependency in the data.…”

Section: Baselines For Comparisonmentioning

confidence: 99%

“…The learnable patch selection mechanism eliminates the need for full spatial processing of SITS, thereby reducing the amount of processing by a factor of p with some additional overheads and still achieves SOTA results for end tasks. Existing methods deal with the processing challenges by transforming the images into histograms You et al 2017;Sun et al 2020;Kaur et al 2022). A few researchers have also tried to transform images into singlevalue numeric vegetation indices (Sakamoto 2020;Skakun et al 2021;Ji et al 2022;Choudhary et al 2019).…”

Section: Introductionmentioning

confidence: 99%

“…Code is available at (Poonam Goyal 2022) Generative models such as Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) (Gao et al 2006), Robust Spatiotemporal Fusion Network (RSFN) (Tan et al 2022), and GAN ( (Bouabid et al 2020) have been used to generate Landsat-8 like images at higher temporal granularity. These models are constrained by the requirement of availability of Landsat & MODIS images captured on the same day and may also propagate existing noise.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Representation Learning of Satellite Image Time Series and Their Fusion for Spatiotemporal Applications

Goyal,

Kaur,

Ram

et al. 2024

AAAI

View full text Add to dashboard Cite

Satellite data bolstered by their increasing accessibility is leading to many endeavors of automated monitoring of the earth's surface for various applications. Such applications demand high spatial resolution images at a temporal resolution of a few days which entails the challenge of processing a huge volume of image time series data. To overcome this computing bottleneck, we present PatchNet, a bespoke adaptation of beam search and attention mechanism. PatchNet is an automated patch selection neural network that requires only a partial spatial traversal of an image time series and yet achieves impressive results. Satellite systems face a trade-off between spatial and temporal resolutions due to budget/technical constraints e.g., Landsat-8/9 or Sentinel-2 have high spatial resolution whereas, MODIS has high temporal resolution. To deal with the limitation of coarse temporal resolution, we propose FuSITSNet, a twofold feature-based generic fusion model with multimodal learning in a contrastive setting. It produces a learned representation after fusion of two satellite image time series leveraging finer spatial resolution of Landsat and finer temporal resolution of MODIS. The patch alignment module of FuSITSNet aligns the PatchNet processed patches of Landsat-8 with the corresponding MODIS regions to incorporate its finer resolution temporal features. The untraversed patches are handled by the cross-modality attention which highlights additional hot spot features from the two modalities. We conduct extensive experiments on more than 2000 counties of US for crop yield, snow cover, and solar energy prediction and show that even one-fourth spatial processing of image time series produces state-of-the-art results. FuSITSNet outperforms the predictions of single modality and data obtained using existing generative fusion models and allows for monitoring of dynamic phenomena using freely accessible images, thereby unlocking new opportunities.

show abstract

Section: Baselines For Comparisonmentioning

confidence: 99%