MapReduce functions to remote sensing distributed data processing—Global vegetation drought monitoring as example

Abstract

Global change models for different applications are developed, according to the principle of remote sensing technology. Data for these models are generally remote sensing image, which is multiplatform, multidimentional, multiband, and multisource. Moreover, such data may be in different parts of the world and perhaps up to terabyte or petabyte level. Therefore, a data-intensive computing problem in the global change has emerged. Distributed computing infrastructures are suitable to store large-scale datalike satellite images that have to be written only once and read frequently. The emergence of the cloud computing technology brings new information architecture, and global change models implemented in the cloud platform provide users with stable, effective, on-demand cloud computing services. In this paper, the experiment is carried out on the cloud framework based on open cloud computing platform-Hadoop.In addition, on this framework, it achieves a prototype example for monitoring global vegetation drought conditions. Oriented to a variety of remote sensing data, we propose an abstract data format to achieve the unified description of remote sensing data. The data abstraction is to discretize the multidimensional remote sensing data for easy-distributed storage and computation. Using MapReduce paradigm, the complexity of remote sensing algorithms is resolved.Experimental results show that based on the parallel programming method, good scalability changing with the amount of processed data in the Hadoop distributed environment.In the era of big data, there are plenty of remote sensing information to choose. Reasonable choice and extraction of the most useful value are necessary requirements for the development of computer technology and network technology. It captures data from a huge remote sensing database then processes, transmits, analyzes these data, and extracts useful information to help users make a decision within a reasonable time, and if necessary, provides and shares the visualizations. The purpose of big data 1 technology is to extract intrinsic information from a huge data collection. The global change model is put forward for the global change issues, such as the vegetation drought-monitoring model, the dust storm monitoring model, the aerosol optical depth inversion model, and so on. From the data perspective, the global change model usually requires multisource, 2-4 multiplatform, 5,6 multisensor, 7 multiresolution, 8 and multispectral 9 remote sensing image data located across the globe, and a large number of gigabytes even terabytes of remote sensing image data are being generated every day over the world. From the computing point of view, global change models are usually composed of several remote sensing image processing algorithms. Thus, the simulation of global change models are typical data-intensive computing, which have frequent data input/output (I/O) but a relatively simple computing process.Today, data-intensive computing researches are often about distributed storage and computing sy...

How can Big Data and machine learning benefit environment and water management: a survey of methods, applications, and future directions

“…Drought analyses focus on early warning and impact assessment. Although the velocity of observation data is less of a concern and most of the analyses are performed offline, scalable distributed platforms are desirable to manage and synthesize information from multisources and multi-sensors [77]. Main data analytics challenges are related to (a) fusion of multi-sensor data to derive drought information, (b) development of robust long-term climatology for drought assessment, (c) development of robust change detection methods for drought warning, and (d) enabling self-service, region-specific drought analyses at different user-specified resolution or scales.…”

Cloud Computing Based on Computational Characteristics for Disaster Monitoring

“…The large amount of remote-sensing data received from satellites poses a substantial challenge for remote-sensing application scientists to effectively manage and analyze big data. To perform query analysis on remote-sensing data, scientists first decrypt the standard format of the original satellite data and localize the distributed preprocessing steps on the equal split of datasets in Hadoop [17,34]. They then use cloud computing technology to efficiently query the data.…”

“…First, there are various data formats and multi-source heterogeneity in remote-sensing image data. This issue has been addressed in previous studies [34]. Secondly, cloud computing programming for remote-sensing images is tightly coupled with image-processing algorithms, and often can only be used for a specific algorithm, i.e., parallel strategies are not reusable.…”

“…Generally, disaster-monitoring processing with remote-sensing data process starts with data acquisition, passes through remote-sensing image preprocessing, information extraction, and global monitoring applications, and ends with product visualization [34]. The remote-sensing data processing model based on MapReduce can appear complicated, but this complexity is usually because there are many MapReduce jobs, not the complexity of the individual Map or Reduce operations themselves.…”

Thirty-Year Dynamics of LULC at the Dong Thap Muoi Area, Southern Vietnam, Using Google Earth Engine

“…To date, remote sensing images have been the most widely and efficiently used data [29]. The recent developments in artificial intelligence and cloud computing, i.e., Microsoft Azure Cloud [30], Google Earth Engine (GEE) [31], Geospatial Amazon Web Services [32,33], and Open Data Cube [34], have made it easier to derive LULC classes from remotely sensed data. Herein, using GEE, historical imagery data are accessed and processed online using semi-automatic and automatic workflows [35].…”