Hadoop Paradigm for Satellite Environmental Big Data Processing

Abstract

The important growth of industrial, transport, and agriculture activities, has not led only to the air quality and climate changes issues, but also to the increase of the potential natural disasters. The emission of harmful gases, particularly: the Vertical Column Density (VCD) of CO, SO2 and NOx, is one of the major factors causing the aforementioned environmental problems. Our research aims to contribute finding solution to this hazardous phenomenon, by using remote sensing (RS) techniques to monitor air quality which may help decision makers. However, RS data is not easy to manage, because of their huge amount, high complexity, variety, and velocity, Thus, our manuscript explains the different aspects of the used satellite data. Furthermore, this article has proven that RS data could be regarded as big data. Accordingly, we have adopted the Hadoop big data architecture and explained how to process efficiently RS environmental data.

Big data and remote sensing: A new software of ingestion

“…Consequently, the processing is challenging and also take a vital execution time. For this aim, we have designed an original BD architecture to split and facilitate problem-solving the problems of RS BD [11].…”

Agricultural Big Data Architectures in the Context of Climate Change: A Systematic Literature Review

“…For example, Lopez et al, used HIVE, XPath, and XQuery [49]. On the other hand, Semlali et al, usde Map-Reduce with Spark and Storm [40]. Finally, Bendre et al, used Mahaut, Drill, and Storm [67].…”

“…In the case of climate management, a fairly widely used option was satellite remote sensing [40] and data from meteorological services. Another natural source of data was provided by government ministries, communities, and farmers [66].…”

“…At this stage, the Big Data technologies used were Raspberry Pi, BUFRextract (Python), pybufr_ecmwf (python), and wget, curl; and as data repositories, the MySQL database engine for structured data, Hadoop HDFS for unstructured data. In addition, processing algorithms developed in Java, Python, and Bash processes were used [40].…”

Research Progress on Models, Algorithms, and Systems for Remote Sensing Spatial-Temporal Big Data Processing

“…Task parallelism, also known as function parallelism or control parallelism, can be further divided into processes parallelism, thread parallelism and instruction parallelism according to the granularity of task parallelism. High performance computing of RS cloud platform generally adopts multi-computer cluster (such as Hadoop [90], and MapReduce [91]), multi-process parallelism (such as MPI [92], and Spark [93]), multi-core or multi-thread parallelism (such as OpenMP [94] etc. ), heterogeneous parallelism (such as GPU [95]), and other parallel processing technologies.…”