SAT-ETL-Integrator: an extract-transform-load software for satellite big data ingestion

Semlali, Badr-Eddine Boudriki; Amrani, Chaker El; Ortiz, Guadalupe

doi:10.1117/1.jrs.14.018501

Cited by 14 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparing this study to the SAT-ETL-Integrator software [21], we confirm that the two software have the same RS data input and output specifications. However, they differ in their processing architecture.…”

Section: Comparison With Related Worksupporting

confidence: 61%

“…Some appropriate methods that focus on novel architectures for RS processing are explained as follows: Boudriki Semlali [20] developed Java-based application software to collect, process, and visualize numerous environmental data acquired from the EUMETSAT datacenter. Boudriki Semlali et al [21] also proposed software as an extract-transform-load tool for satellite data pre-processing that allows effective RSBD integration. Thus, the developed software layer gathers data unceasingly and eliminates about 86% of the unemployed files.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SAT-Hadoop-Processor: A Distributed Remote Sensing Big Data Processing Software for Earth Observation Applications

Semlali

Freitag²

2021

Applied Sciences

View full text Add to dashboard Cite

Nowadays, several environmental applications take advantage of remote sensing techniques. A considerable volume of this remote sensing data occurs in near real-time. Such data are diverse and are provided with high velocity and variety, their pre-processing requires large computing capacities, and a fast execution time is critical. This paper proposes a new distributed software for remote sensing data pre-processing and ingestion using cloud computing technology, specifically OpenStack. The developed software discarded 86% of the unneeded daily files and removed around 20% of the erroneous and inaccurate datasets. The parallel processing optimized the total execution time by 90%. Finally, the software efficiently processed and integrated data into the Hadoop storage system, notably the HDFS, HBase, and Hive.

show abstract

Section: Comparison With Related Worksupporting

confidence: 61%

Section: Related Workmentioning

confidence: 99%

SAT-Hadoop-Processor: A Distributed Remote Sensing Big Data Processing Software for Earth Observation Applications

Semlali

Freitag²

2021

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…The ingestion layer is an inherent part of the proposed BD architecture. It is responsible for preprocessing RS data [27]. Thus, the acquired data are firstly stored and then processed.…”

Section: The Data Ingestionmentioning

confidence: 99%

Big data and remote sensing: A new software of ingestion

Semlali

Amrani

2021

IJECE

Self Cite

View full text Add to dashboard Cite

Currently, remote sensing is widely used in environmental monitoring applications, mostly air quality mapping and climate change supervision. However, satellite sensors occur massive volumes of data in near-real-time, stored in multiple formats and are provided with high velocity and variety. Besides, the processing of satellite big data is challenging. Thus, this study aims to approve that satellite data are big data and proposes a new big data architecture for satellite data processing. The developed software is enabling an efficient remote sensing big data ingestion and preprocessing. As a result, the experiment results show that 86 percent of the unnecessary daily files are discarded with a data cleansing of 20 percent of the erroneous and inaccurate plots. The final output is integrated into the Hadoop system, especially the HDFS, HBase, and Hive, for extra calculation and processing.

show abstract

“…Climate-friendly urban planning (Milojevic-Dupont et al, 2020), monitoring the energy demand of buildings (Gouveia and Palma, 2019), and defining disaster resilience (Sasaki et al, 2020) play an important role in achieving sustainable cities and communities (SDG11). The Climate Action goal (SDG13) tackles most data gaps, so research such as linking satellite images to Semlali and El Amrani (2021) with air quality, preprocessing them (Meraner et al, 2020;Qin and Chi, 2020;Semlali et al, 2020), the analysis of time series data (Ise et al, 2020) and its exploration (Joshi et al, 2019), downscaling (Wang Q. et al, 2020) techniques, enrichment of precipitation and temperature data (Jimenez et al, 2019), tracking the movement of clouds (Xie Y. et al, 2019), or just using IoT sensors (Mabrouki et al, 2021) are all key in creating a strategy to support the achievement of the climate goal. For the sustainability of life below water (SDG14), marine life prediction models (Coro et al, 2020) and human coastal activity (Kubo et al, 2020) can be integrated.…”

Section: The Importance Of the System Of Systems Approachmentioning

confidence: 99%

The Applicability of Big Data in Climate Change Research: The Importance of System of Systems Thinking

Sebestyén

Czvetkó

Abonyi

2021

Front. Environ. Sci.

View full text Add to dashboard Cite

The aim of this paper is to provide an overview of the interrelationship between data science and climate studies, as well as describes how sustainability climate issues can be managed using the Big Data tools. Climate-related Big Data articles are analyzed and categorized, which revealed the increasing number of applications of data-driven solutions in specific areas, however, broad integrative analyses are gaining less of a focus. Our major objective is to highlight the potential in the System of Systems (SoS) theorem, as the synergies between diverse disciplines and research ideas must be explored to gain a comprehensive overview of the issue. Data and systems science enables a large amount of heterogeneous data to be integrated and simulation models developed, while considering socio-environmental interrelations in parallel. The improved knowledge integration offered by the System of Systems thinking or climate computing has been demonstrated by analysing the possible inter-linkages of the latest Big Data application papers. The analysis highlights how data and models focusing on the specific areas of sustainability can be bridged to study the complex problems of climate change.

show abstract

SAT-ETL-Integrator: an extract-transform-load software for satellite big data ingestion

Cited by 14 publications

References 29 publications

SAT-Hadoop-Processor: A Distributed Remote Sensing Big Data Processing Software for Earth Observation Applications

SAT-Hadoop-Processor: A Distributed Remote Sensing Big Data Processing Software for Earth Observation Applications

Big data and remote sensing: A new software of ingestion

The Applicability of Big Data in Climate Change Research: The Importance of System of Systems Thinking

Contact Info

Product

Resources

About