A New Initiative for Tiling, Stitching and Processing Geospatial Big Data in Distributed Computing Environments

Abstract

Within recent years, several new approaches and solutions for Big Data processing have been developed. The Geospatial world is still facing the lack of well-established distributed processing solutions tailored to the amount and heterogeneity of geodata, especially when fast data processing is a must. The goal of such systems is to improve processing time by distributing data transparently across processing (and/or storage) nodes. These types of methodology are based on the concept of divide and conquer. Nevertheless, in the context of geospatial processing, most of the distributed computing frameworks have important limitations regarding both data distribution and data partitioning methods. Moreover, flexibility and expendability for handling various data types (often in binary formats) are also strongly required. <br><br> This paper presents a concept for tiling, stitching and processing of big geospatial data. The system is based on the IQLib concept (<a href="https://github.com/posseidon/IQLib/"target="_blank">https://github.com/posseidon/IQLib/</a>) developed in the frame of the IQmulus EU FP7 research and development project (<a href="http://www.iqmulus.eu"target="_blank">http://www.iqmulus.eu</a>). The data distribution framework has no limitations on programming language environment and can execute scripts (and workflows) written in different development frameworks (e.g. Python, R or C#). It is capable of processing raster, vector and point cloud data. The above-mentioned prototype is presented through a case study dealing with country-wide processing of raster imagery. Further investigations on algorithmic and implementation details are in focus for the near future.

BigData Analysis in Healthcare: Apache Hadoop , Apache spark and Apache Flink

“…These projects are designed to help researchers and experts in all stages of the analysis and machine learning workflow. The general structure of the ecosystem consists of three layers: the storage layer, the processing layer and the management layer [26][27][28][29][30][31][32][33][34][35][36][37][38][39][40]. MapReduce is a Google-generated programming model for Big Data processing based on the "divide and conquer" method [41,42].…”

Developing the Geospatial Big Data Benchmark: A Comparative Framework for Evaluating Raster Analysis on Big Data Platforms

“…Meta-reviews discuss the platforms and their capabilities. [14][15][16][17][18] These papers are orientated towards the broadest communities and focus on the platform capabilities not platform performance.…”

Hogyan lesz térbeli a Big Data?

“…A kiterjedésük alapján jellemzett algoritmus kategóriák között felállíthatunk egy sorrendet tekintve, azok párhuzamosíthatóságának komplexitását. Az alkalmazott csoportosítás egyértelműen megadja az algoritmusok területhez igazodó adatfelosztási gyakorlatot (Nguyen, 2016;Ma et al, 2015).…”