Elcano: A Geospatial Big Data Processing System based on SparkSQL

Engélinus, Jonathan; Badard, Thierry

doi:10.5220/0006794601190128

Cited by 9 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Few solutions exist that guarrantee scalability. Some promising work has been done for vector data and topological data structure (Engélinus and Badard, 2018) but more specific architectures handling TIN need to be investigated.…”

Section: Discussionmentioning

confidence: 99%

Thalweg Detection for River Network Cartography in Forest From High-Resolution Lidar Data

Guilbert

Jutras

Badard

2018

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

<p><strong>Abstract.</strong> This paper addresses the problem of extracting the drainage network in forested areas. A precise description of the drainage network including intermittent streams is required for the planning of logging operations and environmental conservation. LiDAR provides now high-resolution point clouds from which the terrain is modelled and the drainage extracted but it also brings some challenges for traditional approaches. First, the raster DTM is interpolated from LiDAR ground points and has to be split in tiles for processing, adding approximations. Second, drainage enforcement techniques alter the terrain and rely on parameters difficult to fix and limiting the optimisation of the process. In this context, we discuss a new approach aiming at: (1) Designing a data structure to model the terrain with a Triangulated Irregular Network in order to avoid interpolation. This data structure must enable the distribution of data and processes across several nodes in Big data architectures and eventually, the processing of complete watersheds with no tiling. (2) Modelling the river network through thalwegs and avoiding the filling and breaching operations. Thalweg detection is more robust, removing the need for filling and breaching. However, it yields a very dense network requiring a simplification step. Combining this model and the architecture will enable the design and modelling of a new tool for river network computation directly from LiDAR ground points. In this paper, we mainly discuss the second point and propose to model the drainage by a network of thalwegs computed from the terrain. Thalwegs are extracted from the surface network, a topological structure formed of peaks, pits and saddles as vertices and ridges and thalwegs as vertices. We present preliminary results comparing the thalweg network and the drainage network.</p>

show abstract

Section: Discussionmentioning

confidence: 99%

Thalweg Detection for River Network Cartography in Forest From High-Resolution Lidar Data

Guilbert

Jutras

Badard

2018

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…These SASs are mainly based on Hadoop MapReduce or Spark, and several surveys were recently published to describe and classify them [15][16][17][18]. The most representative Spark-based SASs are SpatialSpark [19], GeoSpark (currently Sedona) [4], Simba [20], LocationSpark [21], STARK [22], SparkGIS [23], Elcano [24] and Beast [25]. These Spark-based SASs and the spatial queries that they support are shown in Table 1.…”

Section: Spatial Query Processing In Apache Sparkmentioning

confidence: 99%

“…RQ KNNQ SJ KNNJQ DJ SpatialSpark [19] GeoSpark [4] Simba [20] LocationSpark [21] STARK [22] SparkGIS [23] Elcano [24] Beast [25] The previous SASs are either using the on-top or the built-in implementation approach for spatial query processing, utilizing well-known partitioning and indexing techniques. However, other spatial queries have been implemented directly in Apache Spark with sophisticated processing strategies.…”

Section: Spatial Analytics Systemmentioning

confidence: 99%

Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

Moutafis,

Mavrommatis,

Vassilakopoulos

et al. 2021

IJGI

View full text Add to dashboard Cite

Aiming at the problem of spatial query processing in distributed computing systems, the design and implementation of new distributed spatial query algorithms is a current challenge. Apache Spark is a memory-based framework suitable for real-time and batch processing. Spark-based systems allow users to work on distributed in-memory data, without worrying about the data distribution mechanism and fault-tolerance. Given two datasets of points (called Query and Training), the group K nearest-neighbor (GKNN) query retrieves (K) points of the Training with the smallest sum of distances to every point of the Query. This spatial query has been actively studied in centralized environments and several performance improving techniques and pruning heuristics have been also proposed, while, a distributed algorithm in Apache Hadoop was recently proposed by our team. Since, in general, Apache Hadoop exhibits lower performance than Spark, in this paper, we present the first distributed GKNN query algorithm in Apache Spark and compare it against the one in Apache Hadoop. This algorithm incorporates programming features and facilities that are specific to Apache Spark. Moreover, techniques that improve performance and are applicable in Apache Spark are also incorporated. The results of an extensive set of experiments with real-world spatial datasets are presented, demonstrating that our Apache Spark GKNN solution, with its improvements, is efficient and a clear winner in comparison to processing this query in Apache Hadoop.

show abstract

“…The most remarkable Hadoop‐based systems are Hadoop‐GIS 3,4 and SpatialHadoop 5‐9 . The state‐of‐the‐art Spark‐based systems are SpatialSpark, 10,11 GeoSpark, 12‐15 GeoMesa Spark, 16,17 Simba, 18,19 LocationSpark, 20,21 STARK, 22‐24 Magellan, 25 SparkGIS, 26 and Elcano 27 . However, each SAS has its own characteristics and introduces different functionalities to deal with spatial data in parallel and distributed environments.…”

Section: Introductionmentioning

confidence: 99%

Analyzing spatial analytics systems based on Hadoop and Spark: A user perspective

2020

View full text Add to dashboard Cite

Summary Spatial analytics systems (SASs) represent a technology capable of managing huge volumes of spatial data using frameworks such as Apache Hadoop and Apache Spark. An increasing number of SASs have been proposed, requiring a comparison among them. However, existing comparisons in the literature provide a system‐centric view based on performance evaluations. Thus, there is a lack of comparisons based on the user‐centric view, that is, comparisons that help users to understand how the characteristics of SASs are useful to meet the specific requirements of their spatial applications. In this article, we provide a user‐centric comparison of the following SASs based on Hadoop and Spark: Hadoop‐GIS, SpatialHadoop, SpatialSpark, GeoSpark, GeoMesa Spark, SIMBA, LocationSpark, STARK, Magellan, SparkGIS, and Elcano. This comparison employs an extensive set of criteria related to the general characteristics of these systems, to the aspects of spatial data handling, and to the aspects inherent to distributed systems. Based on this comparison, we introduce guidelines to help users to choose an appropriate SAS. We also describe two case studies based on real‐world applications to illustrate the use of these guidelines. Finally, we discuss chronological tendencies related to SASs and identify limitations that SASs should address to improve user experience.

show abstract

Elcano: A Geospatial Big Data Processing System based on SparkSQL

Cited by 9 publications

References 0 publications

Thalweg Detection for River Network Cartography in Forest From High-Resolution Lidar Data

Thalweg Detection for River Network Cartography in Forest From High-Resolution Lidar Data

Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

Analyzing spatial analytics systems based on Hadoop and Spark: A user perspective

Contact Info

Product

Resources

About