Spark-based in-memory DEM creation from 3D LiDAR point clouds

Rizki, Permata Nur Miftahur; Eum, Junho; Lee, Heezin; Oh, Sangyoon

doi:10.1080/2150704x.2016.1275053

Cited by 9 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tiling, rasterisation, and voxelisation are commonly seen in MapReduce and Spark applications for point cloud data analysis (e.g. Krishnan et al, 2010;Rizki et al, 2017;Liu et al, 2016;Li et al, 2017). These techniques group points into spatially coherent groups either in 2D (i.e.…”

Section: Related Workmentioning

confidence: 99%

A Parallel Algorithm for Local Point Density Index Computation of Large Point Clouds

Hewage

Le-Khac

et al. 2021

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Abstract. Point density is an important property that dictates the usability of a point cloud data set. This paper introduces an efficient, scalable, parallel algorithm for computing the local point density index, a sophisticated point cloud density metric. Computing the local point density index is non-trivial, because this computation involves a neighbour search that is required for each, individual point in the potentially large, input point cloud. Most existing algorithms and software are incapable of computing point density at scale. Therefore, the algorithm introduced in this paper aims to address both the needed computational efficiency and scalability for considering this factor in large, modern point clouds such as those collected in national or regional scans. The proposed algorithm is composed of two stages. In stage 1, a point-level, parallel processing step is performed to partition an unstructured input point cloud into partially overlapping, buffered tiles. A buffer is provided around each tile so that the data partitioning does not introduce spatial discontinuity into the final results. In stage 2, the buffered tiles are distributed to different processors for computing the local point density index in parallel. That tile-level parallel processing step is performed using a conventional algorithm with an R-tree data structure. While straight-forward, the proposed algorithm is efficient and particularly suitable for processing large point clouds. Experiments conducted using a 1.4 billion point data set acquired over part of Dublin, Ireland demonstrated an efficiency factor of up to 14.8/16. More specifically, the computational time was reduced by 14.8 times when the number of processes (i.e. executors) increased by 16 times. Computing the local point density index for the 1.4 billion point data set took just over 5 minutes with 16 executors and 8 cores per executor. The reduction in computational time was nearly 70 times compared to the 6 hours required without parallelism.

show abstract

Section: Related Workmentioning

confidence: 99%

A Parallel Algorithm for Local Point Density Index Computation of Large Point Clouds

Hewage

Le-Khac

et al. 2021

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…In the 1980s and 1990s, optical imagery photogrammetry was the standard practice used for DSM generation, whereas over the past two decades, advances in LiDAR technology have made it the standard practice for DSM generation. Although very high resolution satellite stereo imagery is still used for DSM generation [73][74], the so-called standard practice for DSM generation is the utilization of LiDAR data [48,[75][76][77][78][79]. Millions or even billions of non-selective sampling points can be produced by using LiDAR scanners, whether airborne LiDAR ALS, Mobile LiDAR MMS or terrestrial LiDAR STLS [34,48].…”

Section: Dsm and Dem Generation Approachesmentioning

confidence: 99%

A Crosscutting Three-Modes-Of-Operation Unique LiDAR-Based 3D Mapping System Generic Framework Architecture, Uncertainty Predictive Model And SfM Augmentation

Elshorbagy¹

2021

Preprint

View full text Add to dashboard Cite

<div>The need for 3D mapping is on the rise to meet the requirements of a growing and diverse group of end-users. Existing 3D mapping systems, which have been classified according to the mode of operation as stationary, mobile and aerial, tend to serve one mode of operation only and are considered cost-prohibitive for many end-users. Unmanned aerial vehicles (UAVs) have experienced rapid growth since their introduction and their usage in 3D mapping is likewise accelerating at a rapid pace. This dissertation presents the design, development and implementation of a LiDAR-based generic 3D mapping system that can be used in the three mapping modes (stationary, mobile and UAV-based). The system provides direct georeferencing capabilities through optimized selected multimodal sensors. A fundamental part of this dissertation is the smart integration of the 3D mapping system components both on the hardware and software levels, along with a new mapping scheme that enables platform-independent deployment ability. </div><div>This research project also presents a rigorous non-linear uncertainty predictive model for the generic developed system and introduces a very low-cost variant of the system to be used in stationary and handheld mode. The developed multipurpose mapping system is tested in different environments for the three modes of operation, demonstrating its practicality, versatility and ease of deployment. To maximize the ease of deployment for diverse end-users, careful consideration is given to the mapping system components so that the developed system is ultra-lightweight, compact, and multipurpose. Additionally, this dissertation proposes a colorization workflow to make use of available optical imagery in the colorization process of the LiDAR point cloud. Lastly, the study compares two different 3D mapping approaches: 3D LiDAR-based mapping and a low-cost optical-based 3D structure from motion (SfM) workflow. The comparison is achieved by performing a real-world case study of digital surface model (DSM) generation by the two aforementioned approaches. Real-world testing that includes qualitative and quantitative validation against accurate state-of-the-art high-end LiDAR equipment proves the successful design, development and deployment of the developed crosscutting LiDAR-based 3D mapping system.</div>

show abstract

“…In recent years, Spark has been explored for processing massive amounts of laser scanning point cloud data (e.g. Li et al, 2017;Rizki et al, 2017).…”

Section: Implementation Detailsmentioning

confidence: 99%

A Big Data Approach for Comprehensive Urban Shadow Analysis From Airborne Laser Scanning Point Clouds

Laefer

2019

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

<p><strong>Abstract.</strong> Because of the importance of access to sunlight, shadow analysis is a common consideration in urban design, especially for dense urban developments. As shadow computation is computationally expensive, most urban shadow analysis tools have to date circumvented the high computational costs by representing urban complexity only through simplified geometric models. The simplification process removes details and adversely affects the level of realism of the ultimate results. In this paper, an alternative approach is presented by utilizing the highest level of detail and resolution captured in the geometric input data source, which is an extremely high-resolution airborne laser scanning point cloud (300 points/m2). To cope with the high computational demand caused by the use of this dense and detailed input data set, the Comprehensive Urban Shadow algorithm is introduced to distribute the computation for parallel processing on a Hadoop cluster. The proposed comprehensive urban shadow analysis solution is scalable, reasonably fast, and capable of preserving the original resolution and geometric detail of the original point cloud data.</p>

show abstract

Spark-based in-memory DEM creation from 3D LiDAR point clouds

Cited by 9 publications

References 12 publications

A Parallel Algorithm for Local Point Density Index Computation of Large Point Clouds

A Parallel Algorithm for Local Point Density Index Computation of Large Point Clouds

A Crosscutting Three-Modes-Of-Operation Unique LiDAR-Based 3D Mapping System Generic Framework Architecture, Uncertainty Predictive Model And SfM Augmentation

A Big Data Approach for Comprehensive Urban Shadow Analysis From Airborne Laser Scanning Point Clouds

Contact Info

Product

Resources

About