HadoopViz: A MapReduce framework for extensible visualization of big spatial data

Eldawy, Ahmed; Mokbel, Mohamed F.; Jonathan, Christopher

doi:10.1109/icde.2016.7498274

Cited by 71 publications

(35 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It extends MapReduce API with two new components, namely SpatialFileSplitter and SpatialRecordReader, for efficient and scalable spatial data processing. SpatialHadoop supports various geometry types, such as polygon, point, line string, multi-point, and others and multiple spatial partitioning techniques including uniform grids, R-Tree, Quad-Tree, KD-Tree, and Hilbert curves [85]. It also comes with several predefined spatial operations including box range queries, kNN queries and spatial joins over geometric objects using conditions such as within and intersect.…”

Section: Spatialhadoopmentioning

confidence: 99%

Review of Big Data and Processing Frameworks for Disaster Response Applications

Cumbane

Gidófalvi

2019

IJGI

View full text Add to dashboard Cite

Natural hazards result in devastating losses in human life, environmental assets and personal, and regional and national economies. The availability of different big data such as satellite imageries, Global Positioning System (GPS) traces, mobile Call Detail Records (CDRs), social media posts, etc., in conjunction with advances in data analytic techniques (e.g., data mining and big data processing, machine learning and deep learning) can facilitate the extraction of geospatial information that is critical for rapid and effective disaster response. However, disaster response systems development usually requires the integration of data from different sources (streaming data sources and data sources at rest) with different characteristics and types, which consequently have different processing needs. Deciding which processing framework to use for a specific big data to perform a given task is usually a challenge for researchers from the disaster management field. Therefore, this paper contributes in four aspects. Firstly, potential big data sources are described and characterized. Secondly, the big data processing frameworks are characterized and grouped based on the sources of data they handle. Then, a short description of each big data processing framework is provided and a comparison of processing frameworks in each group is carried out considering the main aspects such as computing cluster architecture, data flow, data processing model, fault-tolerance, scalability, latency, back-pressure mechanism, programming languages, and support for machine learning libraries, which are related to specific processing needs. Finally, a link between big data and processing frameworks is established, based on the processing provisioning for essential tasks in the response phase of disaster management.

show abstract

Section: Spatialhadoopmentioning

confidence: 99%

Review of Big Data and Processing Frameworks for Disaster Response Applications

Cumbane

Gidófalvi

2019

IJGI

View full text Add to dashboard Cite

show abstract

“…Through the spatial partition to achieve a variety of spatial data query, the results show that the system is superior to the parallel spatial data relational database system. An open source framework, SpatialHadoop covers the high-level programming language based on the Pigeon (Eldawy & Mokbel, 2014), spatial data index, spatial query, and visualization (Eldawy, Mokbel, & Jonathan, 2016), application to solve the basic problems of big spatial vector data management, which has a significant representative and reference value in industry. GeoSpark (Yu, Wu, & Sarwat, 2015) based on HDFS is designed with a three-layer architecture, namely, the Apache Spark layer, spatial data distribution layer, and spatial query operational layer.…”

Section: Distributed File Systemmentioning

confidence: 99%

Big spatial vector data management: a review

Yao

2018

Big Earth Data

View full text Add to dashboard Cite

Spatial vector data with high-precision and wide-coverage has exploded globally, such as land cover, social media, and other datasets, which provides a good opportunity to enhance the national macroscopic decision-making, social supervision, public services, and emergency capabilities. Simultaneously, it also brings great challenges in management technology for big spatial vector data (BSVD). In recent years, a large number of new concepts, parallel algorithms, processing tools, platforms, and applications have been proposed and developed to improve the value of BSVD from both academia and industry. To better understand BSVD and take advantage of its value effectively, this paper presents a review that surveys recent studies and research work in the data management field for BSVD. In this paper, we discuss and itemize this topic from three aspects according to different information technical levels of big spatial vector data management. It aims to help interested readers to learn about the latest research advances and choose the most suitable big data technologies and approaches depending on their system architectures. To support them more fully, firstly, we identify new concepts and ideas from numerous scholars about geographic information system to focus on BSVD scope in the big data era. Then, we conclude systematically not only the most recent published literatures but also a global view of main spatial technologies of BSVD, including data storage and organization, spatial index, processing methods, and spatial analysis. Finally, based on the above commentary and related work, several opportunities and challenges are listed as the future research interests and directions for reference.

show abstract

“…The visualization based on the tile pyramid model is developed by employing a three phase technique [26]: (1) The partitioning phase: here, we use the default Hadoop partitioning to split the total input spatial big data into partitions in HDFS; (2) The plotting phase: through the internal loop, all tiles occupied by each spatial object will be plotted in the tile pyramid model; (3) The merging phase: the partial images having the same index code will be combined into one final image. As shown in Figure 7, each tile in the map pyramid model is indexed according to the level, row, and column as the unique identifier.…”

Section: Data Visualizationmentioning

confidence: 99%

LandQv2: A MapReduce-Based System for Processing Arable Land Quality Big Data

Yao

Mokbel

et al. 2018

IJGI

Self Cite

View full text Add to dashboard Cite

Arable land quality (ALQ) data are a foundational resource for national food security. With the rapid development of spatial information technologies, the annual acquisition and update of ALQ data covering the country have become more accurate and faster. ALQ data are mainly vector-based spatial big data in the ESRI (Environmental Systems Research Institute) shapefile format. Although the shapefile is the most common GIS vector data format, unfortunately, the usage of ALQ data is very constrained due to its massive size and the limited capabilities of traditional applications. To tackle the above issues, this paper introduces LandQ v2 , which is a MapReduce-based parallel processing system for ALQ big data. The core content of LandQ v2 is composed of four key technologies including data preprocessing, the distributed R-tree index, the spatial range query, and the map tile pyramid model-based visualization. According to the functions in LandQ v2 , firstly, ALQ big data are transformed by a MapReduce-based parallel algorithm from the ESRI Shapefile format to the GeoCSV file format in HDFS (Hadoop Distributed File System), and then, the spatial coding-based partition and R-tree index are executed for the spatial range query operation. In addition, the visualization of ALQ big data with a GIS (Geographic Information System) web API (Application Programming Interface) uses the MapReduce program to generate a single image or pyramid tiles for big data display. Finally, a set of experiments running on a live system deployed on a cluster of machines shows the efficiency and scalability of the proposed system. All of these functions supported by LandQ v2 are integrated into SpatialHadoop, and it is also able to efficiently support any other distributed spatial big data systems.

show abstract

HadoopViz: A MapReduce framework for extensible visualization of big spatial data

Abstract: This paper introduces Hadoop Viz; a Map Reduce based framework for visualizing big spatial data. Hadoop Viz has three unique features that distinguish it from other techniques.

Cited by 71 publications

References 25 publications

Review of Big Data and Processing Frameworks for Disaster Response Applications

Review of Big Data and Processing Frameworks for Disaster Response Applications

Big spatial vector data management: a review

LandQv2: A MapReduce-Based System for Processing Arable Land Quality Big Data

Contact Info

Product

Resources

About