Enabling Big Geoscience Data Analytics with a Cloud-Based, MapReduce-Enabled and Service-Oriented Workflow Framework

Abstract

Geoscience observations and model simulations are generating vast amounts of multi-dimensional data. Effectively analyzing these data are essential for geoscience studies. However, the tasks are challenging for geoscientists because processing the massive amount of data is both computing and data intensive in that data analytics requires complex procedures and multiple tools. To tackle these challenges, a scientific workflow framework is proposed for big geoscience data analytics. In this framework techniques are proposed by leveraging cloud computing, MapReduce, and Service Oriented Architecture (SOA). Specifically, HBase is adopted for storing and managing big geoscience data across distributed computers. MapReduce-based algorithm framework is developed to support parallel processing of geoscience data. And service-oriented workflow architecture is built for supporting on-demand complex data analytics in the cloud environment. A proof-of-concept prototype tests the performance of the framework. Results show that this innovative framework significantly improves the efficiency of big geoscience data analytics by reducing the data processing time as well as simplifying data analytical procedures for geoscientists.

CyberGIS‐Jupyter for reproducible and scalable geospatial analytics

“…11,24,25 In order to offer user-friendly access, most science gateways employ the Software as a Service (SaaS) 26 approach, ie, providing applications in the form of interactive web services with Web 2.0 technology and Service-Oriented Architecture (SOA). 27,28 A similar architecture was adopted for CyberGIS Gateway 11 as well. On top of the generic science gateway architecture, most CyberGIS Gateway applications adopt web-based GIS capabilities such as OpenLayers (http://openlayers.org) as frontend interfaces; employ geospatial middleware (eg, GISolve 29 ) to access HPC capabilities; and import/export standard geospatial web services via OGC standards.…”

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

“…Krishnan et al investigated the use of MapReduce to generate DEM by gridding the LIDAR data [22]. Li et al utilized Hadoop MapReduce to enable penalization of big climate data processing [11,13]. Besides these problem-specific approaches focusing on solving specific problems with Hadoop, tools have also been developed to handle general geospatial data processing tasks and are being adopted in GIScience communities.…”

“…Hadoop, a distributed computing platform leveraging commodity computers, is gaining increasing popularity in geoscience communities, as reviewed in Section 2. While a lot of effort was put into investigating how to adapt Hadoop for processing big geospatial data (e.g., [9][10][11][12][13]), how to efficiently handle different geoprocessing workload by dynamically adjusting the amount of computing resources (number of nodes of a Hadoop cluster) was barely explored. The ability to dynamically adjust the computing resources is important because the processing workload of operational geospatial applications is rather dynamic than static [14]; for example, the data processing workload for an emergency response system (such as for wildfires, tsunami, and earthquakes) peaks during the emergency event, which requires adequate computing power to respond promptly [14,15].…”

Cloud big data application for transport

“…Schadt et al (2011) demonstrate the efficiency that cloud computing could have for big data analytics, showing that analysis of 1 Petabyte of data in 350 minutes for 2040 dollars. Li et al (2015) propose a service-oriented architecture for geoscience data where they separate the modelling service for geoscience, the data services, the processing service and the cloud infrastructure.…”