Big Data and cloud computing: innovation opportunities and challenges

Abstract

Big Data has emerged in the past few years as a new paradigm providing abundant data and opportunities to improve and/or enable research and decision-support applications with unprecedented value for digital earth applications including business, sciences and engineering. At the same time, Big Data presents challenges for digital earth to store, transport, process, mine and serve the data. Cloud computing provides fundamental support to address the challenges with shared computing resources including computing, storage, networking and analytical software; the application of these resources has fostered impressive Big Data advancements. This paper surveys the two frontiers-Big Data and cloud computingand reviews the advantages and consequences of utilizing cloud computing to tackling Big Data in the digital earth and relevant science domains. From the aspects of a general introduction, sources, challenges, technology status and research opportunities, the following observations are offered: (i) cloud computing and Big Data enable science discoveries and application developments; (ii) cloud computing provides major solutions for Big Data; (iii) Big Data, spatiotemporal thinking and various application domains drive the advancement of cloud computing and relevant technologies with new requirements; (iv) intrinsic spatiotemporal principles of Big Data and geospatial sciences provide the source for finding technical and theoretical solutions to optimize cloud computing and processing Big Data; (v) open availability of Big Data and processing capability pose social challenges of geospatial significance and (vi) a weave of innovations is transforming Big Data into geospatial research, engineering and business values. This review introduces future innovations and a research agenda for cloud computing supporting the transformation of the volume, velocity, variety and veracity into values of Big Data for local to global digital earth science and applications.

GEO-CEOS stage 4 validation of the Satellite Image Automatic Mapper lightweight computer program for ESA Earth observation level 2 product generation – Part 2: Validation

“…Jointly proposed by the intergovernmental Group on Earth Observations (GEO) and the Committee on Earth Observation Satellites (CEOS), the implementation plan for years 2005–2015 of the Global Earth Observation System of Systems (GEOSS) aimed at systematic transformation of multi-source Earth observation (EO) big data (IBM, 2016; Yang, Huang, Li, Liu, & Hu, 2017) into timely, comprehensive, and operational EO value-adding products and services (GEO, 2005), submitted to the GEO-CEOS Quality Assurance Framework for Earth Observation (QA4EO) calibration/validation ( Cal/Val ) requirements (Group on Earth Observation/Committee on Earth Observation Satellites (GEO/CEOS), 2010). The visionary goal of GEOSS cannot be considered fulfilled by the remote sensing (RS) community to date.…”

“…This working hypothesis postulates that no GEOSS can exist if the necessary not sufficient pre-condition of systematic ESA EO Level 2 product generation is not accomplished in advance at the ground segment. Hence, systematic ESA EO Level 2 product generation is considered a mandatory early stage in a hierarchical EO-IUS workflow, capable of scene-from-image reconstruction and understanding in operating mode to cope with the five Vs of big data, specifically, volume, variety, velocity, veracity, and value (IBM, 2016; Yang et al, 2017). …”

GEO-CEOS stage 4 validation of the Satellite Image Automatic Mapper lightweight computer program for ESA Earth observation level 2 product generation - Part 1: Theory

“…In this definition of GEOSS, term big data identifies “a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The big data challenges include capture, storage, search, sharing, transfer, analysis and visualization” (Wikipedia, 2018a), typically summarized as the five Vs of big data, specifically, volume, variety, velocity, veracity and value (IBM, 2016; Yang, Huang, Li, Liu, & Hu, 2017). …”

“…According to the Pareto formal analysis of multi-objective optimization problems, optimization of an mDMI set of OP-Q 2 Is is an inherently-ill posed problem in the Hadamard sense (Hadamard, 1902), where many Pareto optimal solutions lying on the Pareto efficient frontier can be considered equally good (Boschetti, Flasse, & Brivio, 2004). Any EO-IUS solution lying on the Pareto efficient frontier can be considered in operating mode, therefore suitable to cope with the five Vs of spatio-temporal EO big data (Yang et al, 2017). …”

“…In common practice, EO-IUSs are overwhelmed by sensory data they are unable to transform into EO value-adding information products and services, in compliance with the GEO-CEOS QA4EO Cal/Val guidelines (GEO-CEOS, 2010). If this conjecture holds true, then existing EO-IUSs cannot be considered in operating mode because unsuitable to cope with the five Vs of spatio-temporal EO big data (Yang et al, 2017). Several observations (true-facts) support this thesis.…”