The Matsu Wheel: a reanalysis framework for Earth satellite imagery in data commons

Abstract

Project Matsu is a collaboration between the Open Commons Consortium and NASA focused on developing open source technology for the cloud-based processing of Earth satellite imagery and for detecting fires and floods to help support natural disaster detection and relief. We describe a framework for efficient analysis and reanalysis of large amounts of data called the Matsu "Wheel" and the analytics used to process hyperspectral data produced daily by NASA's Earth Observing-1 (EO-1) satellite. The wheel is designed to be able to support scanning queries using cloud computing applications, such as Hadoop and Accumulo. A scanning query processes all, or most, of the data in a database or data repository. In contrast, standard queries typically process a relatively small percentage of the data. The wheel is a framework in which multiple scanning queries are grouped together and processed in turn, over chunks of data from the database or repository. Over time, the framework brings all data to each group of scanning queries. With this approach, contention and the overall time to process all scanning queries can be reduced. We describe our Wheel analytics, including an anomaly detector for rare spectral signatures or anomalies in hyperspectral data and a land cover classifier that can be This paper is an extension version of the BDS2016 accepted paper "The Matsu Wheel: A Cloud-based Framework for the Efficient Analysis and Reanalysis of Earth Satellite Imagery" [1].

BiGeo: A Foundational PaaS Framework for Efficient Storage, Visualization, Management, Analysis, Service, and Migration of Geospatial Big Data—A Case Study of Sichuan Province, China

“…HadoopGIS provides spatial data partitioning for task parallelization and an index-driven spatial query engine to handle various types of spatial queries. Its implicit query parallelization generates correct results through MapReduce and boundary processing [52][53][54][55][56][57][58].…”

Improved method for correcting sample Mahalanobis distance without estimating population eigenvalues or eigenvectors of covariance matrix

“…The MD can be easily defined using the covariance matrix of learning samples and can be calculated faster and more easily than neural networks (NNs) and support vector machines (SVMs). The MD is still widely used for industrial and various other purposes (see a recent example [3]), and there have been cases in which the MD was said to have better recognition performance than NNs and SVMs if there are few learning samples [4][5][6][7]. Usually, the population covariance matrix Σ of the learning samples is unknown in advance.…”