The Data Big Bang and the Expanding Digital Universe: High‐Dimensional, Complex and Massive Data Sets in an Inflationary Epoch

Abstract

Recent and forthcoming advances in instrumentation, and giant new surveys, are creating astronomical data sets that are not amenable to the methods of analysis familiar to astronomers. Traditional methods are often inadequate not merely because of the size in bytes of the data sets, but also because of the complexity of modern data sets. Mathematical limitations of familiar algorithms and techniques in dealing with such data sets create a critical need for new paradigms for the representation, analysis and scientific visualization (as opposed to illustrative visualization) of heterogeneous, multiresolution data across application domains. Some of the problems presented by the new data sets have been addressed by other disciplines such as applied mathematics, statistics and machine learning and have been utilized by other sciences such as space-based geosciences. Unfortunately, valuable results pertaining to these problems are mostly to be found in publications outside of astronomy. Here we offer brief overviews of a number of concepts, techniques and developments that are vital to the analysis and visualization of complex datasets and images. One of the goals of this paper is to help bridge the gap between applied mathematics and artificial intelligence on the one side and astronomy on the other.

NeedATool: A NEEDLET ANALYSIS TOOL FOR COSMOLOGICAL DATA PROCESSING

“…CMB data analysis is very demanding -both in terms of computational power required and sophistication of the necessary techniques -due to the complexity of the datasets and to the high degree of accuracy one wants to achieve. An overview on the techniques recently applied to astrophysical data analysis can be found in Pesenson et al (2010). This holds in particular for cutting-edge experi-Electronic address: davide.pietrobon@jpl.nasa.gov Electronic address: amedeo.balbi@roma2.infn.it Electronic address: paolo.cabella@roma2.infn.it Electronic address: krzysztof.m.gorski@jpl.nasa.gov ments such as the ongoing Planck satellite 1 .…”

Stepwise Covariance-Free Common Principal Components (CF-CPC) With an Application to Neuroscience

“…The issue of computing covariance for ultra-high dimensional data can occur in other areas of neuroscience (raw EEG, MEG data, neurogenetics Data (Genome-Wide Association Study of Parkinson’s Disease: Genes and Environment, n.d.) and many other fields like microarrays in genetics, data from an array of sensors to capture geological or astronomical data, etc. ( Bonham-Carter et al, 1990 ; Pesenson et al, 2010 ; Alonso-Betanzos et al, 2019 ). In repetitive resampling like bootstrap, where multiple CPCs are required, we can use stepwise CF-CPC.…”

Eight grand challenges in socio-environmental systems modeling

“…We live in an era where data become rapidly available due to technological advances in sensory devices and computational infrastructures for archiving and sharing information. The inflationary epoch of data explosion is also referred to as the "Data Big Bang" (Pesenson et al, 2010). New data sources relevant for SES modeling include non-traditional sources of socio-environmental data.…”