Extracting Knowledge from Massive Astronomical Data Sets

Brescia, M.; Cavuoti, S.; Djorgovski, George; Donalek, C.; Longo, Giuseppe; Paolillo, M.

doi:10.1007/978-1-4614-3323-1_3

Cited by 13 publications

(8 citation statements)

References 23 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The amount of astronomical data is massive and has been growing rapidly in recent years due to improved storage capacity and scientific instruments [1]. The rate of growth of data is outpacing the computational improvements from Moore's Law [2]. As a result, faster algorithms are required to keep up with this growth of data.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Clustered Hierarchical Entropy-Scaling Search of Astronomical and Biological Data

Ishaq

Student

Daniels

2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Both astronomy and biology are experiencing explosive growth of data, resulting in a "big data" problem standing in the way of a "big data" opportunity for discovery. One common task on such data sets is the problem of approximate search, or ρ−nearest neighbors search. We present a hierarchical search algorithm for such data sets that takes advantage of particular geometric properties apparent in both astronomical and biological data sets, namely the metric entropy and fractal dimensionality of the data. We present CHESS (Clustered Hierarchical Entropy-Scaling Search), a GPU-accelerated search tool with no loss in specificity or sensitivity, demonstrating a 6.4× speedup over linear search on the Sloan Digital Sky Survey's APOGEE data set and a 3.97× speedup on the GreenGenes 16S metagenomic data set, as well as asymptotically fewer comparisons on APOGEE when compared to the FALCONN locality-sensitive hashing library. CHESS allows for implicit data compression, which we demonstrate on the APOGEE data set. We also discuss an extension allowing for efficient k-nearest neighbors search.

show abstract

Section: Introductionmentioning

confidence: 99%

“…As a result, faster algorithms are required to keep up with this growth of data. This has prompted the emergence of astroinformatics, a new discipline at the intersection of astronomy and computer science [2].…”

Section: Introductionmentioning

confidence: 99%

Clustered Hierarchical Entropy-Scaling Search of Astronomical and Biological Data

Ishaq

Student

Daniels

2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

“…'Folksonomies' [17] regularly emerge, even outside science, to help people organize and exchange information efficiently. Description and taxonomization of new taxa and corpora are as important as annotation and knowledge dissemination [18] .…”

Section: Discovery and Taxon Formationmentioning

confidence: 99%

Visipedia circa 2015

Belongie

Perona

2016

Pattern Recognition Letters

View full text Add to dashboard Cite

“…This is computationally and algorithmically a highly non-trivial problem, that is now being addressed in the Astroinformatics and related disciplines. An example of a data exploration system along these lines is Data Mining and Exploration (DAME; Brescia et al 2010Brescia et al , 2012; http://dame.dsf.unina.it). A caveat is in order: most of the commonly used DM techniques implicitly or explicitly assume that the data are complete, error-free, and free of artifacts, none of which is true in reality.…”

Section: Figurementioning

confidence: 99%

Sky Surveys

Djorgovski

Mahabal

Drake

et al. 2013

Planets, Stars and Stellar Systems

Self Cite

View full text Add to dashboard Cite

Abstract:Sky surveys represent a fundamental data basis for astronomy. We use them to map in a systematic way the universe and its constituents, and to discover new types of objects or phenomena. We review the subject, with an emphasis on the wide-field, imaging surveys, placing them in a broader scientific and historical context. Surveys are now the largest data generators in astronomy, propelled by the advances in information and computation technology, and have transformed the ways in which astronomy is done. This trend is bound to continue, especially with the new generation of synoptic sky surveys that cover wide areas of the sky repeatedly, and open a new time domain of discovery. We describe the variety and the general properties of surveys, illustrated by a number of examples, the ways in which they may be quantified and compared, and offer some figures of merit that can be used to compare their scientific discovery potential. Surveys enable a very wide range of science, and that is perhaps their key unifying characteristic. As new domains of the observable parameter space open up thanks to the advances in technology, surveys are often the initial step in their exploration. Some science can be done with the survey data alone (or a combination of data from different surveys), and some requires a targeted follow-up of potentially interesting sources selected from surveys. Surveys can be used to generate large, statistical samples of objects that can be studied as populations, or as tracers of larger structures to which they belong. They can be also used to discover or generate samples of rare or unusual objects, and may lead to discoveries of some previously unknown types. We discuss a general framework of parameter spaces that can be used for an assessment and comparison of different surveys, and the strategies for their scientific exploration. As we are moving into the Petascale regime and beyond, an effective processing and scientific exploitation of such large data sets and data streams poses many challenges, some of which are specific to any given survey, and some of which may be addressed in the framework of Virtual Observatory and Astroinformatics. The exponential growth of data volumes and complexity makes a broader application of data mining and knowledge discovery technologies critical in order to take a full advantage of this wealth of information. Finally, we discuss some outstanding challenges and prospects for the future.* We dedicate this chapter to the memory of three pioneers of sky surveys, Fritz Zwicky (1898-1974), Bogdan Paczynski (1940), and John Huchra (1948

show abstract

Extracting Knowledge from Massive Astronomical Data Sets

Cited by 13 publications

References 23 publications

Clustered Hierarchical Entropy-Scaling Search of Astronomical and Biological Data

Clustered Hierarchical Entropy-Scaling Search of Astronomical and Biological Data

Visipedia circa 2015

Sky Surveys

Contact Info

Product

Resources

About