Approximating Photo-z PDFs for Large Surveys

Malz, Alex I.; Marshall, Phil; DeRose, Joseph J.; Graham, M. L.; Schmidt, Samuel J.

doi:10.3847/1538-3881/aac6b5

Cited by 30 publications

(23 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The validity of this assumption in our scheme is tested in § 5.2. For an ensemble of N galaxies, an estimator (see Malz et al 2018, for a discussion) of the redshift distribution, dN/dz ∝ p(z), is given by…”

Section: Formalismmentioning

confidence: 99%

Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing

Buchs¹,

Davis²,

Gruen³

et al. 2019

Monthly Notices of the Royal Astronomical Society

View full text Add to dashboard Cite

Wide-field imaging surveys such as the Dark Energy Survey (DES) rely on coarse measurements of spectral energy distributions in a few filters to estimate the redshift distribution of source galaxies. In this regime, sample variance, shot noise, and selection effects limit the attainable accuracy of redshift calibration and thus of cosmological constraints. We present a new method to combine wide-field, few-filter measurements with catalogs from deep fields with additional filters and sufficiently low photometric noise to break degeneracies in photometric redshifts. The multi-band deep field is used as an intermediary between wide-field observations and accurate redshifts, greatly reducing sample variance, shot noise, and selection effects. Our implementation of the method uses self-organizing maps to group galaxies into phenotypes based on their observed fluxes, and is tested using a mock DES catalog created from N-body simulations. It yields a typical uncertainty on the mean redshift in each of five tomographic bins for an idealized simulation of the DES Year 3 weak-lensing tomographic analysis of σ ∆z = 0.007, which is a 60% improvement compared to the Year 1 analysis. Although the implementation of the method is tailored to DES, its formalism can be applied to other large photometric surveys with a similar observing strategy.

show abstract

Section: Formalismmentioning

confidence: 99%

Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing

Buchs¹,

Davis²,

Gruen³

et al. 2019

Monthly Notices of the Royal Astronomical Society

View full text Add to dashboard Cite

show abstract

“…Within few years, the trend to provide both photo-z point estimates and PDFs has now become a consolidated practice (Sheldon et al, 2012;Carrasco Kind and Brunner, 2013;Carrasco Kind and Brunner, 2014a,b;Bonnett, 2015;Cavuoti et al, 2017a;Malz et al, 2018;Tanaka et al, 2018;Amaro et al, 2019;Mucesh et al, 2020;Nishizawa et al, 2020). The idea is that a PDF should be able to provide a more complete information than the point estimation of the redshift.…”

Section: Is the Photo-z Point Estimate Enough?mentioning

confidence: 99%

Photometric Redshifts With Machine Learning, Lights and Shadows on a Complex Data Science Use Case

Brescia

Cavuoti

Razim

et al. 2021

Front. Astron. Space Sci.

View full text Add to dashboard Cite

The importance of the current role of data-driven science is constantly increasing within Astrophysics, due to the huge amount of multi-wavelength data collected every day, characterized by complex and high-volume information requiring efficient and, as much as possible, automated exploration tools. Furthermore, to accomplish main and legacy science objectives of future or incoming large and deep survey projects, such as James Webb Space Telescope (JWST), James Webb Space Telescope (LSST), and Euclid, a crucial role is played by an accurate estimation of photometric redshifts, whose knowledge would permit the detection and analysis of extended and peculiar sources by disentangling low-z from high-z sources and would contribute to solve the modern cosmological discrepancies. The recent photometric redshift data challenges, organized within several survey projects, like LSST and Euclid, pushed the exploitation of the observed multi-wavelength and multi-dimensional data or ad hoc simulated data to improve and optimize the photometric redshifts prediction and statistical characterization based on both Spectral Energy Distribution (SED) template fitting and machine learning methodologies. They also provided a new impetus in the investigation of hybrid and deep learning techniques, aimed at conjugating the positive peculiarities of different methodologies, thus optimizing the estimation accuracy and maximizing the photometric range coverage, which are particularly important in the high-z regime, where the spectroscopic ground truth is poorly available. In such a context, we summarize what was learned and proposed in more than a decade of research.

show abstract

“…As is shown in Schmidt et al (2019), the myriad methods for deriving photo-z PDFs yield radically different results, motivating a desire to store the results of more than one algorithm in the absence of an obvious best choice. For the photo-z PDFs of most codes, one may need to seek a clever storage parameterization to meet LSST's constraints (Carrasco Kind & Brunner 2014;Malz et al 2018), but FlexCode is virtually immune to this restriction. Since FlexCode relies on a basis expansion, one only needs to store n basis coefficients per target density for a lossless compression of the estimated PDF with no need for binning.…”

Section: Flexcodementioning

confidence: 99%

Conditional density estimation tools in python and R with applications to photometric redshifts and likelihood-free cosmological inference

Dalmasso

Pospisil

Lee

et al. 2020

Astronomy and Computing

View full text Add to dashboard Cite

It is well known in astronomy that propagating non-Gaussian prediction uncertainty in photometric redshift estimates is key to reducing bias in downstream cosmological analyses. Similarly, likelihoodfree inference approaches, which are beginning to emerge as a tool for cosmological analysis, require the full uncertainty landscape of the parameters of interest given observed data. However, most machine learning (ML) based methods with open-source software target point prediction or classification, and hence fall short in quantifying uncertainty in complex regression and parameter inference settings such as the applications mentioned above. As an alternative to methods that focus on predicting the response (or parameters) y from features x, we provide nonparametric conditional density estimation (CDE) tools for approximating and validating the entire probability density p(y | x) given training data for x and y. This density approach offers a more nuanced accounting of uncertainty in situations with, e.g., nonstandard error distributions and multimodal or heteroskedastic response variables that are often present in astronomical data sets. As there is no one-size-fits-all CDE method, and the ultimate choice of model depends on the application and the training sample size, the goal of this work is to provide a comprehensive range of statistical tools and open-source software for nonparametric CDE and method assessment which can accommodate different types of settings -involving, e.g., mixedtype input from multiple sources, functional data, and image covariates -and which in addition can easily be fit to the problem at hand. Specifically, we introduce CDE software packages in Python and R based on four ML prediction methods adapted and optimized for CDE: NNKCDE, RFCDE, FlexCode, and DeepCDE. Furthermore, we present the cdetools package, which includes functions for computing a CDE loss function for model selection and tuning of parameters, together with diagnostic functions for computing posterior quantiles and coverage probabilities. We provide sample code in Python and R as well as examples of applications to photometric redshift estimation and likelihood-free cosmology via CDE.

show abstract

Approximating Photo-z PDFs for Large Surveys

Cited by 30 publications

References 40 publications

Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing

Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing

Photometric Redshifts With Machine Learning, Lights and Shadows on a Complex Data Science Use Case

Conditional density estimation tools in python and R with applications to photometric redshifts and likelihood-free cosmological inference

Contact Info

Product

Resources

About