Artificial bias typically neglected in comparisons of uncertain atmospheric data

Pitkänen, Mikko R. A.; Mikkonen, Santtu; Lehtinen, K. E. J.; Lipponen, Antti; Arola, Antti

doi:10.1002/2016gl070852

Cited by 34 publications

(38 citation statements)

References 20 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the regression method suffers from the regression dilution effect, which means that the regression slope tends to be biased toward zero, which causes an underestimated scaling factor [64]. The underestimated scaling factor will in turn produce a "flat" time series.…”

Section: Comparison With Other Fusion Methodsmentioning

confidence: 99%

Merging Satellite Retrievals and Reanalyses to Produce Global Long-Term and Consistent Surface Incident Solar Radiation Datasets

Feng

Wang

2018

Remote Sensing

View full text Add to dashboard Cite

Surface incident solar radiation (R s ) is a key parameter in many climatic and ecological processes. The data from satellites and reanalysis have been widely used. However, for reanalysis, R s data has been shown to have substantial spatial bias, and the time span of reliable satellite R s is too short for climatic and ecological studies. Combining reanalysis and satellite data would be an effective method for generating long-term and consistent R s datasets. Here, we apply a cumulative probability density function-based (CPDF) method to merge eight reanalyses with the latest available satellite R s data from Clouds and Earth's Radiant Energy System Energy Balanced and Filled (CERES EBAF) surface retrievals. The CPDF method not only reduces the spatial bias of the reanalysis R s data, but also makes the R s datasets in a global, long-term and consistent way. The observed R s data collected at 54 Baseline Surface Radiation Network (BSRN) stations from 1992 to 2016 are used to evaluate the method. Results show that the CPDF method could reduce the mean absolute biases (MAB) of the reanalysis R s effectively by 21.24-64.36%. The European Centre for Medium-Range Weather Forecasts Re-Analysis interim (ERA-interim) reanalysis R s data, which are available for 1979 onward, perform the best before MAB = 13.20 W·m −2 and after MAB = 10.40 W·m −2 merging. This small post-merging MAB of the ERA-interim reanalysis is caused by the MAB of 9.90 W·m −2 in the satellite R s retrievals. The Japanese 55-year reanalysis provides R s values back to 1958, and CPDF can reduce its MAB by 32.87%, to 11.17 W·m −2 . The National Oceanic and Atmospheric Administration (NOAA)-CIRES twentieth-century reanalysis (CIRES) and the ECMWF twentieth-century reanalysis (ERA20CM) provide century-long R s estimates. CIRES performs better after merging. The MAB of CIRES can be reduced by 32.10%, to 12.99 W·m −2 , while ERA20CM's can be reduced by 12.51%, to 16.40 W·m −2 .

show abstract

Section: Comparison With Other Fusion Methodsmentioning

confidence: 99%

Merging Satellite Retrievals and Reanalyses to Produce Global Long-Term and Consistent Surface Incident Solar Radiation Datasets

Feng

Wang

2018

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…However, when validating satellite products against AERONET one should bear in mind that the AERONET data are not errorless, and even small uncertainties in the reference data may cause biases and affect the conclusions, especially when using regression analysis, as recently discussed by Pitkänen et al (2016). In this paper we show linear regression lines on some plots but avoid making far-reaching conclusions based on these.…”

Section: T H Virtanen Et Al: Collocation Mismatch Uncertaintiesmentioning

confidence: 97%

Collocation mismatch uncertainties in satellite aerosol retrieval validation

et al. 2018

View full text Add to dashboard Cite

Abstract. Satellite-based aerosol products are routinely validated against ground-based reference data, usually obtained from sun photometer networks such as AERONET (AEROsol RObotic NETwork). In a typical validation exercise a spatial sample of the instantaneous satellite data is compared against a temporal sample of the point-like ground-based data. The observations do not correspond to exactly the same column of the atmosphere at the same time, and the representativeness of the reference data depends on the spatiotemporal variability of the aerosol properties in the samples. The associated uncertainty is known as the collocation mismatch uncertainty (CMU). The validation results depend on the sampling parameters. While small samples involve less variability, they are more sensitive to the inevitable noise in the measurement data. In this paper we study systematically the effect of the sampling parameters in the validation of AATSR (Advanced Along-Track Scanning Radiometer) aerosol optical depth (AOD) product against AERONET data and the associated collocation mismatch uncertainty. To this end, we study the spatial AOD variability in the satellite data, compare it against the corresponding values obtained from densely located AERONET sites, and assess the possible reasons for observed differences.We find that the spatial AOD variability in the satellite data is approximately 2 times larger than in the ground-based data, and the spatial variability correlates only weakly with that of AERONET for short distances. We interpreted that only half of the variability in the satellite data is due to the natural variability in the AOD, and the rest is noise due to retrieval errors. However, for larger distances (∼ 0.5 • ) the correlation is improved as the noise is averaged out, and the day-to-day changes in regional AOD variability are well captured. Furthermore, we assess the usefulness of the spatial variability of the satellite AOD data as an estimate of CMU by comparing the retrieval errors to the total uncertainty estimates including the CMU in the validation. We find that accounting for CMU increases the fraction of consistent observations.

show abstract

“…(Note that a quadratic fit is not required; it provides barely any improvement.) The relatively small value of the slope is the result of an effect termed "regression dilution" or "regression attenuation" (Cantrell, 2008;Pitkäinen et al, 2016), which results from neglecting the measurement error in the x component of the regression pair. This problem can be overcome using a bivariate regression which accounts for errors in both components of the data pairs (see Appendix).…”

Section: Regression-based Intercalibrationmentioning

confidence: 99%

“…Bivariate regression is a method to avoid regression dilution (Cantrell, 2008;Pitkäinen et al, 2016), but not for the derivation of corrections. Further information on bivariate regression for general cases can be found in York et al (2004), who provide unified equations for general least squares regression methods.…”

Section: Appendix A: Bivariate Regressionmentioning

confidence: 99%

Technical note: On the intercalibration of HIRS channel 12 brightness temperatures following the transition from HIRS 2 to HIRS 3/4 for ice saturation studies

Gierens

Eleftheratos

2017

Atmos. Meas. Tech.

View full text Add to dashboard Cite

Abstract. In the present study we explore the capability of the intercalibrated HIRS brightness temperature data at channel 12 (the HIRS water vapour channel; T 12 ) to reproduce ice supersaturation in the upper troposphere during the period 1979-2014. Focus is given on the transition from the HIRS 2 to the HIRS 3 instrument in the year 1999, which involved a shift of the central wavelength in channel 12 from 6.7 to 6.5 µm. It is shown that this shift produced a discontinuity in the time series of low T 12 values (< 235 K) and associated cases of high upper-tropospheric humidity with respect to ice (UTHi > 70 %) in the year 1999 which prevented us from maintaining a continuous, long-term time series of ice saturation throughout the whole record . We show that additional corrections are required to the low T 12 values in order to bring HIRS 3 levels down to HIRS 2 levels. The new corrections are based on the cumulative distribution functions of T 12 from NOAA 14 and 15 satellites (that is, when the transition from HIRS 2 to HIRS 3 occurred). By applying these corrections to the low T 12 values we show that the discontinuity in the time series caused by the transition of HIRS 2 to HIRS 3 is not apparent anymore when it comes to calculating extreme UTHi cases. We come up with a new time series for values found at the low tail of the T 12 distribution, which can be further exploited for analyses of ice saturation and supersaturation cases. The validity of the new method with respect to typical intercalibration methods such as regression-based methods is presented and discussed.

show abstract

Artificial bias typically neglected in comparisons of uncertain atmospheric data

Cited by 34 publications

References 20 publications

Merging Satellite Retrievals and Reanalyses to Produce Global Long-Term and Consistent Surface Incident Solar Radiation Datasets

Merging Satellite Retrievals and Reanalyses to Produce Global Long-Term and Consistent Surface Incident Solar Radiation Datasets

Collocation mismatch uncertainties in satellite aerosol retrieval validation

Technical note: On the intercalibration of HIRS channel 12 brightness temperatures following the transition from HIRS 2 to HIRS 3/4 for ice saturation studies

Contact Info

Product

Resources

About