Technical Note: Review of methods for linear least-squares fitting of data and application to atmospheric chemistry problems

Abstract

Abstract. The representation of data, whether geophysical observations, numerical model output or laboratory results, by a best fit straight line is a routine practice in the geosciences and other fields. While the literature is full of detailed analyses of procedures for fitting straight lines to values with uncertainties, a surprising number of scientists blindly use the standard least-squares method, such as found on calculators and in spreadsheet programs, that assumes no uncertainties in the x values. Here, the available procedures for estimating the best fit straight line to data, including those applicable to situations for uncertainties present in both the x and y variables, are reviewed. Representative methods that are presented in the literature for bivariate weighted fits are compared using several sample data sets, and guidance is presented as to when the somewhat more involved iterative methods are required, or when the standard least-squares procedure would be expected to be satisfactory. A spreadsheetbased template is made available that employs one method for bivariate fitting.

The stable isotopic composition of molecular hydrogen in the tropopause region probed by the CARIBIC aircraft

“…This pattern is almost exactly inversely mirrored in the seasonal variation in the δD profiles in Fig. 4d (Cantrell, 2008). Different colors and markers indicate different sampling latitude bands.…”

“…The TRAC samples are kept in glass flasks con- Keeling plot (δD vs. inverse χ(H 2 )) for those samples that were labeled as polluted with the iterative selection procedure. The solid line is a linear fit through all the data with the "WilliamsonYork" (W-Y) algorithm that takes the variance and the estimated errors in both the x-and y-direction into account (Cantrell, 2008). The 1×SEM (standard error) interval around the axis intercept of the fit is indicated in grey on the vertical axis.…”

“…Methane mixing ratio data (χ(CH 4 )) from Rahn et al (2003) and Röckmann et al (2003) were multiplied by a factor of 1.0124 to convert them to the NOAA2004 scale. All fits were made with the Williamson-York algorithm described by Cantrell (2008). No error estimates for the individual measurements were taken into account for the literature values.…”

“…Plumb (2007) showed that tight correlations exist between many long-lived stratospheric species. (Cantrell, 2008), without error estimates of the individual measurements for the literature values. Open symbols denote outlying datapoints that were left out of the fitting procedure.…”

An Uncertainty Model Of Approximating The Analytical Solution To The Real Case In The Field Of Stress Prediction

“…The Williamson-York method is a weighted least square algorithm for calculating the best straight line when the variances in both axes exist − in our case ( ) 2 P u σ and u 2 (4P/πLD) − thus allowing to reach an acceptable value [23].…”