Use and Interpretation of Common Statistical Tests in Method Comparison Studies

Abstract

Feature Article: Westgard JO, Hunt MR. Use and interpretation of common statistical tests in method comparison studies. Clin Chem 1973;19:49 -57. 2 Marian Hunt and I published this paper to improve the understanding and application of commonly used statistics in method validation studies. We specifically wanted to clarify the misuse of the correlation coefficient and t-test statistics, and to point out important factors that affected the reliability of least squares statistics. This paper also provided some direction in making decisions on the acceptability of a method by using statistics to estimate the sizes of analytical errors. A subsequent paper (1 ) provided criteria for comparing those estimates of errors with quality standards that defined the amount of allowable error. This paper was written at a time when I was very involved in the evaluation of new automated analytic systems. At that time little guidance was available as to how to analyze and interpret the data from method evaluation experiments. Decisions on acceptability of new methods seldom provided any rational explanation as to why a method was judged acceptable or not. The correlation coefficient was the statistic most often being used to justify decisions on acceptability, followed by the t-test, in which the decision was based on whether or not the t-value indicated a statistically significant difference (i.e., calculated t-value greater than the critical t-value) between methods. Neither of these approaches took into account the actual quality required for the application of laboratory tests.Our approach was to experiment with different sets of data to see how the statistics responded to different analytical errors. Data sets were constructed to include known types and amounts of analytical errors. This investigation was a simple simulation study, made possible by new computer technology, a Compucorp 344 Statistician desk calculator, which allowed me the luxury of analyzing the data in my office rather than having to prepare punch-cards for use at the university's central mainframe computer.

Comparison of Two Diverse Measurement Methods by Graphical Agreement

“…A simple scatter plot and correlation analysis? 9 Bland and Altman published long ago that correlation and regression analyses are not suitable for method comparison. 10 In this article, we suggest using graphical tests and we believe that we have proposed the most appropriate one.…”

On the Reliability of Automatic Volume Delineation in Low-Contrast [18F]FMISO-PET Imaging

“…One source of this aspect of limited reproducibility in FMISO-PET may be the threshold approach and not solely the imaging technique. Additionally the technique of using voxel-wise Pearson correlation coefficient seems to be inadequate to measure reproducibility of serial PET scans (Schwartz et al 2011;Westgard 2008). The observations of Nehmeh et al that FMISO-PET is of limited reproducibility was furthermore contradicted in a recent study applying similar analysis methods to a different patient cohort (Okamoto et al 2013).…”

Comparison of Methods for Detecting and Quantifying Variation in Copy Numbers of Duplicated Genes

“…Now we need to evaluate the three methods in terms of precision and accuracy. Precision refers to random errors, whereas accuracy refers to systematic errors (Ahn, 2007;Westgard and Hunt, 1973). We will measure accuracy using the bias, which is the difference between the mean of the replicates and the reference point, and measure precision using the standard deviation(SD) of replicates for a reference point, As shown in Table 2 and Bias plots in Figure 6 and SD plots id Figure 7, peak height values in qOLA CNV fit the reference values better and show the least variation.…”