How to Weight Scientists' Probabilities Is Not a Big Problem: Comment on Barnes

Abstract

Assuming it rational to treat other persons' probabilities as epistemically significant, how shall their judgements be weighted (Barnes [1998])? Several plausible methods exist, but theorems in classical psychometrics greatly reduce the importance of the problem. If scientists' judgements tend to be positively correlated, the difference between two randomly weighted composites shrinks as the number of judges rises. Since, for reasons such as representative coverage, minimizing bias, and avoiding elitism, we would rarely employ small numbers of judges (e.g. less than 10), the difference between two weighting systems becomes negligible. Suggestions are made for quantifying verisimilitude, identifying 'types' of scientists or theories (taxometrics), inferring latent factors, and estimating reliability of pooled judgements. A challenging article by Eric Christian Barnes ([1998]) argues for the rationality of treating other persons' probabilities as epistemically significant. I do not address the qualitative merits of this position, about which I have no settled opinion. 1 My purpose here is to offer a psychologist's reassurance to Barnes in respect to one of his major concerns: how to weight the opinions of a group of scientists who will surely disagree in their subjective probabilities. If we do not give equal weight to the subjective probabilities of Scientists X 1 , X 2 ,…X n , we would wish to weight them differentially on the basis of their several 'competencies'. Competence being itself a complex notion, Barnes considers two ways of judging it, which, I gather, he does not claim to be exhaustive. One way ('mapmakers') involves a fairly complex indexing of the judges' knowledge bases. The other ('probabilistic jury') relies on their

A taxometric approach to base-rate estimation and idiographic classification in psycholegal research.

“…Concerns about interrater reliability may naturally arise when considering this manner of coding. However, we note that such content-coding approaches have proven relatively reliable in previous projects (e.g., Amendola & Wixted, 2015; Gould et al, 2014; Horry et al, 2014); further, even relatively poor pairwise interrater reliability can be overcome by modestly increasing the pool of raters (viz., Spearman–Brown prophecy formula; see Meehl, 1999b). Encouragingly, empirical research suggests that aggregating judgments of forensic evidence (even from novices) may be more accurate than the individual judgments of expert evaluators (Tangen et al, 2020).…”

Why Paul Meehl will revolutionize the philosophy of science and why it should matter to psychologists

“…However, such problems are often expected in a new area, essential methodological challenges seem surmountable, and, at least at first, there are good reasons to assume that even relatively unrefined or crude approaches will produce gains (for a fuller discussion of some of these issues, see Faust, 1984;Faust & Meehl, 2002;Meehl, 1992Meehl, , 1999. For example, although one might think that the problems inherent in gathering some of the needed data bases are 2 The boundaries between "regular" science and meta-science can be fuzzy or overlapping, and the second example I will discuss of meta-scientific study might seem to be more regular science than meta-science.…”

2001 award winners.

“…Absent various meta-analyses across diverse literatures, there would be no informed, evidence-based way to establish proportionality in the importance of the risk factors listed in Table 3. Everyone would be prey to scientific salespersons, zealots, and pressure groups of various persuasions and motivations (see Meehl, 1999). R. W. Heinrichs (2001) has shown a way out of the wilderness by conducting meta-analyses of 54 English-language litera-tures reporting the research with many of the risk factors alleged to be important in the etiology of, or correlated to, schizophrenia.…”