The construct–behavior gap in behavioral decision research: A challenge beyond replicability.

Abstract

Behavioral decision research compares theoretical constructs like preferences to behavior such as observed choices. Three fairly common links from constructs to behavior are (1) to tally, across participants and decision problems, the number of choices consistent with one predicted pattern of pairwise preferences; (2) to compare what most people choose in each decision problem against a predicted preference pattern; or (3) to enumerate the decision problems in which two experimental conditions generate a 1-sided significant difference in choice frequency 'consistent' with the theory. Although simple, these theoretical links are heuristics. They are subject to well-known reasoning fallacies, most notably the fallacy of sweeping generalization and the fallacy of composition. No amount of replication can alleviate these fallacies. On the contrary, reiterating logically inconsistent theoretical reasoning over and again across studies obfuscates science. As a case in point, we consider pairwise choices among simple lotteries and the hypotheses of overweighting or underweighting of small probabilities, as well as the description-experience gap. We discuss ways to avoid reasoning fallacies in bridging the conceptual gap between hypothetical constructs, such as, for example, "overweighting" to observable pairwise choice data. Although replication is invaluable, successful replication of hard-to-interpret results is not. Behavioral decision research stands to gain much theoretical and empirical clarity by spelling out precise and formally explicit theories of how hypothetical constructs translate into observable behavior. (PsycINFO Database Record

Multinomial models with linear inequality constraints: Overview and improvements of computational methods for Bayesian inference

“…As illustrated in Section 5.2 for the description-experience gap in the domain of risky gambles (Hertwig et al, 2004), the hypothesis that participants assign more weight to small probabilities results in multiple predicted patterns. The complete set of predicted patterns can be obtained in different ways (Regenwetter and Robinson, 2017), for instance, by (a) translating a verbal theory into predicted patterns, (b) deriving algebraic implications of axioms or formal theories, and (c) brute force enumeration of all of the predictions made by the deterministic theory, typically under a set of theory-specific assumptions (e.g., theory parameter values). 1…”

“…These stimuli were constructed to ensure that over-and underweighting of small probabilities would result in distinct predicted patterns across the six decision problems. However, under the assumption that preferences are heterogeneous across participants and trials, both overand underweighting imply multiple choice patterns that are in line with the psychological theory (Regenwetter and Robinson, 2017). Given such heterogeneous predictions, observed choice frequencies can be modelled by a mixture distribution over the predicted patterns, and thus, by a multinomial model with inequality constraints (cf.…”

“…To derive all patterns that are predicted when assuming heterogeneous preferences, Regenwetter and Robinson (2017) repeatedly simulated preference patterns with cumulative prospective theory (Kahneman and Tversky, 1979) Here, the number of responses n = 25 was identical for each lottery, and thus we can provide only a scalar value instead of a vector. The Gibbs sampler for the V -representation required 62 seconds to sample M = 5000 iterations on each of cpu = 8 processing units in parallel.…”

“…bf <-bf _ binom ( k = c (9 , 16 , 16 , 7 , 12 , 16) versus B u0 ≈ 1, 100, respectively; cf. Table 6 in Regenwetter and Robinson (2017)). Even though the substantive conclusions are similar to those of the original analysis in this specific example, there are examples in the literature where the reliance on multinomial models with inequality constraints can make a big difference (for instance, when testing transitivity of preferences; Regenwetter and Davis-Stober (2012)).…”

“…For our simple dosage example, the set of all extremal points is the set of all vectors, θ, where each entry is equal to 0 or 1 and satisfy the above inequalities, which yields the set: (0, 0, 0), (1, 0, 0), (1, 1, 0), and (1, 1, 1). Section 1.1 shows that it is often relatively easy to derive these vertices by enumerating all patterns that are predicted by a psychological theory even though it may be difficult to specify the corresponding system of inequality constraints (Regenwetter and Robinson, 2017).…”