Avoiding the dangers of averaging across subjects when using multidimensional scaling

Lee, Michael; Pope, Kenneth J.

doi:10.1016/s0022-2496(02)00019-6

Cited by 29 publications

(31 citation statements)

References 31 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Initial investigations with averaged data, of the type considered by Shepard (1991), showed clearly that the repeated measures nature of individual-participant data was important for making sound inferences about the metric structure of the representational space. This is consistent with results showing that averaging similarity data with individual differences can systematically affect the metric structure of MDS spaces (see Ashby, Maddox, & Lee, 1994;Lee & Pope, 2003). Only three data sets could be found for which raw individual-participant data were available and for which reasonable predictions about the separability or integrality of the stimulus domain could be made.…”

Section: Inference From Mds Datasupporting

confidence: 89%

Three case studies in the Bayesian analysis of cognitive models

Lee

2008

Psychonomic Bulletin & Review

149

202

View full text Add to dashboard Cite

Psychology as an empirical science progresses through the development of formal models incorporating theoretical ideas designed to explain and predict observations of psychological phenomena. This means that progress in psychology relies upon the quality and completeness of the methods it uses to relate models and data. There is little point in developing theories and models, on the one hand, and collecting data in the laboratory or the field, on the other, if the two cannot be brought into contact in useful ways.In most empirical sciences, Bayesian methods have been or are rapidly being adopted as the most complete and coherent available way to relate models and data. Psychology has long been aware of problems with traditional frequentist and null hypothesis significance-testing approaches to parameter estimation and model selection, and recognition of the Bayesian alternative has followed from a number of recent articles and special volumes addressing the general issues (e.g., Lee & Wagenmakers, 2005;Myung, Forster, & Browne, 2000;Myung & Pitt, 1997;Pitt, Myung, & Zhang, 2002). Beyond the illustrative applications provided in these general treatments, however, there are few worked examples of Bayesian methods being applied to models at the forefront of modern psychological theorizing. Perhaps one reason is that there has been too great a focus on model selection defined in a narrow sense-particularly through the evaluation of Bayes factors-rather than a full Bayesian analysis. The perception that all that Bayesian methods have to offer for the evaluation of psychological models is a number that quantifies how much more likely one model is than another is dangerously limiting.In this article, three previous cognitive-modeling studies are revisited, in an attempt to demonstrate the generality and usefulness of the Bayesian approach. The three applications involve the multidimensional scaling (MDS) representation of stimulus similarity (Shepard, 1962(Shepard, , 1980, the generalized context model (GCM) account of category learning (Nosofsky, 1984(Nosofsky, , 1986, and a signal detection theory (SDT) account of inductive and deductive reasoning (Heit & Rotello, 2005). These applications were chosen in order to span a range of cognitive phenomena, to involve well-known and influential theories, and to put a focus on the ability of Bayesian methods to provide useful answers to important theoretical and empirical questions. METRIC MULTIDIMENSIONAL SCALING Theoretical BackgroundMDS representations of stimuli use a low-dimensional metric space in which points correspond to stimuli and the distance between points models the (dis)similarity between stimuli (Shepard, 1957(Shepard, , 1962(Shepard, , 1987(Shepard, , 1994. Nonmetric varieties of MDS algorithms for inferring these representations from pairwise similarity data (e.g., Kruskal, 1964) make only weak assumptions about the form of the relationship between distance in the MDS space and stimulus similarity. However, Shepard's (1987) universal law of generalization pr...

show abstract

Section: Inference From Mds Datasupporting

confidence: 89%

Three case studies in the Bayesian analysis of cognitive models

Lee

2008

Psychonomic Bulletin & Review

149

202

View full text Add to dashboard Cite

show abstract

“…For instance, when modeling the similarity between stimuli, it is commonplace to use the statistical technique known as multidimensional scaling. However, it has recently become clear that noisy, averaged data can distort the scaling solution in some ways (Ashby, Maddox & W. Lee, 1994; also see M. Lee & Pope 2003). This kind of problem is particularly pronounced for nonlinear models (e.g., Brown & Heathcote, 2003;Estes 1956;Myung, C. Kim & Pitt 2000), which are increasingly common in psychology.…”

Section: Data-fitting: a Local Model Analysismentioning

confidence: 99%

Assessing the distinguishability of models and the informativeness of data

Navarro

Pitt

Myung

2004

Cognitive Psychology

View full text Add to dashboard Cite

“…We follow Tenenbaum (1996; see also Lee, 2001;Lee & Pope, 2003) in assuming that the empirical dissimilarities follow Gaussian distributions with common variance σ 2 . As has been argued by Lee (2001), the variance quantifies the precision of the data and plays an important role in determining the appropriate balance between fit and complexity.…”

mentioning

confidence: 99%

Modeling individual differences in cognition

Lee

Webb

2005

Psychonomic Bulletin & Review

Self Cite

116

109

View full text Add to dashboard Cite

Many evaluations of cognitive models rely on data that have been averaged or aggregated across all experimental subjects, and so fail to consider the possibility of important individual differences between subjects. Other evaluations are done at the single-subject level, and so fail to benefit from the reduction of noise that data averaging or aggregation potentially provides. To overcome these weaknesses, we have developed a general approach to modeling individual differences using families of cognitive models in which different groups of subjects are identified as having different psychological behavior. Separate models with separate parameterizations are applied to each group of subjects, and Bayesian model selection is used to determine the appropriate number of groups. We evaluate this individual differences approach in a simulation study and show that it is superior in terms of the key modeling goals of prediction and understanding. We also provide two practical demonstrations of the approach, one using the ALCOVE model of category learning with data from four previously analyzed category learning experiments, the other using multidimensional scaling representational models with previously analyzed similarity data for colors. In both demonstrations, meaningful individual differences are found and the psychological models are able to account for this variation through interpretable differences in parameterization. The results highlight the potential of extending cognitive models to consider individual differences.

show abstract

Avoiding the dangers of averaging across subjects when using multidimensional scaling

Cited by 29 publications

References 31 publications

Three case studies in the Bayesian analysis of cognitive models

Three case studies in the Bayesian analysis of cognitive models

Assessing the distinguishability of models and the informativeness of data

Modeling individual differences in cognition

Contact Info

Product

Resources

About