Assessing individual differences in categorical data

Abstract

This article develops statistical methods to detect individual differences in cognitive data in the form of categorical observations from a set of participants, each of whom responds to the same set of item events-for example, memory items, item serial positions, or repeated choice trials. The purpose of these methods is to determine whether there is heterogeneity in either participants or item events that might make it inappropriate to aggregate the data, respectively, over participants or item events. These determinations are especially important in the case where the data will be analyzed with a cognitive model. If heterogeneity in participants or item events is detected, several recent developments in computational statistics make it feasible to augment a parametric model with a hierarchical level that models random effects on the parameters. On the other hand, a model that includes random effects assumptions on the parameters is clearly more complex (e.g., Myung & Pitt, 1997) and capable of accounting for more data than the same model is, assuming that all effects are equal; so the decision of whether or not to include random effects in a model analysis is a crucial one. Our methods for informing this decision, because they are based on the sampling assumptions of the basic data structure itself, are completely agnostic as to the appropriate cognitive model underlying the data. Before developing the methods, it is useful to review the way that most cognitive models have been employed in the past and how researchers have tried to deal with parametric heterogeneity.Parametric stochastic models of cognition are usually assessed with data taken from one or more experimental conditions consisting of a group of participants, each of whom provides responses to the same set of item events (hereafter, items). Within a given experimental condition, subsets of these observations are usually pooled (aggregated) and treated as if they are a sample from the cognitive model. In the terminology of mathematical statistics, the assumption that a sequence of observations constitutes a sample from a parametric model means that the observations arise from a sequence of independent and identically distributed (i.i.d.) random variables, whose common distribution is determined through the model equations by some fixed but unknown set of model parameters (e.g., Hogg, McKean, & Craig, 2005;Lehmann & Romano, 2005). One particular form of this assumption is when data on a particular item are aggregated over participants and analyzed on the assumption of no individual differences-that is, the assumption of participant homogeneity. Another form of the assumption is when the data from each particular participant are treated as a sample over items. It is not uncommon in list-memory experiments (e.g., the middle items of free recall, old or new items in recognition memory) to aggregate the data over both participants and items, thereby treating all the observations in some experimental condition as a sample from the model.Most examples in ...

Is prospective memory related to depression and anxiety? A hierarchical MPT modelling approach

“…If this assumption does not hold, parameter estimates may be biased (Klauer, 2006(Klauer, , 2010Smith & Batchelder, 2008, 2010. In addition, the traditional approach yields only groupwise parameter estimates.…”

“…In traditional MPT modelling, all participants and all items are assumed to be homogeneous, that is, to have the same parameter values (Smith & Batchelder, 2008). If this assumption does not hold, parameter estimates may be biased (Klauer, 2006(Klauer, , 2010Smith & Batchelder, 2008, 2010.…”

Explaining individual differences in cognitive processes underlying hindsight bias

“…Ignoring this heterogeneity in analyses of aggregated data is also problematic from a statistical point of view, as parameter heterogeneity may distort results of parameter estimation and goodness-of-fit tests (Batchelder & Riefer, 1999;Erdfelder et al, 2009;Klauer, 2006Klauer, , 2010Smith & Batchelder, 2008;. Furthermore, models that adequately describe the response structure at an individual level will often be rejected at the aggregate group level because the parameters vary across individuals .…”

“…older adults, see Smith & Batchelder, 2008). Ignoring this heterogeneity in analyses of aggregated data is also problematic from a statistical point of view, as parameter heterogeneity may distort results of parameter estimation and goodness-of-fit tests (Batchelder & Riefer, 1999;Erdfelder et al, 2009;Klauer, 2006Klauer, , 2010Smith & Batchelder, 2008;.…”

multiTree: A computer program for the analysis of multinomial processing tree models

“…Second, it would be useful to link multiTree to more general statistical programming environments such as the R project. Finally, multiTree offers no means to diagnose or handle heterogeneity across items and/or participants (Klauer, 2006;J. B. Smith & Batchelder, 2008).…”