Observation-Oriented Modeling: Going Beyond “Is It All a Matter of Chance”?

Abstract

An alternative to null hypothesis significance testing is presented and discussed. This approach, referred to as observation-oriented modeling, is centered on model building in an effort to explicate the structures and processes believed to generate a set of observations. In terms of analysis, this novel approach complements traditional methods based on means, variances, and covariances with methods of pattern detection and analysis. Using data from a previously published study by Shoda et al., the basic tenets and methods of observation-oriented modeling are demonstrated and compared with traditional methods, particularly with regard to null hypothesis significance testing.

Four Bad Habits of Modern Psychologists

“…By contrast, even the simple integrated models presented above provided the means for differentiating between various cognitive processes and distinct types of causes. Additional examples of such models can be found in the work of Grice and his colleagues [ 11 , 46 , 47 , 48 ], Powers [ 49 , 50 ], and Cevasco and Marmolejo-Ramos [ 51 ].…”

Editorial: Dynamic Personality Science. Integrating between-Person Stability and within-Person Change

“…This proposal is far from being new (e.g., see already Lamiell, 1981 , 2013 , 2014 ). However, there is now an increasing awareness of the necessity to study individuals repeatedly over time in order to adequately describe, explain, and predict the psychological processes underlying behavior (e.g., Roe, 2008 , 2014 ; Grice, 2015 ; Grice et al, in press ). This, together with the availability of new technology (e.g., apps and mobile devices) and statistical advances that enable researchers to collect and model extensive repeated measurement data more efficiently and effectively will allow researchers to make greater progress.…”

Sequential Collaboration: The Accuracy of Dependent, Incremental Judgments

“…To complement the hypothesis tests with linear mixed models, we also rely on observation-oriented modeling. This framework describes a novel perspective on data analysis that goes beyond traditional statistics by focusing on the description, detection, and interpretation of theoretically relevant patterns in the raw data (Grice et al, 2012(Grice et al, , 2017Grice, 2011). Instead of drawing statistical inferences based on hypothesis testing within standard models such as ANOVA or regression modeling, observation-oriented modeling focuses on descriptive measures such as Percent Classification Correct (PCC) which is defined as the relative frequency of observations conforming to a specific pattern predicted by a hypothesis.…”

“…Moreover, it has a direct intuitive interpretation because it is always between 0% and 100% (with larger values indicating that the data better fit a predicted pattern) and can be used as an effect-size measure tailored to a specific hypothesis. Methods of observation-oriented modeling offer the additional benefit of not making distributional assumptions and are thus robust against outliers (Grice et al, 2012(Grice et al, , 2017. Given that observation-oriented modeling is robust to outliers, this allows us to test the robustness of our results by computing the PCC score twice using (1) the data excluding outliers similar as above and (2) the complete data including all outliers.…”