Classifiers as a model-free group comparison test

Abstract

The conventional statistical methods to detect group differences assume correct model specification, including the origin of difference. Researchers should be able to identify a source of group differences and choose a corresponding method. In this paper, we propose a new approach of group comparison without model specification using classification algorithms in machine learning. In this approach, the classification accuracy is evaluated against a binomial distribution using Independent Validation. As an application example, we examined false-positive errors and statistical power of support vector machines to detect group differences in comparison to conventional statistical tests such as t test, Levene's test, K-S test, Fisher's z-transformation, and MANOVA. The SVMs detected group differences regardless of their origins (mean, variance, distribution shape, and covariance), and showed comparably consistent power across conditions. When a group difference originated from a single source, the statistical power of SVMs was lower than the most appropriate conventional test of the study condition; however, the power of SVMs increased when differences originated from multiple sources. Moreover, SVMs showed substantially improved performance with more variables than with fewer variables. Most importantly, SVMs were applicable to any types of data without sophisticated model specification. This study demonstrates a new application of classification algorithms as an alternative or complement to the conventional group comparison test. With the proposed approach, researchers can test two-sample data even when they are not certain which statistical test to use or when data violates the statistical assumptions of conventional methods.

A Practical Guide to Variable Selection in Structural Equation Modeling by Using Regularized Multiple-Indicators, Multiple-Causes Models

“…This allows researchers to uncover nonlinearities and interactions. Additional methods include the use of heuristic search algorithms (e.g., Marcoulides & Ing, 2012), various methods for identifying group differences (Frick, Strobl, & Zeileis, 2015; Kim & von Oertzen, 2018; Tutz & Schauberger, 2015), and the use of graphical models for identifying latent variables (e.g., Epskamp et al, 2017). Given the increasing amounts of data sharing, facilitated by various new tools for data storage and sharing, such as the Open Science Framework (https://osf.io/) and OpenfMRI (https://openfmri.org/), we can envision the utility of testing models much larger than our template simulation model.…”

Decoding and mapping task states of the human brain via deep learning

“…For years, researchers have been attempting to decode and identify functions of the human brain based on functional brain imaging data (Dehaene et al, 1998;Haynes & Rees, 2006;Jang, Plis, Calhoun, & Lee, 2017;Poldrack, Halchenko, & Hanson, 2009;Rubin et al, 2017). The most popular among these brain-decoding methods is the support vector machine (SVM)-based multi-voxel pattern analysis (MVPA), a supervised technology that incorporates information from multiple variables at the same time (Kim & Oertzen, 2018;Kriegeskorte & Bandettini, 2007;Kriegeskorte, Goebel, & Bandettini, 2006;Norman, Polyn, Detre, & Haxby, 2006). Despite its popularity, the SVM struggles to perform well on high-dimensional raw data, and requires the expert use of design techniques for feature selection/extraction (LeCun, Bengio, & Hinton, 2015;Vieira, Pinaya, & Mechelli, 2017).…”

Bringing Back the Person into Behavioural Personality Science Using Big Data

“…Such techniques are particularly helpful to automatically label data (e.g. movement data, to identify which type of movement the participant did at any specific time) but also to identify differences between groups in the prelabelled data (Kim & Von Oertzen, 2017). For example, in the analysis of a specific person, a researcher might be interested whether this person displays different behaviours when in the presence of female participants than with male participants.…”

“…In 1900, William Stern termed individuality as the problem of 20th century, and in the same vein, Allport (1937) pointed out that ‘the outstanding characteristic of man is his individuality’ (p. 3). In addition, Kluckhohn and Murray's (1949) famous dictum that every man is in certain respects like all other men , like some other men , and like no other man reminds us that the uniqueness of the individual person (like no other man) is an important part of personality research.…”