Detecting Compromised Implicit Association Test Results Using Supervised Learning

Abstract

An implicit association test is a human psychological test used to measure subconscious associations. While widely recognized by psychologists as an effective tool in measuring attitudes and biases, the validity of the results can be compromised if a subject does not follow the instructions or attempts to manipulate the outcome. Compared to previous work, we collect training data using a more generalized methodology. We train a variety of different classifiers to identify a participant's first attempt versus a second possibly compromised attempt. To compromise the second attempt, participants are shown their score and are instructed to change it using one of five randomly selected deception methods. Compared to previous work, our methodology demonstrates a more robust and practical framework for accurately identifying a wide variety of deception techniques applicable to the IAT.

Lying on the Dissection Table: Anatomizing Faked Responses

“…Machine Learning is Able to Detect Fakers Boldt et al (2018) used native Bayes, support vector machines, multinomial logistic regression, multilayer perceptron, simple logistic regression, propositional rule learner, and random forest on data from a self-developed IAT and showed that machine learning was able to detect fakers successfully. Machine learning performed better than Agosta et al's (2011) IAT faking index.…”

Lying on the Dissection Table: Anatomizing Faked Responses

“…There is more faking of low scores than of high scores, and thus, classifiers should be better at detecting faked low scores than at detecting faked high scores. However, previous studies have either included only one faking direction (i.e., faking good; Calanna et al, 2020) or did not distinguish between faking directions (Boldt et al, 2018). Third, faking differs between naive and informed conditions (e.g., Röhner, 2013), and there is more evidence of faking when participants have information than when they are naïve (Röhner et al, 2011).…”

“…Taking a look at the importance of the features offers insights into what (most) fakers did and whether their behavior varied across conditions. Boldt et al (2018) used native Bayes, support vector machines, multinomial logistic regression, multilayer perceptron, simple logistic regression, propositional rule learner, and random forest on data from a self-developed IAT and showed that machine learning was able to detect fakers successfully. Machine learning performed better than Agosta et al's (2011) IAT faking index.…”

“…Calanna et al (2020) found that XGBoost worked best in faking detection. Boldt et al (2018) showed that logistic regression worked best. 5 Because these two studies differed with 5 For reasons of comparability, we refer to the unpruned data set from Boldt et al (2018) here.…”

“…Boldt et al (2018) showed that logistic regression worked best. 5 Because these two studies differed with 5 For reasons of comparability, we refer to the unpruned data set from Boldt et al (2018) here. respect to measures, constructs, and faking directions, this difference may be explained by factors in the study designs.…”