Discerning truth from deception: Human judgments and automation efforts

Abstract

Recent improvements in effectiveness and accuracy of the emerging field of automated deception detection and the associated potential of language technologies have triggered increased interest in mass media and general public. Computational tools capable of alerting users to potentially deceptive content in computer–mediated messages are invaluable for supporting undisrupted, computer–mediated communication and information practices, credibility assessment and decision–making. The goal of this ongoing research is to inform creation of such automated capabilities. In this study we elicit a sample of 90 computer–mediated personal stories with varying levels of deception. Each story has 10 associated human deception level judgments, confidence scores, and explanations. In total, 990 unique respondents participated in the study. Three approaches are taken to the data analysis of the sample: human judges, linguistic detection cues, and machine learning. Comparable to previous research results, human judgments achieve 50–63 percent success rates, depending on what is considered deceptive. Actual deception levels negatively correlate with their confident judgment as being deceptive (r = -0.35, df = 88, ρ = 0.008). The highest-performing machine learning algorithms reach 65 percent accuracy. Linguistic cues are extracted, calculated, and modeled with logistic regression, but are found not to be significant predictors of deception level, confidence score, or an authors’ ability to fool a reader. We address the associated challenges with error analysis. The respondents’ stories and explanations are manually content–analyzed and result in a faceted deception classification (theme, centrality, realism, essence, self–distancing) and a stated perceived cue typology. Deception detection remains novel, challenging, and important in natural language processing, machine learning, and the broader library information science and technology community.

Fake News or Truth? Using Satirical Cues to Detect Potentially Misleading News

“…In general, humans are fairly ineffective at recognizing deception (DePaulo, Charlton, Cooper, Lindsay, & Muhlenbruck, 1997;Rubin & Conroy, 2012;Vrij, Mann, & Leal, 2012). A number of factors may explain why.…”

Veracity Roadmap: Is Big Data Objective, Truthful and Credible?

“…Journalism, online marketing, proofreading and political science, to name a few. For example, in political science Politifact (albeit based on man-powered fact-checking) and TruthGoggles sort true facts in politics helping citizens to develop better understanding of politicians statements (Rubin and Conroy, 2012 Rubin and Chen (2012). Certain types of deception strategies cannot be spotted automatically based on underlying linguistic differences between truth-tellers and liars.…”

“…Objectivity-subjectivity variation in many ways depends on its context, since context determines the types of linguistic cues used to express objective or subjective opinions (Hirst, 2007). To quantify deception levels in big data, we propose to use the existing automated tools on deception detection (see overview in (Rubin & Conroy, 2012;Rubin & Lukoianova, Forthcoming;Rubin & Vashchilko, 2012). For credibility assessment, we propose to use blogs that contain trust evaluation of published content or entire websites.…”

Deception Detection and Rumor Debunking for Social Media

“…Humans are notoriously poor lie detectors even when they are alerted to the possibility of being lied to (Vrij, 2004, Vrij, 2000, Vrij et al, 2012. A widely cited source that conducted a meta-analytical review of over 100 experiments with over 1,000 participants (DePaulo et al, 1997), concludes that on average people are able to distinguish a lie from a truthful statement with a mean accuracy rate of 54%, slightly above chance (Rubin and Conroy, 2012).…”