Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition

Abstract

As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.

A Checklist to Combat Cognitive Biases in Crowdsourcing

“…For instance, Otterbacher et al (2019) show how cognitive biases and stereotypes can affect image labeling and Peng et al (2019) discuss at length how cognitive biases may affect the hiring process. Mohanty et al (2019) and Kemmer et al (2020) acknowledge that a variety of biases such as the confirmation bias can lead to low-quality data labels and propose methods to mitigate these effects.…”

AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

“…Accuracy-focused tasks have one correct answer. Some crowdsourcing papers in this space include TurKit [93] and Second Opinion [109]. Examples of these tasks include math problems [38,88,101,145,158], blurry text recognition [92,93], and matching images with the same content [74,109].…”

“…Simplifying tasks can increase quality [28,83,120,142]. Another strategy is to adapt the subtask to fit the worker's capabilities [12,23,109,120]. For example, researchers found that crowdworkers were better at generating predictive features than at estimating if a feature is predictive; so they adapted the workflow accordingly [25].…”

“…This approach surfaces uncertainty in data as a feature for the user to explore, rather than hiding it in aggregation [23,25,139]. They can sort and filter the returned data [25,109,139], use a slider to control output characteristics like text length [15], and consider category labels at multiple levels of precision [23].…”

“…Other subtasks may include rating and ranking among multiple options [62,102,104,118,124,139], a pairwise comparison [20,92,93,104,105,109,116]. Evaluation subtasks may also determine if another round of generation needs to occur [84,88,134,142].…”