Photo Sleuth

Abstract

Identifying people in historical photographs is important for preserving material culture, correcting the historical record, and creating economic value, but it is also a complex and challenging task. In this article, we focus on identifying portraits of soldiers who participated in the American Civil War (1861--65), the first widely photographed conflict. Many thousands of these portraits survive, but only 10%--20% are identified. We created Photo Sleuth, a web-based platform that combines crowdsourced human expertise and automated face recognition to support Civil War portrait identification. Our mixed-methods evaluations of Photo Sleuth one month and 11 months after its public launch showed that it helped users successfully identify unknown portraits and provided a sustainable model for volunteer contribution. We also discuss implications for crowd-AI interaction and person identification pipelines.

Inclusive Portraits: Race-Aware Human-in-the-Loop Technology

“…In previous research, there have been attempts to combine human and machine intelligence to improve facial verification [63,71]. Our study builds on this work by investigating how human-machine collaborations can be improved by taking into account workers' racial backgrounds.…”

“…IP's human component incorporates a racially aware interface to efficiently manage the coordination of crowdworkers for the facial verification of images where the automated component is uncertain. Unlike traditional human-in-the-loop interfaces that assign "anonymous" crowdworkers to verification tasks [63], IP's approach takes into account the diversity of workers' backgrounds and experiences. This results in a more robust and reliable verification process, especially when verifying the faces of people of color.…”

“…Humans may be better at recognizing other humans under a wide range of settings and can also recognize people as they evolve and change through time [61,62]. Facial verification systems may therefore improve further through the integration of human workers [63].…”

“…Such approaches are typically known as "human-in-the-loop" (HITL) [100]. Examples of HITL approaches within this context involve crowdworkers helping to verify unknown soldier portraits [63], or detecting fraudulent passports [96].…”