Photo sleuth

Mohanty, Vikram; Thames, David; Mehta, Sneha; Luther, Kurt

doi:10.1145/3301275.3302301

Cited by 17 publications

(5 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This process of seeking and creating connections requires specialized domain knowledge. Even if matching backdrops are found, successful identification of the photographer or subject is not guaranteed, paralleling observations from other types of Civil War photo investigations (Mohanty et al 2019).…”

Section: Related Work Historical Backdropsmentioning

confidence: 96%

“…CV's ability to recognize scenes have also been used to combat human trafficking (Stylianou et al 2017(Stylianou et al , 2019 as well as assist visually impaired persons explore indoor environments (Afif et al 2020). In the context of historical photo identification, facial recognition combined with crowdsourcing has successfully assisted in identifying soldiers in Civil War photos (Mohanty et al 2019). We take inspiration from these application for Back-Trace, utilizing CV to assist and empower users in painted backdrop research.…”

Section: Human-ai Collaborative Approaches For Visual Investigation T...mentioning

confidence: 99%

“…The issue of missing metadata is frequently cited as a key problem in the curation of rapidly growing cultural heritage image collections (Cordell 2020), with many computer vision (CV) techniques applied towards solving it (Mohanty et al 2019;Chumachenko et al 2020;Zeitlyn, Coto, and Zisserman 2021;Caraffa et al 2020;Resig 2014). One aspect of this problem is inadequate artist or photographer information in visual collections.…”

Section: Introductionmentioning

confidence: 99%

“…For this project, we focused on the American Civil War (1861-1865), the first major conflict to be extensively documented via photography and whose history is closely connected to that of painted backdrops (Military Images 1996; Keller 2021). We chose Civil War Photo Sleuth (CWPS), 1 an online community dedicated to identifying unknown Civil War portraits (Mohanty et al 2019), as the base of our system due to its vast repository of Civil War photos.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

BackTrace: A Human-AI Collaborative Approach to Discovering Studio Backdrops in Historical Photographs

Lim,

Mohanty,

Dodson

et al. 2023

HCOMP

View full text Add to dashboard Cite

In historical photo research, the presence of painted backdrops have the potential to help identify subjects, photographers, locations, and events surrounding certain photographs. However, there are few dedicated tools or resources available to aid researchers in this largely manual task. In this paper, we propose BackTrace, a human-AI collaboration system that employs a three-step workflow to retrieve and organize historical photos with similar backdrops. BackTrace is a content-based image retrieval (CBIR) system powered by deep learning that allows for the iterative refinement of search results via user feedback. We evaluated BackTrace with mixed-methods evaluation and found that it successfully aided users in finding photos with similar backdrops and grouping them into collections. Finally, we discuss how our findings can be applied to other domains, as well as implications of deploying BackTrace as a crowdsourcing system.

show abstract

Section: Related Work Historical Backdropsmentioning

confidence: 96%

Section: Human-ai Collaborative Approaches For Visual Investigation T...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

BackTrace: A Human-AI Collaborative Approach to Discovering Studio Backdrops in Historical Photographs

Lim,

Mohanty,

Dodson

et al. 2023

HCOMP

View full text Add to dashboard Cite

show abstract

“…In parallel, the limitations above have also led researchers to integrate human workers into the machine learning pipeline, to assist the automated methods in the facial verification process [71]. Humans may be better at recognizing other humans under a wide range of settings and can also recognize people as they evolve and change through time [61,62]. Facial verification systems may therefore improve further through the integration of human workers [63].…”

mentioning

confidence: 99%

Inclusive Portraits: Race-Aware Human-in-the-Loop Technology

Flores-Saviaga,

Curtis,

Savage

2023

Equity and Access in Algorithms, Mechanisms, and Optimization

View full text Add to dashboard Cite

Fig. (1) Overview of Inclusive Portait's functionality.AI has revolutionized the processing of various services, including the automatic facial verification of people. Automated approaches have demonstrated their speed and efficiency in verifying a large volume of faces, but they can face challenges when processing content from certain communities, including communities of people of color. This challenge has prompted the adoption of "human-in-the-loop" (HITL) approaches, where human workers collaborate with the AI to minimize errors. However, most HITL approaches do not consider workers' individual characteristics and backgrounds. This paper proposes a new approach, called Inclusive Portraits (IP), that connects with social theories around race to design a racially-aware human-in-the-loop system. Our experiments have provided evidence that incorporating race into human-in-the-loop (HITL) systems for facial verification can significantly enhance performance, especially for services delivered to people of color. Our findings also highlight the importance of considering individual worker characteristics in the design of HITL systems, rather than treating workers as a homogenous group. Our research has significant design implications for developing AI-enhanced services that are more inclusive and equitable.

show abstract

Machine Learning Based Analysis of Finnish World War II Photographers

et al. 2020

View full text Add to dashboard Cite

In this paper, we demonstrate the benefits of using state-of-the-art machine learning methods in the analysis of historical photo archives. Specifically, we analyze prominent Finnish World War II photographers, who have captured high numbers of photographs in the publicly available Finnish Wartime Photograph Archive, which contains 160,000 photographs from Finnish Winter, Continuation, and Lapland Wars captures in 1939-1945. We were able to find some special characteristics for different photographers in terms of their typical photo content and framing (e.g., close-ups vs. overall shots, number of people). Furthermore, we managed to train a neural network that can successfully recognize the photographer from some of the photos, which shows that such photos are indeed characteristic for certain photographers. We further analyzed the similarities and differences between the photographers using the features extracted from the photographer classifier network. We make our annotations and analysis pipeline publicly available, in an effort to introduce this new research problem to the machine learning and computer vision communities and facilitate future research in historical and societal studies over the photo archives.

show abstract

Photo sleuth

Cited by 17 publications

References 33 publications

BackTrace: A Human-AI Collaborative Approach to Discovering Studio Backdrops in Historical Photographs

BackTrace: A Human-AI Collaborative Approach to Discovering Studio Backdrops in Historical Photographs

Inclusive Portraits: Race-Aware Human-in-the-Loop Technology

Machine Learning Based Analysis of Finnish World War II Photographers

Contact Info

Product

Resources

About