Families In Wild Multimedia: A Multimodal Database for Recognizing Kinship

Abstract

Recognizing kinship -a soft biometric with vast applications -in photos has piqued the interest of many machine vision researchers. The large-scale Families In the Wild (FIW) database promoted the problem by supporting annual kinshipbased vision challenges that saw consistent performance improvements. We have now begun to approach performance levels for image-based systems acceptable for practical use -something unforeseeable a decade ago. However, biometric systems can benefit from multi-modal perspectives, as information contained in multimedia can add to and complement that of still images. Thus, we aim to narrow the gap from research-to-reality by extending FIW with multimedia data (i.e., video, audio, and contextual transcripts). Specifically, we introduce the first large-scale dataset for recognizing kinship in multimedia, the FIW in Multimedia (FIW-MM) database. We utilize automated machinery to collect, annotate, and prepare the data with minimal human input and no financial cost. This large-scale, multimedia corpus allows problem formulations to follow more realistic template-based protocols. We show significant improvements in benchmarks for multiple kin-based tasks when additional media-types are added. Experiments provide insights by highlighting edge cases to inspire future research and areas of improvement. Emphasis is put on short and long-term research directions, with the overarching intent to increase the potential of systems built to automatically detect kinship in multimedia. Furthermore, we expect a broader range of researchers with recognition tasks, generative modeling, speech understanding, and nature-based narratives.

One Label, One Billion Faces

“…The field of soft biometrics encompasses notions of identity and traits which are not unique to an individual, but carry information meaningful to society for identification. Canonical examples of soft biometrics are kinship [40], age [8], gender [53], and demographic information [15]. The line of work most relevant to our work is coarse-grained demographic estimation, which aims to predict a broad racial category for a face.…”

“…Datasets are a primary driver of progress in machine learning and computer vision, and each dataset reflects human values. Many of the most visible applications of computer vision require datasets consisting of human faces: face recognition [12], kinship [40], demographic estimation [15], emotion recognition [50], and generative modeling [26]. The datasets driving face-centric computer vision often come with racial annotations of identity, expressed as a race category assigned to each face.…”