Adversary for Social Good: Protecting Familial Privacy through Joint Adversarial Attacks

Kumar, Chandan; Ryan, Riazat; Shao, Ming

doi:10.1609/aaai.v34i07.6791

Cited by 17 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, the robustness of multimodal kinship recognition is also an open issue, such as against adversarial attack [82], spoof attack [83], [84], poor conditions (e.g., modality missing and cross-modal feature learning). Data bias, fairness [79], and privacy-aware studies [85] are also worthy of further attention with the growing concern of data privacy protection. TALKIN-Family can also be helpful in audio-visual studies, such as talking face generation [86] and face-voice matching [87], and human perception studies on kin faces and voices.…”

Section: B Research Opportunities With the Talkin-family Datasetmentioning

confidence: 99%

Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Zhang

López

et al. 2024

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Facial kinship verification refers to automatically determining whether two people have a kin relation from their faces. It has become a popular research topic due to potential practical applications. Over the past decade, many efforts have been devoted to improving the verification performance from human faces only while lacking other biometric information, for example, speaking voice. In this article, to interpret and benefit from multiple modalities, we propose for the first time to combine human faces and voices to verify kinship, which we refer it as the audio-visual kinship verification study. We first establish a comprehensive audio-visual kinship dataset that consists of familial talking facial videos under various scenarios, called TALKIN-Family. Based on the dataset, we present the extensive evaluation of kinship verification from faces and voices. In particular, we propose a deep-learning-based fusion method, called unified adaptive adversarial multimodal learning (UAAML). It consists of the adversarial network and the attention module on the basis of unified multimodal features. Experiments show that audio (voice) information is complementary to facial features and useful for the kinship verification problem. Furthermore, the proposed fusion method outperforms baseline methods. In addition, we also evaluate the human verification ability on a

show abstract

Section: B Research Opportunities With the Talkin-family Datasetmentioning

confidence: 99%

Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Zhang

López

et al. 2024

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…Furthermore, the robustness of multimodal kinship recognition is also an open issue, such as against adversarial attack [71], spoof attack [72], poor conditions (e.g., modality missing, cross-modal feature learning). Data bias, fairness [70] and privacy-aware studies [73] are also worthy of further attention with the growing concern of data privacy protection. TALKIN-Family can also be helpful in audio-visual studies, such as talking face generation [74] and face-voice matching [75], and human perception studies on kin faces and voices.…”

Section: B Research Opportunities With the Talkin-family Datasetmentioning

confidence: 99%

Audio-Visual Kinship Verification: a New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Wu¹,

Feng²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>Facial kinship verification refers to automatically determining whether two people have a kin relation from their faces. It has become a popular research topic due to potential practical applications, such as finding missing children, family photo organization, or criminal investigations. Over the past decade, many efforts have been devoted to improving the verification performance of human faces only while lacking other biometric information, e.g., speaking voice. In this paper, to interpret and benefit from multiple modalities, we propose for the first time to combine human faces and voices to verify kinship, which we refer it as the audio-visual kinship verification study. Since there is still no standard and public audiovisual kinship dataset, we first establish a comprehensive audio-visual kinship dataset that consists of familial talking facial videos under various scenarios, called TALKIN-Family. Based on the dataset, we present the extensive evaluation of kinship verification from faces and voices. In particular, we propose a deep learning-based fusion method, named Unified Adaptive Adversarial Multimodal Learning (UAAML). It consists of the adversarial network and the attention module on the basis of unified multi-modal features. First, the modality adversarial learning eliminates the cross-modality variations by confusing the discriminator. The attention module quantifies the importance of kinship interested features. The overall multimodal fusion network is trained in Siamese fashion to encourage the compactness of kinship and separation of non-kinship. Experiments show that audio (voice) information is complementary to facial features and useful for the kinship verification problem. Further, the proposed fusion method outperforms baseline methods. In addition, we also evaluate the human kinship verification ability on a sub-set of TALKIN-Family. It indicates that human has higher accuracy when they have access to both faces and voice. The machine learning methods could effectively and efficiently outperform human ability. Finally, we include the future work and research opportunities with the TALKIN-Family dataset.</p>

show abstract

“…This was the beginning of big data in kin-based vision tasks-deep learning could then be used to overcome observed failure cases [59], [69]. Furthermore, new applications such as child appearance prediction [20], [22] and familial privacy protection [29] were done recently.…”

Section: A Kinship Recognitionmentioning

confidence: 99%

Families In Wild Multimedia: A Multimodal Database for Recognizing Kinship

Robinson,

Khan,

Yin

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Recognizing kinship -a soft biometric with vast applications -in photos has piqued the interest of many machine vision researchers. The large-scale Families In the Wild (FIW) database promoted the problem by supporting annual kinshipbased vision challenges that saw consistent performance improvements. We have now begun to approach performance levels for image-based systems acceptable for practical use -something unforeseeable a decade ago. However, biometric systems can benefit from multi-modal perspectives, as information contained in multimedia can add to and complement that of still images. Thus, we aim to narrow the gap from research-to-reality by extending FIW with multimedia data (i.e., video, audio, and contextual transcripts). Specifically, we introduce the first large-scale dataset for recognizing kinship in multimedia, the FIW in Multimedia (FIW-MM) database. We utilize automated machinery to collect, annotate, and prepare the data with minimal human input and no financial cost. This large-scale, multimedia corpus allows problem formulations to follow more realistic template-based protocols. We show significant improvements in benchmarks for multiple kin-based tasks when additional media-types are added. Experiments provide insights by highlighting edge cases to inspire future research and areas of improvement. Emphasis is put on short and long-term research directions, with the overarching intent to increase the potential of systems built to automatically detect kinship in multimedia. Furthermore, we expect a broader range of researchers with recognition tasks, generative modeling, speech understanding, and nature-based narratives.

show abstract

Adversary for Social Good: Protecting Familial Privacy through Joint Adversarial Attacks

Cited by 17 publications

References 18 publications

Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Audio-Visual Kinship Verification: a New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Families In Wild Multimedia: A Multimodal Database for Recognizing Kinship

Contact Info

Product

Resources

About