Fairness evaluation in deepfake detection models using metamorphic testing

Pu, Muxin; Kuan, Meng Yi; Lim, Nyee Thoang; Chong, Chun Yong; Lim, Mei Kuan

doi:10.1145/3524846.3527337

Cited by 11 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The advantage of metamorphic testing over traditional testing methods is that there is no need for predefined outputs, and the test can be performed effectively even if the expected results are not clear. It has been found in studies [3][4] that Fault Detection Rate (FDR) is widely used to evaluate the efficiency of metamorphic relations for the detection of image classification procedures. For example, Dwarakanath et al [5] proposed four metamorphic relations to capture errors in a program for support vector machine-based and deep learning-based image classification programs, respectively, which were benchmarked against FDR.…”

Section: Introductionmentioning

confidence: 99%

Metamorphic relationship responsiveness based on improved K-means algorithm

Du,

wang,

wang

et al. 2024

Fourth International Conference on Digital Signal and Computer Communications (DSCC 2024)

View full text Add to dashboard Cite

Testing machine learning programs is challenging and costly using existing methods. Among them, metamorphic testing reduces the cost by verifying that metamorphic relationships are satisfied between test cases and outputs. The core of metamorphic testing is the metamorphic relationship, and the currently used Failure Detecting (FDR) fails to fully reveal its key properties in terms of error detection. Therefore, this paper proposes the concept of metamorphic relationship responsiveness, which comprehensively responds to the characteristics of metamorphic relationships by expressing the error detection rate of each metamorphic relationship as a multi-dimensional vector, thus providing more possibilities for machine learning research. And the optimized K-means algorithm is used for clustering analysis of metamorphic relationships, which mines out the association between metamorphic relationship construction requirements and error discovery ability. Experiments show that the metamorphic relationship reactivity can well support the cluster analysis and mine effective knowledge.

show abstract

Section: Introductionmentioning

confidence: 99%

Metamorphic relationship responsiveness based on improved K-means algorithm

Du,

wang,

wang

et al. 2024

Fourth International Conference on Digital Signal and Computer Communications (DSCC 2024)

View full text Add to dashboard Cite

show abstract

“…Metamorphic testing (MT) has been widely used to test learning-based models in the test oracle problem due to its simplicity of concept and effectiveness in fault detection [1,12,16,23]. MT verifies and validates the robustness of models against metamorphic relations (MRs), which are essential properties of the target algorithms or models in relation to various inputs and their expected outputs [1].…”

Section: Introductionmentioning

confidence: 99%

Robustness Evaluation in Hand Pose Estimation Models using Metamorphic Testing

Pu¹,

Chong²,

Lim³

2023

Preprint

View full text Add to dashboard Cite

Hand pose estimation (HPE) is a task that predicts and describes the hand poses from images or video frames. When HPE models estimate hand poses captured in a laboratory or under controlled environments, they normally deliver good performance. However, the real-world environment is complex, and various uncertainties may happen, which could degrade the performance of HPE models. For example, the hands could be occluded, the visibility of hands could be reduced by imperfect exposure rate, and the contour of hands prone to be blurred during fast hand movements. In this work, we adopt metamorphic testing to evaluate the robustness of HPE models and provide suggestions on the choice of HPE models for different applications. The robustness evaluation was conducted on four state-of-the-art models, namely MediaPipe hands, OpenPose, BodyHands, and NSRM hand. We found that on average more than 80% of the hands could not be identified by BodyHands, and at least 50% of hands could not be identified by MediaPipe hands when diagonal motion blur is introduced, while an average of more than 50% of strongly underexposed hands could not be correctly estimated by NSRM hand. Similarly, applying occlusions on only four hand joints will also largely degrade the performance of these models. The experimental results show that occlusions, illumination variations, and motion blur are the main obstacles to the performance of existing HPE models. These findings may pave the way for researchers to improve the performance and robustness of hand pose estimation models and their applications.

show abstract