Common Evaluation Pitfalls in Touch-Based Authentication Systems

Abstract

In this paper, we investigate common pitfalls affecting the evaluation of authentication systems based on touch dynamics. We consider different factors that lead to misrepresented performance, are incompatible with stated system and threat models or impede reproducibility and comparability with previous work. Specifically, we investigate the effects of (i) small sample sizes (both number of users and recording sessions), (ii) using different phone models in training data, (iii) selecting non-contiguous training data, (iv) inserting attacker samples in training data and (v) swipe aggregation. We perform a systematic review of 30 touch dynamics papers showing that all of them overlook at least one of these pitfalls. To quantify each pitfall's effect, we design a set of experiments and collect a new longitudinal dataset of touch dynamics from 470 users over 31 days comprised of 1,166,092 unique swipes. We make this dataset and our code available online. Our results show significant percentage-point changes in reported mean EER for several pitfalls: including attacker data (2.55%), non-contiguous training data (3.8%), phone model mixing (3.2%-5.8%). We show that, in a common evaluation setting, cumulative effects of these evaluation choices result in a combined difference of 8.9% EER. We also largely observe these effects across the entire ROC curve. Furthermore, we validate the pitfalls on four distinct classifiers -SVM, Random Forest, Neural Network, and kNN. Based on these insights, we propose a set of best practices that, if followed, will lead to more realistic and comparable reporting of results in the field.

Techniques for Continuous Touch-Based Authentication Modeling

“…While some systems might benefit from choosing thresholds for optimizing better FAR or FRR we believe EER is the most representative of the general performance of a system. This is also supported by the related work [14,26,52]. In particular, [26] show that when comparing two touch-based authentication models, the performance differences between them on the ROC curve are largely consistent with the difference at the EER point.…”

“…This is also supported by the related work [14,26,52]. In particular, [26] show that when comparing two touch-based authentication models, the performance differences between them on the ROC curve are largely consistent with the difference at the EER point. The "Other" category means the particular methods have been used less than 3 times in the case of Classification and Metrics and less than 2 times in the case of Aggregation.…”

“…These are required for the majority of feature extraction approaches. The only publicly available datasets which we consider usable under these conditions are Touchalytics [23], Bioident [7] and CEP [26]. We focus on these three datasets in the rest of this study and describe the reason for not using the other datasets in the "Notes" column in Table 1.…”

“…Each session contains only a single swipe CEP [26] 2022 470 (64) 1-30 9 features identified in a single paper is 59 [56], and the smallest is 5 [31].…”

“…However, despite almost a decade of research into this field and positive sentiment from the community for such technology [43], touch-based authentication methods still lack widespread adoption and integration into our daily devices. This issue can be attributed to the way studies are evaluated and the resulting overestimation of performance [26]. However, contributions in this area are also severely fragmented, primarily due to a lack of a common framework to compare and evaluate models against a well-defined benchmark to accurately assess what can be considered state-of-the-art.…”