An Evaluation of "Commercial Off-The-Shelf" Speaker Verification Systems

Abstract

An evaluation of commercial off-the-shelf speaker verification systems is reported. The performance of several systems, which were offered for testing, is analyzed against criteria designed to identify strengths and weaknesses that would determine their suitability for the use by government service agencies. Results for three text-dependent systems by Nuance, Persay and Scansoft are presented in this paper.

Using a predefined passphrase to evaluate a speaker verification system

“…Each participant provided multiple samples, all used to increase the potential value of the enrollment process, and not that of the authentication algorithms themselves. [9] The premise of this paper challenges these results, as the methodology presents a highly artificial testing scenario that could easily invalidate the results as described throughout this study.…”

“…A variety of platforms currently offer support of this software including mobile platforms like iOS devices. It was also a subject of the study conducted by Wagner, et al in 2006, which this paper argues is subject to the experimental flaws described in this research. [9]…”

“…[18] Wagner et al conducted a study in 2006 that included Nuance in their research. [9] As previously stated, the methodology used in this study would be subject to the same experimental flaws that the processes included is this research helps eliminate in the evaluation process. This paper argues that the published results, including those related to the nuance offering, cannot be trusted based on the highly artificial testing environment and overly complex enrollment phases.…”

“…At this time, the only reliable measurement of biometric performance related to speaker verification systems would be the third party evaluations conducted by independent researchers or the proprietary/real-world measurements, which may be insufficient at providing performance details that can be compared to other systems in the com-mercial arena. [3,9] These evaluation methods are similar to those included in this paper, but are presented in a clear and highly repeatable manner in the study included in this research.…”

“…[10,11] Studies like Kato and Shimizu's research on balancing phonemes can be paired with existing evaluations techniques and system architectures to streamline the enrollment and verification process as described by Wagner, et al by using text-dependent phrases to create a powerful and reusable testing methodology. [9,12] Additionally, concepts like phoneme classes are explored by Hebert and Heck and could also be implemented in parallel to a text-dependent evaluation methodology. [7] This study focuses primarily on phonetic research, in addition to common algorithms which evaluate the fundamental frequency of a particular word or phrase spoken by a user.…”