Exploring How Phone Classification Neural Networks Learn Phonetic Information by Visualising and Interpreting Bottleneck Features

Abstract

Neural networks have a reputation for being "black boxes", which it has been suggested that techniques from user interface development, and visualisation in particular, could help lift. In this paper, we explore 9-dimensional bottleneck features (BNFs) that have been shown in our earlier work to well represent speech in the context of speech recognition, and 2-dimensional BNFs directly extracted from bottleneck neural networks. The 9-dimensional BNFs obtained from a phone classification neural network are visualised in 2dimensional spaces using linear discriminant analysis (LDA) and t-distributed stochastic neighbour embedding (t-SNE). The 2-dimensional BNF space is analysed with regard to phonetic features. A back-propagation method is used to create "cardinal" features for each phone under a particular neural network. Both the visualisations of 9-dimensional and 2-dimensional BNFs show distinctions between most phone categories. Particularly, the 2-dimensional BNF space seems to be a union of phonetic category related subspaces that preserve local structures within each subspace where the organisations of phones appear to correspond to phone production mechanisms. By applying LDA to the features of higher dimensional non-bottleneck layers, we observe a triangular pattern which may indicate that silence, friction and voicing are the three main properties learned by the neural networks.

Towards Understanding Attention-Based Speech Recognition Models

“…There have been many explorations toward interpreting neural network outputs in the speech processing area. Bai et al [20] proposed to use linear discriminant analysis (LDA) and t-distributed stochastic neighbor embedding (t-SNE) to analyze 9-dimensional bottleneck features (BNFs). Karita et al [21] use t-SNE to visually show how features are mixed or split with inter-domain loss.…”

Prosodic Representations of Prominence Classification Neural Networks and Autoencoders Using Bottleneck Features

“…Previous work on exploring how neural networks represent different aspects of speech has focused primarily on investigating the learning of phonetic representations [19,20,21]. This typically involves two general approaches: (i) an unsupervised one, where a neural network autoencoder is used in order to investigate whether the network is capable of learning phonemelike representations without explicit labels [22], (ii) a supervised one, where the neural network is trained with phone labels for the task of phoneme recognition [20,19]. In both cases, investigations are focused on analyzing the representational properties of the complex features learned at different layers of the network and also at different nodes.…”