Multi-Task Learning and Weighted Cross-Entropy for DNN-Based Keyword Spotting

Abstract

We propose improved Deep Neural Network (DNN) training loss functions for more accurate single keyword spotting on resource-constrained embedded devices. The loss function modifications consist of a combination of multi-task training and weighted cross entropy. In the multi-task architecture, the keyword DNN acoustic model is trained with two tasks in parallel-the main task of predicting the keyword-specific phone states, and an auxiliary task of predicting LVCSR senones. We show that multi-task learning leads to comparable accuracy over a previously proposed transfer learning approach where the keyword DNN training is initialized by an LVCSR DNN of the same input and hidden layer sizes. The combination of LVCSRinitialization and Multi-task training gives improved keyword detection accuracy compared to either technique alone. We also propose modifying the loss function to give a higher weight on input frames corresponding to keyword phone targets, with a motivation to balance the keyword and background training data. We show that weighted cross-entropy results in additional accuracy improvements. Finally, we show that the combination of 3 techniques-LVCSR-initialization, multi-task training and weighted cross-entropy gives the best results, with significantly lower False Alarm Rate than the LVCSR-initialization technique alone, across a wide range of Miss Rates.

Model Compression Applied to Small-Footprint Keyword Spotting

“…The DNN had a binary output indicating the presence or absence of the keyword (e.g., "Alexa") at the middle frame (i.e., most targets are background except for the ≈70 frame targets from each keyword). We found that adding a simultaneous additional ASR label task improved performance (see [20] for details). So, we used this in all experiments except those with knowledge distillation, where it was unnecessary as long as the teacher model was trained with the multi-task objective.…”

“…Notably, the DNN is trained to optimize a framewise cross-entropy loss, whereas the detection task is truly a sequence level task, however, the two are highly correlated. We trained the DNN using distributed asynchronous SGD [19,20]. We used a performance based learning rate schedule (similar to "newbob"), where the learning rate is halved every time performance degrades on the development set.…”

Artificial-intelligence-driven scanning probe microscopy

“…We trained the classifier CNN using the ADAM 39 optimizer with a cross-entropy loss and L2 weight decay with a value of 5 × 10 −5 and a learning rate of 10 −3 . To account for the imbalance of our training set ("good probe" 25% and "bad probe" 75%), we weighed STM images labeled as "good probe" by a factor of 8 when computing the loss 40 . In addition, we increased the available amount of training data via data augmentation, randomly flipping the input SPM images horizontally or vertically.…”

Learning to Detect Keyword Parts and Whole by Smoothed Max Pooling

“…Following the successes in general ASR [2,3], the neural network based approach has been extensively explored in keyword spotting area with benefits of lowering resource requirements and improving accuracy [4,5,6,7,8,9,10,11]. Such works include DNN + temporal integration [4,5,11,12], and HMM + DNN hybrid approaches [6,7,8,9,10]. Recently introduced end-to-end trainable DNN approaches [1,13] further improved accuracy and lowered resource requirements using highly optimizable system design.…”