Interpretable and Steerable Sequence Learning via Prototypes

Abstract

One of the major challenges in machine learning nowadays is to provide predictions with not only high accuracy but also user-friendly explanations. Although in recent years we have witnessed increasingly popular use of deep neural networks for sequence modeling, it is still challenging to explain the rationales behind the model outputs, which is essential for building trust and supporting the domain experts to validate, critique and refine the model.We propose ProSeNet, an interpretable and steerable deep sequence model with natural explanations derived from case-based reasoning. The prediction is obtained by comparing the inputs to a few prototypes, which are exemplar cases in the problem domain. For better interpretability, we define several criteria for constructing the prototypes, including simplicity, diversity, and sparsity and propose the learning objective and the optimization procedure. ProSeNet also provides a user-friendly approach to model steering: domain experts without any knowledge on the underlying model or parameters can easily incorporate their intuition and experience by manually refining the prototypes.We conduct experiments on a wide range of real-world applications, including predictive diagnostics for automobiles, ECG, and protein sequence classification and sentiment analysis on texts. The result shows that ProSeNet can achieve accuracy on par with state-of-the-art deep learning models. We also evaluate the interpretability of the results with concrete case studies. Finally, through user study on Amazon Mechanical Turk (MTurk), we demonstrate that the model selects high-quality prototypes which align well with human knowledge and can be interactively refined for better interpretability without loss of performance.

A survey of visual analytics techniques for machine learning

“…ProtoSteer [116], a visual analytics system that enables editing prototypes to refine a prototype sequence network named ProSeNet [282]. ProtoSteer uses four coordinated views to present the information about the learned prototypes in ProSeNet.…”

PIPxResNet: Penalty Induced Prototype-Based eXplainable Residual Neural Network for Heartbeat Classification

“…The deep learning models have fared better, but the black-box aspect limits real-world implementation. The PIPxRes-Net and [30] provide prototype based explanations with better performance. Fixed number of individual class prototypes hinders in capturing diversity for majority class beats and creates redundant prototypes for minority class beats with increased inference time [30].…”

“…The PIPxRes-Net and [30] provide prototype based explanations with better performance. Fixed number of individual class prototypes hinders in capturing diversity for majority class beats and creates redundant prototypes for minority class beats with increased inference time [30]. The PIPxResNet prototype generation process is datadriven, requires no parameter adjustment, making it better suited for heartbeat classification.…”

“…These methods are a classical form of case-based reasoning that rely on similarity of a data point to a given prototype. Several such methods have been developed in the past including prototype with autoencoders [21], deep sequence model consisting of sequence encoder and prototype layer for classification and reasoning [30], and explainable deep neural network (xDNN) [3]. This work adopts xDNN [3] for classification and explanation.…”

Visual Reasoning of Feature Attribution with Deep Recurrent Neural Networks

“…As discussed in the introduction, the interpretability is critical to these application domains regarding model fairness, reliability, and trust. However, not much attention has addressed to the visual interpretation of RNN from a more generic perspective except for ProSeNet [26], which proposes an interpretable and steerable deep sequence model. This paper attempts to bridge this gap and propose a visual analysis method that can be applied to broader scenarios of analyses for summarizing contributing sequence patterns in predictions.…”