Rare-Event Simulation for Neural Network and Random Forest Predictors

Abstract

We study rare-event simulation for a class of problems where the target hitting sets of interest are defined via modern machine learning tools such as neural networks and random forests. This problem is motivated from fast emerging studies on the safety evaluation of intelligent systems, robustness quantification of learning models, and other potential applications to large-scale simulation in which machine learning tools can be used to approximate complex rare-event set boundaries. We investigate an importance sampling scheme that integrates the dominating point machinery in large deviations and sequential mixed integer programming to locate the underlying dominating points. Our approach works for a range of neural network architectures including fully connected layers, rectified linear units, normalization, pooling and convolutional layers, and random forests built from standard decision trees. We provide efficiency guarantees and numerical demonstration of our approach using a classification model in the UCI Machine Learning Repository.

Over-Conservativeness of Variance-Based Efficiency Criteria and Probabilistic Efficiency in Rare-Event Simulation

“…In order to design an efficient estimator for this problem, we apply the dominating point scheme introduced in Huang et al ( 2018) and Bai et al (2020) to construct an importance sampler. The scheme sequentially searches for dominating point with highest density using optimization with a cutting-plane algorithm.…”

“…Due to the high-dimensionality of the input space and the complexity of the neural network predictor, the number of dominating points in this problem is huge. We implement the sequential searching algorithm in Bai et al (2020) and it took a week to find the first 100 dominating points.…”

“…Moreover, we suppose that g is a piecewise linear function. Related theoretical guarantees for piecewise linear function of Gaussian input can be found in Bai et al (2020).…”

Efficient Black-Box Importance Sampling for VaR and CVaR Estimation

“…This paper aims to tackle the challenges in marrying efficiency with black-box IS for VaR/CVaR estimation. Restricting to multivariate normal distributions, Bai et al 2020;Arief et al 2021 utilise the machinery of dominating points to algorithmically arrive at efficient IS mixture distributions for estimation of distribution tails of losses that can be either directly written or approximated with a piece-wise linear structure. Assuming only a black box access to the evaluations of loss L(•) and the distribution of the underlying random vector X X X, we present here an efficient IS algorithm (Algorithm 1) to jointly estimate VaR/CVaR of L(X X X).…”