Rare-event Simulation for Neural Network and Random Forest Predictors

Abstract

We study rare-event simulation for a class of problems where the target hitting sets of interest are defined via modern machine learning tools such as neural networks and random forests. This problem is motivated from fast emerging studies on the safety evaluation of intelligent systems, robustness quantification of learning models, and other potential applications to large-scale simulation in which machine learning tools can be used to approximate complex rare-event set boundaries. We investigate an importance sampling scheme that integrates the dominating point machinery in large deviations and sequential mixed integer programming to locate the underlying dominating points. Our approach works for a range of neural network architectures including fully connected layers, rectified linear units, normalization, pooling and convolutional layers, and random forests built from standard decision trees. We provide efficiency guarantees and numerical demonstration of our approach using a classification model in the UCI Machine Learning Repository.

Achieving Efficiency in Black Box Simulation of Distribution Tails with Self-structuring Importance Samplers

“…The self-similarity property serves to guide the automated approach towards selection of IS distribution. While a variety of methods exist for searching optimal parameters within a chosen IS distribution family (see, for eg., [81,1,68,3,10]), to the best of our knowledge, this is the first paper to exhibit an automated approach for tackling the complementary and more challenging problem of selection of IS distribution families with optimal variance reduction properties. 4) (Applications) We demonstrate the utility of the IS scheme and the large deviations characterizations in the evaluation of the probability of i) large losses in a portfolio credit risk setting modeled with a deep neural network and ii) failures in distribution networks (Sections 4.…”

“…In a similar vein, considerations of certifying safety, fairness and robustness in settings deploying automation have led to a number of applications seeking to measure tail risks in avenues extending beyond operations and quantitative risk management as well. Assessing the safety of automation in driving and other intelligent physical systems get naturally cast in terms of evaluating expectations restricted to distribution tails [92,75,88,57,3], as is the case with evaluating severity of algorithmic biases on minority subpopulations [91,35,60]; see also [28,89,90,10,69] and references therein.…”

“…Such explicit reliance on large deviations characterizations to carefully select the IS distribution, while a source of strength, also helps showcase the challenges one may face in estimation tasks pertaining to events in which the value of an optimization problem or that of a performance functional specified in terms of a neural network is unusually high. As we explain in Section 2.2, large deviations characterizations for such events are known only in limited instances and only specific examples, such as in [20,3,10], enjoy efficient samplers as a consequence. Straightforward application of conventional approaches based on exponential twisting are not known to be effective even in elementary instances such as project evaluation (PERT) networks with independent activity durations, and require intricate tailoring based on the objective L(•) and the distribution of X (see [61]).…”

“…Upon identifying an IS distribution family, selection of optimal candidate within the family often requires solving a nontrivial optimization problem whose objective is informed by the distribution of the random vector X and the constraints are informed by the functional L(•). The resulting formulation may be convex, eg., [46,50], or may additionally possess a combinatorial structure [49], or is solved by a mixed integer program [10], in special instances. Such nuanced selection is impractical if the functional L(•) is specified, for example, in terms of a mixed integer linear program with several constraints (or) a deep neural network.…”

“…The proposed IS scheme also serves as an entirely novel addition to the IS methods introduced in [20,10,3], which comprise the first instances of verifiably efficient variance reduction schemes for estimating distribution tails motivated from optimization and learning contexts. As we explain in Section 2.2, the technique used in [20] is specific to a linear program arising in modeling product distribution networks.…”