Achieving Efficiency in Black Box Simulation of Distribution Tails with Self-structuring Importance Samplers

Abstract

Motivated by the increasing adoption of models which facilitate greater automation in risk management and decision-making, this paper presents a novel Importance Sampling (IS) scheme for measuring distribution tails of objectives modeled with enabling tools such as feature-based decision rules, mixed integer linear programs, deep neural networks, etc. Conventional efficient IS approaches suffer from feasibility and scalability concerns due to the need to intricately tailor the sampler to the underlying probability distribution and the objective. This challenge is overcome in the proposed black-box scheme by automating the selection of an effective IS distribution with a transformation that implicitly learns and replicates the concentration properties observed in less rare samples. This novel approach is guided by a large deviations principle that brings out the phenomenon of self-similarity of optimal IS distributions. The proposed sampler is the first to attain asymptotically optimal variance reduction across a spectrum of multivariate distributions despite being oblivious to the underlying structure. The large deviations principle additionally results in new distribution tail asymptotics capable of yielding operational insights. The applicability is illustrated by considering product distribution networks and portfolio credit risk models informed by neural networks as examples.

Certifiable Deep Importance Sampling for Rare-Event Simulation of Black-Box Systems

“…Second, a recent work Deo and Murthy (2021) proposes an elegant IS method based on the self-similarity property of the optimal IS distribution to achieve logarithmic efficiency. They require the imposition of a risk function that is so-called asymptotically homogeneous and has a known order of growth.…”

“…They require the imposition of a risk function that is so-called asymptotically homogeneous and has a known order of growth. Compared to Deo and Murthy (2021), we do not introduce risk functions or impose any accompanying analytical assumptions, but we require light-tailedness and the geometric premise that the rare-event set is orthogonally monotone. Our paper is connected to Deo and Murthy (2021) in that under their asymptotic homogeneity condition, we have, in our notation, x ∈ S γ implies tx ∈ S γ for any t > 1 in R when x is large enough in a suitable sense.…”