Evaluating the stray light cancellation performance of an optical system is an essential step in the search for superior optical systems. However, the existing evaluation methods, such as the Monte Carlo method and the ray tracing method, suffer from the problems of vast arithmetic and cumbersome processes. In this paper, a method for a rapid stray light performance evaluation model and quantitatively determining high-magnitude stray light outside the field of view are proposed by adopting the radiative transfer theory based on the scattering property of the bidirectional scattering distribution function (BSDF). Under the global coordinates, based on the derivation of the light vector variation relationship in the near-linear system, the specific structural properties of the off-axis reflective optical system, and the specular scattering properties, a fast quantitative evaluation model of the optical system’s stray light elimination capability is constructed. A loop nesting procedure was designed based on this model, and its validity was verified by an off-axis reflective optical system. It successfully fitted the point source transmittance (PST) curve in the range of specular radiation reception angles and quantitatively predicted the prominence due to incident stray light outside the field of view. This method does not require multiple software to work in concert and requires only 10–5 orders of magnitude of computing time, which is suitable for the rapid stray light assessment and structural screening of off-axis reflective optical systems with a good symmetry. The method is promising for improving imaging radiation accuracy and developing lightweight space cameras with low stray light effects.