Methodology for Testing and Evaluation of Safety Analytics Approaches

Abstract

There has been a significant increase in the development of data-driven safety analytics (SA) approaches in recent years. In light of these advances it has become imperative to evaluate such approaches in a principled way to determine their merits and limitations. To that end, we propose an evaluation methodology underpinned by a simulated environment that allows for a comprehensive assessment of SA approaches. While assessing such approaches with historical field data is undoubtedly important, such an assessment has limited statistical power because it corresponds to only a few realizations. The proposed methodology enables validation over a large number of realizations, thereby circumventing the statistical limitations of evaluation on historical data. Moreover, a simulated environment allows for a comparison under controlled circumstances, resulting in a fair and systematic assessment of the potential long-term benefits of SA approaches. We demonstrate the utility of the proposed methodology via a case study that compares a few candidate SA approaches, which differ in how they assimilate field data to assess safety risk and suggest mitigative actions. We show that the proposed methodology indeed reveals useful insights and quantifies the relative merits and drawbacks of the different SA approaches, which would be otherwise difficult to objectively determine in a real-world scenario.