Methodology for evaluating and regression testing a conflict probe

Abstract

A conflict probe is an air traffic management decision support tool that predicts aircraft-toaircraft and aircraft-to-airspace conflicts. In order to achieve the confidence of the air traffic controllers who are provided this tool, a conflict probe must accurately predict these events. To ensure their continued confidence, the accuracy should not only be assessed in the laboratory before the probe is deployed but continue to be reassessed as the system undergoes upgrades and software changes. Furthermore, it is desirable to use recorded air traffic data to test these tools in order to preserve real-world errors that affect their performance. This paper utilizes a proven approach that modifies surveillance radar track data in time to create traffic scenarios containing conflicts with characteristic properties similar to those encountered in actual air traffic operations. It is these time shifted traffic scenarios that are used to evaluate the conflict probe.evaluating the missed and false conflict predictions, the calculation of the corresponding error probabilities, and a regression testing methodology to examine two runs of the conflict probe to determine if the conflict prediction accuracy has improved or degraded over time. A detailed flight example is presented which illustrates the specific processing involved in conflict accuracy analysis. Next using a scenario of many flights, a methodology utilizing categorical data analysis techniques is applied to determine if a new version of the conflict probe's software significantly improved or degraded in conflict prediction accuracy. This paper describes the detailed process of

Aircraft conflict probe sensitivity to weather forecasts errors

“…If the hypothesis is true, this ratio will follow a chi-squared distribution or χ 2 with one degree of freedom. This test is described in detail in [1] and later in [17].…”

Lateral Intent Error’s Impact on Aircraft Prediction

“…Metrics have been defined that quantify the errors associated with these conflict predictions [18][19][20]. A missed alert error is a conflict between a pair of aircraft not detected at all or not notified within a minimum warning time prior to the conflict's start time, typically five minutes for strategic conflict predictions.…”

“…The overall performance reported is slightly larger than normal because aircraft deviating significantly from the route are normally excluded from the error event counts for strategic or intent based CPs [19][20]. However, this particular study's objective is to quantify the impact of lateral intent errors so excluding them would not allow their measurement.…”

Performance Metrics for Tactical Aircraft to Aircraft Conflicts

“…The metrics have been applied to the current HCS and will be applied in the future to the new ERAM system when it becomes available. The metrics are similar to those previously used in the FAA acceptance and Lockheed Martin regression testing of the User Request Evaluation (URET) system 2,3 .…”