OPEMI: Online Performance Evaluation Metrics Index for Deep Learning-Based Autonomous Vehicles

Abstract

Vision-based autonomous driving is rapidly growing. There are, however, presently no agreed-upon metrics for assessing how well deep neural network (DNN) models perform in driving. To compare novel approaches and architectures to existing ones, some researchers employed a mean error between labeled and predicted values in a test dataset and others presented a new metric that is designed to match their requirements. The discrepancy in the usage of various performance metrics and lack of objective metrics to judge the driving performance were our primary motives for developing a feasible solution. In this study, we propose online performance evaluation metrics index (OPEMI), an integrated metric that can evaluate the driving capabilities of autonomous driving models in various driving scenarios. To evaluate driving performance precisely and objectively, OPEMI incorporates several variables, including driving control stability, driving trajectory stability, journey duration, travel distance, success rate, and speed. To demonstrate the validity of OPEMI, we first confirmed that the prediction accuracy has a weak correlation with driving performance. Then, we have discussed the constraints in the existing driving performance metrics in certain circumstances, and their failure to assess the driving models. Finally, we conducted experiments with four popular DNN models and two in-house models under three different driving scenarios (generic, urban, and racing). The results show that the proposed evaluation metric, OPEMI, realistically displays driving performance and demonstrates its validity in various driving scenarios.

Comparative Study on Simulated Outdoor Navigation for Agricultural Robots

“…The metrics to define the performance of selected techniques are driving deviation, completion time, and human intervention. We derive these performance metrics from the Online Performance Evaluation Metrics Index (OPEMI) [ 7 ].…”

“…( d ) the trained neural network is deployed to the DNN-based controller who drives the vehicle by using inferred steering angles, throttle, and brakes. Adapted from [ 7 ].…”

“…The comparative assessment of vision-based, laser-based, and DNN-based BC algorithms is grounded in their navigation performance in a simulated orchard farm environment. Performance metrics derived from [ 7 ], encompassing driving deviation, completion time, success rate, and human intervention, constitute the basis for this comprehensive evaluation. The results show that DNN-based BC algorithms have promising capabilities for navigation in the agricultural environment.…”