Self-driving car plays a crucial role in implementing traffic intelligence. Road smoothness in front of self-driving cars has a significant impact on the car's driving safety and comfort. Having potholes on the road may lead to several problems, including car damage and the occurrence of collisions. Therefore, self-driving cars should be able to change their driving behavior based on the real-time detection of road potholes. Various methods are followed to address this problem, including reporting to authorities, employing vibration-based sensors, and 3D laser imaging. However, limitations, such as expensive setup costs and the danger of discovery, affected these methods. Therefore, it is necessary to automate the process of potholes identification with sufficient precision and speed. A novel method based on adaptive mutation and dipper throated optimization (AMDTO) for feature selection and optimization of the random forest (RF) classifier is presented in this paper. In addition, we propose a new adaptive method for dataset balancing, referred to as optimized hashing SMOTE, to boost the performance of the optimized model. Data on potholes in different weather conditions and circumstances were collected and augmented before training the proposed model. The effectiveness of the proposed method is shown in experiments in classifying road potholes accurately. Eleven feature selection methods, including WOA, GWO, and PSO, and three machine learning classifiers were included in the conducted experiments to measure the superiority of the proposed method. The proposed method, AMDTO+RF, achieved a pothole classification accuracy of (99.795%), which outperforms the accuracy achieved by the other approaches, WOA+RF of 97.5%, GWO+RF of 98.6%, PSO+RF of 98.1%, and transfer learning approaches, AlexNet of 86.8%, VGG-19 of 87.3%, GoogLeNet of 90.4%, and ResNet-50 of 93.8%. In addition, an in-depth statistical analysis is performed on the recorded results to study the significance and stability of the proposed method.