Railroad engineering geological survey is a general and important work in the railroad design process. The effective combination of remote sensing and GIS technology has broad application prospects in railroad survey, planning and construction. In this paper, we propose an image classification algorithm in railroad engineering geological survey, and determine the best segmentation scale through several experiments. Based on the spectral and geometric information features of different lithologies, a classification rule set with the KNN and SVM methods are used for lithology classification, and the accuracy evaluation shows that the classification results are more reliable. This work will greatly improve the segmentation efficiency and make the multi-scale segmentation technology of remote sensing images truly automated.