This paper presents a longitudinal speed control algorithm using a model-reference adaptive sliding mode control (ASMC) scheme for an autonomous vehicle in various driving environments using only wheel speed sensors. The proposed algorithm could control the vehicle’s speed not using parameter estimators but using an adaptation technique. The parameter adaptation laws were designed to compensate for the changes in the environmental disturbances and model uncertainties. Moreover, the upper bound of unknown disturbances, that were not compensated by the adaptation algorithm, was estimated using radial basis function neural network (RBFNN). The sliding mode controller updated the upper bound from the RBFNN and obtained robustness without knowing the bound in advance. Adaptive equivalent control input was also defined to compensate for zero-throttle acceleration varying with speed. This input could enhance the mode switch smoothly between throttle and brake control. We conducted computer simulations and vehicle tests under various driving environments to evaluate the performance of the proposed algorithm. In the simulation result, the average tracking error of the proposed algorithm was 0.718 kph, and the maximum change rate of the error due to the disturbances was 11%. The improvements were 55% and 68%, respectively, compared to the PID control. The average error in the vehicle test result was 0.414 kph, which was improved by 48% compared to the PID control in the test track. The results demonstrate that the proposed algorithm ensures desirable tracking performance under environmental disturbances and model uncertainties.