In complex lighting environments, vehicle shadows in video surveillance images severely affect the detection and tracking of vehicles and the recognition of abnormal vehicle behavior. Edge-based shadow removal algorithms remove most parts of the shadow, but they leave fine shadow edges unremoved. Meanwhile, hue-saturation-value (HSV) color space-based shadow removal algorithms may erroneously recognize vehicles as shadows and remove them if a vehicle color is similar to that of the shadow. To address this problem, a shadow removal method that fuses edge-based and HSV color space-based algorithms was proposed. Most of the shadows were first removed by an edge-based shadow removal algorithm, in which subtractions and differential operations were performed between the current frame and background image based on a background model. Then, an exclusive OR (XOR) operation was conducted. The image from the HSV approach was then fused with the image produced by the edge-based method. Finally, mathematical morphology was used to process the combined images, thereby improving the efficacy of shadow removal. Experimental results prove that the proposed shadow removal algorithm can effectively remedy the defects of simple shadow removal algorithms. In normal illumination conditions, the detection rate and resolution of this method can exceed 90%. This study provides an excellent method for optimizing the removal of moving object shadows through the use of multiple feature fusion in video surveillance images, especially in complex lighting environments.