In this paper, we propose the improved vision-based vehicle detection system which is added a hypothesis fitting (HF) step to the typical vehicle detection system consisting of the hypothesis generation (HG) and hypothesis verification (HV) step. In the HG step, the system generates hypotheses using shadow regions appearing under vehicles. In the HV step, the system verifies whether a hypothesis is a vehicle or not by applying a classifier and feature vectors extracted from the hypothesis. The proposed HF step is conducted between HG and HV steps and help to improve performance of the vehicle verification by adapting region of hypotheses. We verify the performance of our proposed HF method through a total of 4,797 hypotheses, 1,606 positive hypotheses, and 3,191 negative hypotheses data set.