Fatigue cracks developed under repetitive loads are one of the major threats to structural integrity of steel bridges. Human inspection is the most commonly applied approach for fatigue crack detection, but is time consuming, labor intensive, and lacks reliability. In this study, we propose a computer vision‐based fatigue crack detection approach using a short video stream taken by a consumer‐grade digital camera. A feature tracking technology is applied to the video for tracking the surface motion of the monitored structure under repetitive load. Then, a crack detection and localization algorithm is established to effectively search differential features at different video frames caused by the crack opening and closing. The effectiveness of the proposed approach is validated through testing two experimental specimens with in‐plane and out‐of‐plane fatigue cracks, respectively. Results indicate that the proposed approach can robustly identify the fatigue crack, even when the crack is under ambient lighting conditions, surrounded by other crack‐like edges, covered by complex surface textures, or invisible to human eyes due to crack closure. Furthermore, our proposed approach enables accurate quantification of the crack opening under fatigue loading with submillimeter accuracy. However, due to the capacity of the camera resolution in this study, accurate detection of crack tip remains challenging.