We propose a real-time infrared pedestrian detection method, which achieves higher performance than the state-of-theart YOLO series in infrared pedestrian detection. Our method is based on the YOLOv6-n with some new technologies. In particular, Mixed Downsampling Module (MDM) and Cross-stage Connection Structure (CSCS) are designed to reduce the inference latency of the network. In addition, Simplified Spatial Attention Module (SimSAM) is proposed to improve detection performance with negligible overheads. We validate our method through extensive experiments on FLIR detection datasets. Experimental results show that our method can achieve 84.86% AP with inference latency of 0.54 ms on an NVIDIA 1080Ti GPU.