We present a semi-automatic annotation method to build a large dataset for traffic landmark detection, where traffic landmarks include traffic signs, traffic lights as well as road markings. Labor-intensive bounding box tagging is a huge challenge to generate a large dataset for detection algorithms. To mitigate the labor, we adopt a high-definition (HD) map and a positioning system. We propose a process to align the HD map and images semi-automatically. Through the registration, the annotations of the HD map can be directly tagged onto traffic landmarks in the images. To make full use of the HD map for the dataset generation, we annotate the traffic landmarks with reference points, following the way that they are represented in the HD map, instead of the bounding boxes. The proposed semi-automatic method speeds up the annotation by a factor of 3.19, as compared to the manual annotation. Our dataset consists of about 150,000 images and includes about 470,000 annotated traffic landmarks. We train a deep neural network on our dataset to detect the traffic landmarks, and its performance is evaluated using a novel evaluation metric. Moreover, we show that the pretrained traffic landmark detection network is effective in detecting traffic landmarks in other countries using the bounding box by fine-tuning.