Patient setup will influence the treatment of the breast cancer in radiation therapy. Improving the accuracy of the tumor target localization is vital for the cancer treatment. In this study, we focus on the breast patient setup and develop an accurate tumor localization method based on the deep learning in radiation therapy. The proposed method used a double residual neural network model to achieve the high precision and efficiency patient tumor localization. In the network training, the model attempt to localize the breast and then detect the landmarks inside the localized region. After the model training, we used an iterative filter scheme for calculating a transformation to the daily CT. Therefore, the gray value distribution can match well with the training image. The final landmark positions were obtained after the iteration. The translation errors in the daily CT were determined using the detected landmarks. We used the digital CT phantom images and the real patient CT images to evaluate the proposed method. Then result of the breast patient setup was shown to be clinically acceptable. The mean and standard deviation setup errors were 0.64 ± 1.40 mm, 0.15 ± 1.28 mm, -0.46 ± 1.17 mm in the anterior-posterior, left-right, and superior-inferior, respectively. In conclusion, we proposed an accurate patient setup method, which shown a very promising alternative for marker-free breast auto-setup.