Non-contact R-test is an instrument to measure the synchronous errors of five-axis machine tools. However, there are still some deficiencies in its researches, such as the difficult and laborious calibrations. How to systematically improve the measurement accuracy with a good balance to minimum cost is a real problem in guiding practice. This paper proposes a new systematic optimization method to solve this problem based on comprehensive understanding of the non-contact R-test in terms of structure parameters and relations. Firstly, the algorithm for sphere center coordinates is established based on the self-adaptive differential evolution algorithm to obtain the definite computational accuracy and efficiency. Secondly, the parameters of the fixture structure are optimized to maximize the measurement stability, measuring space and noninterference space. Thirdly, the on-machine calibration is performed to replace pre-calibration and re-calibration, and to establish the positional relationships between sensors, the fixture and the machine tool simultaneously. It can reduce the difficulties of manufacture, maintenance and application. Fourthly, the measurement accuracy can be evaluated to determine whether the iterative optimization achieves the goal. The proposed method has been verified with case studies to support optimized non-contact R-test setting up, leading to cost-effective and accuracy test on five-axis machine tools.