Waterway traffic monitoring is an important content in waterway traffic management. Taking into account that the number of monitored water areas is growing and that waterway traffic management capabilities are insufficient in the current situation in China, this paper investigates the location optimization of the vessel traffic service (VTS) radar station. During the research process, radar attenuation and environmental occlusion, as well as variable coverage radius and multiple covering are all considered. In terms of the radar attenuation phenomenon in the propagation process and obstacles such as mountains and islands in the real world, judgment and evaluation methods in a three-dimensional space are proposed. Moreover, a bi-objective mathematical model is then developed, as well as a modified adaptive strategy particle swarm optimization algorithm. Finally, a numerical example and a case are given to verify the effectiveness of the proposed methods, model, and algorithm. The results show the methods, model, and algorithm proposed in this paper can solve the model efficiently and provide a method to optimize the VTS radar station location in practice.