Dam deformation monitoring is a critical technical measure to ensure the safe and stable operation of dams. It involves measuring the structural deformation response of engineering dams using monitoring instruments or technological means. By analyzing the regularity and trend of deformation monitoring data, potential safety anomalies can be forecasted and warned against, providing timely and reliable data for the formulation and implementation of risk removal measures. Horizontal displacement, as the most intuitive and effective reflection of the dam's state under the action of internal and external loads and foundation deformation, is an indispensable part of dam safety monitoring. Currently, the plumb line method and the tensioned wire method are mainly used for horizontal displacement monitoring of dams. A plumb line coordinate instrument measures the horizontal deformation in the upstream and downstream directions and the left and right bank directions through two axes, or the radial and tangential horizontal displacements for arch dams. Compared to other principles, optoelectronic plumb line coordinate instruments have better long-term stability and anti-interference ability and are widely used on engineering sites. However, the orthogonality of the two measuring directions of the instrument is often overlooked. This paper starts from the principle of the development of the plumb line coordinate instrument, analyzes the source of instrument orthogonal error, and combines data collection, structural analysis, and experimental verification. By applying methods such as least squares and regression analysis, an effective calibration calculation and error correction method is proposed. This method is then programmed into the developed plumb line coordinate instrument to meet the real-time correction and output of measured values, providing a reliable technical method for the accuracy and continuous real-time remote monitoring of dam horizontal displacement monitoring. It also offers a technical path for the orthogonality testing of plumb line coordinate instruments.