A novel roll angular displacement measurement method based on acousto-optic modulators (AOMs) is presented, where the AOMs are adopted to reduce the frequency-difference to overcome the phase fluctuation. The novel method improves the resolution and simplifies the system configuration by reducing the optical components in the measuring light path. A mathematical model is established, and the impacts of the optical components’ error on measurement errors and magnification are analyzed in detail based on the model. The analysis results can then be used to optimize the system configuration. Experiments are carried out to validate the presented method, which show that a resolution of 0.016 arcsec has eventually been achieved.