The measurement of critical dimensions in the field of integrated circuits has moved from 7 nm to 5 nm. The existing chromium atomic lithography grating has a pitch period of 4700 l mm−1 and uniformity of picometer, and the interferometric signal period based on the above grating is as small as 106.4 nm, which brings new problems and challenges to the accurate processing of the signal. This paper investigates the error characteristics of ultra-high precision grating interferometric signals, establishes a Heydemann correction mathematical model for high inscribed line density grating interferometric signals, corrects the grating interferometer signals based on the random sample consensus (RANSAC), and verifies the effectiveness of the algorithm through simulation. By comparing the repeatability and linearity of the original algorithm and the self-traceable grating interferometric displacement measurement data processed by RANSAC, the conclusion that the standard deviation of the self-traceable grating interferometer repeat measurement after RANSAC is 1.60 nm in a 10 000 nm travel is obtained, and the purpose of improving the stability and uniformity of the signal solution with the algorithm of this paper is achieved, which is important for the study of laser interferometer and grating interferometer The results show that the stability and uniformity of the signal solution can be improved by the algorithm of this paper, which is of great significance for the study of the displacement solution of laser and grating interferometers.