To eliminate the reference mirror (REF) error of the Fizeau interferometer for measuring X-ray mirrors, the reference calibration method of lateral multi-shift measurements at the hundreds-micrometer pixel level is presented. Because of the high aspect ratio of long X-ray mirrors, by shifting the surface under test (SUT) along the tangential direction with integer multiple pixels, we extend the calibration method by using the difference between multiple shifted measurements to build an augmented multi-matrix for extracting the two-dimensional (2D) absolute surface. The method can be applied to arbitrary measurement regions of the test optics, and the measurement for both the benchmark sub-aperture and calibration of the REF is accomplished in a single measuring process. Furthermore, by adjusting the shift to the millimeter scale, reference-subtracted sub-apertures can be stitched to obtain the absolute 2D map of the X-ray mirror. Experimental results show that all the 2D discrepancies reach sub-nanometer repeatability, and comparison results between the long trace profiler (LTP) and the proposed method have been performed. Therefore, the results demonstrated that the proposed method meets the requirements of X-ray mirror measurements.