The rigorous coupled-wave method (RCWM) is extended to calculate the diffraction efficiency of multilayer grating in spectrum with strong absorption. This paper gives a detailed algorithm analysis of the RCWM, an enhanced, numerically stable transmittance matrix approach applied to homogenous layered media is generalized for multilayer grating structure. In order to calculate in the lossy medium, the wave vector of electric field is redefined. This method is programmed to design multilayer grating which is expected to use on Seya-Namioka synchrotron monochromator in vacuum ultraviolet (VUV) spectrum. The influence on diffraction efficiency caused by grating profile and multilayer stack is analyzed theoretically. The diffraction efficiencies of well-designed multilayer gratings are all greater than 10% in 50-110nm region, and they are about two to three times higher than that of the single metallic layer grating which is being used on Seya-Namioka monochromator now.