The mechanism of flat band voltage (VFB) shift for alternate La2O3/Al2O3 multilayer stack structures in different annealing condition is investigated. The samples were prepared for alternate multilayer structures, which were annealed in different conditions. The capacitance-voltage (C-V) measuring results indicate that the VFB of samples shift negatively for thinner bottom Al2O3 layer, increasing annealing temperature or longer annealing duration. Simultaneously, the diffusion of high-k material to interfaces in different multilayer structures and annealing conditions is observed by X-ray photoelectron spectroscopy (XPS). Based on the dipole theory, a correlation between the diffusion effect of La towards bottom Al2O3/Si interface and VFB shift is found. Without changing the dielectric constant k of films, VFB shift can be manipulated by controlling the single-layer cycles and annealing conditions of alternate high-k multilayer stack.