H.264 adopts Variable Block Size Motion Estimation and Mode Decision with Rate-Distortion Optimization to improve video quality and decrease bit rate. However, these techniques increase the computation complexity and the encoding time enormously. An efficient macroblock level fast mode decision algorithm is proposed in this paper. Macroblocks which are likely to choose a large block size during mode decision are detected early and those smaller size modes are rejected for these macroblocks. Experimental results show that 40.52% encoding time is saved with only 0.01dB PSNR drop on average.