Scalable video coding (SVC) can provide different bitrate, resolution, or quality formats of the same video through the one-time encoding process. However, SVC also increases encoding complexity significantly, making real-time implementations difficult especially when applied to high definition videos. In this work, we proposed to utilize a fast mode decision algorithm to speed up the SVC encoder. A simple model of the relationship of the video complexity is proposed, the rate-distortion (RD) cost and the statistical mode distribution of the enhancement layer (EL). The proposed mode decision algorithm evaluates the complexity of the video content, then sets the rate distortion RD cost thresholds for inter coding modes. The method also optimizes the process INTRA_BL for I slices. We demonstrate the method using four different resolutions (1080p, 720p, VGA and QVGA), four temporal layers, and two quality layers. We find that the proposed algorithm can reduce the computational complexity significantly while maintaining or even improving encoder efficiency.