In this paper, two kinds of optimization techniques were proposed to enhance the performance of H.264 decoder that based on a DSP processor. One approach is to transfer reference data via DMA with a novel approach, which can utilize external bus efficiently. The other approach is to perform the block-based de-blocking filter, which minimizes the times of external memory access. The experimental results show that the proposed solution can improve the H.264 decoding speed by almost 58%. The optimized H.264 decoder can achieve the CIF @25Hz video decoding on a Blackfin 533 processor when operating at 526MHz clock.