Asynchronous pipeline structure is adopted for the real-time video decoder design because of its better performance when the stage processing times are irregular. However, the structure requires a lot of memories, the invaluable resource on chip, to buffer data and parameters between modules. To solve this problem, a specially designed switching buffer module is used between stages instead of traditional FIFO, and the module can also take some buffering function in each stage, which helps to reduce the utilization of memory. An H.264 decoder with the proposed structure was implemented. Compared to decoder without improved structure, the experimental decoder can save nearly 50% memory and 31% I/O operations between stages.