This paper mainly focuses on the rain streak removal task from a single image. Based on the observation that the distribution of rain streak in an image is sparse. We propose a two-phase single image deraining method. Firstly, it detects the rain locations with a proposed anisotropic global gradient prior (AGGP) and generates a rain mask for rain streak removal. Then it recovers the information in rain distorted region with AGGP based multi-layer image inpainting model. Furthermore, to solve the multi-variable optimization problems, we develop an alternating half-quadratic algorithm by introducing alternating algorithm and the variable split method. Both experiments on synthesized and realworld images show the proposed method outperforms state-of-the-art methods in terms of rain streak removal and image multi-layer information preservation.