This paper concerns the imaging problem for downward looking sparse linear array three-dimensional synthetic aperture radar (DLSLA 3-D SAR) under the circumstance of sparse and non-uniform cross-track dimensional virtual phase centers configuration. Since the 3-D imaging scene behaves typical sparsity in a certain domain, sparse recovery approaches hold the potential to achieve a better reconstruction performance. However, most of the existing compressive sensing (CS) algorithms assume the scatterers located on the pre-discretized grids, which is often violated by the off-grid effect. By contrast, atomic norm minimization (ANM) deals with sparse recovery problem directly on continuous space instead of discrete grids. This paper firstly analyzes the off-grid effect in DLSLA 3-D SAR sparse image reconstruction, and then introduces an imaging method applied to off-gird targets reconstruction which combines 3-D pseudo-polar formatting algorithm (pseudo-PFA) with ANM. With the proposed method, wave propagation and along-track image reconstruction are operated with pseudo-PFA, then the cross-track reconstruction is implemented with semidefinite programming (SDP) based on the ANM model. The proposed method holds the advantage of avoiding the off-grid effect and managing to locate the off-grid targets to accurate locations in different imaging scenes. The performance of the proposed method is verified and evaluated by the 3-D image reconstruction of different scenes, i.e., point targets and distributed scene.