Efficient and fully automatic building outline extraction and simplification methods are highly demanded for 3D model reconstruction tasks. In spite of the efforts put into developing such methods, the results of the recently proposed methods are still not satisfactory, especially for satellite images, due to object complexities and the presence of noise. Dealing with this problem, in this article, we propose a new approach which detects rough building boundaries (building mask) from Digital Surface Model (DSM) data, and then refines the resulting mask by classifying the geometrical features of the high spatial resolution panchromatic satellite image. The refined mask represents finer details of the building outlines which are close to the original building edges. These outlines are then simplified through a parameterization phase, where a tracing algorithm detects the building boundary points from the refined masks and a set of line segments is fitted to them. After that, for each building, the existing main orientations are determined based on the length and arc lengths of the building's line segments. Our method is able to determine the multiple main orientations of complex buildings. Through a regularization process, the line segments are then aligned and adjusted according to the building's main orientations. Finally, the adjusted line segments are intersected and connected to each other in order to form a polygon representing the building's outlines. Experimental results demonstrate that the computed building outlines are highly accurate and simple, even for large and complex buildings with inner yards.