Depth map texturing is one of the prominent image-based ren dering methods. It requires the condition of epipolar constraint be satistied. However, the cylindrical warping involved in the construction of panoramic view makes it highly challenging for this condition to be strictly met. This paper proposes depth-map based synthesis of virtual environment starting with stereo panoramic views of a scene. A stereo-panoramic view is first generated by mosaicking a set of images containing both the optical and geometric properties of different parts of the scene. A depth map of the scene contained in the stereo-panoramic view is constructed based on the normalization of matching points and the singular value decomposition of fundamental matrix. Finally, the depth map is textured with one of the pano ramic views to obtain a synthesized view. While a good quality depth map can be only generated if and only if epipolar con straint condition for planar rectification is satisfied, it is also the determining factor for the computation of virtual environment from a stereo-panoramic view.Index Terms -Stereo vision, stereo image processing, im age fusion, image sequences, image texture, image registration.