The environment geometry model based-VR teleoperation systems couldn't adapt to the unpredicted, dynamic changing and unstructured environment any more for the development space of mankind is expanding continuously, in which many circumstances are uncertain. The Interactive Partial Modeling (IPM) technique was developed to solve this problem. The establishing of IPM system for the unstructured environment includes three parts: the image information obtaining subsystem of the reality worksite, the designing of 3D virtual manipulation subsystem and the registration between virtual object and real scene. The feedback of the image information is realized through stereo-video and the virtual model of the telerobot is developed with OpenGL. The position and pose of the virtual graphic can be set in the stereo-video with the telerobot as the calibration object. So manipulation of virtual unstructured environments and the real scene with 3D image can be realized, which has expanded the application of robotic teleoperation.