This paper studies the passive impedance control of a robot manipulator with model uncertainty to perform manipulation tasks while interacting with dynamic environment. Impedance control is a powerful approach for the robot to perform mechanical tasks while interacting with dynamic environment. However, in our previous research, it was clarified that, the time varying impedance center as well as the robot's model uncertainty influences the robot's passivity, which may lead to serious safety problem for both the robot as well as its environment. In order for the robot to keep its passivity as well as to realize desired objective impedance, in this paper, a novel observer based control design is proposed. Computer simulations of a 2-link manipulator interacting with a dynamic wall show the effectiveness of our control approach.