This paper describes a method for target-tracking by a moving camera such as a mobile robot based on a scheme of the fusion of the unsynchronous sensor data. To realize the interaction between the person and the robot, we need not only the location of the target but the relation of the target and the robot such as the motion plan of the target. This study focuses on the sensor data fusion scheme to estimate the relation between the robot and the targets. We apply a fuzzy inference for the sensor data fusion since the result of the fuzzy inference is useful for the fuzzy controller of the moving camera. In addition, the paper discusses the implementation of the fuzzy inference system for the target tracking as an RT component for a distributed component middleware. Experimental results show the feasibility of the proposed method.