This paper proposes a novel bilateral control system using environmental data memory and slave system simulator in order to eliminate effect of the time delay, which exist in the communication network. The slave system simulator imitates the real slave system by using the environmental data memory without environmental mathematical model. Thus, non-linear environment and unknown environment can be treated, since raw force information is utilized. In addition, responses of the slave system in the real-world are not returned directly to the master side. By the proposed method, the bilateral controller operates normally under the time delay, which causes vibration and/or overdrive. Therefore, the human operator in the master side is able to grasp the environmental sense through the bilateral controller, even if the environment exists in the distant place. By the experiments, the validity of the proposed method was confirmed in the case that the time delay is set to 0 to 10s.