Image-based control, using a mobile terminal, is a widely used control method for many remote robot applications since it provides valuable visual information from around the remote robot to the operator. However, the computational power of the mobile terminal is normally limited, compared with that of the stationary computing machine, because it has a number of constraints such as size, power, and cost. Since the mobile robot also has its inherent tasks, processing a large amount of data may cause other performance problems as the intelligence level grows. For this reason, it is not good practice to perform image processing via software programs, either in the mobile terminal or the in remote robot. This paper proposes a dedicated hardware architecture which can assist both the mobile terminal and the remote robot by taking complete charge of the visionrelated tasks and thus decreasing the computational burden still to be performed. As a result, the remote robot can fully use its computation power for its main tasks so that the overall performance and efficiency can be increased.