Abstract-Imaging has become an indispensable tool in modern medicine. Various powerful and expensive platforms to study medical imaging applications appear in recent years. In this article, we design and propose a Visual Processing System (ViPS) that processes medical imaging applications efficiently. ViPS provides a user-friendly programming environment and highperformance architecture to perform image analysis, features extraction and object recognition for complex real-time images or videos. The data structure of image or video is described in the program memory using pattern descriptors; ViPS uses specialized 3D memory structure to handle complex images or videos and processes them on microprocessors or application specific hardware accelerators. The proposed system is highly reliable in terms of cost, performance, and power. ViPS based system is implemented and tested on a Xilinx Virtex-7 FPGA VC707 Evaluation Kit. The performance of ViPS is compared with the Intel i7 multi-core, GPU Jetson TK1 Embedded Development Kit with 192 CUDA cores based graphic systems. When compared with the Intel and GPU-based systems, the results show that ViPS performs real-time video reconstruction at 2x and 1.45x of higher frame rate, achieves 14.6x to 4.8x of speedup while executing different image processing applications and 20.3% and 12.6% of speedup for video processing algorithms respectively.