In the real-time control of networked inverted pendulum visual servo control system (NIPVSCS), how to achieve safe and efficient image transmission is an important and challenging problem. Generally, the inverted pendulum images are directly transmitted via the network, while these images are not secure because they may be attacked by the hacker. To solve the problem, a fast image encryption method by combining image scaling and improved self-diffusion image encryption (ISDIE) algorithm is firstly proposed, which is employed to encrypt the captured images after they are scaled. The security performance of the proposed algorithm, the characteristics of different delays, and the control performance of the NIPVSCS are then analyzed. Finally, simulation and real-time control experiments confirm the feasibility and effectiveness of the proposed method.