The spectral resolution of images improves with the continuous development of imaging technology in remote sensing. Traditional image fusion technologies result in problems in hyperspectral image fusion, such as spectral distortion and unclear physical meaning. A fast model based on improved non-negative matrix factorization (INMF) for hyperspectral and panchromatic image fusion was proposed to optimize the results of image fusion. First, the hyperspectral image was decomposed into endmember and abundance matrices using the INMF algorithm. The panchromatic image was then used to sharpen the abundance matrix. The spectral and sparse constraints were imposed on the objective function of the model. Finally, the resultant fused image was obtained by reconstructing the resolved endmember and abundance matrices. Results show that the proposed method is superior to other methods in terms of subjective assessment and objective analysis. As far as the indices of information entropy, correlation coefficient, Q-average, and spectral divergence are concerned, the proposed method surpasses those of second-best methods (Ehler, classical NMF, Ehler and Ehler) by 2.06%, 0.36%, 0.91%, and 56.31%, respectively, in the synthetic data experiment, and exceeds those of the second-best methods (Ehler, high-pass filtering, high-pass filtering and Ehler) by 0.13%, 10.05%, 3.89%, and 7.26%, respectively, in the real data experiment. Moreover, runtime is proportional to data size, and the proposed method takes the least time when image size is between 1 MB and 480 MB. This study provides theoretical reference for fast hyperspectral and panchromatic image fusion.