Recent high-resolution infrared space missions have revealed supernova remnants (SNRs) of diverse morphology in infrared (IR) dust emission that are often very different from their X-ray appearance. The observed range of infrared-to-X-ray (IRX) flux ratios of SNRs is also wide. For a sample of 20 Galactic SNRs, we obtain their IR and X-ray properties and investigate the physical causes for such large differences. We find that the observed IRX flux ratios (R IRX,obs ) are related to the IRX morphology, with SNRs with the largest R IRX,obs showing anticorrelated IRX morphology. By analyzing the relation of R IRX,obs to X-ray and IR parameters, we show that the R IRX,obs of some SNRs agrees with theoretical ratios of SNR shocks in which dust grains are heated and destroyed by collisions with plasma particles. For the majority of SNRs, however, R IRX,obs values are either significantly smaller or significantly larger than the theoretical ratios. The latter SNRs have relatively low dust temperatures. We discuss how the natural and/or environmental properties of SNRs could have affected the IRX flux ratios and the IRX morphology of these SNRs. We conclude that the SNRs with largest R IRX,obs are probably located in a dense environment and that their IR emission is from dust heated by shock radiation rather than by collisions. Our result suggests that the IRX flux ratio, together with dust temperature, can be used to infer the nature of unresolved SNRs in external galaxies.