Novel tetragonal scandium orthovanadate crystals doped with neodymium ions (Nd:ScVO4) have been successfully grown with the floating zone (FZ) method for applications in photonic devices. High-temperature stability studies and X-ray structural studies show that thermally induced vanadium oxide vaporization triggers a series of topological transitions from the rutile lattice to the defect fluorite lattice when crystallization begins in the ScVO4 melt. Besides zircon-type ScVO4 and bixbyite-type Sc2O3, a third form with metallic properties called tetragonal Sc2VO5 was thus obtained. Ab initio calculations further reveal that such structural transitions at high-temperature are directly driven by increasing covalency in the scandium−oxygen bonds and are energetically favored. In addition, the anisotropic mechanical and optical properties of ScVO4, such as elastic stiffness, Young’s modulus, dielectric constants, and refractive index, were calculated using density functional theory and evaluated for future applications. The calculated results tend to support the experimental data.