This work demonstrates the improvement in DC and RF characteristics and a reduction in the gate leakage current for thermally grown Nb2O5 as a gate dielectric in AlGaN/GaN metal–oxide–semiconductor high electron-mobility transistors (MOS-HEMTs). The MOS-HEMTs with an amorphous 10 nm thick Nb2O5 as the gate dielectric show a reduced gate leakage current of 10-9 A/mm. Nb2O5 thin film creates a tensile strain in the AlGaN layer, enhancing the density of two-dimension electron gas (2-DEG). The performance of the device also improves in terms of saturation drain current, peak transconductance, subthreshold swing, and unity current gain frequency. An increase in the source-to-drain ON/OFF current ratio to 108 and a significant reduction in the subthreshold leakage current by at least two orders of magnitude are measured compared to the control HEMTs.