Geometric gates that use the global property of the geometric phase is believed to be a powerful tool to realize fault-tolerant quantum computation. However, for singlet-triplet (ST) qubits in semiconductor quantum dot, the low Rabi frequency of the microwave control leads to overly long gating time, and thus the constructing geometric gate suffers more from the decoherence effect. Here, we investigate the key issue of whether the fast geometric gate can be realized for ST qubits without introducing an extra microwave-driven pulse, while maintain the high-fidelity gate operation at the same time. We surprisingly find that, both the single-and two-qubit geometric gates can be implemented via only modulating the time-dependent exchange interaction of the Hamiltonian, which can typically be on the order of ∼GHz, and thus the corresponding gate-time is of several nanoseconds. Furthermore, the obtained geometric gates are superior to their counterparts, i.e., the conventional dynamical gates for ST qubits, with a relatively high fidelity surpassing 99%. Therefore, our scheme is particularly applied to ST qubits to obtain fast and high-fidelity geometric gates. Our scheme can be also extended to other system without microwave drive.