Quantum critical point is one key concept for studying many-body physics but its investigation may be resource-demanding even for a quantum computer due to the intrinsic complexity. In this paper, we propose an approach based on variational quantum eigensolver(VQE), dubbed as Delta-VQE, for locating the quantum critical point using only shallow quantum circuits. With Delta-VQE, the critical point can be identified as a most confusing point, quantified as zero difference between two variational energies that use two representative reference states of distinct phases of matter. Remarkably, the signature of a critical point as a minimal point can be sharper while using shallower quantum circuits. We test the algorithm for different quantum systems and demonstrate the usefulness of Delta-VQE. The scheme suggests a new avenue for investigating quantum phases of matter on near-term quantum devices with limited quantum resources.