The lattice and elastic instabilities of rocksalt (RS) calcium oxide CaO under pressure are extensively studied to reveal the physically driven mechanism of the phase transition from RS to CsCl structure by using the pseudopotential plane-wave method within density-functional theory. The predicted phase transition pressure is 66.38 GPa, employing the total energy method, which falls in the experimental transition pressure range of 60-70 GPa. A pressure-induced soft transverse acoustic (TA) phonon mode is identified at the zone boundary X point in the Brillouin zone, signifying a structural instability. A predicted charge transfer from Ca to O with pressure might be attributable to the phonon softening. Moreover, a softening behavior in the C(44) shear modulus with pressure is predicted. Analysis of the calculated results suggested that, with increasing pressure, the predicted TA phonon softening behavior, instead of C(44) shear modulus instability, is mainly responsible for the pressure-induced structural phase transition. We also find that the bond between Ca and O becomes more ionic under compression from Mulliken population analysis.