Conventional direction-of-arrival (DOA) estimation methods for multiple-input multiple-output (MIMO) joint sensing and communication system normally pursue high estimation accuracy and resolution by imposing orthogonal waveforms, which however, results in a deterioration of communication performance. In this paper, we propose a nonorthogonal waveform assisted DOA estimation algorithm, where an augmented virtual array is derived for DOA estimation exploiting the nonorthogonal MIMO communication waveforms, while a high communication rate can still be maintained. To estimate the round-trip sensing channels on each subcarrier, we utilize the transmitted symbols as pilot symbols, and obtain all the channel coefficients with a minimum mean square error (MMSE) solver. A virtual channel matrix can be formulated with these channel coefficients, which can be regarded as the samples of the augmented virtual array. Further, the subspace framework of DOA estimation for this nonorthogonal scenario is presented, and the rank deficiency property of the equivalent signal matrix of the virtual array is analyzed when the distance of targets are identical. To address the problem, a suitable Toeplitz reconstruction method is proposed for the rank-deficient equivalent signal matrix. Simulations show that the proposed nonorthogonal waveform assisted DOA estimation algorithm outperforms the conventional DOA estimation methods for joint MIMO sensing and communication system in terms of resolution and accuracy, and is capable of locating targets with both identical and non-identical distance.