We consider the least singular value of M = R * XT + U * Y V , where R, T , U , V are independent Haar-distributed unitary matrices and X, Y are deterministic diagonal matrices. Under weak conditions on X and Y , we show that the limiting distribution of the least singular value of M , suitably rescaled, is the same as the limiting distribution for the least singular value of a matrix of i.i.d. Gaussian random variables. Our proof is based on the dynamical method used by Che and Landon to study the local spectral statistics of sums of Hermitian matrices.