In the exploration of new infrared nonlinear optical (IR NLO) materials, element substitution has been developed as an effective way to adjust the structural features and material performances. A series of new IR NLO materials have been discovered in the I-Ba-M-Q system ( I = Li, Na, Cu, and Ag; M = Si, Ge, and Sn; Q = S and Se), and they undergo interesting structural transformation with different element substitution except Li analogues. Herein, we have successfully synthesized three selenides with different space groups (AgBaSiSe: I4̅2 m; AgBaGeSe and AgBaSnSe: I222) in the above system and studied their properties through experimental and theoretical methods. Remarkably, the detailed analysis on the structural changes and properties comparison was also systematically investigated in the I-Ba-M-Q system and the results indicate that the distortion degrees of different IQ tetrahedra play the critical role to cause the structural transformation with the M or Q elements substitution. More importantly, we have also found that the structural changes have the close relationship with the distance d( I- I) between adjacent I cations in the IBaSnSe system, which makes the four-membered rings formed by edge-sharing BaSe units change from the square to rhombus with the increase of d( I- I). The properties comparisons (band gap and NLO effect) in this system have been also systematically studied.