In this paper, we investigate the optimization problem of joint source and relay beamforming matrices for a two-way amplify-and-forward (AF) multi-input multi-output (MIMO) relay system. The system, consisting of two source nodes and two relay nodes, is considered, and the linear minimum mean-square-error (MMSE) is employed at both receivers. We assume individual relay power constraints and study an important design problem, a so-called determinant maximization (DM) problem. Since this DM problem is nonconvex, we consider an efficient iterative algorithm by using an MSE balancing result to obtain at least a locally optimal solution. The proposed algorithm is developed based on QL, QR, and Choleskey decompositions which differ in complexity and performance. Analytical and simulation results show that the proposed algorithm can significantly reduce computational complexity compared with their existing two-way relay systems and have equivalent bit-error-rate (BER) performance to the singular value decomposition (SVD) based on a regular block diagonal (RBD) scheme.