The properties of nonpolar a-plane GaN layers grown with different Mg doping levels were investigated. With increasing the Mg flow rate, the hole concentration initially increased and then decreased, indicating the formation of compensation centers. The dominant photoluminescence (PL) emission at relatively low Mg doping levels is the blue luminescence (BL) band due to the donor-acceptor pair (DAP) transition with MgGaVN (deep donor) and MgGa (acceptor). In addition to the weak BL band, both the ultraviolet luminescence (UVL) and yellow luminescence (YL) bands are observed at higher Mg doping level. The UVL band, especially dominant at 10 K, can be related to bound excitonic emissions involving Mg-induced extended defects, whereas one or more mechanisms may contribute to the YL band.