Mixing characteristics of under-expanded supersonic jets emerging from plane and notched rectangular nozzles are computationally studied using nozzle exit boundary layer swirl as a mean of passive flow control. The coupling of the rectangular jet instability modes, such as flapping, and the swirl is investigated. A threedimensional unsteady Reynolds-Averaged Navier-Stokes (RANS) code with shock adaptive grids is utilized. For plane rectangular nozzle with boundary layer swirl, the flapping and spanwise oscillations are captured in the jet's small and large dimensions at twice the frequencies of the nozzles without swirl. A symmetrical oscillatory mode is also observed in the jet with double the frequency of spanwise oscillation mode. For the notched rectangular nozzle with boundary layer swirl, the flapping oscillation in the small jet dimension and the spanwise oscillation in the large jet dimension are observed at the same frequency as those without boundary layer swirl. The mass flow rates in jets at 11 and 8 nozzle heights downstream of the nozzles increased by nearly 25% and 41% for the plane and notched rectangular nozzles respectively, due to swirl. The axial gross thrust penalty due to induced swirl was 5.1% for the plane and 4.9% for the notched rectangular nozzle.