In this paper, we investigate a full duplex (FD) multiuser non-orthogonal multiple access (NoMA) communication system, based on the optimization of received signalto-interference-plus-noise ratio (SINR) per unit power. Since the communication system operates in FD mode, co-channel interference (CCI) and self-interference (SI) dominate the system's performance. Accordingly, to combat the CCI, we adopt a gametheoretic approach and propose users clustering algorithms and to suppress the SI, we formulate an optimization problem to maximize the power-normalized SINR (PN-SINR). While the user clustering optimization problem is constrained by i) the successive interference cancellation (SIC) constraint and ii) two binary constraints for the allocations of UL and DL users, the PN-SINR problem is constrained by i) total transmit power budget at the base station and uplink (UL) users, ii) the fundamental condition for the implementation of successive interference cancellation in NoMA, and iii) the minimum fairness condition for UL users. The original PN-SINR problem is non-convex and hence is converted into an equivalent subtractive-form problem, after which we propose an iterative algorithm to find the optimal power allocation policy. Properties of all the proposed algorithms are thoroughly investigated and numerical results are provided. Based on the channel conditions and suppression level of SI and CCI, the superiority of the proposed FD-NoMA system over half duplex NoMA and FD orthogonal multiple access systems is verified.