In this paper, a novel hybrid non-orthogonal multiple access (NOMA) system that combines the cooperative NOMA (C-NOMA), original NOMA, and the orthogonal multiple access (OMA) is investigated. The probability-based user management algorithm (PUMA) is proposed for the considered hybrid NOMA system with regard to fulfilling the requirements of the 5G services, which is a joint transmission mode selection and user pairing scheme. The exact coverage and outage probabilities are derived in closed-forms under more realistic network scenarios, involved with the stochastic characteristics of both channel and distance for users. Furthermore, these probabilities are not only utilized in the proposed PUMA, but also can be applied to measure the stochastic geometric performance of the NOMA-related systems. Monte Carlo simulation is executed to verify the mathematical derivations. Simulation results demonstrate that the proposed PUMA possesses higher spectrum utilization, efficiency, and better outage performance in comparison with the random user coordination scheme.INDEX TERMS 5G, coverage probability, hybrid NOMA, outage probability, user pairing.