In this paper, for the first time, we propose a nonparametric histogram estimation (NPHE) scheme for the nonuniform quantization in intensity modulated direct detection with filter bank multicarrier signal to improve system performance. By combining the NPHE scheme and the common nonlinear programming method, the distribution of offset quadrature amplitude modulation based optical orthogonal frequency division multiplexing (OQAM-OFDM) signal is accurately estimated and then it is used to achieve the optimal nonuniform quantization level. In this way, the low-level signal can achieve much more effective digitalizing bits, thereby significantly reducing the quantization noise. The numerical simulation and experiment with 10-Gb/s 16-order OQAM-OFDM signal over 20 km fiber are conducted to further verify the feasibility of our scheme. The results show that our scheme always outperforms the uniform quantization and Gaussian-like nonuniform schemes in terms of the signal to quantization noise ratio, error-vector-magnitudes, and bit error rate. Compared to the conventional method, our scheme also achieves received sensitivity improvement thereby improving the system power budget. So, it can support the low-bit resolution transmission in intensity modulated direct detection-based OQAM-OFDM system.