To understand the ablation mechanism of materials is crucial for the design of thermal protection system (TPS) for manned reentry vehicles. To investigate the thermal behavior of lightweight quartz fiber reinforced phenolic (LQP) composite subjected to an aerodynamic hyper-thermal environment, a 3D ablation material response model for the LQP composite is developed with surface and volume ablation in consideration. It is used to predict surface and back temperature, linear ablation velocity, and quality loss rate of the material.The 3D temperature field, pyrolysis degree, gas flow, and pressure field are analyzed as well. To evaluate the accuracy of the model, an ablation test of oxyacetylene flame is adopted on the LQP composite. In the ablation process, the pyrolysis gases propagate primarily through the surface primarily and through the sidewall afterwards. The gas pressure peaks at the early stage rather than increasing constantly. The analysis of thermal insulation mechanism of the LQP composite indicates that the radiation and heat blockage effect are the primary factors to improve the ablation resistance, accounting for 50.0% and 37.3% of the improvement, respectively. The low thermal conductivity determines the admirable thermal insulation performance. The surface temperature of material can be reduced by introducing high emissivity powder into the material, preparing coating on the surface, and adjusting the content of phenolic and silica fiber.