SCE-Al2O3was the nano-Al2O3modified by supercritical ethanol and the surface of SCE-Al2O3was coated with active group. 4,4′-diaminodiphenylmethane bismaleimide (MBMI) was used as matrix; 3,3′-diallyl bisphenol A (BBA) and bisphenol-A diallyl ether (BBE) were used as reactive diluent, polyethersulfone (PES) as toughening agent, and SCE-Al2O3as modifier; SCE-Al2O3/PES-MBAE nanocomposite was prepared through in situ sol-gel method. The mechanism of composite toughened by PES was observed and analyzed. FTIR indicated that the reaction between MBMI and allyl compound occurred and SCE-Al2O3had doped into the polymer matrix. SEM showed that PES particle was inlaid in matrix and presented as a two-phase structure in matrix. The heat resistance, dielectric properties, and mechanical properties of SCE-Al2O3/PES-MBAE nanocomposites were evaluated. The results showed that with the incorporation of PES, although the toughness of the material improved, the heat resistance and dielectric properties of material declined, meanwhile. The adulteration of SCE-Al2O3could remedy the harmful effect caused by PES, while the content of SCE-Al2O3was reasonable. The decomposition temperature, dielectric constant, and dielectric loss of composite were 441.23°C, 3.63 (100 Hz), and 1.52 × 10−3(100 Hz); the bending strength and impact strength were 129.22 MPa and 13.19 kJ/mm2, respectively, when the content of SCE-Al2O3was 3 wt% and PES was 5 wt%.