MNSiO2/CN40/PF nanocomposites have been fabricated in this study. Nano‐SiO2 was initially surface‐modified with γ‐methacryloxypropyl trimethoxysilane (MPS). The resulting nano‐SiO2 was denoted as MNSiO2. Core‐shell composites, MNSiO2/CN40, was fabricated by a solution blending method, in which MNSiO2 served as the core and liquid acrylonitrile butadiene rubber (CN40) served as shell. Phenol formaldehyde resins (PF) were prepared by means of condensation polymerization method. MNSiO2/CN40/PF hybrid with a fixed weight ratio (2:98) of MNSiO2/CN40 composites to PF were prepared by a modified routine of preparation of pure PF. MNSiO2/CN40/PF nanocomposites were prepared by incorporating filler particles with MNSiO2/CN40/PF hybrid via melt blending, followed by compression molding method. The impact of MNSiO2 content on the properties of the MNSiO2/CN40/PF nanocomposites was systematically investigated by varying the weight ratio of MNSiO2 to CN40 in MNSiO2/CN40 core‐shell composites. The results revealed that MNSiO2/CN40 significantly enhanced the mechanical and frictional properties of the as‐prepared MNSiO2/CN40/PF nanocomposites. Compared to pure PF, the impact strength of MNSiO2/CN40/PF hybrid containing 1.0 wt% of MNSiO2 in MNSiO2/CN40 composites increased by 29.2%, and the initial storage modulus was enhanced dramatically. In addition, MNSiO2/CN40/PF hybrid demonstrated stable friction coefficient, while the wear mechanism of pure PF was adhesive and fatigue wear. The MNSiO2/CN40/PF nanocomposites exhibited a combination of adhesive, abrasive, and fatigue wear. POLYM. COMPOS., 40:179–186, 2019. © 2017 Society of Plastics Engineers