Hepatocellular carcinoma (HCC) is a common cancer that affects people worldwide with high morbidity and mortality, and its resistance to current chemotherapeutic drugs is a serious concern. Cytotoxicity of silica nanoparticles (Nano-SiO2) towards cancer cells has been reported previously, but the specific mechanism is not fully clear. In this study, Nano-SiO2 showed a remarkable cytotoxic effect against HCC cells, regardless of whether the cells were drug resistant or not. Further study showed that Nano-SiO2 treatment leads to cell cycle arrest, apoptosis enhancement and necroptosis induction in the HCC cells. RNA-seq data, together with bioinformatics analysis, revealed that a series of genes involved in cancer cell death could be regulated by Nano-SiO2, among which ZBP-1 was up-regulated the most by Nano-SiO2 treatment. The siRNA based experiments demonstrated that ZBP-1 might play a key role in mediating Nano-SiO2 cytotoxic functions against HCC cells. These results have evidently signified the anti-tumor potential of Nano-SiO2 in the treatment of HCC.