Background Osteoporosis is prevalent in elderly women, which causes fragility fracture and hence increased mortality and morbidity. Predicting osteoporotic fracture risk is both clinically-beneficial and cost-effective. However, traditional tools using clinical factors and bone mineral density (BMD) fail to reflect bone microstructure. Here we aim to use high-resolution peripheral quantitative CT (HR-pQCT) images to construct deep-learning models which predict fragility fracture history in elderly Chinese women. Methods We used ChiVOS, a community-based national cohort of 2,664 Chinese elderly women. Demographic data, BMD, and HR-pQCT from 216 patients were used to construct three groups of models: BMD, pQCT-index, and DeepQCT. For DeepQCT, we used ResNet34 as classifier, and logistic regression for late fusion. Models were developed using 6-fold cross-validation in development set (90%, N=195), and tested in internal test set (10%, N=21). We applied unsupervised clustering on HR-pQCT indices to derive patient subgroups. Findings DeepQCT (best model AUC 0.86-0.94) was superior or similar to pQCT-index (best model AUC 0.8-0.93), which both outperformed BMD (best model AUC 0.54-0.78). Surprisingly, DeepQCT built from non-weight-bearing bones performed similarly to weight-bearing bones. Furthermore, two distinct patient groups were classified using HR-pQCT indices. The one with higher DeepQCT risk score showed lower volumetric BMD, bone more microarchitectural abnormalities, and had higher probability of osteoporosis and fragility fracture history. Interpretation DeepQCT scores and HR-pQCT-index permit early recognition of patients with high risk of fragility fracture. This established framework can be easily adapted for other diagnostic tasks using HR-pQCT scans, which promotes bone health management via digital medicine.