Following technology scaling, on-chip heterogeneous architecture emerges as a promising solution to combat the power wall of microprocessors. This work presents Harmonica-a framework of heterogeneous computing system enhanced by memristor-based neuromorphic computing accelerators (NCAs). In Harmonica, a conventional pipeline is augmented with a NCA which is designed to speedup artificial neural network (ANN) relevant executions by leveraging the extremely efficient mixed-signal computation capability of nanoscale memristor-based crossbar (MBC) arrays. With the help of a mixed-signal interconnection network (M-Net), the hierarchically arranged MBC arrays can accelerate the computation of a variety of ANNs. Moreover, an inline calibration scheme is proposed to ensure the computation accuracy degradation incurred by the memristor resistance shifting within an acceptable range during NCA executions. Compared to general-purpose processor, Harmonica can achieve on average 27.06× performance speedup and 25.23× energy savings when the NCA is configured with auto-associative memory (AAM) implementation. If the NCA is configured with multilayer perception (MLP) implementation, the performance speedup and energy savings can be boosted to 178.41× and 184.24×, respectively, with slightly degraded computation accuracy. Moreover, the performance and power efficiency of Harmonica are superior to the designs with either digital neural processing units (D-NPUs) or MBC arrays cooperating with a digital interconnection network. Compared to the baseline of general-purpose processor, the classification rate degradation of Harmonica in MLP or AAM is less than 8% or 4%, respectively. . His research interests include large-scale neuromorphic computing circuits and systems, highperformance computing architectures, energy-efficient embedded computing.Jianhua (Joshua) Yang (M'08) received the B.A. . He spent over 8 years at HP Labs before joining UMass in 2015. His current research interests are Nanoelectronics and Nanoionics, especially for unconventional computing applications, where he authored and co-authored over 100 papers in peer-reviewed academic journals and conferences, and holds 61 granted and over 70 pending US Patents.Hai (Helen) Li (M'08-SM'16) received the B.S. and M.S. degrees in microelectronics from Tsinghua University,