The cleavage of amyloid precursor protein (APP) by β-site APP cleaving enzyme 1 (BACE1) is the rate limiting step in the generation of β-amyloid (Aβ) during Alzheimer’s disease (AD) pathogenesis. In AD brains, BACE1 is abnormally accumulated in the endocytic compartments, where the acidic pH is optimal for its activity. However mechanisms that regulate the endosome-to-TGN retrieval of BACE1 under physiological conditions remain unclear. Here we show the Par polarity complex containing Par3 and aPKC facilitates BACE1 retrograde trafficking from the endosomes to the TGN. We further show that Par3 functions through aPKC-mediated phosphorylation of BACE1 on Ser498. Finally, we found that Ser498 phosphorylation promotes the interaction between BACE1 and PACS1, which is necessary for the retrograde trafficking of BACE1 to the TGN. In human AD brains, there is a significant decrease in Ser498 phosphorylation of BACE1 suggesting that defective phosphorylation-dependent retrograde transport of BACE1 is important in the AD pathogenic process. Together, our studies provide mechanistic insight into a novel role for Par3 and aPKC in regulating the retrograde endosome-to-TGN trafficking of BACE1, and shed light on the mechanisms of AD pathogenesis.