The outer layer of purple sweet potato is removed during processing; however, this layer serves as a potential source of phenolics, especially anthocyanins. Herein, the phenolic composition and antioxidant activity were determined for the inner and outer layers of five purple sweet potato cultivars (‘Sinjami’, ‘Jami’, ‘Danjami’, ‘Yeonjami’, and ‘Borami’) harvested in Korea. Anthocyanins were identified using ultra-high-performance liquid chromatography coupled with electrospray ionization quadrupole time-of-flight mass spectrometer (UHPLC-(ESI)-qTOF-MS) and ultra-high-performance liquid chromatography-linear ion trap mass spectrometer (UHPLC-Ion trap-MS), and their composition was quantified using HPLC-coupled with diode array detector (DAD). Non-anthocyanin phenolic compounds (phenolic acids and flavonols) were quantified using UHPLC-(ESI)-triple quadrupole (QqQ). A total of 20 anthocyanins, including non-acylated or acylated peonidin, cyanidin, and pelargonidin glycosides, were identified. Peonidin 3-caffeoyl-p-hydroxybenzoyl sophoroside-5-glucoside was the major anthocyanin, with the highest level in the ‘Sinjami’ cultivar (outer; 12,366 mg/kg DW, inner; 14,832 mg/kg DW). Additionally, 12 phenolic acids and 6 flavonols (quercetin derivatives) were identified, with the outer layers of all cultivars displaying higher total levels than the inner layers. ‘Sinjami’ and ‘Jami’ had higher phenolic acid and quercetin derivative content and antioxidant activities than the other three cultivars (p < 0.05). Thus, the outer layers of ‘Sinjami’ and ‘Jami’ cultivars could be potential sources of anthocyanins and other phenolics.