Cells convert complex metabolic information into stress-adapted autophagy responses. Canonically, multilayered protein kinase networks converge on the conserved Atg1/ULK kinase complex (AKC) to induce non-selective and selective forms of autophagy in response to metabolic changes. Here, we show that, upon phosphate starvation, the metabolite sensor Pho81 interacts with the adaptor subunit Atg11 at the AKC via an Atg11/FIP200 interaction motif to modulate pexophagy by virtue of its conserved phospho-metabolite sensing SPX domain. Notably, we find core AKC components Atg13 and Atg17 are dispensable for phosphate starvation-induced autophagy revealing significant compositional and functional plasticity of the AKC. Our data indicate that, instead of functioning as a selective autophagy receptor, Pho81 compensates for partially inactive Atg13 during pexophagy when TORC1 remains active under phosphate starvation. Our work shows Atg11/FIP200 adaptor subunits not only bind selective autophagy receptors but also modulator subunits that convey metabolic information directly to the AKC for autophagy regulation.