Flavonoids are a class of plant secondary metabolites and among thousands of flavonoids few are considered as dietary flavonoids. Serum albumin (SA), the most abundant protein in plasma, functions as the most important carrier of vital drugs, including dietary flavonoids. The binding affinity of dietary flavonoids to SA is demonstrated to be governed by structure-affinity relationship (SAR) and its bioavailability. The present review summarizes the interactions of flavonoids categorized as flavanol, flavonol, flavone, isoflavone, flavanones, and anthocyanidins with SAs (bovine serum albumin and human serum albumin) in light of SAR. The key findings are: (1) the position and degree of hydroxylation highly influence the affinity of flavonoids to SAs, (2) glycosylation decreases and substitution of methoxy group increases the affinity of flavonoids for SAs, (3) catechin gallates have higher binding affinity to SAs than catechins and gallocatechins, (4) inorganic metal ions modulate the binding affinity of flavonoids to SAs, and (5) hydrophobic interaction plays a major role in the interactions of all flavonoids with SAs.