Glycosylation is a ubiquitous modification that occurs on proteins and lipids in all living cells. Consistent with their high complexity, glycans play crucial biological roles in protein quality control and recognition events. Asparagine-linked protein N-glycosylation, the most complex glycosylation, initiates in the endoplasmic reticulum (ER) and matures in the Golgi apparatus. This process not only requires an accurate distribution of processing machineries, such as glycosyltransferases, glycosidases and nucleotide sugar transporters, but also needs an efficient and well-organized factory that is responsible for the fidelity and quality control of sugar chain processing. In addition, accurate glycosylation must occur in coordination with protein trafficking and sorting. These activities are carried out by the Golgi apparatus, a membrane organelle in the center of the secretory pathway. To accomplish these tasks, the Golgi has developed into a unique stacked structure of closely aligned flattened cisternae in which Golgi enzymes reside; in mammalian cells, dozens of Golgi stacks are often laterally linked into a ribbon-like structure. Here, we review our current knowledge of how the Golgi structure is formed and why its formation is required for accurate glycosylation, with the focus of how the Golgi stacking factors GRASP55 and GRASP65 generate the Golgi structure and how the conserved oligomeric Golgi (COG) complex maintains Golgi enzymes in different Golgi subcompartments by retrograde protein trafficking.