Coarse-grained (CG) modeling is an invaluable tool for the study of polymers and other soft matter systems due to the span of spatiotemporal scales that typify their physics and behavior. Given continuing advancements in experimental synthesis and characterization of such systems, there is ever greater need to leverage and expand CG capabilities to simulate diverse soft matter systems with chemical specificity. In this review, we discuss essential modeling techniques, bottom-up coarse-graining methodologies, and outstanding challenges for the chemically specific CG modeling of polymer-based systems. This methodologically oriented discussion is complemented by representative literature examples for polymer simulation; we also offer some advisory practical considerations that should be useful for new researchers. Given its growing importance in the modeling and polymer science community, we further highlight some recent applications of machine learning that enhance CG modeling strategies. Overall, this review provides comprehensive discussion of methods and prospects for the chemically specific coarse-graining of polymers.