The popularity of concrete has been accompanied with dreadful consumptions that have led to huge carbon footprint in our environment. The exhaustion of natural resources is not yet the problem, but also the energy that is needed for the fabrication of the natural materials, in which this process releases significant amount of carbon dioxide (CO2) emissions into the air. Ordinary Portland Cement (OPC) and natural aggregates, which are the key constituents of concrete, are suggested to be recycled or substituted in order to address the sustainability concern. Here, by-products have been targeted to reduce the carbon footprint, including, but not limited to, fly ash, rice husk ash, silica fume, recycled coarse aggregates, ground granular blast-furnace slag, waste glass, and plastic. Moreover, advanced approaches with an emphasis on sustainability are highlighted, which include the enhancement of the hydration process in cement (calcium-silicate hydrate) and the development of new materials that can be used in concrete (e.g. carbon nanotube). This review paper provides a comprehensive discussion upon the utilization of the reviewed materials, as well as the challenges and the knowledge gaps in producing green and sustainable concrete.