The need to overcome the low energy-absorbing response and low tensile strength of geopolymers and cementitious materials has motivated researchers to develop fiber-reinforced composites. Interestingly, natural cellulosic fibers provide exciting prospects for advancements in bio-based materials compared to synthetic fibers. Incorporating fibers in such binders offer a crack limiting effect by creating multiple cracks than forming one major crack, usually leading to improved toughness and tensile strength of the composites. However, their sensitivity in alkaline media and environmental fluctuations deteriorates their durability and reinforcing capacity in composites. Therefore, exploring suitable mitigation strategies for such drawbacks of natural fibers based on previous research work and figuring out the existing gaps for further developments is extremely important. In this paper, cellulosic fiber degradation in geopolymer and cement-based matrices and the respective remedial actions have been reviewed based on a comprehensive analysis of the available literature. The influence of fiber modification and matrix treatment on the durability and strength characteristics of such composites is thoroughly discussed. Finally, recommendations and suggestions for further research in the field are presented.