Background The COVID-19 pandemic has prompted the decrease of in-person visits to reduce the risk of virus transmission. Telemedicine is an efficient communication tool employed between healthcare providers and patients that prevents the risk of exposure to infected persons. However, telemedicine use is not infallible; its users reported multiple issues that complicated the expansion of this technology. So, this systematic review aimed to explore the barriers and challenges of telemedicine use during the pandemic and to propose solutions for improving future use. Methods A systematic review was conducted following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) statement. PubMed, Scopus, Web of Science, Academic Search Complete, CINAHL, Embase, and Science Direct were used to look for articles addressing barriers and challenges, in addition to articles proposing solutions. Studies were screened by title and abstract, followed by a full-text review. Risk of bias assessment was done using Critical Appraisal Skills Program for qualitative studies, Newcastle–Ottawa Scale for cross-sectional studies, and A MeaSurement Tool to Assess Systematic Reviews for systematic reviews. After the extraction of data, a narrative synthesis and analysis of the outcomes were performed. Results Among 1194 papers identified, only 27 studies were included. Barriers and challenges were assembled under 7 categories: technical aspects, privacy, data confidentiality and reimbursement, physical examination and diagnostics, special populations, training of healthcare providers and patients, doctor-patient relationship, and acceptability. Poor internet connection and lack of universal access to technology were among the technical barriers. Concerns about patient privacy and reimbursement hindered the use of telemedicine too. Physical examination and certain procedures were impossible to perform via telemedicine. Training both healthcare providers and patients was deficient. The doctor-patient relationship was troubled by telemedicine, and both healthcare providers and patients were reluctant to use telemedicine. Conclusion Widespread use of telemedicine is still hampered by various barriers and challenges. Healthcare providers should work with various stakeholders to implement the proposed solutions. More research and policy changes are essential to optimize telemedicine utilization.