Background: The cross-sectional area of skeletal muscles at the level of the third lumbar vertebra (L3) measured from computed tomography (CT) images is an established imaging biomarker used to assess patients’ nutritional status. With the increasing prevalence of low-dose CT scans in clinical practice, accurate and automated skeletal muscle segmentation at the L3 level in low-dose CT images has become an issue to address. This study proposed a lightweight algorithm for automated segmentation of skeletal muscles at the L3 level in low-dose CT images. Methods: This study included 57 patients with rectal cancer, with both low-dose plain and contrast-enhanced pelvic CT image series acquired using a radiotherapy CT scanner. A training set of 30 randomly selected patients was used to develop a lightweight segmentation algorithm, and the other 27 patients were used as the test set. A radiologist selected the most representative axial CT image at the L3 level for both the image series for all the patients, and three groups of observers manually annotated the skeletal muscles in the 54 CT images of the test set as the gold standard. The performance of the proposed algorithm was evaluated in terms of the Dice similarity coefficient (DSC), precision, recall, 95th percentile of the Hausdorff distance (HD95), and average surface distance (ASD). The running time of the proposed algorithm was recorded. An open source deep learning-based AutoMATICA algorithm was compared with the proposed algorithm. The inter-observer variations were also used as the reference. Results: The DSC, precision, recall, HD95, ASD, and running time were 93.2 ± 1.9% (mean ± standard deviation), 96.7 ± 2.9%, 90.0 ± 2.9%, 4.8 ± 1.3 mm, 0.8 ± 0.2 mm, and 303 ± 43 ms (on CPU) for the proposed algorithm, and 94.1 ± 4.1%, 92.7 ± 5.5%, 95.7 ± 4.0%, 7.4 ± 5.7 mm, 0.9 ± 0.6 mm, and 448 ± 40 ms (on GPU) for AutoMATICA, respectively. The differences between the proposed algorithm and the inter-observer reference were 4.7%, 1.2%, 7.9%, 3.2 mm, and 0.6 mm, respectively, for the averaged DSC, precision, recall, HD95, and ASD. Conclusion: The proposed algorithm can be used to segment skeletal muscles at the L3 level in either the plain or enhanced low-dose CT images.