Pepper leaf segmentation plays a crucial role in monitoring pepper leaf diseases in various backgrounds and ensuring the healthy growth of peppers. However, existing transformer-based segmentation methods suffer from computational inefficiency, excessive parameterization, and limited utilization of edge information. To tackle these challenges, we propose an adaptive multi-scale MLP framework, named AMS-MLP, which combines the multi-path aggregation module (MPAM) and the multi-scale context relation mask module (MCRD) to refine the object boundaries in pepper leaf segmentation. AMS-MLP consists of an encoder-based network, an adaptive multi-scale MLP (AM-MLP) module, and a decoder network. In the encoder network, the MPAM module effectively fuses five-scale features to generate a single-channel mask, improving the accuracy of pepper leaf boundary extraction. The AM-MLP module divides the input features into two branches: the global multi-scale MLP branch captures long-range dependencies between image information, while the local multi-scale MLP branch focuses on extracting local feature maps. Adaptive attention mechanism is designed to dynamically adjust the weights of global and local features. The decoder network incorporates the MCRD module into the convolutional layer, enhancing the extraction of boundary features. To verify the performance of the proposed method, we conducted extensive experiments on three pepper leaf datasets with different backgrounds. The results demonstrate mIoU scores of 97.39%, 96.91%, and 97.91%, as well as F1 scores of 98.29%, 97.86%, and 98.51%, respectively. Comparative analysis with U-Net and state-of-the-art models reveals that the proposed method dramatically improves the accuracy and efficiency of pepper leaf image segmentation.