Skin cancer affects people of all skin tones, including those with darker complexions. Melanomas are known as the malignant tumors of skin cancer, resulting in an adverse prognosis, responsible for most deaths relating to skin cancer. Early diagnosis and treatment of skin cancer from dermoscopic images can significantly reduce mortality and save lives. Although several Computer-Aided Diagnosis (CAD) systems with satisfactory performance have been introduced in the literature for skin cancer detection, the high false detection rate has made it inevitable to have an expert physician for more examination. In this paper, a CAD system based on machine learning algorithms is provided to classify various skin cancer types. The proposed method uses the Online Region-based Active Contour Model (ORACM) to extract the Region Of Interest (ROI) of skin lesions. This model uses a new binary level set equation and regularization operation such as morphological opening and closing.Additionally, various combinations of different textures and nonlinear features are extracted for the ROI to show the multiple aspects of skin lesions. Several metaheuristic optimization algorithms are used to remove redundant or irrelevant features and reduce the feature space dimension. These are applied to the combination of the extracted features in which, Non-dominated Sorting Genetic Algorithm (NSGA II) as a multi-objective optimization algorithm has the best performance. Furthermore, various machine learning algorithms include K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Fitting neural network (Fit net), Feed-Forward neural network (FF net), and Pattern recognition network (Pat net) are employed for the classification. Accordingly, the best-obtained precision of 99.24% based on five-fold cross-validation is attained by the selected features of texture and nonlinear indices through NSGA II, applying the pattern net classifier. Also, the comparison between this paper's experimental results and other similar works with the same dataset demonstrates the proposed method's efficiency.