The mounting prevalence of health insurance fraud, propelled by a myriad of socioeconomic factors, presents significant hurdles to insurers, healthcare institutions, and individuals. In an attempt to counter this, insurance companies have begun harnessing the power of advanced technology, utilizing Machine Learning models to distinguish legitimate from fraudulent claims within expansive datasets. The present study conducts an in-depth examination of a health insurance dataset comprising 517,737 records, employing the Extreme Gradient Boosting (XGBoost) model as a potent tool for the detection of deceptive claims. In a noteworthy development, the performance of the model is markedly amplified through the integration of Bayesian optimization techniques, culminating in the Bayesian Optimized XGBoost (BOXGBoost) Model. The BOXGBoost Model is meticulously evaluated against an array of algorithms, which include Naive Bayes, Logistic Regression, Random Forest, K-Nearest Neighbor, and AdaBoost. A comparative analysis, focusing on key performance metrics such as accuracy, precision, recall, F1-Score, and the Area Under the Curve (AUC), is undertaken to discern the most effective algorithm. Remarkably, the proposed BOXGBoost model emerges as the superior performer, achieving an impressive accuracy rate of 98% and an AUC of 0.994. Additionally, the model exhibits high precision (98%), recall (97%), and F1-Score (97.5%), highlighting its exceptional capability in the prediction of health insurance fraud.