Health Insurance Risk Segmentation Using Unsupervised Machine Learning and Clustering Techniques

Abstract

Health insurance databases include key demographic, health-related, lifestyle and socioeconomic risk factors that allow for the discovery of patterns of policyholder risk. This paper applies unsupervised machine learning techniques to analyze an insurance cost and risk dataset of 1,338 rows/12 variables. In particular, the overall goal is to find hidden risk classes among policyholders and gain better insights into determinants of differences in insurance premiums. Initial preprocessing included checking variable types, imputing missing numerical values, encoding categorical variables, and standardizing numerical features. Next, multiple clustering techniques were used which are BIRCH, Gaussian Mixture Model, Agglomerative Clustering (average linkage), Agglomerative Clustering (complete linkage) and DBSCAN. Clusters were evaluated using internal validation metrics, namely Silhouette Score, Davies-Bouldin Index, and Calinski-Harabasz Score. Cluster sizes were examined, and the results were visualized using two-dimensional PCA plots. Results: BIRCH provided the best clustering framework that clustered all data into three identifiable low, medium and high-risk insurance groups. The main differences between insurance risk groups were seen in age, BMI, insurance premiums and associated health-risk factors based on cluster profiling. Compared to the low-risk group, policyholders in the high-risk group had higher average insurance charges, older age, and a greater burden of obesity. The findings nonetheless suggest that techniques for clustering are able to partition health insurance data, thereby enabling decisions based on modelling and evidence on rate