Rejection Threshold Optimization using 3D ROC Curves: Novel Findings on Biomedical Datasets

Abstract

Reject option is introduced in classification tasks to prevent potential misclassifications. Although optimization of error-rejecttrade-off has been widely investigated, it is shown that error rate itself is not an appropriate performance measure, when misclassificationcosts are unequal or class distributions are imbalanced. ROC analysis is proposed as an alternative approach to performance evaluation interms of true positives (TP) and false positives (FP). Considering classification with reject option, we need to represent the tradeoff betweenTP, FP and rejection rates. In this paper, we propose 3D ROC analysis to determine the optimal rejection threshold as an analogy to decisionthreshold optimization in 2D ROC curves. We have demonstrated our proposed method with Naive Bayes classifier on Heart Diseasedataset and validated the efficiency of the method on multiple datasets from UCI Machine Learning Repository. Our experiments revealthat classification with optimized rejection threshold significantly improves true positive rates in biomedical datasets. Furthermore, falsepositive rates remain the same with rejection rates below 10% on average.