Explainable Machine Learning Framework for Predictive Resource Failure Detection in Distributed Computing Environments

Abstract

Large-scale distributed computing systems produce millions of telemetry events per day, yet existing failure prediction depends on static thresholds, ignores feature interdependencies and offers minimal interpretability. This research introduces the Explainable Machine Learning Framework for Predictive Resource Failure Detection (XML-PRFD), which integrates telemetry preprocessing, tuned supervised classification, decision-threshold calibration, and interpretability through SHAP values and permutation importance. The framework is evaluated on a synthetic dataset of 12,000 observations and ten features derived from seven interaction-dependent failure modes, split 80:20 into stratified training and test subsets, with Logistic Regression, Random Forest and XGBoost given equal tuning budgets. XGBoost achieves the best test-set performance, with 84.00% accuracy, 71.16% precision, 54.06% recall, an F1-score of 61.45% and a ROC-AUC of 0.810. SHAP attributes 43.3% of total feature importance to memory utilization and 20.4% to CPU utilization, and permutation importance identifies the same leading predictors (Spearman ρ = 0.78). The results demonstrate that the integration of tree ensemble methodologies with Shapley-value-based explanatory frameworks yields failure predictions characterized by both high accuracy and comprehensive auditability.