Explainable and Robust Machine Learning for Phishing URL Detection: A Cross-Dataset Evaluation

Abstract

Phishing URL detection using machine learning has achieved near-perfect accuracy on popular benchmark datasets, but the extent to which these results generalize across datasets remains unclear. We train a Random Forest classifier on the PhiUSIIL phishing URL dataset (235,795 records) and evaluate it on the UCI Phishing Websites dataset (11,055 records), after mapping a subset of PhiUSIIL features into the UCI feature space. In-domain, our model achieves balanced accuracy of 0.9999 on a domain-separated PhiUSIIL test split and 0.9979 for a URL-lexical-feature-only variant. On the external UCI dataset, balanced accuracy drops to 0.9530 (full numeric) and 0.6753 (transfer experiment using 10 mapped features), with a generalization gap of 0.287. We further analyze 64 feature-vector groups with conflicting labels in the UCI dataset and show that a trained model is near chance-level on those rows, indicating an irreducible label-noise ceiling rather than a model deficiency. Our results argue that high in-domain performance is insufficient evidence of real-world reliability and that multi-dataset evaluation should be standard practice for phishing detectors.