Improving the security of web services by automatic robot detection

Abstract

Abstract Web robots account for over half of all web traffic and typically monitor resources uniformly, unlike humans, who tend to focus on specific topics. This research examines the impact of a machine learning algorithm, known as Extreme Gradient Boosting (XGBoost), on identifying automated robots. XGBoost is notable for its strengths in parallel processing and regularization. The study analyzes and classifies data from 4,091,155 requests, with a primary dataset of 67,352 sessions divided into universal and dependent files. The universal data file contains 30 distinct features, while the dependent file has 5. Alongside XGBoost, two additional algorithms—Random Forest and Hist Gradient Boosting—were utilized to enhance the study’s validity. The three models achieved a comparable accuracy to one another across the datasets for five-fold cross-validation, achieving accuracy figures of 0.8407–0.8413 on the dependent dataset and 0.9671–0.9771 on the universal dataset. Paired tests for significance indicated that the difference in performance of XGBoost to both Random Forest and HistGB was significant on the universal data but not the dependent data ( p < 0.01 on both, vs. p > 0.29 on both respectively). The algorithms also differed somewhat in computational cost, where HistGB took a maximum of 3.1x the training time compared to XGBoost, but then recorded 0.1x the inference cost of both compared to the other models suggesting a trade-off between training time and prediction time that may be important when choosing models for real world implementation.