Boundary of the Maximum Depth Hyperparameters of the Random Forest Leading to the Highest Accuracy Discovered by Single-Tree Techniques

Abstract

Abstract Random Forest (RF) is a widely used ensemble learning method known for its high classification accuracy. However, optimizing its numerous hyperparameters remains a persistent challenge, particularly the Maximum Depth (MaxDepth) of individual trees. This study examines the boundaries of MaxDepth and Bagging Fraction (%bag) values that yield optimal performance in RF models. By conducting exhaustive experiments across ten diverse datasets for heart disease, breast cancer, and therapeutic domains, we define a performance-critical range termed the Highest-Accuracy Maximum-Depth boundary (HAMD boundary) that characterizes the region in which RF models achieve their peak accuracy. The core hypothesis is that the accuracy patterns of single decision trees at varying MaxDepths can be used to approximate this HAMD boundary. Using Scikit-learn’s Gini and entropy criteria in a Python-Colab environment, we analyze RF behavior and compare it with single-tree accuracy trends. Two distinct single-tree patterns emerge: one in which accuracy increases and stabilizes with depth, and another in which accuracy peaks and subsequently declines. These patterns correspond to the shape and extent of the HAMD boundary across RF models that reduce one-third of the entire search space for hyperparameter tuning. Additionally, we test the C4.5 decision tree (criterion = gain ratio) via WEKA integration to further refine the long boundary to a shorter one for practical purposes. Some datasets are validated for the characteristic of the HAMD boundary by various numbers of trees and seeds. The results demonstrate that MaxDepth tuning can be effectively guided using single-tree accuracy curves, providing a principled, non-random approach to define the range of MaxDepth selection. This work contributes a new heuristic search space for RF hyperparameter tuning and opens avenues for integrating single-tree diagnostics into hyperparameter search frameworks.