An Instance Space Analysis of Neural Network Training as a Black-Box Optimisation Problem

Abstract

Training a neural network is an optimisation task that is commonly solved using gradient-based techniques, such as stochastic gradient descent or Adam. There are, however, scenarios in which black-box optimisation approaches, such as evolutionary algorithms, may be appropriate for optimising neural network weights. In this study, we view neural network weight optimisation as a black-box problem and analyse the search space using exploratory landscape analysis (ELA). The purpose is to understand what makes neural network search spaces different from other bound-constrained, continuous-valued problems commonly used to benchmark evolutionary algorithms. We perform an instance space analysis on a combined set of neural network training instances and instances from the widely used Black-Box Optimisation Benchmarking (BBOB) problem suite. The analysis included instances with 41, 261, and 481 decision variables, corresponding with 1-, 3-, and 5-layer networks. We show that neural-network training instances and BBOB instances occupy distinct regions of the ELA feature space. While the landscapes of BBOB instances typically exhibit strong global structure and large differences between local and global optima, neural network instances exhibit ill-conditioned landscapes with very weak global structure. Surprisingly, these instances were easier for black-box algorithms to find acceptable solutions than BBOB instances, likely due to the abundance of high-quality solutions. Another key and unexpected finding was that the features of the neural network error landscapes were more strongly influenced by the choice of activation function and the network size than by the underlying machine learning task, in this case, the regression function to be fitted.