Conditional updates of neural network weights for increased out of training performance

Abstract

This study proposes a method to enhance neural network performance when training data and application data are not very similar, e.g., out of distribution problems, as well as pattern and regime shifts. In contrast to previous approaches to out of distribution problems, which alter the inputs and outputs of an operator, our approach alters the operator itself. Here, we especially aim to alter the weights and biases of a neural network to mitigate the out of distribution problem. The method consists of three main steps: (1) Fine-tune a trained neural network towards reasonable subsets of the training data set and note down the resulting weight anomalies. (2) Choose reasonable predictors and derive a regression between the predictors and the noted weight anomalies. (3) Extrapolate the weights, and thereby the neural network, to the application data. We show and discuss this method in three nonlinear use cases from the climate sciences, which include successful temporal, spatial, and cross-regime extrapolations of neural networks.