Fair backward compatibility: definitions, theoretical framework, and empirical results

Abstract

Machine learning systems often require updates for various reasons, ranging from the availability of new data or models to the necessity of meeting new or revised technical and ethical metrics. Typically, such metrics capture average performance rather than sample-wise behavior. However, improvements in aggregate measures such as accuracy may lead to negative flips, i.e., instances where the updated model produces errors that the previous model did not, thereby introducing backward incompatibilities. Furthermore, if the distribution of these negative flips is biased with respect to sensitive attributes (e.g., sex or race), the resulting models become not only backward incompatible but also unfair. In this paper, we introduce a generalized notion of Fair Backward Compatibility , which extends and unifies prior work on fairness and backward compatibility. Building on this concept, we propose Fair Backward-Compatible Empirical Risk Minimization , a framework that integrates fairness-aware backward compatibility into most modern machine learning algorithms. We establish the statistical consistency of our framework by showing that both the empirical risk and the empirical Fair Backward Compatibility of the learned model converge to their population-level counterparts at the same rate as in classical Empirical Risk Minimization. We demonstrate that Fair Backward-Compatible Empirical Risk Minimization can be deployed through simple modifications of the cost function in standard machine learning models, using different—convex or at least differentiable—relaxations. We propose a new procedure for tuning and evaluating the performance of models that address both risk and Fair Backward Compatibility. Finally, experiments on real-world datasets, employing both shallow and deep architectures, confirm the effectiveness of our proposal.