Relationship between data bias and model fairness in machine learning

Abstract

Ensuring fairness in machine learning (ML) applications remains a critical challenge, particularly in high-stakes domains such as healthcare, finance, and criminal justice. Bias in training data is widely considered a key driver of unfair outcomes. However, the relationship between data bias and fairness violations in ML models is not always straightforward. This study investigates whether biased datasets consistently lead to unfair ML predictions and explores the extent to which bias mitigation strategies influence fairness outcomes. We systematically evaluated 22 publicly available datasets, assessing bias using the Earth Mover’s Distance metric and fairness using three widely recognised fairness definitions: Equalised Odds, Demographic Parity, and Treatment Equality. We applied five ML algorithms and performed Chi-square tests to assess the statistical significance of fairness discrepancies. We further explored the impact of bias mitigation by transforming randomly selected biased datasets into unbiased versions through data preprocessing techniques and evaluating changes in fairness outcomes. Our findings show that datasets identified as biased are significantly more likely to produce models that fail to satisfy all three fairness criteria, with a highly significant association observed across analyses ( p < 0.001). Nonetheless, data bias does not invariably translate into unfair outcomes, as certain ML algorithms can partially mitigate fairness disparities during training. Bias mitigation through data preprocessing guided by the Earth Mover’s Distance led to notable improvements in fairness across multiple ML models, though often accompanied by a reduction in predictive performance, highlighting the trade-off between fairness and accuracy. These results underscore the need for fairness assessments that consider both dataset characteristics and model behaviour, highlighting that achieving fairness in ML requires a combined approach that addresses data bias and algorithmic factors.