Comparison of Methods for Incorporating Related Data When Developing Clinical Prediction Models: A Simulation Study

Abstract

Clinical Prediction Models (CPMs) compute an individual's risk of an outcome, given a set of predictors. Guidance states CPMs should be constructed using data sampled from the target population. However, researchers might have access to ancillary data sets from different time points, countries, or healthcare settings, which could support model development. This study explores in which situations the ancillary data affect CPM performance, given potential heterogeneity. We conducted a simulation study to assess the impact of heterogeneity between the target and the ancillary data sets, and their relative sample size, on CPM performance. Target and ancillary populations were generated with varying degrees of heterogeneity. CPMs were developed using target-only logistic regression, logistic and intercept updating, and importance weighting using propensity scores. These models were evaluated on independent data using calibration, discrimination, and prediction stability. Also, a real-world case study was used as an illustrative example of application using the SWEDEHEART registry. Incorporating ancillary data generally improve CPM performance. Logistic and Intercept Recalibration often outperformed the target-only regression approach. However, Logistic Recalibration showed greater variability and instability in calibration, while Intercept Recalibration performed poorly under predictor-outcome association shift. The importance weighting method demonstrated consistent performance across a wide range of scenarios and appears to be a reliable alternative, particularly in practical settings where the presence and type of data distribution shift is often unknown.