A decision-tree for trend analyses following major survey redesigns: an analysis using the Canadian Community Health Survey (CCHS)

Abstract

Abstract Background Monitoring health inequality trends is essential for assessing progress, stagnation, or regression on the reduction of health inequalities; assessing the impact of certain policies on health inequalities; and identifying emerging areas of concern. Because inequalities can change slowly, long-term data (≥ 15 years) may be required to detect meaningful shifts. However, methodological changes in data sources over time raise concerns about data comparability. Frameworks are needed to guide decisions on whether data should be combined in these instances. Methods We used the Canadian Community Health Survey (CCHS) as a case to explore how sensitivity analyses and a decision-tree could help assess the comparability of survey cycles before and after major CCHS redesigns in 2015 and 2022, balancing considerations of statistical and practical significance, and background trends. Using CCHS cycles 2007–2008 to 2024, we estimated crude prevalence of 20 health indicators across cycles, among adults. We assessed: 1) the relative percent (%) change and 2) absolute percentage-point (PP) change between cycles pre- and post-redesigns (i.e., 2013–2014 vs. 2015–2016, 2019–2021 vs. 2022), including tests of statistical significance. Then, using generalized linear models and their predicted probabilities, we modeled the predicted post-redesign prevalence (for 2015–2016, 2022), based on pre-redesign trends (2007–2014, 2015–2021, respectively). Drawing from literature-based thresholds, the proposed decision-tree was used to flag indicators that met the three criteria described below. These indicators were subjected to additional qualitative investigation. Specifically, the flagged indicators must have demonstrated practically significant change on: A) relative (≥ 10% change) and B) absolute scales (≥ 2.5- or 5-PP change, based on baseline prevalence); and C) statistically significant change (no overlap with the margin of equivalence (95% CI) between observed and predicted post-redesign prevalence). Results No indicators met all three criteria post-2015. Post-2022, food insecurity, and self-rated physical and mental health marginally met all three criteria, signaling a need for additional research for long-term reporting. Conclusions This study presents a novel and reproducible framework for CCHS-specific analyses that others can refine or adapt. As it remains sensitive to assumptions, caution notes regarding the redesigns continue to be warranted for all CCHS indicators.