Bias in Cohort Studies: The Critical Importance of the “Why” and “When” of Treatment Decisions

Abstract

When it comes to high-quality, rigorous research, children generally get the short end of the stick. Randomized controlled trials (RCTs)—the “gold standard” for clinical research1—are rare in this population.2 RCTs are expensive, time-consuming, and often fraught with ethical concerns surrounding the enrollment of children. As large databases—including multicenter research collaborations, electronic health records (EHRs), administrative data sets, and insurance claims—continue to expand and become more accessible, clinical researchers are increasingly turning to these resources to answer important questions in child health. Although these alternatives to randomized trials have obvious appeal, they are susceptible to important biases.3The accompanying article in this issue of Hospital Pediatrics by Yu et al4 reports on data from more than 1500 children hospitalized with orbital infections across multiple Canadian hospitals. The study describes intranasal (IN) corticosteroid use and assessed whether this intervention was associated with hospital length of stay (LOS). The research question is clinically meaningful, and the study attempts to fill important knowledge gaps. Orbital infections are associated with substantial morbidity, including the potential for vision loss, intracranial extension, and surgical intervention. IN steroids are relatively cheap and benign, and there is some biologic plausibility for their benefit.Data were collected via chart review, though the timing of IN steroids was not recorded. IN steroids were administered in 44% of patients. On unadjusted analyses, median LOS was almost twice as long in patients who received this intervention: 118 vs 61 hours, a difference of 67 hours! The difference was much smaller after multivariable adjustment (14.9 hours; 95% CI, 8.4–21.4 hours), but this difference—if truly causally associated with IN steroids—would still be viewed as clinically meaningful. However, it is difficult to envision a biologically plausible mechanism by which IN steroids would prolong hospitalization to this extent. More likely explanations for these findings include confounding by indication and/or immortal time bias.Confounding by indication is a fairly intuitive concept. Sicker and more complicated patients generally have a higher likelihood of receiving treatments and are also more likely to have worse outcomes. This tends to make treatments look falsely bad. (The exception is treatments that are indicated only in healthier patients, like chemotherapy unlikely to be tolerated in the sickest cancer patients.) Uncontrolled confounding by indication is thus a potential explanation for the adjusted increase in LOS in the IN corticosteroids group.Many investigations attempt to address confounding by indication by adjusting for baseline severity of illness. In administrative and insurance claims databases, which lack key markers of illness severity, such as vital signs and physical exam findings, this type of adjustment is likely to be insufficient. Even more granular data from EHR databases may fail to fully capture the baseline severity of illness, such that the potential for residual confounding always looms.Immortal time bias is trickier. This bias gets its name from studies of mortality, but the same mechanism can affect studies of other outcomes, such as LOS, when the event that defines the outcome is hospital discharge rather than death.5,6The problem arises when patients are assigned to the treatment group after “time 0” for the analysis. For example, time 0 for a study of LOS would be the date and time of admission, but patients might not get assigned to the treated group until some days later, when treatment starts. The period between admission and treatment initiation becomes “immortal” because patients who ultimately receive the intervention, by definition, must have remained hospitalized long enough to do so. Thus, treated patients have an automatic minimum LOS equal to the time until treatment was initiated, and untreated patients have no such minimum.In the investigation by Yu et al,4 44% of patients received IN steroids, but the steroids could have been initiated at any time point during the hospitalization. Many interventions in hospitalized children demonstrate variability. Rather than being driven by patient-level factors, often this variability is “unwarranted,” driven by factors that have nothing to do with the patient but instead reflect regional practices, institutional culture, or individual physician preferences. As such, each additional day in a hospital poses more opportunity for a patient to receive the intervention. If treatment is analyzed as a simple “yes/no” exposure, however, all hospital days preceding treatment initiation are effectively attributed to the treatment group.Immortal time bias could easily impact other commonly studied associations between various controversial interventions and LOS in hospitalized children, such as oseltamivir in influenza, a second dose of steroids in croup, or bronchodilators/steroids in bronchiolitis. Table 1 describes a few hypothetical patients to illustrate how immortal time bias and/or confounding by indication could influence the association between IN steroids and LOS in children hospitalized with orbital infections.The fact that the authors explicitly acknowledge immortal time bias in both the abstract and discussion deserves praise. Too often, observational studies present adjusted associations using causal language that exceeds what the data can support. Here, the investigators appropriately frame their findings with caution and transparency.What approaches can investigators use in studies like this to further mitigate confounding by indication and immortal time bias? Although RCTs are ideal, we recommend consideration of target trial emulation and/or instrumental variable (IV) analyses when RCTs are not feasible.Target trial emulation is a causal inference framework in which investigators explicitly design an observational study to mimic a hypothetical RCT that would ideally answer the clinical question of interest.7 Rather than simply comparing patients who did and did not receive a treatment, this approach requires prespecification of eligibility criteria, treatment strategies, the timing of treatment assignment (time 0), follow-up, outcomes, and analytic approaches.8Target trial emulation can help address immortal time bias by aligning the start of follow-up with the treatment decision and defining treatment strategies prospectively (eg, initiation of IN steroids within 24 hours of admission vs no initiation within 24 hours). This methodology can also reduce confounding by indication through careful measurement and adjustment for baseline factors associated with both treatment selection and outcomes, including illness severity, comorbid conditions, and demographic characteristics. However, unlike randomization, target trial emulation cannot eliminate bias from unmeasured confounders; its validity depends on adequately capturing the clinical factors that influence treatment decisions. At least one member of the clinical research team should be someone who makes these treatment decisions and can evaluate the availability of measurements made at baseline to determine whether the data available can provide a credible answer to the research question.IV analysis is another causal inference approach that leverages natural variation in treatment practices to estimate treatment effects in observational studies. Rather than comparing patients who did and did not receive an intervention directly, IV analyses use a factor associated with treatment receipt—but not otherwise related to the outcome—as a proxy for randomization. In pediatric comparative effectiveness studies, hospital-level prescribing preference is a commonly used instrument when substantial variation in practice exists across institutions. For example, hospitals vary in their approach to interventions for various conditions, such as oseltamivir prescribing in children hospitalized with influenza9 or parenteral antibiotic durations for bacteremic urinary tract infections.10 Because this variation does not appear to be driven by patient factors, a “natural experiment” is created to better understand the efficacy of these interventions.These RCT alternatives have promise, but we continue to believe that RCTs should remain the aspirational standard for pediatric hospital medicine researchers. We hope this study team will use their findings to help design and provide equipoise for a future RCT. An important next step for our field is to identify and address the barriers to conducting rigorous pediatric RCTs while continuing to demonstrate their value to patients, families, institutions, and funding agencies.