Higher inter-rater agreement but more low-risk judgments: the influence of rater experience on RoB 2 assessments

Abstract

It remains unclear whether risk-of-bias judgments made using the revised Cochrane risk-of-bias tool for randomized trials (RoB 2) by less experienced raters, even after standard training, systematically differ from those made by experienced raters. Our study recruited 12 inexperienced and 6 experienced raters to assess the risk of bias in 94 results from 37 randomized controlled trials using RoB 2. After the independent assessments were completed, raters were randomly paired within groups to reach consensus judgments. We calculated Gwet’s agreement coefficient 2 to evaluate inter-rater reliability in both phases. After exclusions, 10 inexperienced raters (four pairs) and 6 experienced raters (three pairs) were included. Inexperienced raters demonstrated higher inter-rater agreement across multiple domains in both independent and consensus assessments, whereas experienced raters showed higher agreement only in the “randomization process” domain. Despite this higher agreement, inexperienced raters were more likely to assign “low risk” ratings than experienced raters across most domains. After consensus, agreement in the “randomization process” domain improved in the experienced group, whereas it increased in two domains but decreased in two in the inexperienced group. Overall, rating distributions remained stable among experienced raters but varied across several domains among inexperienced raters. This may suggest that increased reliability among inexperienced raters may reflect systematic differences in interpretation rather than more accurate assessments. These findings highlight potential challenges in applying RoB 2 among less experienced users and underscore the importance of adequate training and calibration.