Issues in the Analysis of Data: A Replication of Zuckerman et al. (1993) and Extension to Contemporary Developmental Research

Abstract

Developmental psychology relies on statistical evidence to draw conclusions about change, mechanisms, and context, but the validity of those conclusions depends on whether researchers interpret that evidence correctly. This article examines whether psychological researchers recognize which conclusions are justified in common analytic scenarios. Study 1 replicated Zuckerman et al.’s survey of five classic statistical issues using a contemporary sample of authors from leading psychology journals. Researchers showed similar accuracy to the original sample on interpreting interaction effects, multivariate analyses of variance, and effect-size comparisons, but lower accuracy on questions involving reliability and sample size. Study 2 extended this approach to issues relevant to contemporary developmental research. Researchers showed strong understanding that opportunistic biases can affect effect-size estimates, mixed accuracy in evaluating the use of covariates in longitudinal mediation analyses, and lower accuracy when interpreting overlapping confidence intervals, separating within- and between-cluster effects, and using multiple imputation for missing outcomes. These findings suggest that advances in developmental methods need to be accompanied by stronger attention to the inferential logic linking statistical results to developmental theory.