Repeated Measure Designs are Superior for (Most) Experimental Survey Research Applications

Abstract

An influential study in the American Political Science Review by Clifford, Sheagley, and Piston (2021) finds that including pre-treatment measures of outcome variables in survey experiments does not bias treatment effect estimates and greatly improves precision, prompting many researchers to adopt repeated measure designs. In a large-scale partial replication, we experimentally manipulate the design of six classic experiments in political science and field all six experiments in three separate samples of US adults (total Ni= 13,163). We also provide three extensions that assess the broader suitability of repeated measure designs, specifically by fielding a larger set of within-subject experimental designs, by manipulating repeated measures’ proximity, and by fielding our experiments on both probability-based and non-probability samples. In contrast to the original study, we find consistent evidence of a small attenuation of treatment effects in repeated measure designs. However, this average attenuation bias is sufficiently small that we largely affirm the original authors’ recommendation to prefer repeated measure designs in most research applications, because the large gains to statistical precision will (in expectation) typically produce a more accurate estimate ATE. Further, we provide robust evidence that repeated measure designs are appropriate for within-subject and between-groups experiments, for extremely short surveys, and for both probability and non-probability samples.