Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement

Abstract

Machine Learning models have many potentially beneficial applications in education settings, but a key barrier to their development is securing enough high-quality, labelled data to train these models. This process has traditionally relied on highly skilled raters using complex, multi-class rubrics, which made labelling expensive and difficult to scale. A more scalable approach would be to use non-expert crowdworkers to evaluate student work, but maintaining high levels of accuracy and inter-rater reliability when using non-expert workers can be challenging. This paper reports on two experiments in which non-expert crowdworkers hired to evaluate (i.e., score) student work and were randomly assigned to one of two conditions: the control, where they were asked to assign a rubrics based score (i.e., a categorical judgement), or the treatment, where they were shown the same student answers, but were asked to decide which of two candidate answers was better (i.e., a comparative/preference-based judgement). We found that using comparative judgement substantially improved inter-rater reliability on both tasks. These results are in-line with well-established literature on the benefits of comparative judgement in the field of educational assessment, as well as with recent trends in artificial intelligence research, where comparative judgement is becoming the preferred method for providing human feedback on model outputs. These results are novel and important in demonstrating the effects of using the combination of comparative judgement and crowdworkers to evaluate educational data