Improving annotation quality: empirical insights into bias, human-AI collaboration, and workflow design

Abstract

High-quality annotated datasets are essential for training machine learning (ML) models. Annotation means assigning a label (such as a category, sentiment score, or classification) to an instance, for example to a piece of text, an image, or a PDF file. Even as training algorithms continue to improve, a model’s real-world performance remains limited by the quality of the training data. While there are many approaches for processing training data, relatively little attention within the ML field has been devoted to annotation quality and the development of best practices for data collection. This thesis contributes to the field through empirical assessments of annotation bias and its implications for training data quality. It further proposes and evaluates strategies to mitigate such biases and enhance annotation outcomes. In addition, it explores the role of large language models (LLMs) in annotation workflows by experimentally assessing their use in fully automated and human-assisted hybrid annotation pipelines.