Impact of annotator demographics on sentiment dataset labeling

Authors: Y Ding, J You, TK Machulla, J Jacobs, P Sen

Published: 2025

Publication: Proceedings of the ..., 2022 - dl.acm.org

Demographic differences among annotators significantly affect sentiment dataset labels, causing up to a 4.5% accuracy difference in sentiment prediction models.

Methods: Crowdsourced annotations from >1000 workers combined with demographic data; analysis of multimodal sentiment datasets and evaluation using machine learning models.

Key Findings: Impact of annotator demographics on sentiment labeling and its effect on model predictions.

Limitations: Potential lack of diversity within the crowdworker pool and the focus solely on sentiment analysis datasets.

Institution: University of California Irvine, University of Florida, State University of New York at Buffalo, University of Waterloo, Virginia Tech

Research Area: Computational Social Science, Human-Computer Interaction (HCI), Sentiment Analysis

Discipline: Computational Social Science

Sample Size: 1000 participants

Citations: 28

DOI: https://doi.org/10.1145/3555632