The origin and value of disagreement among data labelers: A case study of individual differences in hate speech annotation

Authors: Y Sang, J Stanton

Published: 2024

Publication: International Conference on Information, 2022 - Springer

This study explores disagreements among hate speech annotators and proposes a multidimensional scale to analyze individual differences, which could improve the value of minority-vote labels.

Methods: Mixed-method approach including expert interviews, concept mapping exercises, self-reporting questionnaires, and the development/testing of a multidimensional scale.

Key Findings: Individual differences (e.g., age, personality) and their relationship to annotators' label decisions in hate speech tasks.

Limitations: Potential scalability issues and reliance on self-reported data which may introduce biases.

Institution: Syracuse University

Research Area: Hate Speech Annotation, Individual Differences in Data Labeling

Discipline: Computational Social Science

Sample Size: 170 participants

Citations: 46

DOI: https://doi.org/10.1007/978-3-030-96957-8_36